Vision and Language Pre-Trained Models

In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss (formulated as normalized softmax) that pushes the embeddings of the matched image-text pair together and pushing those of non-matched image-text pair apart. The model learns to align visual and language representations of the image and text pairs using the contrastive loss. The representations can be used for vision-only or vision-language task transfer. Without any fine-tuning, ALIGN powers zero-shot visual classification and cross-modal search including image-to-text search, text-to image search and even search with joint image+text queries.

Source: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision


Paper Code Results Date Stars


Task Papers Share
Language Modelling 58 8.12%
Large Language Model 29 4.06%
Retrieval 17 2.38%
Question Answering 15 2.10%
Semantic Segmentation 13 1.82%
Text Generation 13 1.82%
Decision Making 13 1.82%
In-Context Learning 12 1.68%
Object Detection 12 1.68%


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign