Search Results for author:
Found 26 papers, 16 papers with code
On Pre-training of Multimodal Language Models Customized for Chart Understanding
Experimental results show that CHOPINLLM exhibits strong performance in understanding both annotated and unannotated charts across a wide range of types.
ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos
This form of reasoning, requiring advanced understanding of cause-and-effect relationships across video segments, poses significant challenges to even the frontier multimodal large language models.
Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search
To further optimize the LLM-generated programs, we develop a search algorithm named Scheduled Hill Climbing, designed to efficiently explore the programmatic search space to consistently improve the programs.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views
Our strategy unfolds in three steps: (1) We invert the diffusion model for camera pose estimation instead of synthesizing novel views.
LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following
Additional analysis shows that the contrastive objective and meta-actions are complementary in achieving the best results, and the resulting agent better aligns its states with corresponding instructions, making it more suitable for real-world embodied agents.
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
Text-to-Image diffusion models have made tremendous progress over the past two years, enabling the generation of highly realistic images based on open-domain text descriptions.
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis
Diffusion models (DMs) have shown great potential for high-quality image synthesis.
GLIPv2: Unifying Localization and Vision-Language Understanding
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e. g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e. g., VQA, image captioning).
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Test
(using extra training data)
Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks
Experiments demonstrate that MAD leads to consistent gains in the low-shot, domain-shifted, and fully-supervised conditions on VCR, SNLI-VE, and VQA, achieving SOTA performance on VCR compared to other single models pretrained with image-text data.
Ranked #4 on
Visual Question Answering (VQA)
on VCR (Q-A) test
CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51. 9%) and domain-shifted (up to 71. 3%) conditions of VCR, while simultaneously improving performance under standard fully-supervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
Playing Lottery Tickets with Vision and Language
However, we can find "relaxed" winning tickets at 50%-70% sparsity that maintain 99% of the full accuracy.
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Multimodal pre-training has propelled great advancement in vision-and-language research.
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Transformer has become ubiquitous in the deep learning field.
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
Transformer has become ubiquitous in the deep learning field.
Ranked #1 on
Open-Domain Question Answering
on SearchQA
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
We present a large, tunable neural conversational response generation model, DIALOGPT (dialogue generative pre-trained transformer).
Large-Scale Adversarial Training for Vision-and-Language Representation Learning
We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning.
Ranked #7 on
Visual Entailment
on SNLI-VE val
(using extra training data)
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
To reveal the secrets behind the scene of these powerful models, we present VALUE (Vision-And-Language Understanding Evaluation), a set of meticulously designed probing tasks (e. g., Visual Coreference Resolution, Visual Relation Detection, Linguistic Probing Tasks) generalizable to standard pre-trained V+L models, aiming to decipher the inner workings of multimodal pre-training (e. g., the implicit knowledge garnered in individual attention heads, the inherent cross-modal alignment learned through contextualized multimodal embeddings).
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
We present HERO, a novel framework for large-scale video+language omni-representation learning.
Ranked #1 on
Video Retrieval
on TVR
Distilling Knowledge Learned in BERT for Text Generation
Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization.
DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained transformer).
UNITER: UNiversal Image-TExt Representation Learning
Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i. e., masked language/region modeling is conditioned on full observation of image/text).
Ranked #3 on
Visual Question Answering (VQA)
on VCR (Q-A) test
UNITER: Learning UNiversal Image-TExt Representations
Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding.
Explore, Propose, and Assemble: An Interpretable Model for Multi-Hop Reading Comprehension
Multi-hop reading comprehension requires the model to explore and connect relevant information from multiple sentences/documents in order to answer the question about the context.
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Inspired by how humans summarize long documents, we propose an accurate and fast summarization model that first selects salient sentences and then rewrites them abstractively (i. e., compresses and paraphrases) to generate a concise overall summary.
Ranked #7 on
Text Summarization
on CNN / Daily Mail (Anonymized)
