Search Results for author:
Found 20 papers, 13 papers with code
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Today's most advanced vision-language models (VLMs) remain proprietary.
CodeNav: Beyond tool-use to using real-world codebases with LLM agents
We present CodeNav, an LLM agent that navigates and leverages previously unseen code repositories to solve user queries.
m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks
With m&m's, we evaluate 10 popular LLMs with 2 planning strategies (multi-step vs. step-by-step planning), 2 plan formats (JSON vs. code), and 3 types of feedback (parsing/verification/execution).
Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain.
SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
Reinforcement learning (RL) with dense rewards and imitation learning (IL) with human-generated trajectories are the most widely used approaches for training modern embodied agents.
Visual Programming: Compositional visual reasoning without training
We present VISPROG, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions.
GRIT: General Robust Image Task Benchmark
Computer vision models excel at making predictions when the test distribution closely resembles the training distribution.
Webly Supervised Concept Expansion for General Purpose Vision Models
This work presents an effective and inexpensive alternative: learn skills from supervised datasets, learn concepts from web image search, and leverage a key characteristic of GPVs: the ability to transfer visual knowledge across skills.
Ranked #2 on
Visual Question Answering (VQA)
on GRIT
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture
To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process.
Visual Semantic Role Labeling for Video Understanding
We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling.
Towards General Purpose Vision Systems
To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process.
Learning Curves for Analysis of Deep Networks
Learning curves model a classifier's test error as a function of the number of training samples.
Contrastive Learning for Weakly Supervised Phrase Grounding
Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions.
ViCo: Word Embeddings from Visual Co-occurrences
Through unsupervised clustering, supervised partitioning, and a zero-shot-like generalization analysis we show that our word embeddings complement text-only embeddings like GloVe by better representing similarities and differences between visual concepts that are difficult to obtain from text corpora alone.
No-Frills Human-Object Interaction Detection: Factorization, Layout Encodings, and Training Techniques
We show that for human-object interaction detection a relatively simple factorized model with appearance and layout encodings constructed from pre-trained object detectors outperforms more sophisticated approaches.
Imagine This! Scripts to Compositions to Videos
Imagining a scene described in natural language with realistic layout and appearance of entities is the ultimate test of spatial, visual, and semantic world knowledge.
Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning.
3DFS: Deformable Dense Depth Fusion and Segmentation for Object Reconstruction from a Handheld Camera
The resulting depth maps are then fused using a proposed implicit surface function that is robust to estimation error, producing a smooth surface reconstruction of the entire scene.
Completing 3D Object Shape From One Depth Image
Our goal is to recover a complete 3D model from a depth image of an object.
