Search Results for author:
Found 52 papers, 23 papers with code
Can We Generate Visual Programs Without Prompting LLMs?
Visual programming prompts LLMs (large language mod-els) to generate executable code for visual tasks like visual question answering (VQA).
RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations
We present RELOCATE, a simple training-free baseline designed to perform the challenging task of visual query localization in long videos.
Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB
The goal of this paper is to encode a 3D scene into an extremely compact representation from 2D images and to enable its transmittance, decoding and rendering in real-time across various platforms.
Anytime Continual Learning for Open Vocabulary Classification
We propose a dynamic weighting between predictions of a partially fine-tuned model and a fixed open vocabulary model that enables continual improvement when training samples are available for a subset of a task's labels.
MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based Monocular Guidance
The latest regularized Neural Radiance Field (NeRF) approaches produce poor geometry and view extrapolation for large scale sparse view scenes, such as ETH3D.
Region-Based Representations Revisited
We investigate whether region-based representations are effective for recognition.
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action
We present Unified-IO 2 a multimodal and multi-skill unified model capable of following novel instructions.
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action.
ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation
We adopt a weakly-supervised approach to directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs from the web.
WebWISE: Web Interface Control and Sequential Exploration with Large Language Models
The paper investigates using a Large Language Model (LLM) to automatically perform web software tasks using click, scroll, and text input operations.
Continual Learning in Open-vocabulary Classification with Complementary Memory Systems
We introduce a method for flexible and efficient continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in human cognition.
Consistent Multimodal Generation via A Unified GAN Framework
We investigate how to generate multimodal image outputs, such as RGB, depth, and surface normals, with a single generative model.
Make It So: Steering StyleGAN for Any Image Inversion and Editing
StyleGAN's disentangled style representation enables powerful image editing by manipulating the latent variables, but accurately mapping real-world images to their latent variables (GAN inversion) remains a challenge.
QFF: Quantized Fourier Features for Neural Field Representations
Instead, we propose to encode features in bins of Fourier features that are commonly used for positional encoding.
Sparse SPN: Depth Completion from Sparse Keypoints
Our long term goal is to use image-based depth completion to quickly create 3D models from sparse point clouds, e. g. from SfM or SLAM.
Deep PatchMatch MVS with Learned Patch Coplanarity, Geometric Consistency and Adaptive Pixel Sampling
Recent work in multi-view stereo (MVS) combines learnable photometric scores and regularization with PatchMatch-based optimization to achieve robust pixelwise estimates of depth, normals, and visibility.
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction.
GRIT: General Robust Image Task Benchmark
Computer vision models excel at making predictions when the test distribution closely resembles the training distribution.
Webly Supervised Concept Expansion for General Purpose Vision Models
This work presents an effective and inexpensive alternative: learn skills from supervised datasets, learn concepts from web image search, and leverage a key characteristic of GPVs: the ability to transfer visual knowledge across skills.
Ranked #2 on
Visual Question Answering (VQA)
on GRIT
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture
To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process.
PatchMatch-RL: Deep MVS with Pixelwise Depth, Normal, and Visibility
To overcome the challenge of the non-differentiable PatchMatch optimization that involves iterative sampling and hard decisions, we use reinforcement learning to minimize expected photometric cost and maximize likelihood of ground truth depth and normals.
Towards General Purpose Vision Systems
To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process.
Learning Curves for Analysis of Deep Networks
Learning curves model a classifier's test error as a function of the number of training samples.
Contrastive Learning for Weakly Supervised Phrase Grounding
Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions.
Boundary Cues for 3D Object Shape Recovery
Early work in computer vision considered a host of geometric cues for both shape reconstruction and recognition.
Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network.
Manhattan Room Layout Reconstruction from a Single 360 image: A Comparative Study of State-of-the-art Methods
Recent approaches for predicting layouts from 360 panoramas produce excellent results.
ViCo: Word Embeddings from Visual Co-occurrences
Through unsupervised clustering, supervised partitioning, and a zero-shot-like generalization analysis we show that our word embeddings complement text-only embeddings like GloVe by better representing similarities and differences between visual concepts that are difficult to obtain from text corpora alone.
Task-Assisted Domain Adaptation with Anchor Tasks
Our key idea to improve domain adaptation is to introduce a separate anchor task (such as facial landmarks) whose annotations can be obtained at no cost or are already available on both synthetic and real datasets.
Silhouette Guided Point Cloud Reconstruction beyond Occlusion
One major challenge in 3D reconstruction is to infer the complete shape geometry from partial foreground occlusions.
Reducing Overconfident Errors outside the Known Distribution
We compare a number of methods from related fields such as calibration and epistemic uncertainty modeling, as well as two proposed methods that reduce overconfident errors of samples from an unknown novel distribution without drastically increasing evaluation time: (1) G-distillation, training an ensemble of classifiers and then distill into a single model using both labeled and unlabeled examples, or (2) NCR, reducing prediction confidence based on its novelty detection score.
No-Frills Human-Object Interaction Detection: Factorization, Layout Encodings, and Training Techniques
We show that for human-object interaction detection a relatively simple factorized model with appearance and layout encodings constructed from pre-trained object detectors outperforms more sophisticated approaches.
Improved Structure from Motion Using Fiducial Marker Matching
In this paper, we present an incremental structure from motion (SfM) algorithm that signiï¬cantly outperforms existing algorithms when ï¬ducial markers are present in the scene, and that matches the performance of existing algorithms when no markers are present.
Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction
The goal of this paper is to compare surface-based and volumetric 3D object shape representations, as well as viewer-centered and object-centered reference frames for single-view 3D shape prediction.
Imagine This! Scripts to Compositions to Videos
Imagining a scene described in natural language with realistic layout and appearance of entities is the ultimate test of spatial, visual, and semantic world knowledge.
Improving Confidence Estimates for Unfamiliar Examples
In this paper, we compare and evaluate several methods to improve confidence estimates for unfamiliar and familiar samples.
LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image
We propose an algorithm to predict room layout from a single image that generalizes across panoramas and perspective images, cuboid layouts and more general layouts (e. g. L-shape room).
ChromaTag: A Colored Marker and Fast Detection Algorithm
Current fiducial marker detection algorithms rely on marker IDs for false positive rejection.
3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks
The success of various applications including robotics, digital content creation, and visualization demand a structured and abstract representation of the 3D world from limited sensor data.
Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning.
Geometry-Informed Material Recognition
Our goal is to recognize material categories using images and geometry information.
Learning without Forgetting
We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities.
Ranked #4 on
Domain 11-5
on Cityscapes
3DFS: Deformable Dense Depth Fusion and Segmentation for Object Reconstruction from a Handheld Camera
The resulting depth maps are then fused using a proposed implicit surface function that is robust to estimation error, producing a smooth surface reconstruction of the entire scene.
Learning to Localize Little Landmarks
We describe a method to find such landmarks by finding a sequence of latent landmarks, each with a prediction model.
Swapout: Learning an ensemble of deep architectures
When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers.
Where To Look: Focus Regions for Visual Question Answering
We present a method that learns to answer visual questions by selecting image regions relevant to the text-based query.
Part Localization using Multi-Proposal Consensus for Fine-Grained Categorization
We present a simple deep learning framework to simultaneously predict keypoint locations and their respective visibilities and use those to achieve state-of-the-art performance for fine-grained classification.
Completing 3D Object Shape From One Depth Image
Our goal is to recover a complete 3D model from a depth image of an object.
Learning a Sequential Search for Landmarks
We propose a general method to find landmarks in images of objects using both appearance and spatial context.
Predicting Complete 3D Models of Indoor Scenes
One major goal of vision is to infer physical models of objects, surfaces, and their layout from sensors.
Learning Collections of Part Models for Object Recognition
We propose a method to learn a diverse collection of discriminative parts from object bounding box annotations.
