Search Results for author:
Found 39 papers, 22 papers with code
SOHES: Self-supervised Open-world Hierarchical Entity Segmentation
Using raw images as the sole training data, our method achieves unprecedented performance in self-supervised open-world segmentation, marking a significant milestone towards high-quality open-world entity segmentation in the absence of human-annotated masks.
MeshLRM: Large Reconstruction Model for High-Quality Mesh
This allows for end-to-end mesh reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering.
PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition
In contrast to the unidirectional fusion in previous works, we introduce a Dual-Modal Attention (DMA) that enables bidirectional interaction between textual and visual features, yielding context-aware label representations and semantic-related visual representations, which are subsequently used to calculate similarities and generate final predictions for all labels.
Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM
We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image.
Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
To this end, we introduce Carve3D, an improved RLFT algorithm coupled with a novel Multi-view Reconstruction Consistency (MRC) metric, to enhance the consistency of multi-view diffusion models.
Compound Text-Guided Prompt Tuning via Image-Adaptive Cues
We introduce text supervision to the optimization of prompts, which enables two benefits: 1) releasing the model reliance on the pre-defined category names during inference, thereby enabling more flexible prompt generation; 2) reducing the number of inputs to the text encoder, which decreases GPU memory consumption significantly.
PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction
We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing a 3D object from a few unposed images even with little visual overlap, while simultaneously estimating the relative camera poses in ~1. 3 seconds on a single A100 GPU.
DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model
We propose \textbf{DMV3D}, a novel 3D generation approach that uses a transformer-based 3D large reconstruction model to denoise multi-view diffusion.
Federated Skewed Label Learning with Logits Fusion
By fusing the logits of the two models, the private weak learner can capture the variance of different data, regardless of their category.
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model
Text-to-3D with diffusion models has achieved remarkable progress in recent years.
LRM: Large Reconstruction Model for Single Image to 3D
We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds.
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
Boosting Punctuation Restoration with Data Generation and Reinforcement Learning
Punctuation restoration is an important task in automatic speech recognition (ASR) which aim to restore the syntactic structure of generated ASR texts to improve readability.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Learning Navigational Visual Representations with Semantic Map Supervision
Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot.
DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents
In this work, we propose DocumentCLIP, a salience-aware contrastive learning framework to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents.
Graph Propagation Transformer for Graph Representation Learning
The core insight of our method is to fully consider the information propagation among nodes and edges in a graph when building the attention module in the transformer blocks.
Ranked #2 on
Graph Regression
on PCQM4M-LSC
(Validation MAE metric)
Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters
Our tiny-attention adapter learns to modify the hidden states at each position directly conditioned on the hidden states at all the other positions, which is missed by the previously proposed adapters.
Adversarial Attacks on ASR Systems: An Overview
With the development of hardware and algorithms, ASR(Automatic Speech Recognition) systems evolve a lot.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations
Empirically, on the Room-Across-Room dataset, we show that our multilingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation.
EnvEdit: Environment Editing for Vision-and-Language Navigation
Training on these edit-augmented environments prevents the agent from overfitting to existing environments and helps generalize better to new, unseen environments.
Ranked #2 on
Vision and Language Navigation
on RxR
(using extra training data)
How Much Can CLIP Benefit Vision-and-Language Tasks?
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.
Ranked #4 on
Vision and Language Navigation
on RxR
(using extra training data)
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset.
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e. g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.
Ranked #10 on
Action Recognition
on Diving-48
Unifying Vision-and-Language Tasks via Text Generation
On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models.
Ranked #3 on
Image Captioning
on nocaps val
ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
During this task, the agent (similar to a PokeMON GO player) is asked to find and collect different target objects one-by-one by navigating based on natural language instructions in a complex, realistic outdoor environment, but then also ARRAnge the collected objects part-by-part in an egocentric grid-layout environment.
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
We find that the main reason hindering this exploration is the large divergence in magnitude and distributions between the visually-grounded language datasets and pure-language corpora.
MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
Phrase localization is a task that studies the mapping from textual phrases to regions of an image.
RelativeNAS: Relative Neural Architecture Search via Slow-Fast Learning
Despite the remarkable successes of Convolutional Neural Networks (CNNs) in computer vision, it is time-consuming and error-prone to manually design a CNN.
Diagnosing the Environment Bias in Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations.
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve.
Modality-Balanced Models for Visual Dialogue
The Visual Dialog task requires a model to exploit both image and conversational context information to generate the next response to the dialogue.
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder.
Ranked #1 on
Visual Question Answering (VQA)
on VizWiz 2018
Expressing Visual Relationships via Language
To push forward the research in this direction, we first introduce a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions.
Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning
Our results show the feasibility of a robot learning commonsense knowledge automatically from web-based textual corpora, and the power of learned commonsense reasoning models in enabling a robot to autonomously perform tasks based on incomplete natural language instructions.
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions.
Ranked #1 on
Vision-Language Navigation
on Room2Room
Object Ordering with Bidirectional Matchings for Visual Reasoning
Visual reasoning with compositional natural language instructions, e. g., based on the newly-released Cornell Natural Language Visual Reasoning (NLVR) dataset, is a challenging task, where the model needs to have the ability to create an accurate mapping between the diverse phrases and the several objects placed in complex arrangements in the image.
Source-Target Inference Models for Spatial Instruction Understanding
Models that can execute natural language instructions for situated robotic tasks such as assembly and navigation have several useful applications in homes, offices, and remote scenarios.
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions.
