Search Results for author:
Found 66 papers, 30 papers with code
Test-Time Training Done Right
We validate our approach across diverse modalities and tasks, including novel view synthesis with image set, language models, and auto-regressive video diffusion.
Neural BRDF Importance Sampling by Reparameterization
Neural bidirectional reflectance distribution functions (BRDFs) have emerged as popular material representations for enhancing realism in physically-based rendering.
RayZer: A Self-supervised Large View Synthesis Model
The emerging 3D awareness of RayZer is attributed to two key factors.
Gaussian Mixture Flow Matching Models
Diffusion models approximate the denoising distribution as a Gaussian and predict its mean, whereas flow matching models reparameterize the Gaussian mean as flow velocity.
Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transport
Identifying multiple novel classes in an image, known as open-vocabulary multi-label recognition, is a challenging task in computer vision.
VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation
To support diverse tasks and complex instructions, we employ a curriculum learning strategy: first aligning the MLLM and video diffusion model with large-scale instructional image editing data, followed by end-to-end fine-tuning on high-quality multitask video data.
RigAnything: Template-Free Autoregressive Rigging for Diverse 3D Assets
To achieve this, we organize the joints in a breadth-first search (BFS) order, enabling the skeleton to be defined as a sequence of 3D locations and the parent index.
Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models
Recently, Large language models (LLMs) with in-context learning have demonstrated remarkable potential in handling neural machine translation.
Large-scale Multi-view Tensor Clustering with Implicit Linear Kernels
On its basis, a novel large-scale multi-view tensor clustering method is developed by incorporating the pair-wise similarities with implicit linear kernel function.
MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data
We propose scaling up 3D scene reconstruction by training with synthesized data.
LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers
In this paper, we show that performing the full computation of the model at each diffusion step is unnecessary, as some computations can be skipped by lazily reusing the results of previous steps.
Numerical Pruning for Efficient Autoregressive Models
Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing.
Turbo3D: Ultra-fast Text-to-3D Generation
By shifting the Gaussian reconstructor's inputs from pixel space to latent space, we eliminate the extra image decoding time and halve the transformer sequence length for maximum efficiency.
HunyuanVideo: A Systematic Framework For Large Video Generative Models
In this report, we introduce HunyuanVideo, an innovative open-source video foundation model that demonstrates performance in video generation comparable to, or even surpassing, that of leading closed-source models.
RandAR: Decoder-only Autoregressive Visual Generation in Random Orders
We introduce RandAR, a decoder-only visual autoregressive (AR) model capable of generating images in arbitrary token orders.
Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors
We present Buffer Anytime, a framework for estimation of depth and normal maps (which we call geometric buffers) from video that eliminates the need for paired video--depth and video--normal training data.
Generating 3D-Consistent Videos from Unposed Internet Photos
We design a self-supervised method that takes advantage of the consistency of videos and variability of multiview internet photos to train a scalable, 3D-aware video model without any 3D annotations such as camera parameters.
Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats
Unlike previous feed-forward models that are limited to processing 1~4 input images and can only reconstruct a small portion of a large scene, Long-LRM reconstructs the entire scene in a single feed-forward step.
Progressive Autoregressive Video Diffusion Models
In this work, we introduce a more natural formulation of autoregressive long video generation by revisiting the noise level assumption in video diffusion models.
RelitLRM: Generative Relightable Radiance for Large Reconstruction Models
We propose RelitLRM, a Large Reconstruction Model (LRM) for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting.
SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition
With the help of flexible prompting and gated alignments, SSPA is generalizable to specific domains.
Identifying Speakers in Dialogue Transcripts: A Text-based Approach Using Pretrained Language Models
We introduce an approach to identifying speaker names in dialogue transcripts, a crucial task for enhancing content accessibility and searchability in digital media archives.
LRM-Zero: Training Large Reconstruction Models with Synthesized Data
We demonstrate that our LRM-Zero, trained with our fully synthesized Zeroverse, can achieve high visual quality in the reconstruction of real-world objects, competitive with models trained on Objaverse.
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
We propose GS-LRM, a scalable large reconstruction model that can predict high-quality 3D Gaussian primitives from 2-4 posed sparse images in 0. 23 seconds on single A100 GPU.
Pre-trained Vision-Language Models Learn Discoverable Visual Concepts
Do vision-language models (VLMs) pre-trained to caption an image of a "durian" learn visual concepts such as "brown" (color) and "spiky" (texture) at the same time?
MeshLRM: Large Reconstruction Model for High-Quality Meshes
This allows for end-to-end mesh reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering.
SOHES: Self-supervised Open-world Hierarchical Entity Segmentation
Using raw images as the sole training data, our method achieves unprecedented performance in self-supervised open-world segmentation, marking a significant milestone towards high-quality open-world entity segmentation in the absence of human-annotated masks.
PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition
In contrast to the unidirectional fusion in previous works, we introduce a Dual-Modal Attention (DMA) that enables bidirectional interaction between textual and visual features, yielding context-aware label representations and semantic-related visual representations, which are subsequently used to calculate similarities and generate final predictions for all labels.
Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM
We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image.
Building Vision-Language Models on Solid Foundations with Masked Distillation
Recent advancements in Vision-Language Models (VLMs) have marked a significant leap in bridging the gap between computer vision and natural language processing.
Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
To this end, we introduce Carve3D, an improved RLFT algorithm coupled with a novel Multi-view Reconstruction Consistency (MRC) metric, to enhance the consistency of multi-view diffusion models.
Compound Text-Guided Prompt Tuning via Image-Adaptive Cues
We introduce text supervision to the optimization of prompts, which enables two benefits: 1) releasing the model reliance on the pre-defined category names during inference, thereby enabling more flexible prompt generation; 2) reducing the number of inputs to the text encoder, which decreases GPU memory consumption significantly.
PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction
We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing a 3D object from a few unposed images even with little visual overlap, while simultaneously estimating the relative camera poses in ~1. 3 seconds on a single A100 GPU.
DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model
We propose \textbf{DMV3D}, a novel 3D generation approach that uses a transformer-based 3D large reconstruction model to denoise multi-view diffusion.
Federated Skewed Label Learning with Logits Fusion
By fusing the logits of the two models, the private weak learner can capture the variance of different data, regardless of their category.
LRM: Large Reconstruction Model for Single Image to 3D
We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds.
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
Boosting Punctuation Restoration with Data Generation and Reinforcement Learning
Punctuation restoration is an important task in automatic speech recognition (ASR) which aim to restore the syntactic structure of generated ASR texts to improve readability.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Learning Navigational Visual Representations with Semantic Map Supervision
Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot.
DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents
In this work, we propose DocumentCLIP, a salience-aware contrastive learning framework to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents.
Graph Propagation Transformer for Graph Representation Learning
The core insight of our method is to fully consider the information propagation among nodes and edges in a graph when building the attention module in the transformer blocks.
Ranked #2 on
Graph Regression
on PCQM4M-LSC
(Validation MAE metric)
Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters
Our tiny-attention adapter learns to modify the hidden states at each position directly conditioned on the hidden states at all the other positions, which is missed by the previously proposed adapters.
Adversarial Attacks on ASR Systems: An Overview
With the development of hardware and algorithms, ASR(Automatic Speech Recognition) systems evolve a lot.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations
Empirically, on the Room-Across-Room dataset, we show that our multilingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation.
EnvEdit: Environment Editing for Vision-and-Language Navigation
Training on these edit-augmented environments prevents the agent from overfitting to existing environments and helps generalize better to new, unseen environments.
Ranked #2 on
Vision and Language Navigation
on RxR
(using extra training data)
How Much Can CLIP Benefit Vision-and-Language Tasks?
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.
Ranked #4 on
Vision and Language Navigation
on RxR
(using extra training data)
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset.
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e. g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.
Ranked #11 on
Action Recognition
on Diving-48
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information
One key challenge in this task is to ground instructions with the current visual information that the agent perceives.
Unifying Vision-and-Language Tasks via Text Generation
On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models.
Ranked #3 on
Image Captioning
on nocaps val
ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
During this task, the agent (similar to a PokeMON GO player) is asked to find and collect different target objects one-by-one by navigating based on natural language instructions in a complex, realistic outdoor environment, but then also ARRAnge the collected objects part-by-part in an egocentric grid-layout environment.
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
We find that the main reason hindering this exploration is the large divergence in magnitude and distributions between the visually-grounded language datasets and pure-language corpora.
MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
Phrase localization is a task that studies the mapping from textual phrases to regions of an image.
RelativeNAS: Relative Neural Architecture Search via Slow-Fast Learning
Despite the remarkable successes of Convolutional Neural Networks (CNNs) in computer vision, it is time-consuming and error-prone to manually design a CNN.
Diagnosing the Environment Bias in Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations.
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve.
Modality-Balanced Models for Visual Dialogue
The Visual Dialog task requires a model to exploit both image and conversational context information to generate the next response to the dialogue.
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder.
Ranked #1 on
Visual Question Answering (VQA)
on VizWiz 2018
Expressing Visual Relationships via Language
To push forward the research in this direction, we first introduce a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions.
Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning
Our results show the feasibility of a robot learning commonsense knowledge automatically from web-based textual corpora, and the power of learned commonsense reasoning models in enabling a robot to autonomously perform tasks based on incomplete natural language instructions.
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions.
Ranked #1 on
Vision-Language Navigation
on Room2Room
Object Ordering with Bidirectional Matchings for Visual Reasoning
Visual reasoning with compositional natural language instructions, e. g., based on the newly-released Cornell Natural Language Visual Reasoning (NLVR) dataset, is a challenging task, where the model needs to have the ability to create an accurate mapping between the diverse phrases and the several objects placed in complex arrangements in the image.
Source-Target Inference Models for Spatial Instruction Understanding
Models that can execute natural language instructions for situated robotic tasks such as assembly and navigation have several useful applications in homes, offices, and remote scenarios.
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions.
