Search Results for author:
Found 337 papers, 153 papers with code
Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation
Such training objective is sub-optimal when the target sequence is not perfect, e. g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available.
PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly
To close this gap, we introduce PhyBlock, a progressive benchmark designed to assess VLMs on physical understanding and planning through robotic 3D block assembly tasks.
Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs
Remarkable progress in 2D Vision-Language Models (VLMs) has spurred interest in extending them to 3D settings for tasks like 3D Question Answering, Dense Captioning, and Visual Grounding.
TreeRPO: Tree Relative Policy Optimization
To address this, we introduce \textbf{\name}, a novel method that estimates the mathematical expectations of rewards at various reasoning steps using tree sampling.
Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning
Large Vision-Language Models (LVLMs) have demonstrated impressive general capabilities across a wide range of multi-modal tasks.
MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents
In this work, we take a step further to build a comprehensive benchmark called MineAnyBuild, aiming to evaluate the spatial planning ability of open-world AI agents in the Minecraft game.
SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning
We present SeePhys, a large-scale multimodal benchmark for LLM reasoning grounded in physics questions ranging from middle school to PhD qualifying exams.
BridgeIV: Bridging Customized Image and Video Generation through Test-Time Autoregressive Identity Propagation
Both zero-shot and tuning-based customized text-to-image (CT2I) generation have made significant progress for storytelling content creation.
CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics
CombiBench is suitable for testing IMO solving capabilities since it includes all IMO combinatorial problems since 2000 (except IMO 2004 P3 as its statement contain an images).
RoBridge: A Hierarchical Architecture Bridging Cognition and Execution for General Robotic Manipulation
Operating robots in open-ended scenarios with diverse tasks is a crucial research and application direction in robotics.
SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
Evaluating the step-by-step reliability of large language model (LLM) reasoning, such as Chain-of-Thought, remains challenging due to the difficulty and cost of obtaining high-quality step-level supervision.
FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model
By combining the VLM enhanced with fine-grained region tokens and the time-dependent diffusion model, FireEdit demonstrates significant advantages in comprehending editing instructions and maintaining high semantic consistency.
Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models
Recent advancements in Large Video Language Models (LVLMs) have highlighted their potential for multi-modal understanding, yet evaluating their factual grounding in video contexts remains a critical unsolved challenge.
Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation
Experiments on both the discrete environments (R2R, REVERIE, and R4R datasets) and continuous environments (R2R-CE dataset) show the superior performance and impressive generalization ability of our method.
WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation
Experimental results demonstrate that WISA can effectively enhance the compatibility of T2V models with real-world physical laws, achieving a considerable improvement on the VideoPhy benchmark.
Can Atomic Step Decomposition Enhance the Self-structured Reasoning of Multimodal Large Models?
Different from existing methods that rely on structured templates or free-form paradigms, our method can not only generate cognitive CoT structures for various complex tasks but also mitigates the phenomenon of overthinking.
Structured Preference Optimization for Vision-Language Long-Horizon Task Planning
Existing methods for vision-language task planning excel in short-horizon tasks but often fall short in complex, long-horizon planning within dynamic environments.
UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting
However, adopting point clouds as 3D representation fails to fully capture the intricacies of the 3D world and exhibits a noticeable gap between the discrete points and the dense 2D pixels of images.
TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba
Nevertheless, training specialized subquadratic architectures from scratch for certain tasks is both resource-intensive and time-consuming.
ComposeAnyone: Controllable Layout-to-Human Generation with Decoupled Multimodal Conditions
Building on the success of diffusion models, significant advancements have been made in multimodal image generation tasks.
CatV2TON: Taming Diffusion Transformers for Vision-Based Virtual Try-On with Temporal Concatenation
Comprehensive experiments demonstrate that CatV2TON outperforms existing methods in both image and video try-on tasks, offering a versatile and reliable solution for realistic virtual try-ons across diverse scenarios.
DreamFit: Garment-Centric Human Generation via a Lightweight Anything-Dressing Encoder
However, existing methods often face a dilemma: lightweight approaches, such as adapters, are prone to generate inconsistent textures; while finetune-based methods involve high training costs and struggle to maintain the generalization capabilities of pretrained diffusion models, limiting their performance across diverse scenarios.
Dynamic Try-On: Taming Video Virtual Try-on with Dynamic Attention Mechanism
The primary challenges in this domain are twofold: (1) leveraging the garment encoder's capabilities in video try-on while lowering computational requirements; (2) ensuring temporal consistency in the synthesis of human body parts, especially during rapid movements.
RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation
Unlike existing VLN datasets, RoomTour3D leverages the scale and diversity of online videos to generate open-ended human walking trajectories and open-world navigable instructions.
DriveMM: All-in-One Large Multimodal Model for Autonomous Driving
Large Multimodal Models (LMMs) have demonstrated exceptional comprehension and interpretation capabilities in Autonomous Driving (AD) by incorporating large language models.
EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation
Remarkably, EACO also shows the potential critical ability in open-source MLLMs, demonstrating that EACO is a viable path to boost the competence of MLLMs.
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
In this paper, we propose PhysGame as a pioneering benchmark to evaluate physical commonsense violations in gameplay videos.
AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning
In this paper, we address the challenging task of multimodal mathematical reasoning by incorporating the ability of ``slow thinking" into multimodal large language models (MLLMs).
VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation
Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics.
StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration
Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.
Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
In terms of methodology, we propose Continual LLaVA, a rehearsal-free method tailored for continual instruction tuning in LVLMs.
PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation
To address this issue, we propose a PrIrmitive-driVen waypOinT-aware world model for Robotic manipulation (PIVOT-R) that focuses solely on the prediction of task-relevant waypoints.
Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes
Recent advancements in human motion synthesis have focused on specific types of motions, such as human-scene interaction, locomotion or human-human interaction, however, there is a lack of a unified system capable of generating a diverse combination of motion types.
Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars
To address these challenges, we introduce a novel two-stage interaction-aware GS framework that exploits cross-subject hand priors and refines 3D Gaussians in interacting areas.
UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation
We present UncertaintyRAG, a novel approach for long-context Retrieval-Augmented Generation (RAG) that utilizes Signal-to-Noise Ratio (SNR)-based span uncertainty to estimate similarity between text chunks.
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models.
Learning to Generalize Unseen Domains via Multi-Source Meta Learning for Text Classification
In this paper, we study the multi-source Domain Generalization of text classification and propose a framework to use multiple seen domains to train a model that can achieve high accuracy in an unseen domain.
Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models
Unlike prior works reliant on multiple off-the-shelf models, ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models.
Efficient Training of Large Vision Models via Advanced Automated Progressive Learning
This work provides a robust and scalable approach to efficient training of LVMs, with potential applications in a wide range of vision tasks.
Qihoo-T2X: An Efficient Proxy-Tokenized Diffusion Transformer for Text-to-Any-Task
The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity.
Making Large Language Models Better Planners with Reasoning-Decision Alignment
Inspired by the knowledge-driven nature of human driving, recent approaches explore the potential of large language models (LLMs) to improve understanding and decision-making in traffic scenarios.
EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation
Specifically, for the sketch-to-video generation task, EasyControl achieves an improvement of 152. 0 on FVD and 19. 9 on IS, respectively, in UCF101 compared with VideoComposer.
GarmentAligner: Text-to-Garment Generation via Retrieval-augmented Multi-level Corrections
General text-to-image models bring revolutionary innovation to the fields of arts, design, and media.
MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval
Text-Video Retrieval (TVR) aims to align and associate relevant video content with corresponding natural language queries.
FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance
Then, TAR refines the correlation matrix between cross-frame textual conditions and latent features along the time dimension.
APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation
By incorporating emotional support strategies, we aim to enrich the model's capabilities in both cognitive and affective empathy, leading to a more nuanced and comprehensive empathetic response.
DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models
Image-based 3D Virtual Try-ON (VTON) aims to sculpt the 3D human according to person and clothes images, which is data-efficient (i. e., getting rid of expensive 3D data) but challenging.
CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models
Virtual try-on methods based on diffusion models achieve realistic try-on effects but often replicate the backbone network as a ReferenceNet or use additional image encoders to process condition inputs, leading to high training and inference costs.
Contrastive Learning with Counterfactual Explanations for Radiology Report Generation
By leveraging this concept, CoFE can learn non-spurious visual representations by contrasting the representations between factual and counterfactual images.
OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling
Furthermore, to alleviate the data scarcity for optimization problems, and to bridge the gap between open-source LLMs on a small scale (e. g., Llama-3-8b) and closed-source LLMs (e. g., GPT-4), we further propose a data synthesis method namely ReSocratic.
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
High-resolution inputs enable Large Vision-Language Models (LVLMs) to discern finer visual details, enhancing their comprehension capabilities.
OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion
To address these challenges, we propose a novel unified open-vocabulary detection method called OV-DINO, which is pre-trained on diverse large-scale datasets with language-aware selective fusion in a unified framework.
Ranked #5 on
Zero-Shot Object Detection
on MSCOCO
(AP metric, using extra
training data)
Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
In this survey, we give a comprehensive exploration of the latest advancements in Embodied AI.
HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance
To address this issue, we introduce AbHuman, the first large-scale synthesized human benchmark focusing on anatomical anomalies.
Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation
LLM-based agents have demonstrated impressive zero-shot performance in vision-language navigation (VLN) task.
Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs
To address this problem, we propose $\texttt{Web2Code}$, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs.
FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving
In this paper, we propose FVEL, an interactive Formal Verification Environment with LLMs.
Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification
(b) A knowledge association module that fuses linguistic and biomedical knowledge into gene priors by transformer-based graph representation learning, capturing the intrinsic relationships between different genes' mutations.
S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
In this paper, we first summarize the current end-to-end 3D MOT framework by decomposing it into three constituent parts: query initialization, query propagation, and query matching.
AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generation
As cutting-edge Text-to-Image (T2I) generation models already excel at producing remarkable single images, an even more challenging task, i. e., multi-turn interactive image generation begins to attract the attention of related research communities.
RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter
Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries.
Correctable Landmark Discovery via Large Models for Vision-Language Navigation
To mitigate the noise in the priors due to the lack of visual constraints, we introduce a learnable cooccurrence scoring module, which corrects the importance of each cooccurrence according to actual observations for accurate landmark discovery.
VITON-DiT: Learning In-the-Wild Video Try-On from Human Dance Videos via Diffusion Transformers
To faithfully recover the clothing details, the extracted garment features are fused with the self-attention outputs of the denoising DiT and the ControlNet.
The SkatingVerse Workshop & Challenge: Methods and Results
The SkatingVerse Workshop & Challenge aims to encourage research in developing novel and accurate methods for human action understanding.
Proving Theorems Recursively
This approach allows the theorem to be tackled incrementally by outlining the overall theorem at the first level and then solving the intermediate conjectures at deeper levels.
DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data
Proof assistants like Lean have revolutionized mathematical proof verification, ensuring high accuracy and reliability.
Ranked #5 on
Automated Theorem Proving
on miniF2F-test
(using extra training data)
Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs
In this study, we propose an axiomatic system to define and quantify the precise memorization and in-context reasoning effects used by the large language model (LLM) for language generation.
ATG: Benchmarking Automated Theorem Generation for Generative Language Models
We conduct extensive experiments to investigate whether current LMs can generate theorems in the library and benefit the problem theorems proving.
MMTryon: Multi-Modal Multi-Reference Control for High-Quality Fashion Generation
Besides, to remove the segmentation dependency, MMTryon uses a parsing-free garment encoder and leverages a novel scalable data generation pipeline to convert existing VITON datasets to a form that allows MMTryon to be trained without requiring any explicit segmentation.
TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation
To address this issue, we introduce TheaterGen, a training-free framework that integrates large language models (LLMs) and text-to-image (T2I) models to provide the capability of multi-turn image generation.
ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving
ConsistentID comprises two key components: a multimodal facial prompt generator that combines facial features, corresponding facial descriptions and the overall facial context to enhance precision in facial details, and an ID-preservation network optimized through the facial attention localization strategy, aimed at preserving ID consistency in facial regions.
DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection
This is followed by a fine-tuning stage that leverages a small number of high-resolution samples to further enhance detection performance.
Ranked #3 on
Object Detection
on ODinW Full-Shot 13 Tasks
MLP Can Be A Good Transformer Learner
We identify that regarding the attention layer in bottom blocks, their subsequent MLP layers, i. e. two feed-forward layers, can elicit the same entropy quantity.
LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model
Specifically, an inter-layer attention module is designed to encourage information exchange and learning between layers, while a text-guided intra-layer attention module incorporates layer-specific prompts to direct the specific-content generation for each layer.
DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation
However, many of these works face challenges in identifying correct output modalities and generating coherent images accordingly as the number of output modalities increases and the conversations go deeper.
Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
To mitigate these issues, we propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information. Specifically, we leverage class embeddings as anchors due to their discrete and abstract nature, steering vision features toward class embeddings.
NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning
Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.
Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
For encouraging the agent to well capture the difference brought by perturbation, a perturbation-aware contrastive learning mechanism is further developed by contrasting perturbation-free trajectory encodings and perturbation-based counterparts.
DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions
Addressing this problem, we modularize a large search space into blocks with small search spaces and develop a family of models with the distilling neural architecture (DNA) techniques.
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Through extensive experiments across various datasets and scenes, we demonstrate the effectiveness of our approach in facilitating better interaction between LiDAR and camera modalities within a unified neural field.
MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data
Recent large language models (LLMs) have witnessed significant advancement in various tasks, including mathematical reasoning and theorem proving.
GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data
3D Shape represented as point cloud has achieve advancements in multimodal pre-training to align image and language descriptions, which is curial to object identification, classification, and retrieval.
MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
Embodied agents equipped with GPT as their brains have exhibited extraordinary decision-making and generalization abilities across various tasks.
Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models
BEV-InMLLM integrates multi-view, spatial awareness, and temporal semantics to enhance MLLMs' capabilities on NuInstruct tasks.
3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands
Subsequently, a feature fusion module that exploits the visibility of query points and mesh vertices is introduced to adaptively merge features of both hands, enabling the recovery of features in unseen areas.
BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving
Furthermore, there lack of well-formed retrieval datasets for effective evaluation.
Monocular 3D Hand Mesh Recovery via Dual Noise Estimation
However, due to the fixed hand topology and complex hand poses, current models are hard to generate meshes that are aligned with the image well.
Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model
Besides, we introduce a multi-denoiser framework for the advanced diffusion model to ease the learning of high-dimensional model and fully explore the generative potential of the diffusion model.
WarpDiffusion: Efficient Diffusion Model for High-Fidelity Virtual Try-on
Image-based Virtual Try-On (VITON) aims to transfer an in-shop garment image onto a target person.
DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
Especially for fidelity, our model has a powerful image retention ability and delivers the best results in UCF101 compared to other image-to-video models to our best knowledge.
CLOMO: Counterfactual Logical Modification with Large Language Models
In this study, we delve into the realm of counterfactual reasoning capabilities of large language models (LLMs).
AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations
Large Language Models prompting, such as using in-context demonstrations, is a mainstream technique for invoking LLMs to perform high-performance and solid complex reasoning (e. g., mathematical reasoning, commonsense reasoning), and has the potential for further human-machine collaborative scientific findings.
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Automated theorem proving (ATP) has become an appealing domain for exploring the reasoning ability of the recent successful generative language models.
DQ-LoRe: Dual Queries with Low Rank Approximation Re-ranking for In-Context Learning
Dual Queries first query LLM to obtain LLM-generated knowledge such as CoT, then query the retriever to obtain the final exemplars via both question and the knowledge.
LEGO-Prover: Neural Theorem Proving with Growing Libraries
Our ablation study indicates that these newly added skills are indeed helpful for proving theorems, resulting in an improvement from a success rate of 47. 1% to 50. 4%.
Ranked #1 on
Automated Theorem Proving
on miniF2F-valid
(Pass@100 metric)
Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images
In this paper, we propose a text-guided 3D faces generation method, refer as TG-3DFace, for generating realistic 3D faces using text guidance.
DiffCloth: Diffusion Based Garment Synthesis and Manipulation via Structural Cross-modal Semantic Alignment
Cross-modal garment synthesis and manipulation will significantly benefit the way fashion designers generate garments and modify their designs via flexible linguistic interfaces. Current approaches follow the general text-to-image paradigm and mine cross-modal relations via simple cross-attention modules, neglecting the structural correspondence between visual and textual representations in the fashion design domain.
GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training
Compared with the existing methods, GrowCLIP improves 2. 3% average top-1 accuracy on zero-shot image classification of 9 downstream tasks.
Coordinate Transformer: Achieving Single-stage Multi-person Mesh Recovery from Videos
Multi-person 3D mesh recovery from videos is a critical first step towards automatic perception of group behavior in virtual reality, physical therapy and beyond.
DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability
DiffDis first formulates the image-text discriminative problem as a generative diffusion process of the text embedding from the text encoder conditioned on the image.
CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation
Regarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly.
LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts
Thanks to the rapid development of diffusion models, unprecedented progress has been witnessed in image synthesis.
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation
To address this issue, we propose MixReorg, a novel and straightforward pre-training paradigm for semantic segmentation that enhances a model's ability to reorganize patches mixed across images, exploring both local visual relevance and global semantic coherence.
FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration
Specifically, the gradients, produced by the task heads and used to update the shared backbone, will be calibrated at the backbone's last layer to alleviate the task conflict.
Fashion Matrix: Editing Photos by Just Talking
The utilization of Large Language Models (LLMs) for the construction of AI systems has garnered significant attention across diverse fields.
Surfer: Progressive Reasoning with World Models for Robotic Manipulation
To conduct a comprehensive and systematic evaluation of the robot manipulation model in terms of language understanding and physical execution, we also created a robotic manipulation benchmark with progressive reasoning tasks, called SeaWave.
CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation
Understanding and following natural language instructions while navigating through complex, real-world environments poses a significant challenge for general-purpose robots.
UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning
Recent advances in vision-language pre-training have enabled machines to perform better in multimodal object discrimination (e. g., image-text semantic alignment) and image synthesis (e. g., text-to-image generation).
RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment
Recent advances in text-to-image diffusion models have achieved remarkable success in generating high-quality, realistic images from textual descriptions.
Boosting Visual-Language Models by Exploiting Hard Samples
In this work, we present HELIP, a cost-effective strategy tailored to enhance the performance of existing CLIP models without the need for training a model from scratch or collecting additional data.
Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining
In MOTOR, we combine two kinds of basic medical knowledge, i. e., general and specific knowledge, in a complementary manner to boost the general pretraining process.
LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields
We address this challenge by formulating, to the best of our knowledge, the first differentiable end-to-end LiDAR rendering framework, LiDAR-NeRF, leveraging a neural radiance field (NeRF) to facilitate the joint learning of geometry and the attributes of 3D points.
DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment
This paper presents DetCLIPv2, an efficient and scalable training framework that incorporates large-scale image-text pairs to achieve open-vocabulary object detection (OVD).
Ranked #6 on
Object Detection
on ODinW Full-Shot 13 Tasks
GP-VTON: Towards General Purpose Virtual Try-on via Collaborative Local-Flow Global-Parsing Learning
Specifically, compared with the previous global warping mechanism, LFGP employs local flows to warp garments parts individually, and assembles the local warped results via the global garment parsing, resulting in reasonable warped parts and a semantic-correct intact garment even with challenging inputs. On the other hand, our DGT training strategy dynamically truncates the gradient in the overlap area and the warped garment is no more required to meet the boundary constraint, which effectively avoids the texture squeezing problem.
CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks.
Ranked #3 on
Zero-shot 3D Point Cloud Classification
on ScanNetV2
Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation
To address the limitation, we propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning, named DCL.
CapDet: Unifying Dense Captioning and Open-World Detection Pretraining
Besides, our CapDet also achieves state-of-the-art performance on dense captioning tasks, e. g., 15. 44% mAP on VG V1. 2 and 13. 98% on the VG-COCO dataset.
Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving
Multi-task learning has emerged as a powerful paradigm to solve a range of tasks simultaneously with good efficiency in both computation resources and inference time.
Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation
Vision-Language Navigation (VLN) is a challenging task which requires an agent to align complex visual observations to language instructions to reach the goal position.
ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency
Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image.
CLIP2: Contrastive Language-Image-Point Pretraining From Real-World Point Cloud Data
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks.
Learning To Segment Every Referring Object Point by Point
Extensive experiments show that our model achieves 52. 06% in terms of accuracy (versus 58. 93% in fully supervised setting) on RefCOCO+@testA, when only using 1% of the mask annotations.
CTP:Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation
Regarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly.
NLIP: Noise-robust Language-Image Pre-training
Large-scale cross-modal pre-training paradigms have recently shown ubiquitous success on a wide range of downstream tasks, e. g., zero-shot classification, retrieval and image captioning.
UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
Naturally, we also present a unified multi-task Geometric Transformer framework, Geoformer, to tackle calculation and proving problems simultaneously in the form of sequence generation, which finally shows the reasoning ability can be improved on both two tasks by unifying formulation.
Ranked #5 on
Mathematical Reasoning
on PGPS9K
CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target.
3D-TOGO: Towards Text-Guided Cross-Category 3D Object Generation
Here, we make the first attempt to achieve generic text-guided cross-category 3D object generation via a new 3D-TOGO model, which integrates a text-to-views generation module and a views-to-3D generation module.
Towards Hard-pose Virtual Try-on via 3D-aware Global Correspondence Learning
Existing methods are restricted in this setting as they estimate garment warping flows mainly based on 2D poses and appearance, which omits the geometric prior of the 3D human body shape.
Structure-Preserving 3D Garment Modeling with Neural Sewing Machines
In this paper, we propose a novel Neural Sewing Machine (NSM), a learning-based framework for structure-preserving 3D garment modeling, which is capable of learning representations for garments with diverse shapes and topologies and is successfully applied to 3D garment reconstruction and controllable manipulation.
Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pre-trained VLMs and generating pseudo labels for unseen classes in a self-training manner.
MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure
To this end, we propose a comprehensive logical reasoning explanation form.
Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformers
Despite the success, its development and application on self-supervised vision transformers have been hindered by several barriers, including the high search cost, the lack of supervision, and the unsuitable search space.
MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library
A significant challenge facing researchers in the area of multi-agent reinforcement learning (MARL) pertains to the identification of a library that can offer fast and compatible development for multi-agent tasks and algorithm combinations, while obviating the need to consider compatibility issues.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning
Providing Emotional Support (ES) to soothe people in emotional distress is an essential capability in social interactions.
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection
We further design a concept dictionary~(with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept.
Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving
Aiming towards a holistic understanding of multiple downstream tasks simultaneously, there is a need for extracting features with better transferability.
ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
ARMANI discretizes an image into uniform tokens based on a learned cross-modal codebook in its first stage and uses a Transformer to model the distribution of image tokens for a real image given the tokens of the control signals in its second stage.
Composable Text Controls in Latent Space with ODEs
This paper proposes a new efficient approach for composable text operations in the compact latent space of text.
Ranked #2 on
Unsupervised Text Style Transfer
on Yelp
SiRi: A Simple Selective Retraining Mechanism for Transformer-based Visual Grounding
Particularly, SiRi conveys a significant principle to the research of visual grounding, i. e., a better initialized vision-language encoder would help the model converge to a better local minimum, advancing the performance accordingly.
PASTA-GAN++: A Versatile Framework for High-Resolution Unpaired Virtual Try-on
In this work, we take a step forwards to explore versatile virtual try-on solutions, which we argue should possess three main properties, namely, they should support unsupervised training, arbitrary garment categories, and controllable garment editing.
Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding
To bridge the gap between supervised semantic segmentation and real-world applications that acquires one model to recognize arbitrary new concepts, recent zero-shot segmentation attracts a lot of attention by exploring the relationships between unseen and seen object categories, yet requiring large amounts of densely-annotated data with diverse base classes.
Discourse-Aware Graph Networks for Textual Logical Reasoning
In this work, we propose logic structural-constraint modeling to solve the logical reasoning QA and introduce discourse-aware graph networks (DAGNs).
Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval
Our goal in this research is to study a more realistic environment in which we can conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories.
Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation
To endow models with the capability of incorporating expert knowledge, we propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG), in which clinical relation triples are injected into the visual features as prior knowledge to drive the decoding procedure.
Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL
In this study, we quantify the agent's behavior difference and build its relationship with the policy performance via {\bf Role Diversity}, a metric to measure the characteristics of MARL tasks.
ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts
Vision-Language Navigation (VLN) is a challenging task that requires an embodied agent to perform action-level modality alignment, i. e., make instruction-asked actions sequentially in complex visual environments.
Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection
There are two critical sensors for 3D perception in autonomous driving, the camera and the LiDAR.
Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning
In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train a task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs.
Knowledge Distillation via the Target-aware Transformer
To this end, we propose a novel one-to-all spatial matching knowledge distillation approach.
Unbiased Math Word Problems Benchmark for Mitigating Solving Bias
However, current solvers exist solving bias which consists of data bias and learning bias due to biased dataset and improper training strategy.
LogicSolver: Towards Interpretable Math Word Problem Solving with Logical Prompt-enhanced Learning
To address this issue and make a step towards interpretable MWP solving, we first construct a high-quality MWP dataset named InterMWP which consists of 11, 495 MWPs and annotates interpretable logical formulas based on algebraic knowledge as the grounded linguistic logic of each solution equation.
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism
To make ROSETTA automatically determine which experience is available and useful, a prototypical task correlation guided Gating Diversity Controller(GDC) is introduced to adaptively adjust the diversity of gates for the new task based on class-specific prototypes.
"My nose is running.""Are you also coughing?": Building A Medical Diagnosis Agent with Interpretable Inquiry Logics
In this paper, we explore how to bring interpretability to data-driven DSMD.
Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search
It is able to find top 0. 16\% and 0. 29\% architectures on average on two search spaces under the budget of only 50 models.
Dressing in the Wild by Watching Dance Videos
While significant progress has been made in garment transfer, one of the most applicable directions of human-centric image generation, existing works overlook the in-the-wild imagery, presenting severe garment-person misalignment as well as noticeable degradation in fine texture details.
Automated Progressive Learning for Efficient Training of Vision Transformers
First, we develop a strong manual baseline for progressive learning of ViTs, by introducing momentum growth (MoGrow) to bridge the gap brought by model growth.
Beyond Fixation: Dynamic Window Visual Transformer
Recently, a surge of interest in visual transformers is to reduce the computational cost by limiting the calculation of self-attention to a local window.
Laneformer: Object-aware Row-Column Transformers for Lane Detection
We present Laneformer, a conceptually simple yet powerful transformer-based architecture tailored for lane detection that is a long-standing research topic for visual perception in autonomous driving.
elBERto: Self-supervised Commonsense Learning for Question Answering
Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context.
CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving
One main reason that impedes the development of truly reliably self-driving systems is the lack of public datasets for evaluating the performance of object detectors on corner cases.
Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration
To improve the ability of fast cross-domain adaptation, we propose Prompt-based Environmental Self-exploration (ProbES), which can self-explore the environments by sampling trajectories and automatically generates structured instructions via a large-scale cross-modal pretrained model (CLIP).
Modern Augmented Reality: Applications, Trends, and Future Directions
Augmented reality (AR) is one of the relatively old, yet trending areas in the intersection of computer vision and computer graphics with numerous applications in several areas, from gaming and entertainment, to education and healthcare.
Revisiting Over-smoothing in BERT from the Perspective of Graph
Recently over-smoothing phenomenon of Transformer-based models is observed in both vision and language fields.
Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods.
Ranked #6 on
Image Retrieval
on MUGE Retrieval
Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation
Extensive experiments on two vision tasks, includ-ing ImageNet classification and Pascal VOC segmentation, demonstrate the superiority of our ICKD, which consis-tently outperforms many existing methods, advancing thestate-of-the-art in the fields of Knowledge Distillation.
BodyGAN: General-Purpose Controllable Neural Human Body Generation
This is because current methods mainly rely on a single pose/appearance model, which is limited in disentangling various poses and appearance in human images.
Contrastive Instruction-Trajectory Learning for Vision-Language Navigation
The vision-language navigation (VLN) task requires an agent to reach a target with the guidance of natural language instruction.
Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN
Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential.
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective.
UltraPose: Synthesizing Dense Pose with 1 Billion Points by Human-body Decoupling 3D Model
However, the popular DensePose-COCO dataset relies on a sophisticated manual annotation system, leading to severe limitations in acquiring the denser and more accurate annotated pose resources.
Image Comes Dancing with Collaborative Parsing-Flow Video Synthesis
The integration of human parsing and appearance flow effectively guides the generation of video frames with realistic appearance.
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset.
Ranked #1 on
Visual Question Answering (VQA)
on IconQA
DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference.
Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition
Specifically, we unify a pre-trained acoustic model (wav2vec 2. 0) and a language model (BERT) into an end-to-end trainable framework.
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
In this paper, we have a critical insight that improving the feed-forward network (FFN) in BERT has a higher gain than improving the multi-head attention (MHA) since the computational cost of FFN is 2$\sim$3 times larger than MHA.
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Despite the potential of multi-modal pre-training to learn highly discriminative feature representations from complementary data modalities, current progress is being slowed by the lack of large-scale modality-diverse datasets.
Voxel Transformer for 3D Object Detection
We present Voxel Transformer (VoTr), a novel and effective voxel-based Transformer backbone for 3D object detection from point clouds.
Ranked #3 on
3D Object Detection
on waymo vehicle
(L1 mAP metric)
Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection
To resolve the problems, we propose a novel second-stage module, named pyramid RoI head, to adaptively learn the features from the sparse points of interest.
Ranked #2 on
3D Object Detection
on waymo vehicle
(AP metric)
Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
We attribute this ranking correlation problem to the supernet training consistency shift, including feature shift and parameter shift.
FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark
Researchers have explored advanced methods from computer vision and natural language processing to incorporate medical domain knowledge for the generation of readable medical reports.
M3D-VTON: A Monocular-to-3D Virtual Try-On Network
Virtual 3D try-on can provide an intuitive and realistic view for online shopping and has a huge potential commercial value.
Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning
Medical imaging technologies, including computed tomography (CT) or chest X-Ray (CXR), are largely employed to facilitate the diagnosis of the COVID-19.
NASOA: Towards Faster Task-oriented Online Fine-tuning with a Zoo of Models
Fine-tuning from pre-trained ImageNet models has been a simple, effective, and popular approach for various computer vision tasks.
WAS-VTON: Warping Architecture Search for Virtual Try-on Network
Despite recent progress on image-based virtual try-on, current methods are constraint by shared warping networks and thus fail to synthesize natural try-on results when faced with clothing categories that require different warping operations.
Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining
In this paper, we investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval among fine-grained product categories.
Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
Specifically, we propose a Dynamic Reinforced Instruction Attacker (DR-Attacker), which learns to mislead the navigator to move to the wrong target by destroying the most instructive information in instructions at different timesteps.
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural language processing (NLP) tasks.
Ranked #10 on
Semantic Textual Similarity
on MRPC
Deep Learning for Embodied Vision Navigation: A Survey
A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc.
Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks
Previous math word problem solvers following the encoder-decoder paradigm fail to explicitly incorporate essential math symbolic constraints, leading to unexplainable and unreasonable predictions.
Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generation
Such training objective is sub-optimal when the target sequence is not perfect, e. g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available.
One Million Scenes for Autonomous Driving: ONCE Dataset
To facilitate future research on exploiting unlabeled data for 3D detection, we additionally provide a benchmark in which we reproduce and evaluate a variety of self-supervised and semi-supervised methods on the ONCE dataset.
SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
Experiments show that SODA10M can serve as a promising pre-training dataset for different self-supervised learning methods, which gives superior performance when fine-tuning with different downstream tasks (i. e., detection, semantic/instance segmentation) in autonomous driving domain.
Prototypical Graph Contrastive Learning
However, since for a query, its negatives are uniformly sampled from all graphs, existing methods suffer from the critical sampling bias issue, i. e., the negatives likely having the same semantic structure with the query, leading to performance degradation.
Vision-Language Navigation with Random Environmental Mixup
Then, we cross-connect the key views of different scenes to construct augmented scenes.
Ranked #38 on
Vision and Language Navigation
on VLN Challenge
Towards Quantifiable Dialogue Coherence Evaluation
To address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating standards.
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning
Therefore, we propose a Geometric Question Answering dataset GeoQA, containing 4, 998 geometric problems with corresponding annotated programs, which illustrate the solving process of the given problems.
Ranked #6 on
Mathematical Reasoning
on PGPS9K
TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
While existing NAS methods mostly design architectures on a single task, algorithms that look beyond single-task search are surging to pursue a more efficient and universal solution across various tasks.
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
We further propose a novel geometry solving approach with formal language and symbolic reasoning, called Interpretable Geometry Problem Solver (Inter-GPS).
Ranked #1 on
Mathematical Question Answering
on GeoS
SOON: Scenario Oriented Object Navigation with Graph-based Exploration
In this task, an agent is required to navigate from an arbitrary position in a 3D embodied environment to localize a target following a scene description.
Ranked #6 on
Visual Navigation
on SOON Test
