Search Results for author:
Found 269 papers, 121 papers with code
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach
We investigate a principle way to progressively mine discriminative object regions using classification networks to address the weakly-supervised semantic segmentation problems.
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents.
Image-derived generative modeling of pseudo-macromolecular structures - towards the statistical assessment of Electron CryoTomography template matching
Cellular Electron CryoTomography (CECT) is a 3D imaging technique that captures information about the structure and spatial organization of macromolecular complexes within single cells, in near-native state and at sub-molecular resolution.
Dilated Temporal Relational Adversarial Network for Generic Video Summarization
Specifically, DTR-GAN learns a dilated temporal relational generator and a discriminator with three-player loss in an adversarial manner.
ConnNet: A Long-Range Relation-Aware Pixel-Connectivity Network for Salient Segmentation
We argue that semantic salient segmentation can instead be effectively resolved by reformulating it as a simple yet intuitive pixel-pair based connectivity prediction task.
Visual Question Reasoning on General Dependency Tree
This network comprises of two collaborative modules: i) an adversarial attention module to exploit the local visual evidence for each word parsed from the question; ii) a residual composition module to compose the previously mined evidence.
Dynamic-structured Semantic Propagation Network
Moreoever, we demonstrate a universal segmentation model that is jointly trained on diverse datasets can surpass the performance of the common fine-tuning scheme for exploiting multiple domain knowledge.
Ranked #61 on
Semantic Segmentation
on Cityscapes test
Deep Structured Scene Parsing by Learning with Image Descriptions
This paper addresses a fundamental problem of scene understanding: How to parse the scene image into a structured configuration (i. e., a semantic object hierarchy with object interaction relations) that finely accords with human perception.
Learning to Segment Human by Watching YouTube
An intuition on human segmentation is that when a human is moving in a video, the video-context (e. g., appearance and motion clues) may potentially infer reasonable mask information for the whole human body.
Deep learning based supervised semantic segmentation of Electron Cryo-Subtomograms
Cellular Electron Cryo-Tomography (CECT) is a powerful imaging technique for the 3D visualization of cellular structure and organization at submolecular resolution.
Real-to-Virtual Domain Unification for End-to-End Autonomous Driving
In the spectrum of vision-based autonomous driving, vanilla end-to-end models are not interpretable and suboptimal in performance, while mediated perception models require additional intermediate representations such as segmentation masks or detection bounding boxes, whose annotation can be prohibitively expensive as we move to a larger scale.
Unsupervised Object-Level Video Summarization with Online Motion Auto-Encoder
Unsupervised video summarization plays an important role on digesting, browsing, and searching the ever-growing videos every day, and the underlying fine-grained semantic and motion information (i. e., objects of interest and their key motions) in online videos has been barely touched.
Nonparametric Variational Auto-encoders for Hierarchical Representation Learning
In this work, we propose hierarchical nonparametric variational autoencoders, which combines tree-structured Bayesian nonparametric priors with VAEs, to enable infinite flexibility of the latent representation space.
Attention-Aware Face Hallucination via Deep Reinforcement Learning
Face hallucination is a domain-specific super-resolution problem with the goal to generate high-resolution (HR) faces from low-resolution (LR) input images.
Dual Motion GAN for Future-Flow Embedded Video Prediction
To make both synthesized future frames and flows indistinguishable from reality, a dual adversarial training method is proposed to ensure that the future-flow prediction is able to help infer realistic future-frames, while the future-frame prediction in turn leads to realistic optical flows.
Temporal Dynamic Graph LSTM for Action-driven Video Object Detection
A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label".
Ranked #3 on
Weakly Supervised Object Detection
on Charades
Generative Semantic Manipulation with Contrasting GAN
Generative Adversarial Networks (GANs) have recently achieved significant improvement on paired/unpaired image-to-image translation, such as photo$\rightarrow$ sketch and artist painting style transfer.
Ranked #4 on
Facial Expression Translation
on CelebA
Recurrent 3D Pose Sequence Machines
3D human articulated pose recovery from monocular image sequences is very challenging due to the diverse appearances, viewpoints, occlusions, and also the human 3D pose is inherently ambiguous from the monocular imagery.
Ranked #20 on
3D Human Pose Estimation
on HumanEva-I
Perceptual Generative Adversarial Networks for Small Object Detection
In this work, we address the small object detection problem by developing a single architecture that internally lifts representations of small objects to "super-resolved" ones, achieving similar characteristics as large objects and thus more discriminative for detection.
Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
We show that Poseidon enables Caffe and TensorFlow to achieve 15. 5x speed-up on 16 single-GPU machines, even with limited bandwidth (10GbE) and the challenging VGG19-22K network for image classification.
SCAN: Structure Correcting Adversarial Network for Organ Segmentation in Chest X-rays
Through this adversarial process the critic network learns the higher order structures and guides the segmentation model to achieve realistic segmentation outcomes.
Recurrent Topic-Transition GAN for Visual Paragraph Generation
The proposed Recurrent Topic-Transition Generative Adversarial Network (RTT-GAN) builds an adversarial framework between a structured paragraph generator and multi-level paragraph discriminators.
Generative Adversarial Network
Image Paragraph Captioning
+1
ZM-Net: Real-time Zero-shot Image Manipulation Network
We cast this problem as manipulating an input image according to a parametric model whose key parameters can be conditionally generated from any guiding signal (even unseen ones).
Interpretable Structure-Evolving LSTM
Instead of learning LSTM models over the pre-fixed structures, we propose to further learn the intermediate interpretable multi-level graph structures in a progressive and stochastic way from data during the LSTM network optimization.
Tree-Structured Reinforcement Learning for Sequential Object Localization
Therefore, Tree-RL can better cover different objects with various scales which is quite appealing in the context of object proposal.
Peak-Piloted Deep Network for Facial Expression Recognition
Objective functions for training of deep networks for face-related recognition tasks, such as facial expression recognition (FER), usually consider each sample independently.
Ranked #2 on
Facial Expression Recognition (FER)
on Oulu-CASIA
Multi-stage Object Detection with Group Recursive Learning
Most of existing detection pipelines treat object proposals independently and predict bounding box locations and classification scores over them separately.
Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning
In this paper, we propose a novel inference-embedded multi-task learning framework for predicting human pose from still depth images, which is implemented with a deep architecture of neural networks.
Is Faster R-CNN Doing Well for Pedestrian Detection?
Detecting pedestrian has been arguably addressed as a special topic beyond general object detection.
Ranked #19 on
Pedestrian Detection
on Caltech
Scale-aware Pixel-wise Object Proposal Networks
In particular, in order to improve the localization accuracy, a fully convolutional network is employed which predicts locations of object proposals for each pixel.
Scale-aware Fast R-CNN for Pedestrian Detection
Taking pedestrian detection as an example, we illustrate how we can leverage this philosophy to develop a Scale-Aware Fast R-CNN (SAF R-CNN) framework.
Ranked #23 on
Pedestrian Detection
on Caltech
Geometric Scene Parsing with Hierarchical LSTM
This paper addresses the problem of geometric scene parsing, i. e. simultaneously labeling geometric surfaces (e. g. sky, ground and vertical plane) and determining the interaction relations (e. g. layering, supporting, siding and affinity) between main regions.
Attentive Contexts for Object Detection
We provide preliminary answers to these questions through developing a novel Attention to Context Convolution Neural Network (AC-CNN) based object detection model.
Semantic Object Parsing with Graph LSTM
By taking the semantic object parsing task as an exemplar application scenario, we propose the Graph Long Short-Term Memory (Graph LSTM) network, which is the generalization of LSTM from sequential data or multi-dimensional data to general graph-structured data.
Semantic Object Parsing with Local-Global Long Short-Term Memory
The long chains of sequential computation by stacked LG-LSTM layers also enable each pixel to sense a much larger region for inference benefiting from the memorization of previous dependencies in all positions along all dimensions.
Proposal-free Network for Instance-level Object Segmentation
Instance-level object segmentation is an important yet under-explored task.
Computational Baby Learning
Then the concept detector can be fine-tuned based on these new instances.
Matching-CNN Meets KNN: Quasi-Parametric Human Parsing
Under the classic K Nearest Neighbor (KNN)-based nonparametric framework, the parametric Matching Convolutional Neural Network (M-CNN) is proposed to predict the matching confidence and displacements of the best matched region in the testing image for a particular semantic region in one KNN image.
Complex Background Subtraction by Pursuing Dynamic Spatio-Temporal Models
Although it has been widely discussed in video surveillance, background subtraction is still an open problem in the context of complex scenarios, e. g., dynamic backgrounds, illumination variations, and indistinct foreground objects.
Learning Latent Spatio-Temporal Compositional Model for Human Action Recognition
Action recognition is an important problem in multimedia understanding.
Deep Generative Models with Learnable Knowledge Constraints
The broad set of deep generative models (DGMs) has achieved remarkable advances.
CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
To our knowledge, this is the first successful case of the learned driving policy through reinforcement learning in the high-fidelity simulator, which performs better-than supervised imitation learning.
Geometric Generalization Based Zero-Shot Learning Dataset Infinite World: Simple Yet Powerful
Analogously, this paper introduces geometric generalization based zero-shot learning tests to measure the rapid learning ability and the internal consistency of deep generative models.
Unsupervised Domain Adaptation for Automatic Estimation of Cardiothoracic Ratio
Specifically, we propose a model that enforces our intuition that prediction masks should be domain independent.
Query-Conditioned Three-Player Adversarial Network for Video Summarization
Video summarization plays an important role in video understanding by selecting key frames/shots.
Reinforced Auto-Zoom Net: Towards Accurate and Fast Breast Cancer Segmentation in Whole-slide Images
Motivated by the zoom-in operation of a pathologist using a digital microscope, RAZN learns a policy network to decide whether zooming is required in a given region of interest.
Jointly Deep Multi-View Learning for Clustering Analysis
To our best knowledge, this is the first work to model the multi-view clustering in a deep joint framework, which will provide a meaningful thinking in unsupervised multi-view learning.
Interpretable Visual Question Answering by Reasoning on Dependency Trees
Collaborative reasoning for understanding image-question pairs is a very critical but underexplored topic in interpretable visual question answering systems.
Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis
Despite remarkable advances in image synthesis research, existing works often fail in manipulating images under the context of large geometric transformations.
Texar: A Modularized, Versatile, and Extensible Toolbox for Text Generation
The features make Texar particularly suitable for technique sharing and generalization across different text generation applications.
Reinforcement Cutting-Agent Learning for Video Object Segmentation
In this paper, we formulate this problem as a Markov Decision Process, where agents are learned to segment object regions under a deep reinforcement learning framework.
RCAA: Relational Context-Aware Agents for Person Search
In this paper, we address this problem by training relational context-aware agents which learn the actions to localize the target person from the gallery of whole scene images.
Generative Semantic Manipulation with Mask-Contrasting GAN
Despite the promising results on paired/unpaired image-to-image translation achieved by Generative Adversarial Networks (GANs), prior works often only transfer the low-level information (e. g. color or texture changes), but fail to manipulate high-level semantic meanings (e. g., geometric structure or content) of different object regions.
Adversarial Geometry-Aware Human Motion Prediction
We explore an approach to forecasting human motion in a few milliseconds given an input 3D skeleton sequence based on a recurrent encoder-decoder framework.
Graph Transformer
Graph neural networks (GNN) have gained increasing research interests as a mean to the challenging goal of robust and universal graph learning.
Towards Computational Baby Learning: A Weakly-Supervised Approach for Object Detection
Then the concept detector can be fine-tuned based on these new instances.
Human Parsing With Contextualized Convolutional Neural Network
In this work, we address the human parsing task with a novel Contextualized Convolutional Neural Network (Co-CNN) architecture, which well integrates the cross-layer context, global image-level context, within-super-pixel context and cross-super-pixel neighborhood context into a unified network.
Towards Multi-pose Guided Virtual Try-on Network
Given an input person image, a desired clothes image, and a desired pose, the proposed Multi-pose Guided Virtual Try-on Network (MG-VTON) can generate a new person image after fitting the desired clothes into the input image and manipulating human poses.
Ranked #1 on
Virtual Try-on
on Deep-Fashion
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions.
Learning Personalized Modular Network Guided by Structured Knowledge
Learning semantic configurations and activation of modules to align well with structured knowledge can be regarded as a decision-making procedure, which is solved by a new graph-based reinforcement learning algorithm.
Fashion Editing with Adversarial Parsing Learning
Interactive fashion image manipulation, which enables users to edit images with sketches and color strokes, is an interesting research problem with great application value.
Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
Explanation and high-order reasoning capabilities are crucial for real-world visual question answering with diverse levels of inference complexity (e. g., what is the dog that is near the girl playing with?)
Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning
Resembling the rapid learning capability of human, few-shot learning empowers vision systems to understand new concepts by training with few samples.
Ranked #19 on
Few-Shot Object Detection
on MS-COCO (30-shot)
Layout-Graph Reasoning for Fashion Landmark Detection
Each Layout-Graph Reasoning(LGR) layer aims to map feature representations into structural graph nodes via a Map-to-Node module, performs reasoning over structural graph nodes to achieve global layout coherency via a layout-graph reasoning module, and then maps graph nodes back to enhance feature representations via a Node-to-Map module.
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information.
Ranked #13 on
Vision and Language Navigation
on VLN Challenge
SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection
In this paper, we present a two-stage coarse-to-fine searching strategy named Structural-to-Modular NAS (SM-NAS) for searching a GPU-friendly design of both an efficient combination of modules and better modular-level architecture for object detection.
Universal-RCNN: Universal Object Detector via Transferable Graph R-CNN
Finally, an InterDomain Transfer Module is proposed to exploit diverse transfer dependencies across all domains and enhance the regional feature representation by attending and transferring semantic contexts globally.
Towards Causality-Aware Inferring: A Sequential Discriminative Approach for Medical Diagnosis
To eliminate this bias and inspired by the propensity score matching technique with causal diagram, we propose a propensity-based patient simulator to effectively answer unrecorded inquiry by drawing knowledge from the other records; Bias (ii) inherently comes along with the passively collected data, and is one of the key obstacles for training the agent towards "learning how" rather than "remembering what".
ElixirNet: Relation-aware Network Architecture Adaptation for Medical Lesion Detection
Is a hand-crafted detection network tailored for natural image undoubtedly good enough over a discrepant medical lesion domain?
Linguistically Driven Graph Capsule Network for Visual Question Reasoning
Inspired by the property of a capsule network that can carve a tree structure inside a regular convolutional neural network (CNN), we propose a hierarchical compositional reasoning model called the "Linguistically driven Graph Capsule Network", where the compositional process is guided by the linguistic parse tree.
Bidirectional Graph Reasoning Network for Panoptic Segmentation
We introduce a Bidirectional Graph Reasoning Network (BGRNet), which incorporates graph structure into the conventional panoptic segmentation network to mine the intra-modular and intermodular relations within and between foreground things and background stuff classes.
CATCH: Context-based Meta Reinforcement Learning for Transferrable Architecture Search
In spite of its remarkable progress, many algorithms are restricted to particular search spaces.
Ranked #13 on
Neural Architecture Search
on NAS-Bench-201, ImageNet-16-120
(Accuracy (Val) metric)
Erasure for Advancing: Dynamic Self-Supervised Learning for Commonsense Reasoning
To solve this issue, we propose a novel DynamIc Self-sUperviSed Erasure (DISUSE) which adaptively erases redundant and artifactual clues in the context and questions to learn and establish the correct corresponding pair relations between the questions and their clues.
CAT-SAC: Soft Actor-Critic with Curiosity-Aware Entropy Temperature
The curiosity is added to the target entropy to increase the entropy temperature for unfamiliar states and decrease the target entropy for familiar states.
NASOA: Towards Faster Task-oriented Online Fine-tuning
The resulting model zoo is more training efficient than SOTA NAS models, e. g. 6x faster than RegNetY-16GF, and 1. 7x faster than EfficientNetB3.
UPDeT: Universal Multi-agent RL via Policy Decoupling with Transformers
Recent advances in multi-agent reinforcement learning have been largely limited in training one model from scratch for every new task.
Iterative Graph Self-Distillation
Recently, there has been increasing interest in the challenge of how to discriminatively vectorize graphs.
AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning
Synchronization is a key step in data-parallel distributed machine learning (ML).
Towards Robust Partially Supervised Multi-Structure Medical Image Segmentation on Small-Scale Data
To bridge the methodological gaps in partially supervised learning (PSL) under data scarcity, we propose Vicinal Labels Under Uncertainty (VLUU), a simple yet efficient framework utilizing the human structure similarity for partially supervised medical image segmentation.
Ada-Segment: Automated Multi-loss Adaptation for Panoptic Segmentation
Panoptic segmentation that unifies instance segmentation and semantic segmentation has recently attracted increasing attention.
Ranked #17 on
Panoptic Segmentation
on COCO test-dev
Adversarial Meta Sampling for Multilingual Low-Resource Speech Recognition
When sampling tasks in MML-ASR, AMS adaptively determines the task sampling probability for each source language.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
REM-Net: Recursive Erasure Memory Network for Commonsense Evidence Refinement
It is crucial since the quality of the evidence is the key to answering commonsense questions, and even determines the upper bound on the QA systems performance.
Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition
Beyond generating long and topic-coherent paragraphs in traditional captioning tasks, the medical image report composition task poses more task-oriented challenges by requiring both the highly-accurate medical term diagnosis and multiple heterogeneous forms of information including impression and findings.
SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
Experiments show that SODA10M can serve as a promising pre-training dataset for different self-supervised learning methods, which gives superior performance when fine-tuning with different downstream tasks (i. e., detection, semantic/instance segmentation) in autonomous driving domain.
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural language processing (NLP) tasks.
Ranked #10 on
Semantic Textual Similarity
on MRPC
WAS-VTON: Warping Architecture Search for Virtual Try-on Network
Despite recent progress on image-based virtual try-on, current methods are constraint by shared warping networks and thus fail to synthesize natural try-on results when faced with clothing categories that require different warping operations.
NASOA: Towards Faster Task-oriented Online Fine-tuning with a Zoo of Models
Fine-tuning from pre-trained ImageNet models has been a simple, effective, and popular approach for various computer vision tasks.
Deep Learning for Embodied Vision Navigation: A Survey
A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc.
Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning
Medical imaging technologies, including computed tomography (CT) or chest X-Ray (CXR), are largely employed to facilitate the diagnosis of the COVID-19.
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Despite the potential of multi-modal pre-training to learn highly discriminative feature representations from complementary data modalities, current progress is being slowed by the lack of large-scale modality-diverse datasets.
Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition
Specifically, we unify a pre-trained acoustic model (wav2vec 2. 0) and a language model (BERT) into an end-to-end trainable framework.
Linguistically Routing Capsule Network for Out-of-Distribution Visual Question Answering
The experimental results show that our proposed method can improve current VQA models on OOD split without losing performance on the in-domain test data.
Self-Motivated Communication Agent for Real-World Vision-Dialog Navigation
Vision-Dialog Navigation (VDN) requires an agent to ask questions and navigate following the human responses to find target objects.
Exploring Geometry-Aware Contrast and Clustering Harmonization for Self-Supervised 3D Object Detection
Here we present a novel self-supervised 3D Object detection framework that seamlessly integrates the geometry-aware contrast and clustering harmonization to lift the unsupervised 3D representation learning, named GCC-3D.
Image Comes Dancing with Collaborative Parsing-Flow Video Synthesis
The integration of human parsing and appearance flow effectively guides the generation of video frames with realistic appearance.
AutoLoss: Learning Discrete Schedule for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters.
Revisiting Over-smoothing in BERT from the Perspective of Graph
Recently over-smoothing phenomenon of Transformer-based models is observed in both vision and language fields.
Modern Augmented Reality: Applications, Trends, and Future Directions
Augmented reality (AR) is one of the relatively old, yet trending areas in the intersection of computer vision and computer graphics with numerous applications in several areas, from gaming and entertainment, to education and healthcare.
CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving
One main reason that impedes the development of truly reliably self-driving systems is the lack of public datasets for evaluating the performance of object detectors on corner cases.
elBERto: Self-supervised Commonsense Learning for Question Answering
Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context.
Laneformer: Object-aware Row-Column Transformers for Lane Detection
We present Laneformer, a conceptually simple yet powerful transformer-based architecture tailored for lane detection that is a long-standing research topic for visual perception in autonomous driving.
Dressing in the Wild by Watching Dance Videos
While significant progress has been made in garment transfer, one of the most applicable directions of human-centric image generation, existing works overlook the in-the-wild imagery, presenting severe garment-person misalignment as well as noticeable degradation in fine texture details.
ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts
Vision-Language Navigation (VLN) is a challenging task that requires an embodied agent to perform action-level modality alignment, i. e., make instruction-asked actions sequentially in complex visual environments.
BodyGAN: General-Purpose Controllable Neural Human Body Generation
This is because current methods mainly rely on a single pose/appearance model, which is limited in disentangling various poses and appearance in human images.
Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation
To endow models with the capability of incorporating expert knowledge, we propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG), in which clinical relation triples are injected into the visual features as prior knowledge to drive the decoding procedure.
Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval
Our goal in this research is to study a more realistic environment in which we can conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories.
Discourse-Aware Graph Networks for Textual Logical Reasoning
In this work, we propose logic structural-constraint modeling to solve the logical reasoning QA and introduce discourse-aware graph networks (DAGNs).
Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL
In this study, we quantify the agent's behavior difference and build its relationship with the policy performance via {\bf Role Diversity}, a metric to measure the characteristics of MARL tasks.
Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding
To bridge the gap between supervised semantic segmentation and real-world applications that acquires one model to recognize arbitrary new concepts, recent zero-shot segmentation attracts a lot of attention by exploring the relationships between unseen and seen object categories, yet requiring large amounts of densely-annotated data with diverse base classes.
PASTA-GAN++: A Versatile Framework for High-Resolution Unpaired Virtual Try-on
In this work, we take a step forwards to explore versatile virtual try-on solutions, which we argue should possess three main properties, namely, they should support unsupervised training, arbitrary garment categories, and controllable garment editing.
ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
ARMANI discretizes an image into uniform tokens based on a learned cross-modal codebook in its first stage and uses a Transformer to model the distribution of image tokens for a real image given the tokens of the control signals in its second stage.
Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving
Aiming towards a holistic understanding of multiple downstream tasks simultaneously, there is a need for extracting features with better transferability.
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection
We further design a concept dictionary~(with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept.
Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pre-trained VLMs and generating pseudo labels for unseen classes in a self-training manner.
Structure-Preserving 3D Garment Modeling with Neural Sewing Machines
In this paper, we propose a novel Neural Sewing Machine (NSM), a learning-based framework for structure-preserving 3D garment modeling, which is capable of learning representations for garments with diverse shapes and topologies and is successfully applied to 3D garment reconstruction and controllable manipulation.
3D-TOGO: Towards Text-Guided Cross-Category 3D Object Generation
Here, we make the first attempt to achieve generic text-guided cross-category 3D object generation via a new 3D-TOGO model, which integrates a text-to-views generation module and a views-to-3D generation module.
CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target.
NLIP: Noise-robust Language-Image Pre-training
Large-scale cross-modal pre-training paradigms have recently shown ubiquitous success on a wide range of downstream tasks, e. g., zero-shot classification, retrieval and image captioning.
Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation
Vision-Language Navigation (VLN) is a challenging task which requires an agent to align complex visual observations to language instructions to reach the goal position.
Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving
Multi-task learning has emerged as a powerful paradigm to solve a range of tasks simultaneously with good efficiency in both computation resources and inference time.
CapDet: Unifying Dense Captioning and Open-World Detection Pretraining
Besides, our CapDet also achieves state-of-the-art performance on dense captioning tasks, e. g., 15. 44% mAP on VG V1. 2 and 13. 98% on the VG-COCO dataset.
CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks.
Ranked #3 on
Zero-shot 3D Point Cloud Classification
on ScanNetV2
DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment
This paper presents DetCLIPv2, an efficient and scalable training framework that incorporates large-scale image-text pairs to achieve open-vocabulary object detection (OVD).
CLIP2: Contrastive Language-Image-Point Pretraining From Real-World Point Cloud Data
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks.
UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning
Recent advances in vision-language pre-training have enabled machines to perform better in multimodal object discrimination (e. g., image-text semantic alignment) and image synthesis (e. g., text-to-image generation).
Surfer: Progressive Reasoning with World Models for Robotic Manipulation
To conduct a comprehensive and systematic evaluation of the robot manipulation model in terms of language understanding and physical execution, we also created a robotic manipulation benchmark with progressive reasoning tasks, called SeaWave.
CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation
Understanding and following natural language instructions while navigating through complex, real-world environments poses a significant challenge for general-purpose robots.
FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration
Specifically, the gradients, produced by the task heads and used to update the shared backbone, will be calibrated at the backbone's last layer to alleviate the task conflict.
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation
To address this issue, we propose MixReorg, a novel and straightforward pre-training paradigm for semantic segmentation that enhances a model's ability to reorganize patches mixed across images, exploring both local visual relevance and global semantic coherence.
LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts
Thanks to the rapid development of diffusion models, unprecedented progress has been witnessed in image synthesis.
DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability
DiffDis first formulates the image-text discriminative problem as a generative diffusion process of the text embedding from the text encoder conditioned on the image.
Coordinate Transformer: Achieving Single-stage Multi-person Mesh Recovery from Videos
Multi-person 3D mesh recovery from videos is a critical first step towards automatic perception of group behavior in virtual reality, physical therapy and beyond.
GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training
Compared with the existing methods, GrowCLIP improves 2. 3% average top-1 accuracy on zero-shot image classification of 9 downstream tasks.
DiffCloth: Diffusion Based Garment Synthesis and Manipulation via Structural Cross-modal Semantic Alignment
Cross-modal garment synthesis and manipulation will significantly benefit the way fashion designers generate garments and modify their designs via flexible linguistic interfaces. Current approaches follow the general text-to-image paradigm and mine cross-modal relations via simple cross-attention modules, neglecting the structural correspondence between visual and textual representations in the fashion design domain.
Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images
In this paper, we propose a text-guided 3D faces generation method, refer as TG-3DFace, for generating realistic 3D faces using text guidance.
CLOMO: Counterfactual Logical Modification with Large Language Models
In this study, we delve into the realm of counterfactual reasoning capabilities of large language models (LLMs).
WarpDiffusion: Efficient Diffusion Model for High-Fidelity Virtual Try-on
Image-based Virtual Try-On (VITON) aims to transfer an in-shop garment image onto a target person.
DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
Especially for fidelity, our model has a powerful image retention ability and delivers the best results in UCF101 compared to other image-to-video models to our best knowledge.
Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model
Besides, we introduce a multi-denoiser framework for the advanced diffusion model to ease the learning of high-dimensional model and fully explore the generative potential of the diffusion model.
MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
Embodied agents equipped with GPT as their brain have exhibited extraordinary decision-making and generalization abilities across various tasks.
GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data
3D Shape represented as point cloud has achieve advancements in multimodal pre-training to align image and language descriptions, which is curial to object identification, classification, and retrieval.
Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
For encouraging the agent to well capture the difference brought by perturbation, a perturbation-aware contrastive learning mechanism is further developed by contrasting perturbation-free trajectory encodings and perturbation-based counterparts.
Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
To mitigate these issues, we propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information. Specifically, we leverage class embeddings as anchors due to their discrete and abstract nature, steering vision features toward class embeddings.
LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model
Specifically, an inter-layer attention module is designed to encourage information exchange and learning between layers, while a text-guided intra-layer attention module incorporates layer-specific prompts to direct the specific-content generation for each layer.
DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection
This is followed by a fine-tuning stage that leverages a small number of high-resolution samples to further enhance detection performance.
MMTryon: Multi-Modal Multi-Reference Control for High-Quality Fashion Generation
Besides, to remove the segmentation dependency, MMTryon uses a parsing-free garment encoder and leverages a novel scalable data generation pipeline to convert existing VITON datasets to a form that allows MMTryon to be trained without requiring any explicit segmentation.
Unbiased Math Word Problems Benchmark for Mitigating Solving Bias
However, current solvers exist solving bias which consists of data bias and learning bias due to biased dataset and improper training strategy.
Monocular 3D Hand Mesh Recovery via Dual Noise Estimation
However, due to the fixed hand topology and complex hand poses, current models are hard to generate meshes that are aligned with the image well.
Contrastive Instruction-Trajectory Learning for Vision-Language Navigation
The vision-language navigation (VLN) task requires an agent to reach a target with the guidance of natural language instruction.
Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search
It is able to find top 0. 16\% and 0. 29\% architectures on average on two search spaces under the budget of only 50 models.
Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformers
Despite the success, its development and application on self-supervised vision transformers have been hindered by several barriers, including the high search cost, the lack of supervision, and the unsuitable search space.
ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency
Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image.
Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp
Although deep reinforcement learning (RL) has been successfully applied to a variety of robotic control tasks, it's still challenging to apply it to real-world tasks, due to the poor sample efficiency.
Boosting Visual-Language Models by Exploiting Hard Samples
In this work, we present HELIP, a cost-effective strategy tailored to enhance the performance of existing CLIP models without the need for training a model from scratch or collecting additional data.
Multivariate-Information Adversarial Ensemble for Scalable Joint Distribution Matching
A broad range of cross-$m$-domain generation researches boil down to matching a joint distribution by deep generative models (DGMs).
Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods.
Ranked #6 on
Image Retrieval
on MUGE Retrieval
SiRi: A Simple Selective Retraining Mechanism for Transformer-based Visual Grounding
Particularly, SiRi conveys a significant principle to the research of visual grounding, i. e., a better initialized vision-language encoder would help the model converge to a better local minimum, advancing the performance accordingly.
3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands
Subsequently, a feature fusion module that exploits the visibility of query points and mesh vertices is introduced to adaptively merge features of both hands, enabling the recovery of features in unseen areas.
Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generation
Such training objective is sub-optimal when the target sequence is not perfect, e. g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available.
LogicSolver: Towards Interpretable Math Word Problem Solving with Logical Prompt-enhanced Learning
To address this issue and make a step towards interpretable MWP solving, we first construct a high-quality MWP dataset named InterMWP which consists of 11, 495 MWPs and annotates interpretable logical formulas based on algebraic knowledge as the grounded linguistic logic of each solution equation.
Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation
Such training objective is sub-optimal when the target sequence is not perfect, e. g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available.
Towards Hard-pose Virtual Try-on via 3D-aware Global Correspondence Learning
Existing methods are restricted in this setting as they estimate garment warping flows mainly based on 2D poses and appearance, which omits the geometric prior of the 3D human body shape.
Learning To Segment Every Referring Object Point by Point
Extensive experiments show that our model achieves 52. 06% in terms of accuracy (versus 58. 93% in fully supervised setting) on RefCOCO+@testA, when only using 1% of the mask annotations.
Speak Like a Native: Prompting Large Language Models in a Native Style
Specifically, with AlignedCoT, we observe an average +3. 2\% improvement for \texttt{gpt-3. 5-turbo} compared to the carefully handcrafted CoT on multi-step reasoning benchmarks. Furthermore, we use AlignedCoT to rewrite the CoT text style in the training set, which improves the performance of Retrieval Augmented Generation by 3. 6\%. The source code and dataset is available at https://github. com/yangzhch6/AlignedCoT
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism
To make ROSETTA automatically determine which experience is available and useful, a prototypical task correlation guided Gating Diversity Controller(GDC) is introduced to adaptively adjust the diversity of gates for the new task based on class-specific prototypes.
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Automated theorem proving (ATP) has become an appealing domain for exploring the reasoning ability of the recent successful generative language models.
MLP Can Be A Good Transformer Learner
We identify that regarding the attention layer in bottom blocks, their subsequent MLP layers, i. e. two feed-forward layers, can elicit the same entropy quantity.
MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure
To this end, we propose a comprehensive logical reasoning explanation form.
TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation
To address this issue, we introduce TheaterGen, a training-free framework that integrates large language models (LLMs) and text-to-image (T2I) models to provide the capability of multi-turn image generation.
Vision-Language Navigation with Random Environmental Mixup
Then, we cross-connect the key views of different scenes to construct augmented scenes.
Ranked #38 on
Vision and Language Navigation
on VLN Challenge
Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
Specifically, we propose a Dynamic Reinforced Instruction Attacker (DR-Attacker), which learns to mislead the navigator to move to the wrong target by destroying the most instructive information in instructions at different timesteps.
Recognizing Focal Liver Lesions in Contrast-Enhanced Ultrasound with Discriminatively Trained Spatio-Temporal Model
The aim of this study is to provide an automatic computational framework to assist clinicians in diagnosing Focal Liver Lesions (FLLs) in Contrast-Enhancement Ultrasound (CEUS).
DQ-LoRe: Dual Queries with Low Rank Approximation Re-ranking for In-Context Learning
Dual Queries first query LLM to obtain LLM-generated knowledge such as CoT, then query the retriever to obtain the final exemplars via both question and the knowledge.
NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning
Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.
Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems
A practical automatic textual math word problems (MWPs) solver should be able to solve various textual MWPs while most existing works only focused on one-unknown linear MWPs.
Ranked #10 on
Math Word Problem Solving
on ALG514
Data-to-Text Generation with Style Imitation
That is, the model learns to imitate the writing style of any given exemplar sentence, with automatic adaptions to faithfully describe the content record.
"My nose is running.""Are you also coughing?": Building A Medical Diagnosis Agent with Interpretable Inquiry Logics
In this paper, we explore how to bring interpretability to data-driven DSMD.
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Through extensive experiments across various datasets and scenes, we demonstrate the effectiveness of our approach in facilitating better interaction between LiDAR and camera modalities within a unified neural field.
LSTM-CF: Unifying Context Modeling and Fusion with LSTMs for RGB-D Scene Labeling
Another long short-term memorized fusion layer is set up to integrate the contexts along the vertical direction from different channels, and perform bi-directional propagation of the fused vertical contexts along the horizontal direction to obtain true 2D global contexts.
SparseBERT: Rethinking the Importance Analysis in Self-attention
A surprising result is that diagonal elements in the attention map are the least important compared with other attention positions.
SOON: Scenario Oriented Object Navigation with Graph-based Exploration
In this task, an agent is required to navigate from an arbitrary position in a 3D embodied environment to localize a target following a scene description.
Ranked #5 on
Visual Navigation
on SOON Test
Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation
Extensive experiments on two vision tasks, includ-ing ImageNet classification and Pascal VOC segmentation, demonstrate the superiority of our ICKD, which consis-tently outperforms many existing methods, advancing thestate-of-the-art in the fields of Knowledge Distillation.
Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining
In MOTOR, we combine two kinds of basic medical knowledge, i. e., general and specific knowledge, in a complementary manner to boost the general pretraining process.
RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment
Recent advances in text-to-image diffusion models have achieved remarkable success in generating high-quality, realistic images from textual descriptions.
AutoLoss: Learning Discrete Schedules for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters.
Symbolic Graph Reasoning Meets Convolutions
To cooperate with local convolutions, each SGR is constituted by three modules: a) a primal local-to-semantic voting module where the features of all symbolic nodes are generated by voting from local representations; b) a graph reasoning module propagates information over knowledge graph to achieve global semantic coherency; c) a dual semantic-to-local mapping module learns new associations of the evolved symbolic nodes with local representations, and accordingly enhances local features.
Ranked #81 on
Semantic Segmentation
on ADE20K val
Vision-Dialog Navigation by Exploring Cross-modal Memory
Benefiting from the collaborative learning of the L-mem and the V-mem, our CMN is able to explore the memory about the decision making of historical navigation actions which is for the current step.
Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
In this work, we propose an efficient, cooperative and highly automated framework to simultaneously search for all main components including backbone, segmentation branches, and feature fusion module in a unified panoptic segmentation pipeline based on the prevailing one-shot Network Architecture Search (NAS) paradigm.
Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
We attribute this ranking correlation problem to the supernet training consistency shift, including feature shift and parameter shift.
Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration
To improve the ability of fast cross-domain adaptation, we propose Prompt-based Environmental Self-exploration (ProbES), which can self-explore the environments by sampling trajectories and automatically generates structured instructions via a large-scale cross-modal pretrained model (CLIP).
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In this paper, we develop a general framework for interpretable natural language understanding that requires only a small set of human annotated explanations for training.
Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks
Previous math word problem solvers following the encoder-decoder paradigm fail to explicitly incorporate essential math symbolic constraints, leading to unexplainable and unreasonable predictions.
Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning
In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train a task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs.
TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
While existing NAS methods mostly design architectures on a single task, algorithms that look beyond single-task search are surging to pursue a more efficient and universal solution across various tasks.
Knowledge Distillation via the Target-aware Transformer
To this end, we propose a novel one-to-all spatial matching knowledge distillation approach.
Prototypical Graph Contrastive Learning
However, since for a query, its negatives are uniformly sampled from all graphs, existing methods suffer from the critical sampling bias issue, i. e., the negatives likely having the same semantic structure with the query, leading to performance degradation.
MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data
Recent large language models (LLMs) have witnessed significant advancement in various tasks, including mathematical reasoning and theorem proving.
