Search Results for author:
Found 83 papers, 51 papers with code
PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning
Offline-to-online reinforcement learning (RL), by combining the benefits of offline pretraining and online finetuning, promises enhanced sample efficiency and policy performance.
Breaching FedMD: Image Recovery via Paired-Logits Inversion Attack
Federated Learning with Model Distillation (FedMD) is a nascent collaborative learning paradigm, where only output logits of public datasets are transmitted as distilled knowledge, instead of passing on private model parameters that are susceptible to gradient inversion attacks, a known privacy risk in federated learning.
K-means Clustering Based Feature Consistency Alignment for Label-free Model Evaluation
The label-free model evaluation aims to predict the model performance on various test sets without relying on ground truths.
VIMI: Vehicle-Infrastructure Multi-view Intermediate Fusion for Camera-based 3D Object Detection
In autonomous driving, Vehicle-Infrastructure Cooperative 3D Object Detection (VIC3D) makes use of multi-view cameras from both vehicles and traffic infrastructure, providing a global vantage point with rich semantic context of road conditions beyond a single vehicle viewpoint.
SpiderMesh: Spatial-aware Demand-guided Recursive Meshing for RGB-T Semantic Segmentation
For semantic segmentation in urban scene understanding, RGB cameras alone often fail to capture a clear holistic topology, especially in challenging lighting conditions.
Ranked #2 on
Thermal Image Segmentation
on PST900
Calibration-free BEV Representation for Infrastructure Perception
Effective BEV object detection on infrastructure can greatly improve traffic scenes understanding and vehicle-toinfrastructure (V2I) cooperative perception.
Ranked #3 on
3D Object Detection
on DAIR-V2X-I
Metropolis Theorem and Its Applications in Single Image Detail Enhancement
Traditional image detail enhancement is local filter-based or global filter-based.
Multimodal Federated Learning via Contrastive Representation Ensemble
In this work, we propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL), a multimodal federated learning framework that enables training larger server models from clients with heterogeneous model architectures and data modalities, while only communicating knowledge on public dataset.
Mind the Gap: Offline Policy Optimization for Imperfect Rewards
RGM is formulated as a bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitation distribution matching w. r. t.
ADAPT: Action-aware Driving Caption Transformer
To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action.
OrthoGAN:High-Precision Image Generation for Teeth Orthodontic Visualization
We design a multi-modal encoder-decoder based generative model to synthesize identity-preserving frontal facial images with aligned teeth.
Correcting the Sub-optimal Bit Allocation
In this paper, we investigate the problem of bit allocation in Neural Video Compression (NVC).
Bit Allocation using Optimization
In this paper, we consider the problem of bit allocation in Neural Video Compression (NVC).
Adversarial Contrastive Learning via Asymmetric InfoNCE
Contrastive learning (CL) has recently been applied to adversarial learning tasks.
When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning
In offline reinforcement learning (RL), one detrimental issue to policy learning is the error accumulation of deep Q function in out-of-distribution (OOD) areas.
TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation
With the recent exponential increase in applying Vision Transformer (ViT) to vision tasks, the capability of ViT in adapting cross-domain knowledge, however, remains unexplored in the literature.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We hope our Adversarial VQA dataset can shed new light on robustness study in the community and serve as a valuable benchmark for future work.
Playing Lottery Tickets with Vision and Language
However, we can find "relaxed" winning tickets at 50%-70% sparsity that maintain 99% of the full accuracy.
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language.
CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
This work concerns video-language pre-training and representation learning.
The Elastic Lottery Ticket Hypothesis
Based on these results, we articulate the Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or dropping) and re-ordering layers for one network, its corresponding winning ticket could be stretched (or squeezed) into a subnetwork for another deeper (or shallower) network from the same family, whose performance is nearly the same competitive as the latter's winning ticket directly found by IMP.
Adversarial Feature Augmentation and Normalization for Visual Recognition
Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models.
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Multimodal pre-training has propelled great advancement in vision-and-language research.
Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models.
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle.
Ranked #20 on
Visual Question Answering (VQA)
on MSRVTT-QA
(using extra training data)
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Transformer has become ubiquitous in the deep learning field.
Adversarial Masking: Towards Understanding Robustness Trade-off for Generalization
By incorporating different feature maps after the masking, we can distill better features to help model generalization.
ALFA: Adversarial Feature Augmentation for Enhanced Image Recognition
Adversarial training is an effective method to combat adversarial attacks in order to create robust neural networks.
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks.
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
Large-scale pre-trained multimodal transformers, such as ViLBERT and UNITER, have propelled the state of the art in vision-and-language (V+L) research to a new level.
Wasserstein Contrastive Representation Distillation
The primary goal of knowledge distillation (KD) is to encapsulate the information of a model learned from a teacher network into a student network, with the latter being more compact than the former.
Ranked #7 on
Knowledge Distillation
on CIFAR-100
Cross-Thought for Sentence Encoder Pre-training
In this paper, we propose Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering.
Multi-Fact Correction in Abstractive Text Summarization
Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE.
InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks.
Ranked #1 on
Natural Language Inference
on ANLI test
(using extra training data)
Efficient Robust Training via Backward Smoothing
In this work, we develop a new understanding towards Fast Adversarial Training, by viewing random initialization as performing randomized smoothing for better optimization of the inner maximization problem.
Contrastive Distillation on Intermediate Representations for Language Model Compression
Existing language model compression methods mostly use a simple L2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one.
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
Transformer has become ubiquitous in the deep learning field.
Ranked #1 on
Open-Domain Question Answering
on SearchQA
Accelerating Real-Time Question Answering via Question Generation
Although deep neural networks have achieved tremendous success for question answering (QA), they are still suffering from heavy computational and energy cost for real product deployment.
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding
During inference, the model makes predictions based on the text input in the target language and its translation in the source language.
Ranked #18 on
Zero-Shot Cross-Lingual Transfer
on XTREME
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
We present a large, tunable neural conversational response generation model, DIALOGPT (dialogue generative pre-trained transformer).
Graph Optimal Transport for Cross-Domain Alignment
In GOT, cross-domain alignment is formulated as a graph matching problem, by representing entities into a dynamically-constructed graph.
MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients
Adaptive gradient methods such as RMSProp and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in face of noisy objectives.
Large-Scale Adversarial Training for Vision-and-Language Representation Learning
We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning.
Ranked #7 on
Referring Expression Comprehension
on RefCOCOg-test
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
To reveal the secrets behind the scene of these powerful models, we present VALUE (Vision-And-Language Understanding Evaluation), a set of meticulously designed probing tasks (e. g., Visual Coreference Resolution, Visual Relation Detection, Linguistic Probing Tasks) generalizable to standard pre-trained V+L models, aiming to decipher the inner workings of multimodal pre-training (e. g., the implicit knowledge garnered in individual attention heads, the inherent cross-modal alignment learned through contextualized multimodal embeddings).
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
We present HERO, a novel framework for large-scale video+language omni-representation learning.
Ranked #1 on
Video Retrieval
on TVR
Contextual Text Style Transfer
To realize high-quality style transfer with natural context preservation, we propose a Context-Aware Style Transfer (CAST) model, which uses two separate encoders for each input sentence and its surrounding context.
APo-VAE: Text Generation in Hyperbolic Space
In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations.
BachGAN: High-Resolution Image Synthesis from Salient Object Layout
We propose a new task towards more practical application for image generation - high-quality image synthesis from salient object layout.
VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text.
Self-Guided Adaptation: Progressive Representation Alignment for Domain Adaptive Object Detection
In this work, we propose a Self-Guided Adaptation (SGA) model, target at aligning feature representation and transferring object detection models across domains while considering the instantaneous alignment difficulty.
Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
Transformer has been successfully applied to many natural language processing tasks.
Distilling Knowledge Learned in BERT for Text Generation
Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization.
Hierarchical Graph Network for Multi-hop Question Answering
In this paper, we present Hierarchical Graph Network (HGN) for multi-hop question answering.
Ranked #34 on
Question Answering
on HotpotQA
DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained transformer).
Discourse-Aware Neural Extractive Text Summarization
Recently BERT has been adopted for document encoding in state-of-the-art text summarization models.
Meta Module Network for Compositional Visual Reasoning
To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically.
FreeLB: Enhanced Adversarial Training for Natural Language Understanding
Adversarial training, which minimizes the maximal risk for label-preserving input perturbations, has proved to be effective for improving the generalization of language models.
UNITER: Learning UNiversal Image-TExt Representations
Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding.
UNITER: UNiversal Image-TExt Representation Learning
Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i. e., masked language/region modeling is conditioned on full observation of image/text).
Ranked #2 on
Visual Question Answering (VQA)
on VCR (Q-A) test
Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation
Recent unsupervised approaches to domain adaptation primarily focus on minimizing the gap between the source and the target domains through refining the feature generator, in order to learn a better alignment between the two domains.
What Makes A Good Story? Designing Composite Rewards for Visual Storytelling
Previous storytelling approaches mostly focused on optimizing traditional metrics such as BLEU, ROUGE and CIDEr.
Patient Knowledge Distillation for BERT Model Compression
Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks.
Adversarial Domain Adaptation for Machine Reading Comprehension
In this paper, we focus on unsupervised domain adaptation for Machine Reading Comprehension (MRC), where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain.
A Hybrid Retrieval-Generation Neural Conversation Model
In this paper, we propose a hybrid neural conversation model that combines the merits of both response retrieval and generation methods.
Unsupervised Deep Structured Semantic Models for Commonsense Reasoning
Commonsense reasoning is fundamental to natural language understanding.
Ranked #3 on
Natural Language Understanding
on PDP60
Relation-Aware Graph Attention Network for Visual Question Answering
In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the interactive dynamics between different objects.
Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
We present the Frontier Aware Search with backTracking (FAST) Navigator, a general framework for action decoding, that achieves state-of-the-art results on the Room-to-Room (R2R) Vision-and-Language navigation challenge of Anderson et.
Ranked #3 on
Vision-Language Navigation
on Room2Room
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image.
Sequential Attention GAN for Interactive Image Editing
The main challenges in this sequential and interactive image generation task are two-fold: 1) contextual consistency between a generated image and the provided textual description; 2) step-by-step region-level modification to maintain visual consistency across the generated image sequence in each session.
StoryGAN: A Sequential Conditional GAN for Story Visualization
We therefore propose a new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework.
Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning
Training task-completion dialogue agents with reinforcement learning usually requires a large number of real user experiences.
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
We present a large-scale dataset, ReCoRD, for machine reading comprehension requiring commonsense reasoning.
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
We propose a multi-task learning framework to learn a joint Machine Reading Comprehension (MRC) model that can be applied to a wide range of MRC tasks in different domains.
Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning
This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to improving the effectiveness and robustness of Deep Dyna-Q (DDQ), a recently proposed framework that extends the Dyna-Q algorithm to integrate planning for task-completion dialogue policy learning.
Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems
This proposal introduces a Dialogue Challenge for building end-to-end task-completion dialogue systems, with the goal of encouraging the dialogue research community to collaborate and benchmark on standard datasets and unified experimental environment.
Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning
During dialogue policy learning, the world model is constantly updated with real user experience to approach real user behavior, and in turn, the dialogue agent is optimized using both real experience and simulated experience.
Reinforcement Learning (RL)
Task-Completion Dialogue Policy Learning
Language-Based Image Editing with Recurrent Attentive Models
First, we introduce a synthetic dataset, called CoSaL, to evaluate the end-to-end performance of our LBIE system.
Dynamic Fusion Networks for Machine Reading Comprehension
This paper presents a novel neural model - Dynamic Fusion Network (DFN), for machine reading comprehension (MRC).
Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning
This paper presents a new method --- adversarial advantage actor-critic (Adversarial A2C), which significantly improves the efficiency of dialogue policy learning in task-completion dialogue systems.
Image Disguise based on Generative Model
To protect image contents, most existing encryption algorithms are designed to transform an original image into a texture-like or noise-like image, which is, however, an obvious visual sign indicating the presence of an encrypted image, results in a significantly large number of attacks.
Multispectral Deep Neural Networks for Pedestrian Detection
Multispectral pedestrian detection is essential for around-the-clock applications, e. g., surveillance and autonomous driving.
3D Face Tracking and Multi-Scale, Spatio-temporal Analysis of Linguistically Significant Facial Expressions and Head Positions in ASL
Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body.
