Search Results for author:
Found 195 papers, 74 papers with code
ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment
To address these challenges, we propose BiHumanML3D, a novel bilingual human motion dataset, which establishes a crucial benchmark for bilingual text-to-motion generation models.
Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Interactive 3D segmentation has emerged as a promising solution for generating accurate object masks in complex 3D scenes by incorporating user-provided clicks.
Large Reasoning Models in Agent Scenarios: Exploring the Necessity of Reasoning Capabilities
The rise of Large Reasoning Models (LRMs) signifies a paradigm shift toward advanced computational reasoning.
DreamInsert: Zero-Shot Image-to-Video Object Insertion from A Single Image
Recent developments in generative diffusion models have turned many dreams into realities.
A Survey on Post-training of Large Language Models
The emergence of Large Language Models (LLMs) has fundamentally transformed natural language processing, making them indispensable across domains ranging from conversational systems to scientific exploration.
Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation
Multimodal retrieval-augmented generation (RAG) enhances the visual reasoning capability of vision-language models (VLMs) by dynamically accessing information from external knowledge bases.
CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom
Therefore, we investigate more diverse signals to capture comprehensive instruction-response pair characteristics and propose three foundational metrics that leverage Multi-LLM wisdom, informed by (1) diverse LLM responses and (2) reward model assessment.
Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence
In this paper, to overcome the limitation, we propose CS-Aligner, a novel and straightforward framework that performs distributional vision-language alignment by integrating Cauchy-Schwarz (CS) divergence with mutual information.
HPS: Hard Preference Sampling for Human Preference Alignment
Aligning Large Language Model (LLM) responses with human preferences is vital for building safe and controllable AI systems.
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding
Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously.
Towards Understanding Why Data Augmentation Improves Generalization
In this work, we present a unified theoretical framework that elucidates how data augmentation enhances generalization through two key effects: partial semantic feature removal and feature mixing.
EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification
In this paper, we revisit existing gradient-based circuit identification methods and find that their performance is either affected by the zero-gradient problem or saturation effects, where edge attribution scores become insensitive to input changes, resulting in noisy and unreliable attribution evaluations for circuit components.
A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models
This game aligns well with the input-output structure of modern multimodal LLMs and benefits from a rich repository of high-quality, human-annotated creative responses, making it an ideal platform for studying LLM creativity.
Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network
Given some video-query pairs with untrimmed videos and sentence queries, temporal sentence grounding (TSG) aims to locate query-relevant segments in these videos.
Memory-Efficient 4-bit Preconditioned Stochastic Optimization
To our knowledge, this is the first quantization approach applied to Cholesky factors of preconditioners.
Grimm: A Plug-and-Play Perturbation Rectifier for Graph Neural Networks Defending against Poisoning Attacks
Recent studies have revealed the vulnerability of graph neural networks (GNNs) to adversarial poisoning attacks on node classification tasks.
Cooperative SQL Generation for Segmented Databases By Using Multi-functional LLM Agents
In the first stage, agents analyze their respective schema and communicate with each other to collect the schema information relevant to the question.
SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization
This method leverages both online and offline DPO, allowing each to compensate for the other's limitations.
Hard-Label Black-Box Attacks on 3D Point Clouds
With the maturity of depth sensors in various 3D safety-critical applications, 3D point cloud models have been shown to be vulnerable to adversarial attacks.
Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph
The former retrieves question-relevant domain knowledge from DKG and uses it to prompt LLM to enhance the reasoning capability for domain-specific tasks; the latter leverages LLM to generate new domain knowledge from processed tasks and use it to evolve DKG.
Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment
While compositional approaches that combine separate language and image models show a 111% improvement over unified models at the holistic level, their performance remains suboptimal at both block and image levels.
Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning
Manipulating the interaction trajectories between the intelligent agent and the environment can control the agent's training and behavior, exposing the potential vulnerabilities of reinforcement learning (RL).
CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperation
To solve this issue, we propose Cooperative Plan Optimization (CaPo) to enhance the cooperation efficiency of LLM-based embodied agents.
Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector
With the new RADAR dataset, we further develop a novel and effective iN-time Embedding-based AdveRSarial Image DEtection (NEARSIDE) method, which exploits a single vector that distilled from the hidden states of VLMs, which we call the attacking direction, to achieve the detection of adversarial images against benign ones in the input.
Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation
To address this issue, we propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task which utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities.
Two are better than one: Context window extension with multi-grained self-injection
The limited context window of contemporary large language models (LLMs) remains a huge barrier to their broader application across various domains.
Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning
Semi-supervised learning (SSL), exemplified by FixMatch (Sohn et al., 2020), has shown significant generalization advantages over supervised learning (SL), particularly in the context of deep neural networks (DNNs).
Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
Fine-tuning Large Language Models (LLMs) has proven effective for a variety of downstream tasks.
Towards Natural Image Matting in the Wild via Real-Scenario Prior
For training objectives, the proposed regularization and trimap loss aim to retain the prior from the pre-trained model and push the matting logits extracted from the mask decoder to contain trimap-based semantic information.
The Impact of Large Language Models in Academia: from Writing to Speaking
Large language models (LLMs) are increasingly impacting human society, particularly in textual information.
A Deep Dive into Fairness, Bias, Threats, and Privacy in Recommender Systems: Insights and Future Research
This study explores fairness, bias, threats, and privacy in recommender systems.
LPT++: Efficient Training on Mixture of Long-tailed Experts
We introduce LPT++, a comprehensive framework for long-tailed classification that combines parameter-efficient fine-tuning (PEFT) with a learnable model ensemble.
MoExtend: Tuning New Experts for Modality and Task Extension
To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models.
Can Large Language Models Automatically Jailbreak GPT-4V?
In our study, we introduce AutoJailbreak, an innovative automatic jailbreak technique inspired by prompt optimization.
A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
Compared to traditional Large Language Models (LLMs), LVLMs present great potential and challenges due to its closer proximity to the multi-resource real-world applications and the complexity of multi-modal processing.
LoCo: Low-Bit Communication Adaptor for Large-scale Model Training
Experimental results show that across large-scale model training frameworks like Megatron-LM and PyTorch's FSDP, LoCo significantly improves communication efficiency, e. g., improving Adam's training speed by 14% to 40% without performance degradation on large language models like LLAMAs and MoE.
Self-Cognition in Large Language Models: An Exploratory Study
Additionally, we also explore the utility and trustworthiness of LLM in the self-cognition state, revealing that the self-cognition state enhances some specific tasks such as creative writing and exaggeration.
A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning
Chain-of-Thought (CoT) holds a significant place in augmenting the reasoning performance for large language models (LLMs).
GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
We evaluate the capabilities of current state-of-the-art MLLMs, including Image LLMs and Video LLMs, in understanding various types of GUI content, especially dynamic and sequential content.
Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?
Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world.
MVGamba: Unify 3D Content Generation as State Space Sequence Modeling
Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors.
4-bit Shampoo for Memory-Efficient Network Training
In this paper, we propose the first 4-bit second-order optimizers, exemplified by 4-bit Shampoo, maintaining performance similar to that of 32-bit ones.
LOVA3: Learning to Visual Question Answering, Asking and Assessment
Our results demonstrate consistent performance gains, underscoring the critical role of these additional tasks in fostering comprehensive intelligence in MLLMs.
Ranked #158 on
Visual Question Answering
on MM-Vet
Physical Backdoor: Towards Temperature-based Backdoor Attacks in the Physical World
In the digital realm, we evaluate our approach using benchmark datasets for TIOD, achieving an Attack Success Rate (ASR) of up to 98. 21%.
CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code
Experiments conducted on a real-world dataset across five programming languages demonstrate the effectiveness of CodeIP in watermarking LLMs for code generation while maintaining the syntactical correctness of code.
Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach
Subsequently, the membership classifier can be effectively employed to deduce the membership status of a given code sample based on the output of a target code completion model.
Diffusion Time-step Curriculum for One Image to 3D Generation
Score distillation sampling~(SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a \textbf{single} image.
Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction
We tackle the challenge of efficiently reconstructing a 3D asset from a single image at millisecond speed.
Optimization-based Prompt Injection Attack to LLM-as-a-Judge
In this work, we propose JudgeDeceiver, an optimization-based prompt injection attack to LLM-as-a-Judge.
Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models
We conduct GenAP on three popular code intelligence PLMs with three canonical code intelligence tasks including defect prediction, code summarization, and code translation.
Friendly Sharpness-Aware Minimization
By decomposing the adversarial perturbation in SAM into full gradient and stochastic gradient noise components, we discover that relying solely on the full gradient component degrades generalization while excluding it leads to improved performance.
What Makes Good Collaborative Views? Contrastive Mutual Information Maximization for Multi-Agent Perception
The core philosophy of CMiMC is to preserve discriminative information of individual views in the collaborative view by maximizing mutual information between pre- and post-collaboration features while enhancing the efficacy of collaborative views by minimizing the loss function of downstream tasks.
Few-shot Learner Parameterization by Diffusion Time-steps
To this end, we find an inductive bias that the time-steps of a Diffusion Model (DM) can isolate the nuanced class attributes, i. e., as the forward diffusion adds noise to an image at each time-step, nuanced attributes are usually lost at an earlier time-step than the spurious attributes that are visually prominent.
MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
Drawing inspiration from the concept of LLM-as-a-Judge within LLMs, this paper introduces a novel benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting judges across diverse modalities, encompassing three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking.
Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior
To solve this issue, we first deeply analyze the SDS and find that its distillation sampling process indeed corresponds to the trajectory sampling of a stochastic differential equation (SDE): SDS samples along an SDE trajectory to yield a less noisy sample which then serves as a guidance to optimize a 3D model.
The NPU-ASLP-LiAuto System Description for Visual Speech Recognition in CNVSRC 2023
This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and open tracks of Single-Speaker VSR Task, and the open track of Multi-Speaker VSR Task.
MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition
While automatic speech recognition (ASR) systems degrade significantly in noisy environments, audio-visual speech recognition (AVSR) systems aim to complement the audio stream with noise-invariant visual cues and improve the system's robustness.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+4
ICMC-ASR: The ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition Challenge
To promote speech processing and recognition research in driving scenarios, we build on the success of the Intelligent Cockpit Speech Recognition Challenge (ICSRC) held at ISCSLP 2022 and launch the ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) Challenge.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
U2-KWS: Unified Two-pass Open-vocabulary Keyword Spotting with Keyword Bias
Open-vocabulary keyword spotting (KWS), which allows users to customize keywords, has attracted increasingly more interest.
Towards Inductive Robustness: Distilling and Fostering Wave-induced Resonance in Transductive GCNs Against Graph Adversarial Attacks
We then prove that merely three MP iterations within GCNs can induce signal resonance between nodes and edges, manifesting as a coupling between nodes and their distillable surrounding local subgraph.
Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator
Multimodal Large Language Models (MLLMs) demonstrate exceptional problem-solving capabilities, but few research studies aim to gauge the ability to generate visual instruction tuning data.
Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation
To this end, we study LLMs on the popular Oogiri game which needs participants to have good creativity and strong associative thinking for responding unexpectedly and humorously to the given image, text, or both, and thus is suitable for LoT study.
Exploring the Robustness of Decentralized Training for Large Language Models
Decentralized training of large language models has emerged as an effective way to democratize this technology.
MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learning
To solve these challenges, we propose MetaCloak, which solves the bi-level poisoning problem with a meta-learning framework with an additional transformation sampling process to craft transferable and robust perturbation.
Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts
This finding indicates potential exploitable security risks in MLLMs; 2) Based on the acquired system prompts, we propose a novel MLLM jailbreaking attack method termed SASP (Self-Adversarial Attack via System Prompt).
Instant3D: Instant Text-to-3D Generation
We achieve this remarkable speed by devising a new network that directly constructs a 3D triplane from a text prompt.
F$^2$AT: Feature-Focusing Adversarial Training via Disentanglement of Natural and Perturbed Patterns
We propose a Feature-Focusing Adversarial Training (F$^2$AT), which differs from previous work in that it enforces the model to focus on the core features from natural patterns and reduce the impact of spurious features from perturbed patterns.
ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection
Besides, we also observe the theoretical benefits of the LSC coefficient scaling of UNet in the stableness of hidden features and gradient and also robustness.
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests.
3DHacker: Spectrum-based Decision Boundary Generation for Hard-label 3D Point Cloud Attack
With the maturity of depth sensors, the vulnerability of 3D point cloud models has received increasing attention in various applications such as autonomous driving and robot navigation.
Graph Agent Network: Empowering Nodes with Inference Capabilities for Adversarial Resilience
In this paper, we propose the Graph Agent Network (GAgN) to address the aforementioned vulnerabilities of GNNs.
Fast Diffusion Model
In this paper, we propose a Fast Diffusion Model (FDM) to significantly speed up DMs from a stochastic optimization perspective for both faster training and sampling.
Transform-Equivariant Consistency Learning for Temporal Sentence Grounding
This paper addresses the temporal sentence grounding (TSG).
InceptionNeXt: When Inception Meets ConvNeXt
Inspired by the long-range modeling ability of ViTs, large-kernel convolutions are widely studied and adopted recently to enlarge the receptive field and improve model performance, like the remarkable work ConvNeXt which employs 7x7 depthwise convolution.
MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer
To solve this issue, we propose a Masked Diffusion Transformer (MDT) that introduces a mask latent modeling scheme to explicitly enhance the DPMs' ability to contextual relation learning among object semantic parts in an image.
Ranked #23 on
Image Generation
on ImageNet 256x256
You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos
To handle the raw video bit-stream input, we propose a novel Three-branch Compressed-domain Spatial-temporal Fusion (TCSF) framework, which extracts and aggregates three kinds of low-level visual features (I-frame, motion vector and residual features) for effective and efficient grounding.
Unlearnable Graph: Protecting Graphs from Unauthorized Exploitation
While the use of graph-structured data in various fields is becoming increasingly popular, it also raises concerns about the potential unauthorized exploitation of personal data for training commercial graph neural network (GNN) models, which can compromise privacy.
Jointly Visual- and Semantic-Aware Graph Memory Networks for Temporal Sentence Localization in Videos
Temporal sentence localization in videos (TSLV) aims to retrieve the most interested segment in an untrimmed video according to a given sentence query.
Contrastive Video Question Answering via Video Graph Transformer
CoVGT's uniqueness and superiority are three-fold: 1) It proposes a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations and dynamics, for complex spatio-temporal reasoning.
Ranked #34 on
Video Question Answering
on NExT-QA
(using extra training data)
Tracking Objects and Activities with Attention for Temporal Sentence Grounding
Temporal sentence grounding (TSG) aims to localize the temporal segment which is semantically aligned with a natural language query in an untrimmed video. Most existing methods extract frame-grained features or object-grained features by 3D ConvNet or detection network under a conventional TSG framework, failing to capture the subtle differences between frames or to model the spatio-temporal behavior of core persons/objects.
BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT
Recently, ChatGPT has gained significant attention in research due to its ability to interact with humans effectively.
STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognition
For the first time, we introduce vision Transformers into PPAR by treating a video as a tubelet sequence, and accordingly design two complementary mechanisms, i. e., sparsification and anonymization, to remove privacy from a spatio-temporal perspective.
Hypotheses Tree Building for One-Shot Temporal Sentence Localization
Given an untrimmed video, temporal sentence localization (TSL) aims to localize a specific segment according to a given sentence query.
Rethinking the Video Sampling and Reasoning Strategies for Temporal Sentence Grounding
All existing works first utilize a sparse sampling strategy to extract a fixed number of video frames and then conduct multi-modal interactions with query sentence for reasoning.
You Are Catching My Attention: Are Vision Transformers Bad Learners Under Backdoor Attacks?
Vision Transformers (ViTs), which made a splash in the field of computer vision (CV), have shaken the dominance of convolutional neural networks (CNNs).
Position-guided Text Prompt for Vision-Language Pre-training
In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP.
Ranked #5 on
Zero-Shot Cross-Modal Retrieval
on COCO 2014
Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble
Distantly-Supervised Named Entity Recognition (DS-NER) effectively alleviates the data scarcity problem in NER by automatically generating training samples.
MetaFormer Baselines for Vision
By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85. 5% at 224x224 resolution, under normal supervised training without external data or distillation.
Ranked #2 on
Domain Generalization
on ImageNet-C
(using extra training data)
Towards Sustainable Self-supervised Learning
In this work, we explore a sustainable SSL framework with two major challenges: i) learning a stronger new SSL model based on the existing pretrained SSL model, also called as "base" model, in a cost-friendly manner, ii) allowing the training of the new model to be compatible with various base models.
Ranked #1 on
Semantic Segmentation
on ImageNet-S
LPT: Long-tailed Prompt Tuning for Image Classification
For better effectiveness, we divide prompts into two groups: 1) a shared prompt for the whole long-tailed dataset to learn general features and to adapt a pretrained model into target domain; and 2) group-specific prompts to gather group-specific features for the samples which have similar features and also to empower the pretrained model with discrimination ability.
Ranked #1 on
Long-tail Learning
on CIFAR-100-LT (ρ=100)
(using extra training data)
Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval
In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop.
Hierarchical Local-Global Transformer for Temporal Sentence Grounding
To address this issue, in this paper, we propose a novel Hierarchical Local-Global Transformer (HLGT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities for learning more fine-grained multi-modal representations.
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point.
Video Graph Transformer for Video Question Answering
VGT's uniqueness are two-fold: 1) it designs a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations, and dynamics for complex spatio-temporal reasoning; and 2) it exploits disentangled video and text Transformers for relevance comparison between the video and text to perform QA, instead of entangled cross-modal Transformer for answer classification.
Ranked #5 on
Video Question Answering
on IntentQA
Backdoor Attacks on Crowd Counting
In this paper, we propose two novel Density Manipulation Backdoor Attacks (DMBA$^{-}$ and DMBA$^{+}$) to attack the model to produce arbitrarily large or small density estimations.
Gaussian Kernel-based Cross Modal Network for Spatio-Temporal Video Grounding
Spatial-Temporal Video Grounding (STVG) is a challenging task which aims to localize the spatio-temporal tube of the interested object semantically according to a natural language query.
Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks
To solve these problems, we first theoretically show that on an auto-encoder of a two/one-layered convolution encoder/decoder, MRP can capture all discriminative features of each potential semantic class in the pretraining dataset.
Inception Transformer
Recent studies show that Transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information.
Bandits for Structure Perturbation-based Black-box Attacks to Graph Neural Networks with Theoretical Guarantees
We then propose an online attack based on bandit optimization which is proven to be {sublinear} to the query number $T$, i. e., $\mathcal{O}(\sqrt{N}T^{3/4})$ where $N$ is the number of nodes in the graph.
Mugs: A Multi-Granular Self-Supervised Learning Framework
It provides complementary instance supervision to IDS via an extra alignment on local neighbors, and scatters different local-groups separately to increase discriminability.
Ranked #13 on
Self-Supervised Image Classification
on ImageNet
Contrastive Learning
Self-Supervised Image Classification
+3
Self-Promoted Supervision for Few-Shot Transformer
The few-shot learning ability of vision transformers (ViTs) is rarely investigated though heavily desired.
Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding
Temporal sentence grounding aims to localize a target segment in an untrimmed video semantically according to a given sentence query.
End-to-end contextual asr based on posterior distribution adaptation for hybrid ctc/attention system
End-to-end (E2E) speech recognition architectures assemble all components of traditional speech recognition system into a single model.
Unsupervised Temporal Video Grounding with Deep Semantic Clustering
Temporal video grounding (TVG) aims to localize a target segment in a video according to a given sentence query.
Exploring Motion and Appearance Information for Temporal Sentence Grounding
Then, we develop separate motion and appearance branches to learn motion-guided and appearance-guided object relations, respectively.
Memory-Guided Semantic Learning Network for Temporal Sentence Grounding
To tackle this issue, we propose a memory-augmented network, called Memory-Guided Semantic Learning Network (MGSL-Net), that learns and memorizes the rarely appeared content in TSG tasks.
DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition
While transformers have shown great potential on video recognition with their strong capability of capturing long-range dependencies, they often suffer high computational costs induced by the self-attention to the huge number of 3D tokens.
SNEAK: Synonymous Sentences-Aware Adversarial Attack on Natural Language Video Localization
Natural language video localization (NLVL) is an important task in the vision-language understanding area, which calls for an in-depth understanding of not only computer vision and natural language side alone, but more importantly the interplay between both sides.
Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond
Specifically, we prove that lookahead using SGD as its inner-loop optimizer can better balance the optimization error and generalization error to achieve smaller excess risk error than vanilla SGD on (strongly) convex problems and nonconvex problems with Polyak-{\L}ojasiewicz condition which has been observed/proved in neural networks.
Unsupervised Domain Adaptive Person Re-Identification via Human Learning Imitation
Unsupervised domain adaptive person re-identification has received significant attention due to its high practical value.
Domain Adaptive Person Re-Identification
Person Re-Identification
MetaFormer Is Actually What You Need for Vision
Based on this observation, we hypothesize that the general architecture of the Transformers, instead of the specific token mixer module, is more essential to the model's performance.
Ranked #9 on
Semantic Segmentation
on DensePASS
Lottery Image Prior
Drawing inspirations from the recently prosperous research on lottery ticket hypothesis (LTH), we conjecture and study a novel “lottery image prior” (LIP), stated as: given an (untrained or trained) DNN-based image prior, it will have a sparse subnetwork that can be training in isolation, to match the original DNN’s performance when being applied as a prior to various image inverse problems.
Bandits for Black-box Attacks to Graph Neural Networks with Structure Perturbation
However, many recent works have demonstrated that an attacker can mislead GNN models by slightly perturbing the graph structure.
Backdoor Attacks on Federated Learning with Lottery Ticket Hypothesis
Edge devices in federated learning usually have much more limited computation and communication resources compared to servers in a data center.
Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition
Specifically, we unify a pre-trained acoustic model (wav2vec 2. 0) and a language model (BERT) into an end-to-end trainable framework.
Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos
However, the performance of bottom-up model is inferior to the top-down counterpart as it fails to exploit the segment-level interaction.
Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding
A key solution to temporal sentence grounding (TSG) exists in how to learn effective alignment between vision and language features extracted from an untrimmed video and a sentence description.
Coarse to Fine: Domain Adaptive Crowd Counting via Adversarial Scoring Network
In specific, at the coarse-grained stage, we design a dual-discriminator strategy to adapt source domain to be close to the targets from the perspectives of both global and local feature space via adversarial learning.
A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
Although intuitive, such a native label assignment strategy cannot reveal the underlying semantic similarity between a query and its positives and negatives, and impairs performance, since some negatives are semantically similar to the query or even share the same semantic class as the query.
Prototypical Graph Contrastive Learning
However, since for a query, its negatives are uniformly sampled from all graphs, existing methods suffer from the critical sampling bias issue, i. e., the negatives likely having the same semantic structure with the query, leading to performance degradation.
Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction
The consistency of a response to a given post at semantic-level and emotional-level is essential for a dialogue system to deliver human-like interactions.
Exploiting Global Contextual Information for Document-level Named Entity Recognition
To this end, we propose a model called Global Context enhanced Document-level NER (GCDoc) to leverage global contextual information from two levels, i. e., both word and sentence.
TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness
To answer these questions, in this work we first theoretically analyze and outline sufficient conditions for adversarial transferability between models; then propose a practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness.
RBNN: Memory-Efficient Reconfigurable Deep Binary Neural Network with IP Protection for Internet of Things
To this end, a 1-bit quantized DNN model or deep binary neural network maximizes the memory efficiency, where each parameter in a BNN model has only 1-bit.
Towards Adversarial Patch Analysis and Certified Defense against Crowd Counting
Crowd counting has drawn much attention due to its importance in safety-critical surveillance systems.
WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition
Recently, attention-based encoder-decoder (AED) end-to-end (E2E) models have drawn more and more attention in the field of automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
TRS: Transferability Reduced Ensemble via Encouraging Gradient Diversity and Model Smoothness
To answer these questions, in this work we first theoretically analyze and outline sufficient conditions for adversarial transferability between models; then propose a practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness.
Context-aware Biaffine Localizing Network for Temporal Sentence Grounding
This paper addresses the problem of temporal sentence grounding (TSG), which aims to identify the temporal boundary of a specific segment from an untrimmed video by a sentence query.
DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
Deep learning techniques have achieved remarkable performance in wide-ranging tasks.
Progressive Localization Networks for Language-based Moment Localization
The language-based setting of this task allows for an open set of target activities, resulting in a large variation of the temporal lengths of video moments.
Erasure for Advancing: Dynamic Self-Supervised Learning for Commonsense Reasoning
To solve this issue, we propose a novel DynamIc Self-sUperviSed Erasure (DISUSE) which adaptively erases redundant and artifactual clues in the context and questions to learn and establish the correct corresponding pair relations between the questions and their clues.
Adversarial Meta Sampling for Multilingual Low-Resource Speech Recognition
When sampling tasks in MML-ASR, AMS adaptively determines the task sampling probability for each source language.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation
Besides, we develop a Graph-Evolving Meta-Learning (GEML) framework that learns to evolve the commonsense graph for reasoning disease-symptom correlations in a new disease, which effectively alleviates the needs of a large number of dialogues.
Spatiotemporal Graph Neural Network based Mask Reconstruction for Video Object Segmentation
To capture temporal information from previous frames, we use a memory network to refine the mask of current frame by retrieving historic masks in a temporal graph.
F2Net: Learning to Focus on the Foreground for Unsupervised Video Object Segmentation
Specifically, our proposed network consists of three main parts: Siamese Encoder Module, Center Guiding Appearance Diffusion Module, and Dynamic Information Fusion Module.
Ranked #10 on
Unsupervised Video Object Segmentation
on FBMS test
Semantic Segmentation
Unsupervised Video Object Segmentation
+1
Reasoning Step-by-Step: Temporal Sentence Localization in Videos via Deep Rectification-Modulation Network
In this paper, we propose a novel deep rectification-modulation network (RMN), transforming this task into a multi-step reasoning process by repeating rectification and modulation.
V3H: View Variation and View Heredity for Incomplete Multi-view Clustering
Inspired by the variation and the heredity in genetics, V3H first decomposes each subspace into a variation matrix for the corresponding view and a heredity matrix for all the views to represent the unique information and the consistent information respectively.
Unbalanced Incomplete Multi-view Clustering via the Scheme of View Evolution: Weak Views are Meat; Strong Views do Eat
However, different views often have distinct incompleteness, i. e., unbalanced incompleteness, which results in strong views (low-incompleteness views) and weak views (high-incompleteness views).
ANIMC: A Soft Framework for Auto-weighted Noisy and Incomplete Multi-view Clustering
In these scenarios, original image data often contain missing instances and noises, which is ignored by most multi-view clustering methods.
User-based Network Embedding for Collective Opinion Spammer Detection
Due to the huge commercial interests behind online reviews, a tremendousamount of spammers manufacture spam reviews for product reputation manipulation.
Target Guided Emotion Aware Chat Machine
The consistency of a response to a given post at semantic-level and emotional-level is essential for a dialogue system to deliver human-like interactions.
Video-based Facial Expression Recognition using Graph Convolutional Networks
In terms of video based FER task, it is sensible to capture the dynamic expression variation among the frames to recognize facial expression.
Facial Expression Recognition
Facial Expression Recognition (FER)
Iterative Graph Self-Distillation
Recently, there has been increasing interest in the challenge of how to discriminatively vectorize graphs.
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
The result shows that (1) the escaping time of both SGD and ADAM~depends on the Radon measure of the basin positively and the heaviness of gradient noise negatively; (2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly because (a) the geometry adaptation in ADAM~via adaptively scaling each gradient coordinate well diminishes the anisotropic structure in gradient noise and results in larger Radon measure of a basin; (b) the exponential gradient average in ADAM~smooths its gradient and leads to lighter gradient noise tails than SGD.
How Important is the Train-Validation Split in Meta-Learning?
A common practice in meta-learning is to perform a train-validation split (\emph{train-val method}) where the prior adapts to the task on one split of the data, and the resulting predictor is evaluated on another split.
Hybrid Stochastic-Deterministic Minibatch Proximal Gradient: Less-Than-Single-Pass Optimization with Nearly Optimal Generalization
Particularly, in the case of $\epsilon=\mathcal{O}\big(1/\sqrt{n}\big)$ which is at the order of intrinsic excess error bound of a learning model and thus sufficient for generalization, the stochastic gradient complexity bounds of HSDMPG for quadratic and generic loss functions are respectively $\mathcal{O} (n^{0. 875}\log^{1. 5}(n))$ and $\mathcal{O} (n^{0. 875}\log^{2. 25}(n))$, which to our best knowledge, for the first time achieve optimal generalization in less than a single pass over data.
Reinforcement Learning-based Black-Box Evasion Attacks to Link Prediction in Dynamic Graphs
Link prediction in dynamic graphs (LPDG) is an important research problem that has diverse applications such as online recommendations, studies on disease contagion, organizational studies, etc.
Efficient, Direct, and Restricted Black-Box Graph Evasion Attacks to Any-Layer Graph Neural Networks via Influence Function
Specifically, we first introduce two influence functions, i. e., feature-label influence and label influence, that are defined on GNNs and label propagation (LP), respectively.
Identity-Aware Attribute Recognition via Real-Time Distributed Inference in Mobile Edge Clouds
In this paper, we design novel models for pedestrian attribute recognition with re-ID in an MEC-enabled camera monitoring system.
Fine-grained Iterative Attention Network for TemporalLanguage Localization in Videos
In this paper, we propose a Fine-grained Iterative Attention Network (FIAN) that consists of an iterative attention module for bilateral query-video in-formation extraction.
Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localization
To this end, we propose a novel Cross- and Self-Modal Graph Attention Network (CSMGAN) that recasts this task as a process of iterative messages passing over a joint graph.
Theory-Inspired Path-Regularized Differential Network Architecture Search
Then we propose a theory-inspired path-regularized DARTS that consists of two key modules: (i) a differential group-structured sparse binary gate introduced for each operation to avoid unfair competition among operations, and (ii) a path-depth-wise regularization used to incite search exploration for deep architectures that often converge slower than shallow ones as shown in our theory and are not well explored during the search.
Federated Mutual Learning
The experiments show that FML can achieve better performance than alternatives in typical FL setting, and clients can be benefited from FML with different models and tasks.
Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
Despite success on a wide range of problems related to vision, generative adversarial networks (GANs) often suffer from inferior performance due to unstable training, especially for text generation.
Ranked #2 on
Text Generation
on EMNLP2017 WMT
Prototypical Contrastive Learning of Unsupervised Representations
This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning.
Ranked #5 on
Contrastive Learning
on imagenet-1k
AN-GCN: An Anonymous Graph Convolutional Network Defense Against Edge-Perturbing Attack
In this paper, we first generalize the formulation of edge-perturbing attacks and strictly prove the vulnerability of GCNs to such attacks in node classification tasks.
Data Augmentation Imbalance For Imbalanced Attribute Classification
Pedestrian attribute recognition is an important multi-label classification problem.
Crowd Counting via Hierarchical Scale Recalibration Network
Extensive experiments on crowd counting datasets (ShanghaiTech, MALL, WorldEXPO'10, and UCSD) show that our HSRNet can deliver superior results over all state-of-the-art approaches.
Dynamic Graph Correlation Learning for Disease Diagnosis with Incomplete Labels
In this work, we propose a Disease Diagnosis Graph Convolutional Network (DD-GCN) that presents a novel view of investigating the inter-dependency among different diseases by using a dynamic learnable adjacency matrix in graph structure to improve the diagnosis accuracy.
Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach
Peer-to-peer knowledge transfer in distributed environments has emerged as a promising method since it could accelerate learning and improve team-wide performance without relying on pre-trained teachers in deep reinforcement learning.
Prophet: Proactive Candidate-Selection for Federated Learning by Predicting the Qualities of Training and Reporting Phases
In the dynamic environment, the mobile devices selected by the existing reactive candidate-selection algorithms very possibly fail to complete the training and reporting phases of FL, because the FL parameter server only knows the currently-observed resources of all candidates.
Efficient Meta Learning via Minibatch Proximal Update
We address the problem of meta-learning which learns a prior over hypothesis from a sample of meta-training tasks for fast adaptation on meta-testing tasks.
Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embedding
Transformer has shown promising results in many sequence to sequence transformation tasks recently.
Tell-the-difference: Fine-grained Visual Descriptor via a Discriminating Referee
In this paper, we investigate a novel problem of telling the difference between image pairs in natural language.
Generating Robust Audio Adversarial Examples using Iterative Proportional Clipping
We then impose a constraint on the perturbation at the positions with lower sound intensity across the time domain to eliminate the perceptible noise during the silent periods or pauses.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Position-Aware Self-Attention based Neural Sequence Labeling
Sequence labeling is a fundamental task in natural language processing and has been widely studied.
Enhanced 3D convolutional networks for crowd counting
Recently, convolutional neural networks (CNNs) are the leading defacto method for crowd counting.
Attend To Count: Crowd Counting with Adaptive Capacity Multi-scale CNNs
ACM-CNN consists of three types of modules: a coarse network, a fine network, and a smooth network.
Joint Coverage and Power Control in Highly Dynamic and Massive UAV Networks: An Aggregative Game-theoretic Learning Approach
Unmanned aerial vehicles (UAV) ad-hoc network is a significant contingency plan for communication after a natural disaster, such as typhoon and earthquake.
Exact Recovery of Tensor Robust Principal Component Analysis under Linear Transforms
This work studies the Tensor Robust Principal Component Analysis (TRPCA) problem, which aims to exactly recover the low-rank and sparse components from their sum.
EnlightenGAN: Deep Light Enhancement without Paired Supervision
Deep learning-based methods have achieved remarkable success in image restoration and enhancement, but are they still competitive when there is a lack of paired training data?
Adversarial Category Alignment Network for Cross-domain Sentiment Classification
Cross-domain sentiment classification aims to predict sentiment polarity on a target domain utilizing a classifier learned from a source domain.
MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation
Extensive experiments on challenging datasets demonstrate the effectiveness of the proposed method, especially in the case of object missing.
Ranked #40 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
A Stochastic Trust Region Method for Non-convex Minimization
We target the problem of finding a local minimum in non-convex finite-sum minimization.
Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training
Distributed asynchronous offline training has received widespread attention in recent years because of its high performance on large-scale data and complex models.
New Insight into Hybrid Stochastic Gradient Descent: Beyond With-Replacement Sampling and Convexity
In this paper, we affirmatively answer this open question by showing that under WoRS and for both convex and non-convex problems, it is still possible for HSGD (with constant step-size) to match full gradient descent in rate of convergence, while maintaining comparable sample-size-independent incremental first-order oracle complexity to stochastic gradient descent.
Efficient Stochastic Gradient Hard Thresholding
To address these deficiencies, we propose an efficient hybrid stochastic gradient hard thresholding (HSG-HT) method that can be provably shown to have sample-size-independent gradient evaluation and hard thresholding complexity bounds.
Bayesian Cycle-Consistent Generative Adversarial Networks via Marginalizing Latent Sampling
We evaluate the proposed Bayesian CycleGAN on multiple benchmark datasets, including Cityscapes, Maps, and Monet2photo.
Exploring RNN-Transducer for Chinese Speech Recognition
End-to-end approaches have drawn much attention recently for significantly simplifying the construction of an automatic speech recognition (ASR) system.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Modality Attention for End-to-End Audio-visual Speech Recognition
In this paper, we propose a novel multimodal attention based method for audio-visual speech recognition which could automatically learn the fused representation from both modalities based on their importance.
Audio-Visual Speech Recognition
Robust Speech Recognition
+2
An Online Attention-based Model for Speech Recognition
In previous work, researchers have shown that such architectures can acquire comparable results to state-of-the-art ASR systems, especially when using a bidirectional encoder and global soft attention (GSA) mechanism.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Spatio-temporal Edge Service Placement: A Bandit Learning Approach
To solve this problem and optimize the edge computing performance, we propose SEEN, a Spatial-temporal Edge sErvice placemeNt algorithm.
Spatial-Temporal Synergic Residual Learning for Video Person Re-Identification
We present a novel method - Spatial-Temporal Synergic Residual Network (STSRN) for this problem.
Deep Adversarial Subspace Clustering
To solve this issue, we propose a novel deep adversarial subspace clustering (DASC) model, which learns more favorable sample representations by deep learning for subspace clustering, and more importantly introduces adversarial learning to supervise sample representation learning and subspace clustering.
Ranked #2 on
Image Clustering
on coil-40
Understanding Generalization and Optimization Performance of Deep CNNs
Besides, we prove that for an arbitrary gradient descent algorithm, the computed approximate stationary point by minimizing empirical risk is also an approximate stationary point to the population risk.
Pedestrian-Synthesis-GAN: Generating Pedestrian Data in Real Scene and Beyond
The results show that our framework can smoothly synthesize pedestrians on background images of variations and different levels of details.
Ranked #2 on
Scene Text Recognition
on MSDA
Empirical Risk Landscape Analysis for Understanding Deep Neural Networks
This work aims to provide comprehensive landscape analysis of empirical risk in deep neural networks (DNNs), including the convergence behavior of its gradient, its stationary points and the empirical risk itself to their corresponding population counterparts, which reveals how various network parameters determine the convergence performance.
A Survey of Model Compression and Acceleration for Deep Neural Networks
Methods of parameter pruning and quantization are described first, after that the other techniques are introduced.
Jointly Attentive Spatial-Temporal Pooling Networks for Video-based Person Re-Identification
Person Re-Identification (person re-id) is a crucial task as its applications in visual surveillance and human-computer interaction.
Outlier-Robust Tensor PCA
Low-rank tensor analysis is important for various real applications in computer vision.
The Landscape of Deep Learning Algorithms
For an $l$-layer linear neural network, we prove its empirical risk uniformly converges to its population risk at the rate of $\mathcal{O}(r^{2l}\sqrt{d\log(l)}/\sqrt{n})$ with training sample size of $n$, the total weight dimension of $d$ and the magnitude bound $r$ of weight of each layer.
Context-Aware Online Learning for Course Recommendation of MOOC Big Data
In this respect, the key challenge is how to realize personalized course recommendation as well as to reduce the computing and storage costs for the tremendous course data.
Distributed Private Online Learning for Social Big Data Computing over Data Center Networks
An increasing number of people participate in social networks and massive online social data are obtained.
Differentially Private Online Learning for Cloud-Based Video Recommendation with Multimedia Big Data in Social Networks
In addition, none of them has considered both the privacy of users' contexts (e, g., social status, ages and hobbies) and video service vendors' repositories, which are extremely sensitive and of significant commercial value.
Differentially Private Distributed Online Learning
Thus, we use differential privacy to preserve the privacy of learners, and study the influence of guaranteeing differential privacy on the utility of the distributed online learning algorithm.
