Search Results for author:
Found 100 papers, 38 papers with code
TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition
Considering this, we introduce an innovative PEFT method, TeamLoRA, consisting of a collaboration and competition module for experts, and thus achieving the right balance of effectiveness and efficiency: (i) For collaboration, a novel knowledge-sharing and -organizing mechanism is devised to appropriately reduce the scale of matrix operations, thereby boosting the training and inference speed.
E-CGL: An Efficient Continual Graph Learner
In the realm of continual graph learning, where graphs continuously evolve based on streaming graph data, continual graph learning presents unique challenges that require adaptive and efficient graph learning methods in addition to the problem of catastrophic forgetting.
Graph Retrieval-Augmented Generation: A Survey
Recently, Retrieval-Augmented Generation (RAG) has achieved remarkable success in addressing the challenges of Large Language Models (LLMs) without necessitating retraining.
Logic Distillation: Learning from Code Function by Function for Planning and Decision-making
Subsequently, based on the function base, LD fine-tunes S-LLMs to learn the logic employed by L-LLMs in planning and decision-making.
IDEAL: Leveraging Infinite and Dynamic Characterizations of Large Language Models for Query-focused Summarization
Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization.
Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference
The visual question generation (VQG) task aims to generate human-like questions from an image and potentially other side information (e. g. answer type).
Bridging Local Details and Global Context in Text-Attributed Graphs
Representation learning on text-attributed graphs (TAGs) is vital for real-world applications, as they combine semantic textual and contextual structural information.
Improving Large Models with Small models: Lower Costs and Better Performance
On the other hand, certain tasks can be broken down into multiple subtasks, some of which can be completed without powerful capabilities.
T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text
In this work, we propose a two-stage sign language production (SLP) paradigm that first encodes sign language sequences into discrete codes and then autoregressively generates sign language from text based on the learned codebook.
DuetRAG: Collaborative Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) methods augment the input of Large Language Models (LLMs) with relevant retrieved passages, reducing factual errors in knowledge-intensive tasks.
Auto-Encoding Morph-Tokens for Multimodal LLM
For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge.
WorldGPT: Empowering LLM as Multimodal World Model
As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios.
LASER: Tuning-Free LLM-Driven Attention Control for Efficient Text-conditioned Image-to-Animation
Revolutionary advancements in text-to-image models have unlocked new dimensions for sophisticated content creation, e. g., text-conditioned image editing, allowing us to edit the diverse images that convey highly complex visual concepts according to the textual guidance.
Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales
Their ability to execute intricate compositional reasoning tasks is also constrained, culminating in a stagnation of learning progression for these models.
HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models
Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks.
Ranked #112 on
Visual Question Answering
on MM-Vet
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
Specifically, 1) we design a neural codec with factorized vector quantization (FVQ) to disentangle speech waveform into subspaces of content, prosody, timbre, and acoustic details; 2) we propose a factorized diffusion model to generate attributes in each subspace following its corresponding prompt.
Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning
Large Language Models (LLMs) demonstrate remarkable proficiency in comprehending and handling text-based tasks.
Efficient Tuning and Inference for Large Language Models on Textual Graphs
In light of recent advancements in large language models (LLMs), it is apparent that integrating LLMs for enhanced textual encoding can substantially improve the performance of textual graphs.
DIEM: Decomposition-Integration Enhancing Multimodal Insights
On MM-Vet our method achieves an improvement in MM-Vet scores increasing from 31. 1 to 32. 4.
Language Model is a Branch Predictor for Simultaneous Machine Translation
Subsequently, we utilize the predicted source words to decode the output in advance.
HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data
Multi-modal Large Language Models (MLLMs) tuned on machine-generated instruction-following data have demonstrated remarkable performance in various multi-modal understanding and generation tasks.
De-fine: Decomposing and Refining Visual Programs with Auto-Feedback
Visual programming, a modular and generalizable paradigm, integrates different modules and Python operators to solve various vision-language tasks.
Revisiting the Domain Shift and Sample Uncertainty in Multi-source Active Domain Transfer
Active Domain Adaptation (ADA) aims to maximally boost model adaptation in a new target domain by actively selecting a limited number of target data to annotate. This setting neglects the more practical scenario where training data are collected from multiple sources.
Negative Sampling with Adaptive Denoising Mixup for Knowledge Graph Embedding
Most existing negative sampling methods assume that non-existent triples with high scores are high-quality negative triples.
GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning
These pre-trained models can be applied to various downstream Web applications, saving training time and improving downstream (target) performance.
Improving Vision Anomaly Detection with the Guidance of Language Modality
To learn a more compact latent space for the vision anomaly detector, CMLE learns a correlation structure matrix from the language modality, and then the latent space of vision modality will be learned with the guidance of the matrix.
I3: Intent-Introspective Retrieval Conditioned on Instructions
Recent studies indicate that dense retrieval models struggle to perform well on a wide variety of retrieval tasks that lack dedicated training data, as different retrieval tasks often entail distinct search intents.
Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model
Our approach employs a pretrained T2I diffusion model to generate each video frame in an autoregressive fashion.
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task.
Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion
In this work, we propose a novel strategy named \textbf{Degeneration-Tuning (DT)} to shield contents of unwanted concepts from SD weights.
MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning
In this work, we investigate the problem of out-of-distribution (OOD) generalization for unsupervised learning methods on graph data.
Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Document
GOSE initiates by generating preliminary relation predictions on entity pairs extracted from a scanned image of the document.
Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaboration
Recent text-to-image generation models have shown promising results in generating high-fidelity photo-realistic images.
InstructVid2Vid: Controllable Video Editing with Natural Language Instructions
We introduce InstructVid2Vid, an end-to-end diffusion-based methodology for video editing guided by human language instructions.
Continual Vision-Language Representation Learning with Off-Diagonal Information
Moreover, we empirically and theoretically demonstrate how SD leads to a performance decline for CLIP on cross-modal retrieval tasks.
SkillQG: Learning to Generate Question for Reading Comprehension Assessment
We present $\textbf{$\texttt{SkillQG}$}$: a question generation framework with controllable comprehension types for assessing and improving machine reading comprehension models.
Learning in Imperfect Environment: Multi-Label Classification with Long-Tailed Distribution and Partial Labels
Conventional multi-label classification (MLC) methods assume that all samples are fully labeled and identically distributed.
Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zero-shot manner.
Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization
Though effective, prompt tuning under few-shot settings on the one hand heavily relies on a good initialization of soft prompts.
SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
In this paper, we propose a novel dynamic early exiting combined with layer skipping for BERT inference named SmartBERT, which adds a skipping gate and an exiting operator into each layer of BERT.
Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by learning the ``soft prompts'' to condition frozen pre-training models.
Structure-Aware Group Discrimination with Adaptive-View Graph Encoder: A Fast Graph Contrastive Learning Framework
Albeit having gained significant progress lately, large-scale graph representation learning remains expensive to train and deploy for two main reasons: (i) the repetitive computation of multi-hop message passing and non-linearity in graph neural networks (GNNs); (ii) the computational cost of complex pairwise contrastive learning loss.
Lformer: Text-to-Image Generation with L-shape Block Parallel Decoding
To better accelerate the generative transformers while keeping good generation quality, we propose Lformer, a semi-autoregressive text-to-image generation model.
SGL-PT: A Strong Graph Learner with Graph Prompt Tuning
And aiming for graph classification task, we unify pre-training and fine-tuning by designing a novel verbalizer-free prompting function, which reformulates the downstream task in a similar format as pretext task.
A Study on ReLU and Softmax in Transformer
This paper sheds light on the following points: 1) Softmax and ReLU use different normalization methods over elements which lead to different variances of results, and ReLU is good at dealing with a large number of key-value slots; 2) FFN and key-value memory are equivalent, and thus the Transformer can be viewed as a memory network where FFNs and self-attention networks are both key-value memories.
Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding
To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.
DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention
Furthermore, we show that the hidden state dimension can be approximated by extending the Johnson-Lindenstrauss lemma, optimizing the attention in bilinear form.
Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction
Since the error rate of the incorrect sentence is usually low (e. g., 10\%), the correction model can only learn to correct on limited error tokens but trivially copy on most tokens (correct tokens), which harms the effective training of error correction.
Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical Weighted Self-contrastive Learning
Novel category discovery aims at adapting models trained on known categories to novel categories.
Distilling Task-specific Logical Rules from Large Pre-trained Models
Specifically, we borrow recent prompt-based language models as the knowledge expert to yield initial seed rules, and based on the formed high-quality instance pool that acts as an intermediary role, we keep teaching the expert to fit our task and learning task-specific logical rules.
Citation Trajectory Prediction via Publication Influence Representation Using Temporal Knowledge Graph
In this paper, we propose CTPIR, a new citation trajectory prediction framework that is able to represent the influence (the momentum of citation) of either new or existing publications using the history information of all their attributes.
Fine-Grained Semantically Aligned Vision-Language Pre-Training
Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks.
Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos
In this paper, we introduce a new task, named Temporal Emotion Localization in videos~(TEL), which aims to detect human emotions and localize their corresponding temporal boundaries in untrimmed videos with aligned subtitles.
BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval
In this scenario, the input image serves as an intuitive context and background for the search, while the corresponding language expressly requests new traits on how specific characteristics of the query image should be modified in order to get the intended target image.
Collaborative Intelligence Orchestration: Inconsistency-Based Fusion of Semi-Supervised Learning and Active Learning
Motivated by the industry practice of labeling data, we propose an innovative Inconsistency-based virtual aDvErsarial Active Learning (IDEAL) algorithm to further investigate SSL-AL's potential superiority and achieve mutual enhancement of AL and SSL, i. e., SSL propagates label information to unlabeled samples and provides smoothed embeddings for AL, while AL excludes samples with inconsistent predictions and considerable uncertainty for SSL.
Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptile
Moreover, when handling the data with noisy labels, the meta-learner could be extremely sensitive to label noise on a corrupted dataset.
RoSA: A Robust Self-Aligned Framework for Node-Node Graph Contrastive Learning
To the best of our awareness, RoSA is the first work focuses on the non-aligned node-node graph contrastive learning problem.
QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance
Existing metrics for assessing question generation not only require costly human reference but also fail to take into account the input context of generation, rendering the lack of deep understanding of the relevance between the generated questions and input contexts.
Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning
To systematically measure the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.
Feeding What You Need by Understanding What You Learned
In this paper, we argue that a deep understanding of model capabilities and data properties can help us feed a model with appropriate training data based on its learning status.
Learning To Learn by Jointly Optimizing Neural Architecture and Weights
Existing NAS-based meta-learning methods apply a two-stage strategy, i. e., first searching architectures and then re-training meta-weights on the searched architecture.
Boosting RGB-D Saliency Detection by Leveraging Unlabeled RGB Images
The depth estimation branch is trained with RGB-D images and then used to estimate the pseudo depth maps for all unlabeled RGB images to form the paired data.
Consensus Graph Representation Learning for Better Grounded Image Captioning
The contemporary visual captioning models frequently hallucinate objects that are not actually in a scene, due to the visual misclassification or over-reliance on priors that resulting in the semantic inconsistency between the visual information and the target lexical words.
Relational Graph Learning for Grounded Video Description Generation
Such a setting can help explain the decisions of captioning models and prevents the model from hallucinating object words in its description.
Learning to Generate Visual Questions with Noisy Supervision
The task of visual question generation (VQG) aims to generate human-like neural questions from an image and potentially other side information (e. g., answer type or the answer itself).
Self-Supervised Class Incremental Learning
Existing Class Incremental Learning (CIL) methods are based on a supervised classification framework sensitive to data labels.
Towards Communication-Efficient and Privacy-Preserving Federated Representation Learning
This paper investigates the feasibility of federated representation learning under the constraints of communication cost and privacy protection.
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals
It is a consensus that small models perform quite poorly under the paradigm of self-supervised contrastive learning.
Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inference
Secondly, we introduce semantic coherence learning to explicitly encourage the semantic coherence of the adaptive hierarchical graph network from three hierarchies.
Revisiting Catastrophic Forgetting in Class Incremental Learning
After investigating existing strategies, we observe that there is a lack of study on how to prevent the inter-phase confusion.
CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction
The journey of reducing noise from distant supervision (DS) generated training data has been started since the DS was first introduced into the relation extraction (RE) task.
Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training
In this paper, we propose collaborative adversarial training to improve the data utilization, which coordinates virtual adversarial training (VAT) and adversarial training (AT) at different levels.
Improving Weakly-supervised Object Localization via Causal Intervention
The recent emerged weakly supervised object localization (WSOL) methods can learn to localize an object in the image only using image-level labels.
Disentangled Motif-aware Graph Learning for Phrase Grounding
In this paper, we propose a novel graph learning framework for phrase grounding in the image.
Ranked #6 on
Phrase Grounding
on Flickr30k Entities Test
Differentiable Graph Optimization for Neural Architecture Search
To this end, we learn a differentiable graph neural network as a surrogate model to rank candidate architectures, which enable us to obtain gradient w. r. t the input architectures.
Ask Question with Double Hints: Visual Question Generation with Answer-awareness and Region-reference
The task of visual question generation~(VQG) aims to generate human-like questions from an image and potentially other side information (e. g. answer type or the answer itself).
Semi-Supervised Active Learning for Semi-Supervised Models: Exploit Adversarial Examples With Graph-Based Virtual Labels
Although current mainstream methods begin to combine SSL and AL (SSL-AL) to excavate the diverse expressions of unlabeled samples, these methods' fully supervised task models are still trained only with labeled data.
Robust Meta-learning with Noise via Eigen-Reptile
In particular, we first cast the meta-overfitting problem (overfitting on sampling and label noise) as a gradient noise problem since few available samples cause meta-learner to overfit on existing examples (clean or corrupted) of an individual task at every gradient step.
Run Away From your Teacher: a New Self-Supervised Approach Solving the Puzzle of BYOL
Recently, a newly proposed self-supervised framework Bootstrap Your Own Latent (BYOL) seriously challenges the necessity of negative samples in contrastive-based learning frameworks.
Connection-Adaptive Meta-Learning
In this paper, we aim to obtain better meta-learners by co-optimizing the architecture and meta-weights simultaneously.
Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
Recently, a newly proposed self-supervised framework Bootstrap Your Own Latent (BYOL) seriously challenges the necessity of negative samples in contrastive learning frameworks.
MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination
In this paper, we investigate the problem of text-to-pedestrian synthesis, which has many potential applications in art, design, and video surveillance.
Two Step Joint Model for Drug Drug Interaction Extraction
When patients need to take medicine, particularly taking more than one kind of drug simultaneously, they should be alarmed that there possibly exists drug-drug interaction.
Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling
In this paper, we focus on enhancing the generalization ability of the VIST model by considering the few-shot setting.
Neural-DINF: A Neural Network based Framework for Measuring Document Influence
Measuring the scholarly impact of a document without citations is an important and challenging problem.
Deep Sequential Feature Learning in Clinical Image Classification of Infectious Keratitis
Infectious keratitis is the most common entities of corneal diseases, in which pathogen grows in the cornea leading to inflammation and destruction of the corneal tissues.
NTIRE 2020 Challenge on Real Image Denoising: Dataset, Methods and Results
This challenge is based on a newly collected validation and testing image datasets, and hence, named SIDD+.
Quda: Natural Language Queries for Visual Data Analytics
To address this challenge, we present a new dataset, called Quda, that aims to help V-NLIs recognize analytic tasks from free-form natural language by training and evaluating cutting-edge multi-label classification models.
Generating Natural Language Adversarial Examples on a Large Scale with Generative Models
It can attack text classification models with a higher success rate than existing methods, and provide acceptable quality for humans in the meantime.
Grounded and Controllable Image Completion by Incorporating Lexical Semantics
Existing image completion procedure is highly subjective by considering only visual context, which may trigger unpredictable results which are plausible but not faithful to a grounded knowledge.
Deep Neural Network for Fast and Accurate Single Image Super-Resolution via Channel-Attention-based Fusion of Orientation-aware Features
Firstly, we proposed a novel Orientation-Aware feature extraction and fusion Module (OAM), which contains a mixture of 1D and 2D convolutional kernels (i. e., 5 x 1, 1 x 5, and 3 x 3) for extracting orientation-aware features.
Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
Visual navigation is a task of training an embodied agent by intelligently navigating to a target object (e. g., television) using only visual observations.
Learning Dynamic Context Augmentation for Global Entity Linking
Despite of the recent success of collective entity linking (EL) methods, these "global" inference methods may yield sub-optimal results when the "all-mention coherence" assumption breaks, and often suffer from high computational cost at the inference stage, due to the complex search space.
Ranked #5 on
Entity Disambiguation
on AIDA-CoNLL
Walking with MIND: Mental Imagery eNhanceD Embodied QA
The experimental results and further analysis prove that the agent with the MIND module is superior to its counterparts not only in EQA performance but in many other aspects such as route planning, behavioral interpretation, and the ability to generalize from a few examples.
Informative Visual Storytelling with Cross-modal Rules
To solve this problem, we propose a method to mine the cross-modal rules to help the model infer these informative concepts given certain visual input.
Ranked #11 on
Visual Storytelling
on VIST
KCAT: A Knowledge-Constraint Typing Annotation Tool
Fine-grained Entity Typing is a tough task which suffers from noise samples extracted from distant supervision.
Posterior-regularized REINFORCE for Instance Selection in Distant Supervision
This paper provides a new way to improve the efficiency of the REINFORCE training process.
Improving Distantly-supervised Entity Typing with Compact Latent Space Clustering
Recently, distant supervision has gained great success on Fine-grained Entity Typing (FET).
Cross-relation Cross-bag Attention for Distantly-supervised Relation Extraction
Distant supervision leverages knowledge bases to automatically label instances, thus allowing us to train relation extractor without human annotations.
NITE: A Neural Inductive Teaching Framework for Domain Specific NER
In domain-specific NER, due to insufficient labeled training data, deep models usually fail to behave normally.
