Search Results for author:
Found 97 papers, 40 papers with code
M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
Instruction tuning has emerged as an effective strategy for achieving zero-shot generalization by finetuning pretrained models on diverse multimodal tasks.
ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understanding
Understanding biological processes, drug development, and biotechnological advancements requires detailed analysis of protein structures and sequences, a task in protein research that is inherently complex and time-consuming when performed manually.
Visual Agents as Fast and Slow Thinkers
With this novel design, we advocate a flexible system, hierarchical reasoning capabilities, and a transparent decision-making pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence.
Ranked #133 on
Visual Question Answering
on MM-Vet
W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering
In knowledge-intensive tasks such as open-domain question answering (OpenQA), Large Language Models (LLMs) often struggle to generate factual answers relying solely on their internal (parametric) knowledge.
Radiance Field Learners As UAV First-Person Viewers
First-Person-View (FPV) holds immense potential for revolutionizing the trajectory of Unmanned Aerial Vehicles (UAVs), offering an exhilarating avenue for navigating complex building structures.
Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models
We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output.
Inertial Confinement Fusion Forecasting via LLMs
Controlled fusion energy is deemed pivotal for the advancement of human civilization.
AMD: Automatic Multi-step Distillation of Large-scale Vision Models
Transformer-based architectures have become the de-facto standard models for diverse vision tasks owing to their superior performance.
Lateralization LoRA: Interleaved Instruction Tuning with Modality-Specialized Adaptations
Lateralization LoRA employs a hybrid approach, combining the traditional linear LoRA and a Convolutional LoRA for generating text and images, enabling the generation of high-quality text and images by leveraging modality-specific structures and parameter sets.
Direct Multi-Turn Preference Optimization for Language Agents
Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents.
InternalInspector $I^2$: Robust Confidence Estimation in LLMs through Internal States
Despite their vast capabilities, Large Language Models (LLMs) often struggle with generating reliable outputs, frequently producing high-confidence inaccuracies known as hallucinations.
CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAG
To address this issue, we explore the task of "credibility-aware RAG", in which LLMs automatically adjust the influence of retrieved documents based on their credibility scores to counteract misinformation.
EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction
Specifically, we employ a heavy encoder to separately encode the product context and attribute.
Self-supervised Adversarial Training of Monocular Depth Estimation against Physical-World Attacks
Our experiments with two representative MDE networks demonstrate improved robustness against various adversarial attacks, with minimal impact on benign performance.
ProMotion: Prototypes As Motion Learners
We capitalize on a dual mechanism involving the feature denoiser and the prototypical learner to decipher the intricacies of motion.
Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
The recent advancements in large language models (LLMs) have been extraordinary, yet the escalating inference costs associated with them present challenges in real-world applications.
HYDRA: Model Factorization Framework for Black-Box LLM Personalization
To address these challenges, we propose HYDRA, a model factorization framework that captures both user-specific behavior patterns from historical data and shared general knowledge among all users to deliver personalized generation.
Prototypical Transformer as Unified Motion Learners
In this work, we introduce the Prototypical Transformer (ProtoFormer), a general and unified framework that approaches various motion tasks from a prototype perspective.
User Welfare Optimization in Recommender Systems with Competing Content Creators
Driven by the new economic opportunities created by the creator economy, an increasing number of content creators rely on and compete for revenue generated from online content recommendation platforms.
Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
In this paper, we propose a novel parallel decoding approach, namely \textit{hidden transfer}, which decodes multiple successive tokens simultaneously in a single forward pass.
Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models
Parameter Efficient Fine-Tuning (PEFT) methods have been extensively utilized in Large Language Models (LLMs) to improve the down-streaming tasks without the cost of fine-tuing the whole LLMs.
Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection
Self-detection for Large Language Models (LLMs) seeks to evaluate the trustworthiness of the LLM's output by leveraging its own capabilities, thereby alleviating the issue of output hallucination.
FewFedPIT: Towards Privacy-preserving and Few-shot Federated Instruction Tuning
Federated instruction tuning (FedIT) has emerged as a promising solution, by consolidating collaborative training across multiple data owners, thereby resulting in a privacy-preserving learning model.
Large Language Models are Learnable Planners for Long-Term Recommendation
To this end, we propose a Bi-level Learnable LLM Planner framework, which consists of a set of LLM instances and breaks down the learning process into macro-learning and micro-learning to learn macro-level guidance and micro-level personalized recommendation policies, respectively.
PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the perspective of partial differential equations
Recent advancements in deep learning have led to the development of various models for long-term multivariate time-series forecasting (LMTF), many of which have shown promising results.
Multimodal Instruction Tuning with Conditional Mixture of LoRA
Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zero-shot generalization capabilities for unseen multimodal tasks.
Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
Despite vision-language models' (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining and visual instruction tuning, and (2) annotation error and bias in GPT-4 synthesized instruction tuning data.
C-ICL: Contrastive In-context Learning for Information Extraction
There has been increasing interest in exploring the capabilities of advanced large language models (LLMs) in the field of information extraction (IE), specifically focusing on tasks related to named entity recognition (NER) and relation extraction (RE).
Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models
Our in-depth study advocates more careful use of ME in real-world scenarios.
Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?
As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning.
Image Translation as Diffusion Visual Programmers
We introduce the novel Diffusion Visual Programmer (DVP), a neuro-symbolic image translation framework.
Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model
In this work we introduce VistaLLM a powerful visual system that addresses coarse- and fine grained VL tasks over single and multiple input images using a unified framework.
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
In this work, we introduce VistaLLM, a powerful visual system that addresses coarse- and fine-grained VL tasks over single and multiple input images using a unified framework.
RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training
Under a unified evaluation of fine-tuned LMs by incorporating four representative perspectives of model robustness, we demonstrate the effectiveness of RoAST compared to state-of-the-art fine-tuning methods on six different types of LMs, which indicates its usefulness in practice.
MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
Specifically, an adversarial LLM and a target LLM interplay with each other in an iterative manner, where the adversarial LLM aims to generate challenging prompts that elicit unsafe responses from the target LLM, while the target LLM is fine-tuned with safety aligned data on these adversarial prompts.
PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection
Drawing inspiration from Psychological Questionnaires, which are carefully designed by psychologists to evaluate individual personality traits through a series of targeted items, we argue that these items can be regarded as a collection of well-structured chain-of-thought (CoT) processes.
Improving Input-label Mapping with Demonstration Replay for In-context Learning
The effectiveness of ICL can be attributed to the strong language modeling capabilities of large language models (LLMs), which enable them to learn the mapping between input and labels based on in-context demonstrations.
CoLLM: Integrating Collaborative Embeddings into Large Language Models for Recommendation
In pursuit of superior recommendations for both cold and warm start scenarios, we introduce CoLLM, an innovative LLMRec methodology that seamlessly incorporates collaborative information into LLMs for recommendation.
Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression
In particular, our approach extracts knowledge from LLMs to construct a knowledge store, from which the small-scale model can retrieve relevant information and leverage it for effective inference.
Dual-Feedback Knowledge Retrieval for Task-Oriented Dialogue Systems
Efficient knowledge retrieval plays a pivotal role in ensuring the success of end-to-end task-oriented dialogue systems by facilitating the selection of relevant information necessary to fulfill user requests.
Attack Prompt Generation for Red Teaming and Defending Large Language Models
Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks.
Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients
Federated Learning (FL) has surged in prominence due to its capability of collaborative model training without direct data sharing.
Resprompt: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models
Breakdown analysis further highlights RESPROMPT particularly excels in complex multi-step reasoning: for questions demanding at least five reasoning steps, RESPROMPT outperforms the best CoT based benchmarks by a remarkable average improvement of 21. 1% on LLaMA-65B and 14. 3% on LLaMA2-70B.
Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variance
Central to this process is the technique of unbiased client sampling, which ensures a representative selection of clients.
FedAWARE: Maximizing Gradient Diversity for Heterogeneous Federated Server-side Optimization
Furthermore, our results show that \textsc{FedAWARE} can enhance the performance of FL algorithms as a plug-in module.
On the Equivalence of Graph Convolution and Mixup
We establish this equivalence mathematically by demonstrating that graph convolution networks (GCN) and simplified graph convolution (SGC) can be expressed as a form of Mixup.
ClusterFormer: Clustering As A Universal Visual Learner
This paper presents CLUSTERFORMER, a universal vision model that is based on the CLUSTERing paradigm with TransFORMER.
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
As a result, their performance suffers drastically on inputs longer than those encountered during training, substantially limiting their applications in real-world tasks involving long contexts such as encoding scientific articles, code repositories, or long dialogues.
mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning
A multi-view contrastive learning framework is introduced to encompass semantic contrasts between source, codeswitched, and target sentences, as well as contrasts among token-to-token relations.
E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning
Specifically, we introduce a set of learnable key-value prompts and visual prompts into self-attention and input layers, respectively, to improve the effectiveness of model fine-tuning.
LLM-Rec: Personalized Recommendation via Prompting Large Language Models
Notably, the success of LLM-Rec lies in its prompting strategies, which effectively tap into the language model's comprehension of both general and specific item characteristics.
Recommendation Unlearning via Influence Function
In the light that recent recommender models use historical data for both the constructions of the optimization loss and the computational graph (e. g., neighborhood aggregation), IFRU jointly estimates the direct influence of unusable data on optimization loss and the spillover influence on the computational graph to pursue complete unlearning.
MixPAVE: Mix-Prompt Tuning for Few-shot Product Attribute Value Extraction
In this work, we propose a novel prompt tuning approach with Mixed Prompts for few-shot Attribute Value Extraction, namely MixPAVE.
Meta-training with Demonstration Retrieval for Efficient Few-shot Learning
Meta-training allows one to leverage smaller models for few-shot generalization in a domain-general and task-agnostic manner; however, these methods alone results in models that may not have sufficient parameterization or knowledge to adapt quickly to a large variety of tasks.
FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods
This paper introduces the Fair Fairness Benchmark (\textsf{FFB}), a benchmarking framework for in-processing group fairness methods.
PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models
While transformer-based pre-trained language models (PLMs) have dominated a number of NLP applications, these models are heavy to deploy and expensive to use.
RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank
In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework.
AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes
We propose attribute-aware multimodal entity linking, where the input is a mention described with a text and image, and the goal is to predict the corresponding target entity from a multimodal knowledge base (KB) where each entity is also described with a text description, a visual image and a set of attributes and values.
Disentangled Phonetic Representation for Chinese Spelling Correction
Chinese Spelling Correction (CSC) aims to detect and correct erroneous characters in Chinese texts.
The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models
Chain-of-Thought (CoT) prompting enables large language models to solve complex reasoning problems by generating intermediate steps.
Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality
Along with this, we propose novel negative mining techniques in the scene graph space for improving attribute binding and relation understanding.
Ranked #1 on
Image Retrieval
on CREPE (Compositional REPresentation Evaluation)
(Recall@1 (HN-Comp, UC) metric)
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression
To enhance the knowledge transfer of model reasoning and generalization, we further explore multi-view attribution distillation on all potential decisions of the teacher.
Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales
Existing metrics like task performance of the LM generating the rationales, or similarity between generated and gold rationales are not good indicators of their human utility.
MMViT: Multiscale Multiview Vision Transformers
Our model encodes different views of the input signal and builds several channel-resolution feature stages to process the multiple views of the input at different resolutions in parallel.
Prediction then Correction: An Abductive Prediction Correction Method for Sequential Recommendation
To improve the accuracy of these models, some researchers have attempted to simulate human analogical reasoning to correct predictions for testing data by drawing analogies with the prediction errors of similar training data.
TransFlow: Transformer as Flow Learner
Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement.
Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning
It was shown that an adversary can poison a small part of the unlabeled data so that when a victim trains an SSL model on it, the final model will have a backdoor that the adversary can exploit.
SVT: Supertoken Video Transformer for Efficient Video Understanding
Whether by processing videos with fixed resolution from start to end or incorporating pooling and down-scaling strategies, existing video transformers process the whole video content throughout the network without specially handling the large portions of redundant information.
Multi-task Transformer with Relation-attention and Type-attention for Named Entity Recognition
There are three types of NER tasks, including flat, nested and discontinuous entity recognition.
Stochastic Clustered Federated Learning
Federated learning is a distributed learning framework that takes full advantage of private data samples kept on edge devices.
Representation Deficiency in Masked Language Modeling
In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing $\texttt{[MASK]}$ tokens, resulting in a representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted to downstream data without $\texttt{[MASK]}$ tokens.
Retiring $Δ$DP: New Distribution-Level Metrics for Demographic Parity
Unfortunately, in this paper, we reveal that the fairness metric $\Delta DP$ can not precisely measure the violation of demographic parity, because it inherently has the following drawbacks: i) zero-value $\Delta DP$ does not guarantee zero violation of demographic parity, ii) $\Delta DP$ values can vary with different classification thresholds.
Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration
Instance segmentation in videos, which aims to segment and track multiple objects in video frames, has garnered a flurry of research attention in recent years.
Orders Are Unwanted: Dynamic Deep Graph Convolutional Network for Personality Detection
Predicting personality traits based on online posts has emerged as an important task in many fields such as social network analysis.
Improved Adaptive Algorithm for Scalable Active Learning with Weak Labeler
We design a novel algorithmic template, Weak Labeler Active Cover (WL-AC), that is able to robustly leverage the lower quality weak labelers to reduce the query complexity while retaining the desired level of accuracy.
COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation
However, the bias inherent in user written text, often used for PTG model training, can inadvertently associate different levels of linguistic quality with users' protected attributes.
AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuning
Fine-tuning large pre-trained language models on downstream tasks is apt to suffer from overfitting when limited training data is available.
Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling
Transformer-based models have achieved great success on sentence pair modeling tasks, such as answer selection and natural language inference (NLI).
XPrompt: Exploring the Extreme of Prompt Tuning
Prompt tuning learns soft prompts to condition frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner.
Rethinking Missing Data: Aleatoric Uncertainty-Aware Recommendation
A standard choice is treating the missing data as negative training samples and estimating interaction likelihood between user-item pairs along with the observed interactions.
Autoregressive Entity Generation for End-to-End Task-Oriented Dialog
In either approach, the systems may generate a response with conflicting entity information.
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations
However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words.
Fall Detection from Audios with Audio Transformers
Our paper provides a novel, non-wearable, non-intrusive, and scalable solution for fall detection, deployed on an autonomous mobile robot equipped with a microphone.
Towards Unbiased Label Distribution Learning for Facial Pose Estimation Using Anisotropic Spherical Gaussian
In this paper, we propose an Anisotropic Spherical Gaussian (ASG)-based LDL approach for facial pose estimation.
MiniDisc: Minimal Distillation Schedule for Language Model Compression
In particular, motivated by the finding that the performance of the student is positively correlated to the scale-performance tradeoff of the teacher assistant, MiniDisc is designed with a $\lambda$-tradeoff to measure the optimality of the teacher assistant without trial distillation to the student.
GL-RG: Global-Local Representation Granularity for Video Captioning
Video captioning is a challenging task as it needs to accurately transform visual understanding into natural language description.
Deep Partial Multiplex Network Embedding
Network embedding is an effective technique to learn the low-dimensional representations of nodes in networks.
WebFormer: The Web-page Transformer for Structure Information Extraction
Structure information extraction refers to the task of extracting structured text fields from web pages, such as extracting a product offer from a shopping page including product title, description, brand and price.
MAVE: A Product Dataset for Multi-source Attribute Value Extraction
Attribute value extraction refers to the task of identifying values of an attribute of interest from product information.
Ranked #1 on
Attribute Value Extraction
on MAVE
Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
To this end, we analyze the working mechanism of GCN with causal graph, estimating the causal effect of a node's local structure for the prediction.
CatGCN: Graph Convolutional Networks with Categorical Node Features
In this paper, we propose a new GCN model named CatGCN, which is tailored for graph learning when the node features are categorical.
Learning to Extract Attribute Value from Product via Question Answering: A Multi-task Approach
In particular, we build a question answering model which treats each attribute as a question and identifies the answer span corresponding to the attribute value in the product context.
Big Bird: Transformers for Longer Sequences
To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear.
Ranked #1 on
Text Classification
on Arxiv HEP-TH citation graph
ETC: Encoding Long and Structured Inputs in Transformers
Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks.
Ranked #3 on
Question Answering
on ConditionalQA
Graph Convolution Machine for Context-aware Recommender System
The encoder projects users, items, and contexts into embedding vectors, which are passed to the GC layers that refine user and item embeddings with context-aware graph convolutions on user-item graph.
Ranking Preserving Hashing for Fast Similarity Search
But in many real world applications, ranking measure is important for evaluating the quality of hashing codes. In this paper, we propose a novel Ranking Preserving Hashing (RPH) approach that directly optimizes a popular ranking measure, Normalized Discounted Cumulative Gain (NDCG), to obtain effective hashing codes with high ranking accuracy.
