Search Results for author:
Found 73 papers, 32 papers with code
MIRTT: Learning Multimodal Interaction Representations from Trilinear Transformers for Visual Question Answering
On the other hand, trilinear models such as the CTI model efficiently utilize the inter-modality information between answers, questions, and images, while ignoring intra-modality information.
Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems
Joint-GCG's innovative unification of gradient-based attacks across retrieval and generation stages fundamentally reshapes our understanding of vulnerabilities within RAG systems.
AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery
In this paper, we propose AdInject, a novel and real-world black-box attack method that leverages the internet advertising delivery to inject malicious content into the Web Agent's environment.
Fast and Accurate Power Load Data Completion via Regularization-optimized Low-Rank Factorization
Low-rank representation learning has emerged as a powerful tool for recovering missing values in power load data due to its ability to exploit the inherent low-dimensional structures of spatiotemporal measurements.
DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain ``content'' and ``context'' features respectively.
PIV-FlowDiffuser:Transfer-learning-based denoising diffusion models for PIV
To reduce the special noise step-by-step, we employ a denoising diffusion model~(FlowDiffuser) for PIV analysis.
YuE: Scaling Open Foundation Models for Long-Form Music Generation
We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture.
Vulnerability of Text-to-Image Models to Prompt Template Stealing: A Differential Evolution Approach
Prompt trading has emerged as a significant intellectual property concern in recent years, where vendors entice users by showcasing sample images before selling prompt templates that can generate similar images.
Explore-Construct-Filter: An Automated Framework for Rich and Reliable API Knowledge Graph Construction
The API Knowledge Graph (API KG) is a structured network that models API entities and their relations, providing essential semantic insights for tasks such as API recommendation, code generation, and API misuse detection.
Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System
Inspired by the concept of mimicking the familiar, AutoCMD is capable of inferring the information utilized by upstream tools in the toolchain through learning on open-source systems and reinforcement with target system examples, thereby generating more targeted commands for information theft.
OntoTune: Ontology-Driven Self-training for Aligning Large Language Models
Compared to existing domain LLMs based on newly collected large-scale domain-specific corpora, our OntoTune, which relies on the existing, long-term developed ontology and LLM itself, significantly reduces data maintenance costs and offers improved generalization ability.
Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective
Large Language Models (LLMs) have made notable progress in mathematical reasoning, yet often rely on single-paradigm reasoning, limiting their effectiveness across diverse tasks.
Joint Secrecy Rate Achieving and Authentication Enhancement via Tag-based Encoding in Chaotic UAV Communication Environment
To protect legitimate communication in a chaotic UAV environment, where both eavesdropping and jamming become straightforward from multiple adversaries with line-of-sight signal propagation, a new reliable and integrated physical layer security mechanism is proposed in this paper for a massive multiple-input-multiple-output (MIMO) UAV system.
Identity-Clothing Similarity Modeling for Unsupervised Clothing Change Person Re-Identification
Clothing change person re-identification (CC-ReID) aims to match different images of the same person, even when the clothing varies across images.
Understanding Individual Agent Importance in Multi-Agent System via Counterfactual Reasoning
Inspired by counterfactual reasoning, a larger change in reward caused by the randomized action of agent indicates its higher importance.
What External Knowledge is Preferred by LLMs? Characterizing and Exploring Chain of Evidence in Imperfect Context
Incorporating external knowledge into large language models (LLMs) has emerged as a promising approach to mitigate outdated knowledge and hallucination in LLMs.
Multi-Domain Features Guided Supervised Contrastive Learning for Radar Target Detection
In this letter, we propose a multi-domain features guided supervised contrastive learning (MDFG_SCL) method, which integrates statistical features derived from multi-domain differences with deep features obtained through supervised contrastive learning, thereby capturing both low-level domain-specific variations and high-level semantic information.
From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection
To fill this gap, we present ToolCommander, a novel framework designed to exploit vulnerabilities in LLM tool-calling systems through adversarial tool injection.
CodePurify: Defend Backdoor Attacks on Neural Code Models via Entropy-based Purification
In this study, we propose CodePurify, a novel defense against backdoor attacks on code models through entropy-based purification.
Multi-path Exploration and Feedback Adjustment for Text-to-Image Person Retrieval
Text-based person retrieval aims to identify the specific persons using textual descriptions as queries.
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
By stacking training optimizations, we demonstrate accelerations of 65. 08% with 1D parallelism at the 128-GPU scale (Llama 3. 1 8B), an additional 12. 59% with 2D parallelism at the 256-GPU scale (Llama 3. 1 70B), and an additional 30% with 3D parallelism at the 512-GPU scale (Llama 3. 1 405B) on NVIDIA H100 GPUs over optimized baselines.
Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models
We introduce the MTRB (massive tool retrieval benchmark) to evaluate real-world tool-augmented LLM scenarios with a large number of tools.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Question Answering
on PeerQA
ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
To address this challenge, we introduce a comprehensive diagnostic benchmark, ToolBH.
TrustUQA: A Trustful Framework for Unified Structured Data Question Answering
Further more, we have demonstrated the potential of our method for more general QA tasks, QA over mixed structured data and QA across structured data.
GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality
The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process.
Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement
The most prominent issue among these semantic inconsistencies is catastrophic-neglect, where the images generated by T2I DMs miss key objects mentioned in the prompt.
Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs
LLMs fine-tuned with this data have improved planning capabilities, better equipping them to handle complex QA tasks that involve retrieval.
PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documents
Recent advancements in Large Multimodal Models (LMMs) have leveraged extensive multimodal datasets to enhance capabilities in complex knowledge-driven tasks.
HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playing
Generative AI has demonstrated unprecedented creativity in the field of computer vision, yet such phenomena have not been observed in natural language processing.
ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation
We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs).
Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data.
MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning
To address these issues, a natural idea is to enhance the independence and diversity of the learning process for the low-rank matrices.
OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision
However, existing open-vocabulary detectors trained on base category data tend to assign higher confidence to trained categories and confuse novel categories with the background.
Ranked #3 on
Open Vocabulary Object Detection
on LVIS v1.0
VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing
Visual entailment (VE) is a multimodal reasoning task consisting of image-sentence pairs whereby a promise is defined by an image, and a hypothesis is described by a sentence.
Adversarial Testing for Visual Grounding via Image-Aware Property Reduction
To this end, we propose PEELING, a text perturbation approach via image-aware property reduction for adversarial testing of the VG model.
Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems
Subsequently, the Consistency Check is applied to determine the presence of non-optimal PPDs by comparing the driving paths in the original and mutated scenarios.
StructLM: Towards Building Generalist Models for Structured Knowledge Grounding
Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Mistral and the CodeLlama model family, ranging from 7B to 34B parameters.
Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues
With the development of LLMs, the security threats of LLMs are getting more and more attention.
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context.
Know Your Needs Better: Towards Structured Understanding of Marketer Demands with Analogical Reasoning Augmented LLMs
To stimulate the LLMs' reasoning ability, the chain-of-thought (CoT) prompting method is widely used, but existing methods still have some limitations in our scenario: (1) Previous methods either use simple "Let's think step by step" spells or provide fixed examples in demonstrations without considering compatibility between prompts and concrete questions, making LLMs ineffective when the marketers' demands are abstract and diverse.
AdapterDistillation: Non-Destructive Task Composition with Knowledge Distillation
In the second stage, we distill the knowledge from the existing teacher adapters into the student adapter to help its inference.
A Survey on Query-based API Recommendation
We also investigate the typical data extraction procedures and collection approaches employed by the existing approaches.
From Beginner to Expert: Modeling Medical Knowledge into General LLMs
(3) Specifically for multi-choice questions in the medical domain, we propose a novel Verification-of-Choice approach for prompting engineering, which significantly enhances the reasoning ability of LLMs.
GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions
Specifically, we first extract the region of interest (RoI) corresponding to the text instruction, aligning it to 3D Gaussians.
Reliable Academic Conference Question Answering: A Study Based on Large Language Model
Specifically, for each conference, we first organize academic conference data in a tree-structured format through a semi-automated method.
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
In recent times, the generation of 3D assets from text prompts has shown impressive results.
EALM: Introducing Multidimensional Ethical Alignment in Conversational Information Retrieval
Artificial intelligence (AI) technologies should adhere to human norms to better serve our society and avoid disseminating harmful or misleading information, particularly in Conversational Information Retrieval (CIR).
UniEX: An Effective and Efficient Framework for Unified Information Extraction via a Span-extractive Perspective
We propose a new paradigm for universal information extraction (IE) that is compatible with any schema format and applicable to a list of IE tasks, such as named entity recognition, relation extraction, event extraction and sentiment analysis.
NER-to-MRC: Named-Entity Recognition Completely Solving as Machine Reading Comprehension
Named-entity recognition (NER) detects texts with predefined semantic labels and is an essential building block for natural language processing (NLP).
Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning
The latent world model provides a promising way to learn policies in a compact latent space for tasks with high-dimensional observations, however, its generalization across diverse environments with unseen dynamics remains challenging.
Model-based Reinforcement Learning
reinforcement-learning
+1
Solving Math Word Problems via Cooperative Reasoning induced Language Models
This inspires us to develop a cooperative reasoning-induced PLM for solving MWPs, called Cooperative Reasoning (CoRe), resulting in a human-like reasoning architecture with system 1 as the generator and system 2 as the verifier.
Ranked #114 on
Arithmetic Reasoning
on GSM8K
Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks
The noise is due to a fatal bug in the backend of our relevance assessment interface.
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective
We propose a new paradigm for zero-shot learners that is format agnostic, i. e., it is compatible with any format and applicable to a list of language tasks, such as text classification, commonsense reasoning, coreference resolution, and sentiment analysis.
LeVoice ASR Systems for the ISCSLP 2022 Intelligent Cockpit Speech Recognition Challenge
This paper describes LeVoice automatic speech recognition systems to track2 of intelligent cockpit speech recognition challenge 2022.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+8
MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model
Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets.
TC-SKNet with GridMask for Low-complexity Classification of Acoustic scene
In our experiments, the performance gain brought by GridMask is stronger than spectrum augmentation in ASCs.
Im2Oil: Stroke-Based Oil Painting Rendering with Linearly Controllable Fineness Via Adaptive Sampling
Comparison with existing state-of-the-art oil painting techniques shows that our results have higher fidelity and more realistic textures.
Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence
We hope that this project will be the foundation of Chinese cognitive intelligence.
Towards No.1 in CLUE Semantic Matching Challenge: Pre-trained Language Model Erlangshen with Propensity-Corrected Loss
This report describes a pre-trained language model Erlangshen with propensity-corrected loss, the No. 1 in CLUE Semantic Matching Challenge.
Change Detection from Synthetic Aperture Radar Images via Dual Path Denoising Network
We also propose the distinctive patch convolution for feature representation learning to reduce the time consumption.
Change Detection from Synthetic Aperture Radar Images via Graph-Based Knowledge Supplement Network
Synthetic aperture radar (SAR) image change detection is a vital yet challenging task in the field of remote sensing image analysis.
Adaptive DropBlock Enhanced Generative Adversarial Networks for Hyperspectral Image Classification
Second, an adaptive DropBlock (AdapDrop) is proposed as a regularization method employed in the generator and discriminator to alleviate the mode collapse issue.
Low-Latency Online Speaker Diarization with Graph-Based Label Generation
To ensure the low latency in the online setting, we introduce a variant of AHC, namely chkpt-AHC, to cluster the speakers.
Benchmarking Lane-changing Decision-making for Deep Reinforcement Learning
We train several state-of-the-art deep reinforcement learning methods in the designed training scenarios and provide the benchmark metrics evaluation results of the trained models in the test scenarios.
The DKU-DukeECE-Lenovo System for the Diarization Task of the 2021 VoxCeleb Speaker Recognition Challenge
This report describes the submission of the DKU-DukeECE-Lenovo team to the VoxCeleb Speaker Recognition Challenge (VoxSRC) 2021 track 4.
Sparsely Overlapped Speech Training in the Time Domain: Joint Learning of Target Speech Separation and Personal VAD Benefits
This paper proposes the weighted SI-SNR loss, together with the joint learning of target speech separation and personal VAD.
Change Detection from SAR Images Based on Deformable Residual Convolutional Neural Networks
Convolutional neural networks (CNN) have made great progress for synthetic aperture radar (SAR) images change detection.
TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling
Also, the TransfoRNN model was applied on the LibriSpeech speech recognition task and has shown comparable results with the Transformer models.
Towards Alleviating the Modeling Ambiguity of Unsupervised Monocular 3D Human Pose Estimation
Furthermore, two temporal constraints are proposed to alleviate the scale and pose ambiguity respectively.
Monocular 3D Human Pose Estimation
Unsupervised 3D Human Pose Estimation
Skeleton2Mesh: Kinematics Prior Injected Unsupervised Human Mesh Recovery
The challenges of the latter task are two folds: (1) pose failure (i. e., pose mismatching -- different skeleton definitions in dataset and SMPL , and pose ambiguity -- endpoints have arbitrary joint angle configurations for the same 3D joint coordinates).
Training Wake Word Detection with Synthesized Speech Data on Confusion Words
Confusing-words are commonly encountered in real-life keyword spotting applications, which causes severe degradation of performance due to complex spoken terms and various kinds of words that sound similar to the predefined keywords.
