Search Results for author:
Found 460 papers, 159 papers with code
Speeding up Transformer Decoding via an Attention Refinement Network
Extensive experiments on ten WMT machine translation tasks show that the proposed model yields an average of 1. 35x faster (with almost no decrease in BLEU) over the state-of-the-art inference implementation.
Rare and Zero-shot Word Sense Disambiguation using Z-Reweighting
However, the imbalanced training dataset leads to poor performance on rare senses and zero-shot senses.
Toward Knowledge-Enriched Conversational Recommendation Systems
Conversational Recommendation Systems recommend items through language based interactions with users. In order to generate naturalistic conversations and effectively utilize knowledge graphs (KGs) containing background information, we propose a novel Bag-of-Entities loss, which encourages the generated utterances to mention concepts related to the item being recommended, such as the genre or director of a movie.
Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training
PRIOR introduces a reference model-a text-only large language model (LLM) trained on the captions without image inputs, to weight each token based on its probability for LVLMs training.
HuB: Learning Extreme Humanoid Balance
The human body demonstrates exceptional motor capabilities-such as standing steadily on one foot or performing a high kick with the leg raised over 1. 5 meters-both requiring precise balance control.
DeepSORT-Driven Visual Tracking Approach for Gesture Recognition in Interactive Systems
It can accurately capture and track the user's gesture trajectory and is superior to traditional tracking methods in terms of real-time and accuracy.
RM-R1: Reward Modeling as Reasoning
The training of M-R1 consists of two key stages: (1) distillation of high-quality reasoning chains and (2) reinforcement learning with verifiable rewards.
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
Chain-of-thought (CoT) reasoning in large language models (LLMs) can be formalized as a latent variable problem, where the model needs to generate intermediate reasoning steps.
Multi-Step Consistency Models: Fast Generation with Theoretical Guarantees
Additionally, under minimal assumptions on the data distribution an increasingly common setting in recent diffusion model analyses we show that a similar KL convergence guarantee can be obtained, with the number of steps scaling as $ O\left(d \log\left(\frac{d}{\varepsilon}\right)\right) $.
Capturing Conditional Dependence via Auto-regressive Diffusion Models
On the other hand, when there is no obvious conditional dependence across patches of the data, AR diffusion does not outperform DDPM.
STCL:Curriculum learning Strategies for deep learning image steganography models
Aiming at the problems of poor quality of steganographic images and slow network convergence of image steganography models based on deep learning, this paper proposes a Steganography Curriculum Learning training strategy (STCL) for deep learning image steganography models.
MMHCL: Multi-Modal Hypergraph Contrastive Learning for Recommendation
For a comprehensive information exploration from user-product relations, we construct two hypergraphs, i. e. a user-to-user (u2u) hypergraph and an item-to-item (i2i) hypergraph, to mine shared preferences among users and intricate multimodal semantic resemblance among items, respectively.
CLPSTNet: A Progressive Multi-Scale Convolutional Steganography Model Integrating Curriculum Learning
In recent years, a large number of works have introduced Convolutional Neural Networks (CNNs) into image steganography, which transform traditional steganography methods such as hand-crafted features and prior knowledge design into steganography methods that neural networks autonomically learn information embedding.
EarthGPT-X: Enabling MLLMs to Flexibly and Comprehensively Understand Multi-Source Remote Sensing Imagery
Recent advances in the visual-language area have developed natural multi-modal large language models (MLLMs) for spatial reasoning through visual prompting.
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
In this work, we revisit GRPO from a reinforce-like algorithm perspective and analyze its core components.
Multi-Modal Hypergraph Enhanced LLM Learning for Recommendation
To this end, we propose a novel framework, Hypergraph Enhanced LLM Learning for multimodal Recommendation (HeLLM), designed to equip LLMs with the capability to capture intricate higher-order semantic correlations by fusing graph-level contextual signals with sequence-level behavioral patterns.
AdaptRec: A Self-Adaptive Framework for Sequential Recommendations with Large Language Models
AdaptRec employs a two-phase user selection mechanism -- User Similarity Retrieval and Self-Adaptive User Selection -- to efficiently identify relevant user sequences in large-scale datasets from multi-metric evaluation.
Refining CLIP's Spatial Awareness: A Visual-Centric Perspective
Contrastive Language-Image Pre-training (CLIP) excels in global alignment with language but exhibits limited sensitivity to spatial information, leading to strong performance in zero-shot classification tasks but underperformance in tasks requiring precise spatial understanding.
VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models
Our results reveal that even the state-of-the-art LVLMs struggle with these puzzles, highlighting fundamental limitations in their puzzle-solving capabilities.
ASGO: Adaptive Structured Gradient Optimization
Training deep neural networks (DNNs) is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than simple vectors.
FACE: Few-shot Adapter with Cross-view Fusion for Cross-subject EEG Emotion Recognition
This article introduces the few-shot adapter with a cross-view fusion method called FACE for cross-subject EEG emotion recognition, which leverages dynamic multi-view fusion and effective subject-specific adaptation.
Dynamic Topic Analysis in Academic Journals using Convex Non-negative Matrix Factorization Method
In Stage 2, a convex optimization algorithm refines the dynamic topic structure using the convex NMF (cNMF) model, further enhancing topic integration and stability.
Generating Multimodal Driving Scenes via Next-Scene Prediction
Generative models in Autonomous Driving (AD) enable diverse scene creation, yet existing methods fall short by only capturing a limited range of modalities, restricting the capability of generating controllable scenes for comprehensive evaluation of AD systems.
Monte Carlo Diffusion for Generalizable Learning-Based RANSAC
Random Sample Consensus (RANSAC) is a fundamental approach for robustly estimating parametric models from noisy data.
ROCM: RLHF on consistency models
Diffusion models have revolutionized generative modeling in continuous domains like image, audio, and video synthesis.
Integrating network pharmacology, metabolomics, and gut microbiota analysis to explore the effects of Jinhong tablets on chronic superficial gastritis
Differential metabolites in plasma were determined by untargeted metabolomics, and gut microbiota diversity/composition in fecal and cecal samples was assessed via 16S rRNA sequencing.
SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection
To address these two problems above, we propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection (SEOE) by constructing a more representative evaluation benchmark and introducing a semantic evaluation metric.
MA-LoT: Multi-Agent Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Proving
To solve these issues, we propose MA-LoT: Multi-Agent Lean-based Long Chain-of-Thought framework, (to the best of our knowledge), the first multi-agent framework for Lean4 theorem proving that balance high-level NL reasoning and FL verification in Long CoT.
FANS -- Formal Answer Selection for Natural Language Math Reasoning Using Lean4
To the best of our knowledge, it is the first framework that utilizes Lean4 to enhance LLMs' NL math reasoning ability.
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
However, in most existing methods for LLM alignment, all tokens in the response are optimized using a sparse, response-level reward or preference annotation.
MICINet: Multi-Level Inter-Class Confusing Information Removal for Reliable Multimodal Classification
MICINet achieves the reliable removal of both types of noise by unifying them into the concept of Inter-class Confusing Information (\textit{ICI}) and eliminating it at both global and individual levels.
Self-rewarding correction for mathematical reasoning
We study self-rewarding reasoning large language models (LLMs), which can simultaneously generate step-by-step reasoning and evaluate the correctness of their outputs during the inference time-without external feedback.
Reverse Markov Learning: Multi-Step Generative Models for Complex Distributions
We propose a framework that defines a general forward process transitioning from the target distribution to a known distribution (e. g., Gaussian) and then learns a reverse Markov process using multiple engression models.
EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Leveraging Multi-modal Large Language Models (MLLMs) to create embodied agents offers a promising avenue for tackling real-world tasks.
Crime Forecasting: A Spatio-temporal Analysis with Deep Learning Models
We conducted a comparative analysis to access the effects of various data sequences, including raw and binned data, on the prediction errors of four deep learning forecasting models.
Logarithmic Regret for Online KL-Regularized Reinforcement Learning
Recent advances in Reinforcement Learning from Human Feedback (RLHF) have shown that KL-regularization plays a pivotal role in improving the efficiency of RL fine-tuning for large language models (LLMs).
Demystifying Singular Defects in Large Language Models
Large transformer models are known to produce high-norm tokens.
Nearly Optimal Sample Complexity of Offline KL-Regularized Contextual Bandits under Single-Policy Concentrability
KL-regularized policy optimization has become a workhorse in learning-based decision making, while its theoretical understanding is still very limited.
Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training
Small language models (SLMs) have attracted considerable attention from both academia and industry due to their broad range of applications in edge devices.
Volumetric Temporal Texture Synthesis for Smoke Stylization using Neural Cellular Automata
Artistic stylization of 3D volumetric smoke data is still a challenge in computer graphics due to the difficulty of ensuring spatiotemporal consistency given a reference style image, and that within reasonable time and computational resources.
Catoni Contextual Bandits are Robust to Heavy-tailed Rewards
When the variance of the reward at each round is known, we use a variance-weighted regression approach and establish a regret bound that depends only on the cumulative reward variance and logarithmically on the reward range $R$ as well as the number of rounds $T$.
Reformulation is All You Need: Addressing Malicious Text Features in DNNs
Human language encompasses a wide range of intricate and diverse implicit features, which attackers can exploit to launch adversarial or backdoor attacks, compromising DNN models for NLP tasks.
Online-BLS: An Accurate and Efficient Online Broad Learning System for Data Stream Classification
Different from employing existing incremental broad learning algorithms for online learning tasks, which tend to incur degraded accuracy and expensive online update overhead, we design an effective weight estimation algorithm and an efficient online updating strategy to remedy the above two deficiencies, respectively.
Divergence-Augmented Policy Optimization
In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data.
RAG-Reward: Optimizing RAG with Reward Modeling and RLHF
Using \textbf{RAG-Reward}, we train reward models and apply reinforcement learning with human feedback (RLHF) to improve LLMs' effectiveness in RAG.
Computer Vision-Driven Gesture Recognition: Toward Natural and Intuitive Human-Computer
This paper proposes a gesture recognition method based on a three-dimensional hand skeleton model.
Multi-QuAD: Multi-Level Quality-Adaptive Dynamic Network for Reliable Multimodal Classification
This paper finds that existing reliable multimodal classification methods not only fail to provide robust estimation of data quality, but also lack dynamic networks for sample-specific depth and parameters to achieve reliable inference.
A New Perspective on Time Series Anomaly Detection: Faster Patch-based Broad Learning System
Time series anomaly detection (TSAD) has been a research hotspot in both academia and industry in recent years.
Mixture of Physical Priors Adapter for Parameter-Efficient Fine-Tuning
Most parameter-efficient fine-tuning (PEFT) methods rely on low-rank representations to adapt models.
MatchDiffusion: Training-free Generation of Match-cuts
Match-cuts are powerful cinematic tools that create seamless transitions between scenes, delivering strong visual and metaphorical connections.
Scaling Mesh Generation via Compressive Tokenization
We propose a compressive yet effective mesh representation, Blocked and Patchified Tokenization (BPT), facilitating the generation of meshes exceeding 8k faces.
Alopex: A Computational Framework for Enabling On-Device Function Calls with LLMs
The rapid advancement of Large Language Models (LLMs) has led to their increased integration into mobile devices for personalized assistance, which enables LLMs to call external API functions to enhance their performance.
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF
To understand the fundamental distinction between policy learning objectives with KL-regularization and ones without KL-regularization, we are the first to theoretically demonstrate the power of KL-regularization by providing a sharp analysis for KL-regularized contextual bandits and RLHF, revealing an $\mathcal{O}(1 / \epsilon)$ sample complexity when $\epsilon$ is sufficiently small.
SEE-DPO: Self Entropy Enhanced Direct Preference Optimization
Direct Preference Optimization (DPO) has been successfully used to align large language models (LLMs) according to human preferences, and more recently it has also been applied to improving the quality of text-to-image diffusion models.
Self-Ensembling Gaussian Splatting for Few-Shot Novel View Synthesis
In this paper, we address this overfitting issue by introducing Self-Ensembling Gaussian Splatting (SE-GS).
Unlocking Comics: The AI4VA Dataset for Visual Understanding
This dataset is a crucial component of the AI4VA Workshop Challenges~\url{https://sites. google. com/view/ai4vaeccv2024}, where we specifically explore depth and saliency.
Fully First-Order Methods for Decentralized Bilevel Optimization
This paper focuses on decentralized stochastic bilevel optimization (DSBO) where agents only communicate with their neighbors.
Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code
The underlying cause of this issue is the gap between natural language to programming language gap (NL-PL Gap), which is especially pronounced in LRPLs due to limited aligned data.
Transcriptome and Redox Proteome Reveal Temporal Scales of Carbon Metabolism Regulation in Model Cyanobacteria Under Light Disturbance
We develop a systems approach based on an energy-landscape concept to differentiate interactions involving redox activities and conformational changes of proteins and nucleic acids interactions in multi-layered protein-DNA regulatory networks under light disturbance.
Personalized Visual Instruction Tuning
In this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image and engage in personalized and coherent dialogues.
Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference
Based on observing activations from large language models, outliers can be classified into channel-wise and spike outliers.
From Lists to Emojis: How Format Bias Affects Model Alignment
In this work, we extend the study of biases in preference learning beyond the commonly recognized length bias, offering a comprehensive analysis of a wider range of format biases.
Data Augmentation via Latent Diffusion for Saliency Prediction
We propose a novel data augmentation method for deep saliency prediction that edits natural images while preserving the complexity and variability of real-world scenes.
On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
These findings highlight that DPORM has limited generalization ability and substantiates the integration of an explicit reward model in iterative DPO approaches.
Building Math Agents with Multi-Turn Iterative Preference Learning
Recent studies have shown that large language models' (LLMs) mathematical problem-solving capabilities can be enhanced by integrating external tools, such as code interpreters, and employing multi-turn Chain-of-Thought (CoT) reasoning.
A Homogeneous Graph Neural Network for Precoding and Power Allocation in Scalable Wireless Networks
Lastly, using ICGNN as the core algorithm, we tailor the neural network's input and output for specific problem requirements and validate its performance in two scenarios: 1) in cellular networks, we develop a matrix-inverse-free multi-user multi-input multi-output (MU-MIMO) precoding scheme using the conjugate gradient (CG) method, adaptable to varying user and antenna numbers; 2) in a cell-free network, facing dynamic variations in the number of users served by APs, the number of APs serving each user, and the number of antennas per AP, we propose a universal power allocation scheme.
Learning-based Multi-View Stereo: A Survey
3D reconstruction aims to recover the dense 3D structure of a scene.
Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic
Our algorithm works in two steps: i) Localization: identify tiny ($1\%$ of the total parameters) localized regions in the finetuned models containing essential skills for the downstream tasks, and ii) Stitching: reintegrate only these essential regions back into the pretrained model for task synergy.
Practical Video Object Detection via Feature Selection and Aggregation
In principle, the detection in a certain frame of a video can benefit from information in other frames.
Ranked #1 on
Video Object Detection
on ImageNet VID
(using extra training data)
Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
This paper investigates multi-objective reinforcement learning (MORL), which focuses on learning Pareto optimal policies in the presence of multiple reward functions.
Multi-Objective Reinforcement Learning
reinforcement-learning
+1
SINDER: Repairing the Singular Defects of DINOv2
Vision Transformer models trained on large-scale datasets, although effective, often exhibit artifacts in the patch token they extract.
TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data
Achieving this goal poses non-trivial challenges: 1) data selection requires accurate data representations that reflect the training samples' quality, 2) considering the diverse nature of instruction datasets, and 3) ensuring the efficiency of the coreset selection algorithm for large models.
EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensing
Specifically, a shared visual encoding method is developed to establish the spatial pattern interpretation relationships between the multi-scale representations of input images and various visual prompts.
SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization
Recent studies have revealed that, during the inference on generative AI models such as transformer, the importance of different weights exhibits substantial context-dependent variations.
Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsize
In this paper, we revisit \textsf{ROOT-SGD}, an innovative method for stochastic optimization to bridge the gap between stochastic optimization and statistical efficiency.
Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density
We introduce an approach to bias deep generative models, such as GANs and diffusion models, towards generating data with either enhanced fidelity or increased diversity.
Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
We study risk-sensitive reinforcement learning (RL), a crucial field due to its ability to enhance decision-making in scenarios where it is essential to manage uncertainty and minimize potential adverse outcomes.
Coherent and Multi-modality Image Inpainting via Latent Space Optimization
Our method searches latent spaces capable of generating inpainted regions that exhibit high fidelity to user-provided prompts while maintaining coherence with the background.
TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts
However, due to the scarcity of aligned NL and Formal Language (FL) theorem-proving data most modern LLMs exhibit suboptimal performance. This scarcity results in a paucity of methodologies for training LLMs and techniques to fully utilize their capabilities in composing formal proofs.
ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting
Bilevel optimization has shown its utility across various machine learning settings, yet most algorithms in practice require second-order information, making it challenging to scale them up.
AdaGrad under Anisotropic Smoothness
Despite the huge success in practice, their theoretical advantages over classical gradient methods with uniform step sizes across all coordinates (e. g. SGD) have not been fully understood, especially in the large batch-size setting commonly used in practice.
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
The trained RM serves as a proxy for human preferences.
Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation
SGRv2 excels in RLBench tasks with keyframe control using merely 5 demonstrations and surpasses the RVT baseline in 23 of 26 tasks.
Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
Reward models trained on human preference data have been proven to effectively align Large Language Models (LLMs) with human intent within the framework of reinforcement learning from human feedback (RLHF).
VeraCT Scan: Retrieval-Augmented Fake News Detection with Justifiable Reasoning
In response to this challenge, we introduce VeraCT Scan, a novel retrieval-augmented system for fake news detection.
AdaNCA: Neural Cellular Automata As Adaptors For More Robust Vision Transformer
In this paper, we propose Adaptor Neural Cellular Automata (AdaNCA) for Vision Transformers that uses NCA as plug-and-play adaptors between ViT layers, thus enhancing ViT's performance and robustness against adversarial samples as well as out-of-distribution inputs.
Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions
Image description datasets play a crucial role in the advancement of various applications such as image understanding, text-to-image generation, and text-image retrieval.
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments
Large Language Models (LLMs) have advanced rapidly but face significant memory demands.
PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance
Generating compact and sharply detailed 3D meshes poses a significant challenge for current 3D generative models.
Faster Sampling via Stochastic Gradient Proximal Sampler
Stochastic gradients have been widely integrated into Langevin-based methods to improve their scalability and efficiency in solving large-scale sampling problems.
Reverse Transition Kernel: A Flexible Framework to Accelerate Diffusion Inference
To generate data from trained diffusion models, most inference algorithms, such as DDPM, DDIM, and other variants, rely on discretizing the reverse SDEs or their equivalent ODEs.
Gaussian Time Machine: A Real-Time Rendering Methodology for Time-Variant Appearances
In this paper, we present Gaussian Time Machine (GTM) which models the time-dependent attributes of Gaussian primitives with discrete time embedding vectors decoded by a lightweight Multi-Layer-Perceptron(MLP).
CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models
To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy.
Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation
Then a two-stage fine-tuning on LLaMA 2 is performed on the generated data and the real data for the DST prediction.
RLHF Workflow: From Reward Modeling to Online RLHF
We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature.
SC-HVPPNet: Spatial and Channel Hybrid-Attention Video Post-Processing Network with CNN and Transformer
Convolutional Neural Network (CNN) and Transformer have attracted much attention recently for video post-processing (VPP).
Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange
Subsequently, we introduce a context-aware feature learning strategy, which encodes object patterns without relying on their specific context by aggregating object features across various scenes.
Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithm
Unlike previous work, which relies on a generative model or a pre-collected offline dataset enjoying good coverage of the deployment environment, we tackle robust RL via interactive data collection, where the learner interacts with the training environment only and refines the policy through trial and error.
Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorization
In addition, it is still unclear how well an OpenQA model can transfer to completely new knowledge domains.
On the Benefits of Over-parameterization for Out-of-Distribution Generalization
We demonstrate that in this scenario, further increasing the model's parameterization can significantly reduce the OOD loss.
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Attempting to complement this deficiency, we investigate the layerwise properties of LoRA on fine-tuning tasks and observe an unexpected but consistent skewness of weight norms across different layers.
DVMNet++: Rethinking Relative Pose Estimation for Unseen Objects
Determining the relative pose of a previously unseen object between two images is pivotal to the success of generalizable object pose estimation.
A Sober Look at the Robustness of CLIPs to Spurious Features
Large vision language models, such as CLIP, demonstrate impressive robustness to spurious features than single-modal models trained on ImageNet.
Gradient based Feature Attribution in Explainable AI: A Technical Review
Due to the lack of a rigorous definition of explainable AI (XAI), a plethora of research related to explainability, interpretability, and transparency has been developed to explain and analyze the model from various perspectives.
Desigen: A Pipeline for Controllable Design Template Generation
In this paper, we present Desigen, an automatic template creation pipeline which generates background images as well as harmonious layout elements over the background.
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself.
Ranked #102 on
Visual Question Answering
on MM-Vet
Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
To mitigate these issues, we propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information. Specifically, we leverage class embeddings as anchors due to their discrete and abstract nature, steering vision features toward class embeddings.
Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation
We investigate non-collaborative dialogue agents, which are expected to engage in strategic conversations with diverse users, for securing a mutual agreement that leans favorably towards the system's objectives.
OMH: Structured Sparsity via Optimally Matched Hierarchy for Unsupervised Semantic Segmentation
Our OMH yields better unsupervised segmentation performance compared to existing USS methods.
An Improved Analysis of Langevin Algorithms with Prior Diffusion for Non-Log-Concave Sampling
Along this line, Freund et al. (2022) suggest that the modified Langevin algorithm with prior diffusion is able to converge dimension independently for strongly log-concave target distributions.
Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagery
Ship detection needs to identify ship locations from remote sensing (RS) scenes.
Edge Computing Enabled Real-Time Video Analysis via Adaptive Spatial-Temporal Semantic Filtering
This paper proposes a novel edge computing enabled real-time video analysis system for intelligent visual devices.
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
Additionally, DPA models user preferences as directions (i. e., unit vectors) in the reward space to achieve user-dependent preference control.
EntailE: Introducing Textual Entailment in Commonsense Knowledge Graph Completion
In this paper, we propose to adopt textual entailment to find implicit entailment relations between CSKG nodes, to effectively densify the subgraph connecting nodes within the same conceptual class, which indicates a similar level of plausibility.
Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption
We also prove a lower bound to show that the additive dependence on $C$ is optimal.
Model-based Reinforcement Learning
reinforcement-learning
+2
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle.
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
In this paper, we identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers, causing MLLMs to suffer from visual illusion.
PipeNet: Question Answering with Semantic Pruning over Knowledge Graphs
To facilitate the evaluation of pruned subgraphs, we also propose a graph attention network (GAT) based module to reason with the subgraph data.
EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain
Multi-modal large language models (MLLMs) have demonstrated remarkable success in vision and visual-language tasks within the natural image domain.
General Flow as Foundation Affordance for Scalable Robot Learning
Therefore, we propose to utilize 3D flow, which represents the future trajectories of 3D points on objects of interest, as an ideal prediction target.
The Surprising Harmfulness of Benign Overfitting for Adversarial Robustness
Recent empirical and theoretical studies have established the generalization capabilities of large machine learning models that are trained to (approximately or exactly) fit noisy data.
Faster Sampling without Isoperimetry via Diffusion-based Monte Carlo
Specifically, DMC follows the reverse SDE of a diffusion process that transforms the target distribution to the standard Gaussian, utilizing a non-parametric score estimation.
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs.
AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets
We explore the potential of enhancing LLM performance in astronomy-focused question-answering through targeted, continual pre-training.
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs).
Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise
This means SGD with heavy-ball momentum is useful in the large-batch settings such as distributed machine learning or federated learning, where a smaller number of iterations can significantly reduce the number of communication rounds, leading to acceleration in practice.
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
We investigate its behavior in three distinct settings -- offline, online, and hybrid -- and propose efficient algorithms with finite-sample theoretical guarantees.
An Incremental Unified Framework for Small Defect Inspection
Artificial Intelligence (AI)-driven defect inspection is pivotal in industrial manufacturing.
Improving Medical Report Generation with Adapter Tuning and Knowledge Enhancement in Vision-Language Foundation Models
Medical report generation demands automatic creation of coherent and precise descriptions for medical images.
Adaptive Multi-step Refinement Network for Robust Point Cloud Registration
To achieve this, we introduce a training procedure and a refinement network.
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
In this study, we are interested in imbuing robots with the capability of physically-grounded task planning.
Beyond Pixels: Exploring Human-Readable SVG Generation for Simple Images with Vision Language Models
Our method focuses on producing SVGs that are both accurate and simple, aligning with human readability and understanding.
R-Tuning: Instructing Large Language Models to Say `I Don't Know'
This approach is formalized by first identifying the disparity in knowledge encompassed by pre-trained parameters compared to that of instruction tuning data.
Plum: Prompt Learning using Metaheuristic
Since the emergence of large language models, prompt learning has become a popular method for optimizing and customizing these models.
PerceptionGPT: Effectively Fusing Visual Perception into LLM
In this paper, we present a novel end-to-end framework named PerceptionGPT, which efficiently and effectively equips the VLLMs with visual perception abilities by leveraging the representation power of LLMs' token embedding.
CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer
Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR.
Mesh Neural Cellular Automata
We propose Mesh Neural Cellular Automata (MeshNCA), a method that directly synthesizes dynamic textures on 3D meshes without requiring any UV maps.
Corruption-Robust Offline Reinforcement Learning with General Function Approximation
Notably, under the assumption of single policy coverage and the knowledge of $\zeta$, our proposed algorithm achieves a suboptimality bound that is worsened by an additive factor of $\mathcal{O}(\zeta (C(\widehat{\mathcal{F}},\mu)n)^{-1})$ due to the corruption.
Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment.
Consistent123: Improve Consistency for One Image to 3D Object Synthesis
Large image diffusion models enable novel view synthesis with high quality and excellent zero-shot capability.
3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation
Our goal then is to estimate the relative object pose between this reference view and a query image that depicts the object in a different pose.
Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own
We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models.
Spurious Feature Diversification Improves Out-of-distribution Generalization
Contrary to the conventional wisdom that focuses on learning invariant features for better OOD performance, our findings suggest that incorporating a large number of diverse spurious features weakens their individual contributions, leading to improved overall OOD generalization performance.
May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability
With this paper, we investigate if free-form conversations can enhance users' comprehension of static explanations, improve acceptance and trust in the explanation methods, and facilitate human-AI collaboration.
MEDL-U: Uncertainty-aware 3D Automatic Annotation based on Evidential Deep Learning
In this paper, we present the first approach that addresses the inherent ambiguities present in pseudo labels by introducing an Evidential Deep Learning (EDL) based uncertainty estimation framework.
Mitigating the Alignment Tax of RLHF
Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different alignment-forgetting trade-offs, we propose Heterogeneous Model Averaging (HMA) to Heterogeneously find various combination ratios of model layers.
UniKG: A Benchmark and Universal Embedding for Large-Scale Knowledge Graphs
To perform effective learning on the large-scale UniKG, two key measures are taken, including (i) the semantic alignment strategy for multi-attribute entities, which projects the feature description of multi-attribute nodes into a common embedding space to facilitate node aggregation in a large receptive field; (ii) proposing a novel plug-and-play anisotropy propagation module (APM) to learn effective multi-hop anisotropy propagation kernels, which extends methods of large-scale homogeneous graphs to heterogeneous graphs.
Integrated Robotics Networks with Co-optimization of Drone Placement and Air-Ground Communications
Terrestrial robots, i. e., unmanned ground vehicles (UGVs), and aerial robots, i. e., unmanned aerial vehicles (UAVs), operate in separate spaces.
Optimal Sample Selection Through Uncertainty Estimation and Its Application in Deep Learning
Our proposed method, COPS (unCertainty based OPtimal Sub-sampling), is designed to minimize the expected loss of a model trained on subsampled data.
Self-Reference Deep Adaptive Curve Estimation for Low-Light Image Enhancement
In this paper, we propose a 2-stage low-light image enhancement method called Self-Reference Deep Adaptive Curve Estimation (Self-DACE).
Joint Salient Object Detection and Camouflaged Object Detection via Uncertainty-aware Learning
In this case, salient objects are typically non-camouflaged, and camouflaged objects are usually not salient.
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
As the number of available foundation models and specialized tasks keeps growing, the job of training scientific language models becomes highly nontrivial.
A Universal Semantic-Geometric Representation for Robotic Manipulation
To this end, we present $\textbf{Semantic-Geometric Representation} (\textbf{SGR})$, a universal perception module for robotics that leverages the rich semantic information of large-scale pre-trained 2D models and inherits the merits of 3D spatial reasoning.
Structure-Sensitive Graph Dictionary Embedding for Graph Classification
Instead of a plain use of a base graph dictionary, we propose the variational graph dictionary adaptation (VGDA) to generate a personalized dictionary (named adapted graph dictionary) for catering to each input graph.
Dual Adaptive Representation Alignment for Cross-domain Few-shot Learning
Recent progress in this setting assumes that the base knowledge and novel query samples are distributed in the same domains, which are usually infeasible for realistic applications.
Customizing General-Purpose Foundation Models for Medical Report Generation
In this work, we propose customizing off-the-shelf general-purpose large-scale pre-trained models, i. e., foundation models (FMs), in computer vision and natural language processing with a specific focus on medical report generation.
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memories
Pre-trained language models (PLMs) demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain.
What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?
In this paper, we study out-of-distribution (OOD) generalization of offline GCRL both theoretically and empirically to identify factors that are important.
InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields
We propose InNeRF360, an automatic system that accurately removes text-specified objects from 360-degree Neural Radiance Fields (NeRF).
DetGPT: Detect What You Need via Reasoning
Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines.
Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition
Recognizing novel sub-categories with scarce samples is an essential and challenging research topic in computer vision.
Adapter Learning in Pretrained Feature Extractor for Continual Learning of Diseases
In particular, updating an intelligent diagnosis system with training data of new diseases would cause catastrophic forgetting of old disease knowledge.
Hierarchical Interactive Reconstruction Network For Video Compressive Sensing
Deep network-based image and video Compressive Sensing(CS) has attracted increasing attentions in recent years.
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently enhancing the model by fine-tuning on these filtered samples.
Crowd Counting with Sparse Annotation
This paper presents a new annotation method called Sparse Annotation (SA) for crowd counting, which reduces human labeling efforts by sparsely labeling individuals in an image.
ConvBLS: An Effective and Efficient Incremental Convolutional Broad Learning System for Image Classification
Third, the TSMS feature fusion layer is proposed to extract more effective multi-scale features through the integration of CF layers and CE layers.
De-coupling and De-positioning Dense Self-supervised Learning
Dense Self-Supervised Learning (SSL) methods address the limitations of using image-level feature representations when handling images with multiple objects.
Spatiotemporal Self-supervised Learning for Point Clouds in the Wild
In this paper, we introduce an SSL strategy that leverages positive pairs in both the spatial and temporal domain.
NEMTO: Neural Environment Matting for Novel View and Relighting Synthesis of Transparent Objects
We propose NEMTO, the first end-to-end neural rendering pipeline to model 3D transparent objects with complex geometry and unknown indices of refraction.
Environment Invariant Linear Least Squares
To the best of our knowledge, this paper is the first to realize statistically efficient invariance learning in the general linear model.
PAPAL: A Provable PArticle-based Primal-Dual ALgorithm for Mixed Nash Equilibrium
To circumvent this difficulty, we examine the problem of identifying a mixed Nash equilibrium, where strategies are randomized and characterized by probability distributions over continuous domains.
Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
However, most CoT studies rely on carefully designed human-annotated rational chains to prompt LLMs, posing challenges for real-world applications where labeled data is available without rational chains.
Active Prompting with Chain-of-Thought for Large Language Models
For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful ones to annotate from a pool of task-specific queries.
Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiency
We propose a variance-adaptive algorithm for linear mixture MDPs, which achieves a problem-dependent horizon-free regret bound that can gracefully reduce to a nearly constant regret for deterministic MDPs.
Hashtag-Guided Low-Resource Tweet Classification
Social media classification tasks (e. g., tweet sentiment analysis, tweet stance detection) are challenging because social media posts are typically short, informal, and ambiguous.
On the Convergence of Federated Averaging with Cyclic Client Participation
Federated Averaging (FedAvg) and its variants are the most popular optimization algorithms in federated learning (FL).
History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System
HAHT maintains a long-term memory of history conversations and utilizes history information to understand current conversation context and generate well-informed and context-relevant responses.
ADAPT: Action-aware Driving Caption Transformer
To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action.
Learning in POMDPs is Sample-Efficient with Hindsight Observability
POMDPs capture a broad class of decision making problems, but hardness results suggest that learning is intractable even in simple settings due to the inherent partial observability.
Probabilistic Bilevel Coreset Selection
The goal of coreset selection in supervised learning is to produce a weighted subset of data, so that training only on the subset achieves similar performance as training on the entire dataset.
Model Agnostic Sample Reweighting for Out-of-Distribution Learning
The overfitting issue is addressed by considering a bilevel formulation to search for the sample reweighting, in which the generalization complexity depends on the search space of sample weights instead of the model size.
TempSAL -- Uncovering Temporal Information for Deep Saliency Prediction
Deep saliency prediction algorithms complement the object recognition features, they typically rely on additional information, such as scene context, semantic relationships, gaze direction, and object dissimilarity.
TempSAL - Uncovering Temporal Information for Deep Saliency Prediction
Deep saliency prediction algorithms complement the object recognition features, they typically rely on additional information such as scene context, semantic relationships, gaze direction, and object dissimilarity.
Ranked #3 on
Saliency Prediction
on SALICON
DSI2I: Dense Style for Unpaired Image-to-Image Translation
To stylize the source content with the exemplar style, we extract unsupervised cross-domain semantic correspondences and warp the exemplar style to the source content.
VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction
The success of the Neural Radiance Fields (NeRF) in novel view synthesis has inspired researchers to propose neural implicit scene reconstruction.
VO$Q$L: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximation
We study time-inhomogeneous episodic reinforcement learning (RL) under general function approximation and sparse rewards.
Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processes
In this paper, we consider the contextual bandit with general function approximation and propose a computationally efficient algorithm to achieve a regret of $\tilde{O}(\sqrt{T}+\zeta)$.
ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT
In this paper, we present ExtremeBERT, a toolkit for accelerating and customizing BERT pretraining.
On Robust Observer Design for System Motion on SE(3) Using Onboard Visual Sensors
The key design idea for this observer is to estimate the visible set and identify the mis-identified features from the measurements.
Particle-based Variational Inference with Preconditioned Functional Gradient Flow
With the popularity of Stein variational gradient descent (SVGD), the focus of particle-based VI algorithms has been on the properties of functions in Reproducing Kernel Hilbert Space (RKHS) to approximate the gradient flow.
DyNCA: Real-time Dynamic Texture Synthesis Using Neural Cellular Automata
Current Dynamic Texture Synthesis (DyTS) models can synthesize realistic videos.
Normalizing Flow with Variational Latent Representation
The resulting method is significantly more powerful than the standard normalization flow approach for generating data distributions with multiple modes.
FAF: A novel multimodal emotion recognition approach integrating face, body and text
In this paper, we developed a large multimodal emotion dataset, named "HED" dataset, to facilitate the emotion recognition task, and accordingly propose a multimodal emotion recognition method.
Prior-mean-assisted Bayesian optimization application on FRIB Front-End tunning
Bayesian optimization~(BO) is often used for accelerator tuning due to its high sample efficiency.
GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond
The proposed algorithm modifies the standard posterior sampling algorithm in two aspects: (i) we use an optimistic prior distribution that biases towards hypotheses with higher values and (ii) a loglikelihood function is set to be the empirical loss evaluated on the historical data, where the choice of loss function supports both model-free and model-based learning.
