Search Results for author:
Found 319 papers, 174 papers with code
Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach
Existing tracking algorithms typically rely on low-frame-rate RGB cameras coupled with computationally intensive deep neural network architectures to achieve effective tracking.
Dynamic Graph Induced Contour-aware Heat Conduction Network for Event-based Object Detection
To address these issues, this paper proposes a novel dynamic graph induced contour-aware heat conduction network for event stream based object detection, termed CvHeat-DET.
Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs
Large language models (LLMs) excel at mathematical reasoning and logical problem-solving.
SLOT: Sample-specific Language Model Optimization at Test-time
We propose SLOT (Sample-specific Language Model Optimization at Test-time), a novel and parameter-efficient test-time inference approach that enhances a language model's ability to more accurately respond to individual prompts.
Lightweight Spatio-Temporal Attention Network with Graph Embedding and Rotational Position Encoding for Traffic Forecasting
Additionally, the optimal frequency for rotational position encoding is determined through a grid search approach in both the spatial and temporal attention mechanisms.
Describe Anything in Medical Images
Localized image captioning has made significant progress with models like the Describe Anything Model (DAM), which can generate detailed region-specific descriptions without explicit region-text supervision.
ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling
It supports downscaling to 0. 9 km global resolution and processes sequences up to 4. 2 billion tokens.
4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis
Multimodal neuroimaging provides complementary structural and functional insights into both human brain organization and disease-related dynamics.
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
Currently, existing surveys on LLM safety primarily focus on specific stages of the LLM lifecycle, e. g., deployment phase or fine-tuning phase, lacking a comprehensive understanding of the entire "lifechain" of LLMs.
Adversarial Attack for RGB-Event based Visual Object Tracking
To bridge this gap, in this paper, we propose a cross-modal adversarial attack algorithm for RGB-Event visual tracking.
CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework
Event cameras have attracted increasing attention in recent years due to their advantages in high dynamic range, high temporal resolution, low power consumption, and low latency.
Zero-shot Autonomous Microscopy for Scalable and Intelligent Characterization of 2D Materials
This work represents the implementation of foundation models to achieve autonomous analysis, establishing a scalable and data-efficient characterization paradigm that fundamentally transforms the approach to nanoscale materials research.
RGB-Event based Pedestrian Attribute Recognition: A Benchmark Dataset and An Asymmetric RWKV Fusion Framework
In this paper, we revisit these issues and propose a novel multi-modal RGB-Event attribute recognition task by drawing inspiration from the advantages of event cameras in low-light, high-speed, and low-power consumption.
Bregman Linearized Augmented Lagrangian Method for Nonconvex Constrained Stochastic Zeroth-order Optimization
In particular, starting from a near-feasible initial point and using Rademacher smoothing, the oracle complexity is in order \(O(p d^{2/p} \epsilon^{-3})\) for \(p \in [2, 2 \ln d]\), and \(O(\ln d \cdot \epsilon^{-3})\) for \(p > 2 \ln d\), where \(d\) denotes the problem dimension.
Human Activity Recognition using RGB-Event based Sensors: A Multi-modal Heat Conduction Model and A Benchmark Dataset
In this work, we rethink human activity recognition by combining the RGB and event cameras.
ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluation
In this work, we introduce ContrastScore, a contrastive evaluation metric designed to enable higher-quality, less biased, and more efficient assessment of generated text.
Visual Variational Autoencoder Prompt Tuning
Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large vision transformers to downstream tasks without the prohibitive computational costs of full fine-tuning.
Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations
One of the most effective techniques for enhancing robustness, Noisy Training, introduces noise during the training phase to reinforce the model against disturbances encountered during inference.
Optimal Complexity in Byzantine-Robust Distributed Stochastic Optimization with Data Heterogeneity
We establish the lower bounds on the Byzantine error and on the minimum number of queries to a stochastic gradient oracle required to achieve an arbitrarily small optimization error.
Pseudo-Relevance Feedback Can Improve Zero-Shot LLM-Based Dense Retrieval
Pseudo-relevance feedback (PRF) refines queries by leveraging initially retrieved documents to improve retrieval effectiveness.
TGBFormer: Transformer-GraphFormer Blender Network for Video Object Detection
Video object detection has made significant progress in recent years thanks to convolutional neural networks (CNNs) and vision transformers (ViTs).
Ranked #5 on
Video Object Detection
on ImageNet VID
DehazeMamba: SAR-guided Optical Remote Sensing Image Dehazing with Adaptive State Space Model
Optical remote sensing image dehazing presents significant challenges due to its extensive spatial scale and highly non-uniform haze distribution, which traditional single-image dehazing methods struggle to address effectively.
AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding
Multimodal Large Language Models (MLLMs) have revolutionized video understanding, yet are still limited by context length when processing long videos.
CAD-VAE: Leveraging Correlation-Aware Latents for Comprehensive Fair Disentanglement
While deep generative models have significantly advanced representation learning, they may inherit or amplify biases and fairness issues by encoding sensitive attributes alongside predictive features.
Large Language Model Guided Progressive Feature Alignment for Multimodal UAV Object Detection
To address this problem, we propose a Large Language Model (LLM) guided Progressive feature Alignment Network called LPANet, which leverages the semantic features extracted from a large language model to guide the progressive semantic and spatial alignment between modalities for multimodal UAV object detection.
Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms
Accurate sign language understanding serves as a crucial communication channel for individuals with disabilities.
AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot
The social robot's open API allows users to customize open-domain interactions.
Split Adaptation for Pre-trained Vision Transformers
SA, inspired by split learning (SL), segments the pre-trained ViT into a frontend and a backend, with only the frontend shared with the client for data representation extraction.
HAIC: Improving Human Action Understanding and Generation with Better Captions for Multi-modal Large Language Models
Recent Multi-modal Large Language Models (MLLMs) have made great progress in video understanding.
Langevin Multiplicative Weights Update with Applications in Polynomial Portfolio Management
We consider nonconvex optimization problem over simplex, and more generally, a product of simplices.
Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric
To address this, we systematically analyze 11 existing diversity measurement methods by assessing their correlation with model performance through extensive fine-tuning experiments.
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
We introduce SigLIP 2, a family of new multilingual vision-language encoders that build on the success of the original SigLIP.
RAG-Gym: Optimizing Reasoning and Search Agents with Process Supervision
Retrieval-augmented generation (RAG) has shown great potential for knowledge-intensive tasks, but its traditional architectures rely on static retrieval, limiting their effectiveness for complex questions that require sequential information-seeking.
Censor Dependent Variational Inference
This paper provides a comprehensive analysis of variational inference in latent variable models for survival analysis, emphasizing the distinctive challenges associated with applying variational methods to survival data.
EventSTR: A Benchmark Dataset and Baselines for Event Stream based Scene Text Recognition
Mainstream Scene Text Recognition (STR) algorithms are developed based on RGB cameras which are sensitive to challenging factors such as low illumination, motion blur, and cluttered backgrounds.
Scaling Pre-training to One Hundred Billion Data for Vision Language Models
We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples.
Event Stream-based Visual Object Tracking: HDETrack V2 and A High-Definition Benchmark
We then introduce a novel hierarchical knowledge distillation strategy that incorporates the similarity matrix, feature representation, and response map-based distillation to guide the learning of the student Transformer network.
Sparse Measurement Medical CT Reconstruction using Multi-Fused Block Matching Denoising Priors
Instead of directly using a 3D prior, this work proposes a BM3D Multi Slice Fusion (BM3D-MSF) prior that uses multiple 2D image denoisers fused to act as a fully 3D prior model in Plug and Play reconstruction approach.
On Hardening DNNs against Noisy Computations
This work investigates the effectiveness of training neural networks with quantization to increase the robustness against noise.
Fast-RF-Shimming: Accelerate RF Shimming in 7T MRI using Deep Learning
This approach offers a faster and more efficient solution to RF shimming challenges in UHF MRI.
Spatio-temporal Graph Learning on Adaptive Mined Key Frames for High-performance Multi-Object Tracking
In the realm of multi-object tracking, the challenge of accurately capturing the spatial and temporal relationships between objects in video sequences remains a significant hurdle.
Scale-up Unlearnable Examples Learning with High-Performance Computing
A prominent method within this area, called Unlearnable Clustering (UC), has shown improved UE performance with larger batch sizes but was previously limited by computational resources.
Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation
X-ray image based medical report generation achieves significant progress in recent years with the help of the large language model, however, these models have not fully exploited the effective information in visual image regions, resulting in reports that are linguistically sound but insufficient in describing key diseases.
Rethinking Byzantine Robustness in Federated Recommendation from Sparse Aggregation Perspective
Unlike FL, FR has a unique sparse aggregation mechanism, where the embedding of each item is updated by only partial clients, instead of full clients in a dense aggregation of general FL.
Unsupervised dense retrieval with conterfactual contrastive learning
In response to these challenges, we propose to improve the robustness of dense retrieval models by enhancing their sensitivity of fine-graned relevance signals.
ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding
Video Large Language Models (VideoLLMs) have made significant strides in video understanding but struggle with long videos due to the limitations of their backbone LLMs.
VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognition
Inspired by the successful application of large models, the introduction of such large models can also be considered to further enhance the performance of multi-modal tasks.
Bi-directional Mapping of Morphology Metrics and 3D City Blocks for Enhanced Characterization and Generation of Urban Form
This paper highlights the importance of establishing a bi-directional mapping between morphology metrics and complex urban form to enable the integration of urban form generation with performance evaluation.
DocFusion: A Unified Framework for Document Parsing Tasks
Document parsing is essential for analyzing complex document structures and extracting fine-grained information, supporting numerous downstream applications.
Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset
Object detection in event streams has emerged as a cutting-edge research area, demonstrating superior performance in low-light conditions, scenarios with motion blur, and rapid movements.
Blockchain Data Analysis in the Era of Large-Language Models
Blockchain data analysis is essential for deriving insights, tracking transactions, identifying patterns, and ensuring the integrity and security of decentralized networks.
PaliGemma 2: A Family of Versatile VLMs for Transfer
PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models.
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
Experiments show that the method improves the actor's exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger reasoning model.
Nimbus: Secure and Efficient Two-Party Inference for Transformers
For the linear layer, we propose a new 2PC paradigm along with an encoding approach to securely compute matrix multiplications based on an outer-product insight, which achieves $2. 9\times \sim 12. 5\times$ performance improvements compared to the state-of-the-art (SOTA) protocol.
FM-TS: Flow Matching for Time Series Generation
FM-TS is more efficient in terms of training and inference.
A Data-driven Crowd Simulation Framework Integrating Physics-informed Machine Learning with Navigation Potential Fields
Our approach leverages the strengths of both physical models and PIML.
Anderson Acceleration in Nonsmooth Problems: Local Convergence via Active Manifold Identification
Anderson acceleration is an effective technique for enhancing the efficiency of fixed-point iterations; however, analyzing its convergence in nonsmooth settings presents significant challenges.
Multi-modal Fusion based Q-distribution Prediction for Controlled Nuclear Fusion
Q-distribution prediction is a crucial research direction in controlled nuclear fusion, with deep learning emerging as a key approach to solving prediction challenges.
Exploiting Memory-aware Q-distribution Prediction for Nuclear Fusion via Modern Hopfield Network
This study addresses the critical challenge of predicting the Q-distribution in long-term stable nuclear fusion task, a key component for advancing clean energy solutions.
HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter
To address these issues, in this paper, we propose a novel Heterogeneous Graph Adapter to achieve tuning VLMs for the downstream tasks.
SNN-PAR: Energy Efficient Pedestrian Attribute Recognition via Spiking Neural Networks
To address this issue, in this paper, we propose a Spiking Neural Network (SNN) based framework for energy-efficient attribute recognition.
CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset
Thus, we conduct a comprehensive benchmarking of existing mainstream X-ray report generation models and large language models (LLMs), on the CheXpert Plus dataset.
Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding
For iterative refinement, we first leverage a video-language model to generate synthetic annotations, resulting in a refined dataset.
Adaptive Learning of the Latent Space of Wasserstein Generative Adversarial Networks
Generative models based on latent variables, such as generative adversarial networks (GANs) and variational auto-encoders (VAEs), have gained lots of interests due to their impressive performance in many fields.
Ig3D: Integrating 3D Face Representations in Facial Expression Inference
Our experimental results demonstrate that our proposed method outperforms the state-of-the-art AffectNet VA estimation and RAF-DB classification tasks.
Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detection
In this paper, we propose a hierarchical graph interaction network termed HGINet for camouflaged object detection, which is capable of discovering imperceptible objects via effective graph interaction among the hierarchical tokenized features.
VFM-Det: Towards High-Performance Vehicle Detection via Large Foundation Models
In this work, we propose a new vehicle detection paradigm based on a pre-trained foundation vehicle model (VehicleMAE) and a large language model (T5), termed VFM-Det.
Optimizing Transmit Field Inhomogeneity of Parallel RF Transmit Design in 7T MRI using Deep Learning
We propose a two-step deep learning strategy.
MambaEVT: Event Stream based Visual Object Tracking using State Space Model
More importantly, we consider introducing a dynamic template update strategy into the tracking framework using the Memory Mamba network.
Event Stream based Sign Language Translation: A High-Definition Benchmark Dataset and A New Algorithm
Based on this dataset and several other large-scale datasets, we propose a novel baseline method that fully leverages the Mamba model's ability to integrate temporal information of CNN features, resulting in improved sign language translation outcomes.
Pedestrian Attribute Recognition: A New Benchmark Dataset and A Large Language Model Augmented Framework
To address this issue, this paper proposes a new large-scale, cross-domain pedestrian attribute recognition dataset to fill the data gap, termed MSP60K.
Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms
In this paper, we propose a large-scale, high-definition ($1280 \times 800$) human action recognition dataset based on the CeleX-V event camera, termed CeleX-HAR.
R2GenCSR: Retrieving Context Samples for Large Language Model based X-ray Medical Report Generation
They usually adopt a Transformer to extract the visual features of a given X-ray image, and then, feed them into the LLM for text generation.
Can Large Language Models Improve the Adversarial Robustness of Graph Neural Networks?
By presenting the empirical results, we find that despite that LLMs can improve the robustness of GNNs, there is still an average decrease of 23. 1% in accuracy, implying that the GNNs remain extremely vulnerable against topology attacks.
Treat Stillness with Movement: Remote Sensing Change Detection via Coarse-grained Temporal Foregrounds Mining
We also introduce a motion augmented strategy that leverages motion cues as an additional output to aggregate with the spatial features for improved results.
SMILES-Mamba: Chemical Mamba Foundation Models for Drug ADMET Prediction
In drug discovery, predicting the absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of small-molecule drugs is critical for ensuring safety and efficacy.
AI Foundation Models in Remote Sensing: A Survey
Artificial Intelligence (AI) technologies have profoundly transformed the field of remote sensing, revolutionizing data collection, processing, and analysis.
EMatch: A Unified Framework for Event-based Optical Flow and Stereo Matching
Specifically, our method utilizes a Temporal Recurrent Network to aggregate event features across temporal or spatial domains, and a Spatial Contextual Attention to enhance knowledge transfer across event flows via temporal or spatial interactions.
Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction
However, current methods often prioritize the extraction of temporal information from continuous event flow, leading to an overemphasis on low-frequency texture features in the scene, resulting in over-smoothing and blurry artifacts.
An Empirical Study of Mamba-based Pedestrian Attribute Recognition
To further tap into the potential of the novel Mamba architecture for PAR tasks, this paper designs and adapts Mamba into two typical PAR frameworks, i. e., the text-image fusion approach and pure vision Mamba multi-label recognition framework.
On Large Language Model Continual Unlearning
The results indicate that OOO consistently achieves the best unlearning effectiveness and utility preservation, especially when facing continuous unlearning requests.
PaliGemma: A versatile 3B VLM for transfer
PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model.
Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition
In this paper, we propose a novel dual-stream framework for event stream-based pattern recognition via differentiated fusion, termed EFV++.
SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance
As the development of large language models (LLMs) rapidly advances, securing these models effectively without compromising their utility has become a pivotal area of research.
TemPrompt: Multi-Task Prompt Learning for Temporal Relation Extraction in RAG-based Crowdsourcing Systems
Temporal relation extraction (TRE) aims to grasp the evolution of events or actions, and thus shape the workflow of associated tasks, so it holds promise in helping understand task requests initiated by requesters in crowdsourcing systems.
InternLM-Law: An Open Source Chinese Legal Large Language Model
While large language models (LLMs) have showcased impressive capabilities, they struggle with addressing legal queries due to the intricate complexities and specialized expertise required in the legal field.
ECAFormer: Low-light Image Enhancement using Cross Attention
Low-light image enhancement (LLIE) is critical in computer vision.
Beyond Boundaries: Learning a Universal Entity Taxonomy across Datasets and Languages for Open Named Entity Recognition
First, we detect inconsistent entity definitions across datasets and clarify them by distinguishable label names to construct a universal taxonomy of 400+ entity types.
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
In aligning large language models (LLMs), utilizing feedback from existing advanced AI rather than humans is an important method to scale supervisory signals.
Last-iterate Convergence Separation between Extra-gradient and Optimism in Constrained Periodic Games
Last-iterate behaviors of learning algorithms in repeated two-player zero-sum games have been extensively studied due to their wide applications in machine learning and related tasks.
Inductive Global and Local Manifold Approximation and Projection
Nonlinear dimensional reduction with the manifold assumption, often called manifold learning, has proven its usefulness in a wide range of high-dimensional data analysis.
CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models
Artificial intelligence has significantly impacted medical applications, particularly with the advent of Medical Large Vision Language Models (Med-LVLMs), sparking optimism for the future of automated and personalized healthcare.
Explaining the Contributing Factors for Vulnerability Detection in Machine Learning
A fundamental but unresolved research question is: how do different factors in the mining and learning process impact the accuracy of identifying vulnerabilities in software projects of varying characteristics?
RAG-based Crowdsourcing Task Decomposition via Masked Contrastive Learning with Prompts
Therefore, we present a Prompt-Based Contrastive learning framework for TD (PBCT), which incorporates a prompt-based trigger detector to overcome dependence.
No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs.
AMCEN: An Attention Masking-based Contrastive Event Network for Two-stage Temporal Knowledge Graph Reasoning
To achieve more accurate TKG reasoning, we propose an attention masking-based contrastive event network (AMCEN) with local-global temporal patterns for the two-stage prediction of future events.
SignAvatar: Sign Language 3D Motion Reconstruction and Generation
Achieving expressive 3D motion reconstruction and automatic generation for isolated sign words can be challenging, due to the lack of real-world 3D sign-word data, the complex nuances of signing motions, and the cross-modal understanding of sign language semantics.
Less is More: on the Over-Globalizing Problem in Graph Transformers
Then we propose a novel Bi-Level Global Graph Transformer with Collaborative Training (CoBFormer), including the inter-cluster and intra-cluster Transformers, to prevent the over-globalizing problem while keeping the ability to extract valuable information from distant nodes.
Mamba-FETrack: Frame-Event Tracking via State Space Model
Specifically, our Mamba-based tracker achieves 43. 5/55. 6 on the SR/PR metric, while the ViT-S based tracker (OSTrack) obtains 40. 0/50. 9.
Pre-training on High Definition X-ray Images: An Experimental Study
Existing X-ray based pre-trained vision models are usually conducted on a relatively small-scale dataset (less than 500k samples) with limited resolution (e. g., 224 $\times$ 224).
Spatio-Temporal Side Tuning Pre-trained Foundation Models for Video-based Pedestrian Attribute Recognition
Specifically, we formulate the video-based PAR as a vision-language fusion problem and adopt a pre-trained foundation model CLIP to extract the visual features.
Implicit Generative Prior for Bayesian Neural Networks
The results of our experiments highlight the superiority of our proposed framework over existing methods, such as sparse variational Bayesian and generative models, in terms of prediction accuracy and uncertainty quantification.
ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability
As the largest model of its kind, ORBIT surpasses the current climate AI foundation model size by a thousandfold.
S4TP: Social-Suitable and Safety-Sensitive Trajectory Planning for Autonomous Vehicles
To effectively assess the risks prevailing in the vicinity of AVs in social interactive traffic scenarios and achieve safe autonomous driving, this article proposes a social-suitable and safety-sensitive trajectory planning (S4TP) framework.
Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning
The open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress.
State Space Model for New-Generation Network Alternative to Transformers: A Survey
In this paper, we give the first comprehensive review of these works and also provide experimental comparisons and analysis to better demonstrate the features and advantages of SSM.
Adaptive Patching for High-resolution Image Segmentation with Transformers
For high-resolution images, e. g. microscopic pathology images, the quadratic compute and memory cost prohibits the use of an attention-based model, if we are to use smaller patch sizes that are favorable in segmentation.
LocCa: Visual Pretraining with Location-aware Captioners
In this paper, we propose a simple visual pretraining method with location-aware captioners (LocCa).
Exploring Task Unification in Graph Representation Learning via Generative Approach
Specifically, GA^2E proposes to use the subgraph as the meta-structure, which remains consistent across all graph tasks (ranging from node-, edge-, and graph-level to transfer learning) and all stages (both during training and inference).
EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models
This paper introduces EasyJailbreak, a unified framework simplifying the construction and evaluation of jailbreak attacks against LLMs.
Finding Visual Saliency in Continuous Spike Stream
Our framework exhibits a substantial margin of improvement in capturing and highlighting visual saliency in the spike stream, which not only provides a new perspective for spike-based saliency segmentation but also shows a new paradigm for full SNN-based transformer models.
Long-term Frame-Event Visual Tracking: Benchmark Dataset and Baseline
Current event-/frame-event based trackers undergo evaluation on short-term tracking datasets, however, the tracking of real-world scenarios involves long-term tracking, and the performance of existing tracking algorithms in these scenarios remains unclear.
CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?
Interestingly, data and architectural improvements seem to mitigate the negative impact of data balancing on performance; e. g. applying M4 to SigLIP-B/16 with data quality filters improves COCO image-to-text retrieval @5 from 86% (without data balancing) to 87% and ImageNet 0-shot classification from 77% to 77. 5%!
Can Small Language Models be Good Reasoners for Sequential Recommendation?
We encode the generated rationales from the student model into a dense vector, which empowers recommendation in both ID-based and ID-agnostic scenarios.
Intent-aware Recommendation via Disentangled Graph Contrastive Learning
The other is different behaviors have different intent distributions, so how to establish their relations for a more explainable recommender system.
Learning Invariant Representations of Graph Neural Networks via Cluster Generalization
To address this challenge, we propose the Cluster Information Transfer (CIT) mechanism (Code available at https://github. com/BUPT-GAMMA/CITGNN), which can learn invariant representations for GNNs, thereby improving their generalization ability to various and unknown test graphs with structure shift.
Multi-Scale Subgraph Contrastive Learning
By an experimental analysis, we discover the semantic information of an augmented graph structure may be not consistent as original graph structure, and whether two augmented graphs are positive or negative pairs is highly related with the multi-scale structures.
Minimum Topology Attacks for Graph Neural Networks
To break this dilemma, we propose a new type of topology attack, named minimum-budget topology attack, aiming to adaptively find the minimum perturbation sufficient for a successful attack on each node.
Small, Versatile and Mighty: A Range-View Perception Framework
Our proposed Small, Versatile, and Mighty (SVM) network utilizes a pure convolutional architecture to fully unleash the efficiency and multi-tasking potentials of the range view representation.
CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models
Adversarial misuse, particularly through `jailbreaking' that circumvents a model's safety and ethical protocols, poses a significant challenge for Large Language Models (LLMs).
RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions
In this paper, drawing inspiration from recent works that LLMs are sensitive to the design of the instructions, we utilize instructions in code style, which are more structural and less ambiguous, to replace typically natural language instructions.
Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning
In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models.
A Lightweight Inception Boosted U-Net Neural Network for Routability Prediction
As the modern CPU, GPU, and NPU chip design complexity and transistor counts keep increasing, and with the relentless shrinking of semiconductor technology nodes to nearly 1 nanometer, the placement and routing have gradually become the two most pivotal processes in modern very-large-scale-integrated (VLSI) circuit back-end design.
Federated Learning with New Knowledge: Fundamentals, Advances, and Futures
Federated Learning (FL) is a privacy-preserving distributed learning approach that is rapidly developing in an era where privacy protection is increasingly valued.
StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
The advancement of large language models (LLMs) has significantly propelled the field of code generation.
Navigating the OverKill in Large Language Models
Large language models are meticulously aligned to be both helpful and harmless.
Graph Fairness Learning under Distribution Shifts
Recently, there has been an increasing interest in ensuring fairness on GNNs, but all of them are under the assumption that the training and testing data are under the same distribution, i. e., training data and testing data are from the same graph.
Two Heads Are Better Than One: Integrating Knowledge from Knowledge Graphs and Large Language Models for Entity Alignment
The final prediction of the equivalent entity is derived from the LLM's output.
Graph Contrastive Invariant Learning from the Causal Perspective
How can we fix it and encourage the current GCL to learn better invariant representations?
Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
This reliance limits the applicability of RLHF and hinders the development of professional assistants tailored to diverse human preferences.
Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition
The mainstream human activity recognition (HAR) algorithms are developed based on RGB cameras, which are easily influenced by low-quality images (e. g., low illumination, motion blur).
Secrets of RLHF in Large Language Models Part II: Reward Modeling
We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras
In this work, we formally propose the task of object tracking using unaligned neuromorphic and visible cameras.
On Scaling Up a Multilingual Vision and Language Model
We explore the boundaries of scaling up a multilingual vision and language model both in terms of size of the components and the breadth of its training task mixture.
GenoCraft: A Comprehensive, User-Friendly Web-Based Platform for High-Throughput Omics Data Analysis and Visualization
With GenoCraft, researchers and data scientists have access to an array of cutting-edge bioinformatics tools under a user-friendly interface, making it a valuable resource for managing and analyzing large-scale omics data.
Federated Continual Novel Class Learning
In a privacy-focused era, Federated Learning (FL) has emerged as a promising machine learning technique.
Optimizing Distributed Training on Frontier for Large Language Models
For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved $100\%$ weak scaling efficiency on 1024 and 3072 MI250X GPUs, respectively.
Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video Recognition
It is intuitive to combine them for high-performance RGB-Event based video recognition, however, existing works fail to achieve a good balance between the accuracy and model parameters, as shown in Fig.~\ref{firstimage}.
Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion
In this paper, we formulate PAR as a vision-language fusion problem and fully exploit the relations between pedestrian images and attribute labels.
Structural Information Guided Multimodal Pre-training for Vehicle-centric Perception
To address this issue, we propose a novel vehicle-centric pre-training framework called VehicleMAE, which incorporates the structural information including the spatial structure from vehicle profile information and the semantic structure from informative high-level natural language descriptions for effective masked vehicle appearance reconstruction.
LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks.
A Generalized Neural Diffusion Framework on Graphs
In this paper, we propose a general diffusion equation framework with the fidelity term, which formally establishes the relationship between the diffusion process with more GNNs.
RTQ: Rethinking Video-language Understanding Based on Image-text Model
Recent advancements in video-language understanding have been established on the foundation of image-text models, resulting in promising outcomes due to the shared knowledge between images and videos.
Ranked #9 on
Video Captioning
on MSR-VTT
(using extra training data)
Semantic-Aware Frame-Event Fusion based Pattern Recognition via Large Vision-Language Models
Current methods typically employ backbone networks to individually extract the features of RGB frames and event streams, and subsequently fuse these features for pattern recognition.
Ultra-Long Sequence Distributed Transformer
This paper presents a novel and efficient distributed training method, the Long Short-Sequence Transformer (LSS Transformer), for training transformer with long sequences.
Orthogonal Subspace Learning for Language Model Continual Learning
In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks.
Improving Generalization of Alignment with Human Preferences through Group Invariant Learning
In this work, we propose a novel approach that can learn a consistent policy via RL across various data groups or domains.
VcT: Visual change Transformer for Remote Sensing Image Change Detection
Then, each pixel of feature map is regarded as a graph node and the graph neural network is proposed to model the structured information for coarse change map prediction.
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger.
Ranked #2 on
Temporal/Casual QA
on NExT-QA
(using extra training data)
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
In this paper, we introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences.
Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models
This study serves as a clarion call for a collective effort to overhaul and fortify the safety of open-source LLMs against malicious attackers.
Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline
Tracking using bio-inspired event cameras has drawn more and more attention in recent years.
Provable Training for Graph Contrastive Learning
To this end, we propose the PrOvable Training (POT) for GCL, which regularizes the training of GCL to encode node embeddings that follows the GCL principle better.
The Rise and Potential of Large Language Model Based Agents: A Survey
Many efforts have been made to develop intelligent agents, but they mainly focus on advancement in algorithms or training strategies to enhance specific capabilities or performance on particular tasks.
A stochastic block model for community detection in attributed networks
Therefore, a stochastic block model that integrates betweenness centrality and clustering coefficient of nodes for community detection in attributed networks, named BCSBM, is proposed in this paper.
Unified and Dynamic Graph for Temporal Character Grouping in Long Videos
In this paper, we present a unified and dynamic graph (UniDG) framework for temporal character grouping.
Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classification
Additionally, a bottleneck Transformer is introduced to facilitate the fusion of the dual-stream information.
Temporal Sentence Grounding in Streaming Videos
The goal of TSGSV is to evaluate the relevance between a video stream and a given sentence query.
High-performance Data Management for Whole Slide Image Analysis in Digital Pathology
The performance evaluation encompasses two key scenarios: (1) a pure CPU-based image analysis scenario ("CPU scenario"), and (2) a GPU-based deep learning framework scenario ("GPU scenario").
SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition
Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition.
Generative Query Reformulation for Effective Adhoc Search
GenQR directly reformulates the user's input query, while GenPRF provides additional context for the query by making use of pseudo-relevance feedback information.
DSRM: Boost Textual Adversarial Training with Distribution Shift Risk Minimization
Adversarial training is one of the best-performing methods in improving the robustness of deep language models.
Finding the Missing-half: Graph Complementary Learning for Homophily-prone and Heterophily-prone Graphs
Real-world graphs generally have only one kind of tendency in their connections.
Point-Voxel Absorbing Graph Representation Learning for Event Stream based Recognition
To address these issues, we propose a novel dual point-voxel absorbing graph representation learning for event stream data representation.
AMatFormer: Efficient Feature Matching via Anchor Matching Transformer
Second, AMatFormer adopts a shared FFN module to further embed the features of two images into the common domain and thus learn the consensus feature representations for the matching problem.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
Three Towers: Flexible Contrastive Learning with Pretrained Image Models
With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training.
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model
Pretrained language models have achieved remarkable success in various natural language processing tasks.
A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition
Particularly, we propose a Confidence-based Partial Label Learning (CPLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER.
ReGeneration Learning of Diffusion Models with Rich Prompts for Zero-Shot Image Translation
To ensure consistent preservation of the shape during image editing, we propose cross-attention guidance based on regeneration learning.
Hierarchical Contrastive Learning Enhanced Heterogeneous Graph Neural Network
Specifically, two views of a HIN (network schema and meta-path views) are proposed to learn node embeddings, so as to capture both of local and high-order structures simultaneously.
Learning CLIP Guided Visual-Text Fusion Transformer for Video-based Pedestrian Attribute Recognition
The averaged visual tokens and text tokens are concatenated and fed into a fusion Transformer for multi-modal interactive learning.
InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction
Large language models have unlocked strong multi-task capabilities from reading instructive prompts.
Ranked #3 on
Zero-shot Named Entity Recognition (NER)
on CrossNER
(using extra training data)
Curricular Object Manipulation in LiDAR-based Object Detection
For the loss design, we propose the COMLoss to dynamically predict object-level difficulties and emphasize objects of different difficulties based on training stages.
Efficient Multimodal Sampling via Tempered Distribution Flow
Sampling from high-dimensional distributions is a fundamental problem in statistical research and practice.
A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision
A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well.
RGBT Tracking via Progressive Fusion Transformer with Dynamically Guided Learning
In addition, existing learning methods of RGBT trackers either fuse multimodal features into one for final classification, or exploit the relationship between unimodal branches and fused branch through a competitive learning strategy.
Micro-video Tagging via Jointly Modeling Social Influence and Tag Relation
Existing methods mostly focus on analyzing video content, neglecting users' social influence and tag relation.
AdPE: Adversarial Positional Embeddings for Pretraining Vision Transformers via MAE+
A criterion in unsupervised pretraining is the pretext task needs to be sufficiently hard to prevent the transformer encoder from learning trivial low-level features not generalizable well to downstream tasks.
Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey
We also give visualization and analysis of the model parameters and results on representative downstream tasks.
A Survey on Spectral Graph Neural Networks
Graph neural networks (GNNs) have attracted considerable attention from the research community.
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Zero-Shot Transfer Image Classification
on ObjectNet
Machine Learning for Synthetic Data Generation: A Review
In light of these challenges, the concept of synthetic data generation emerges as a promising alternative that allows for data sharing and utilization in ways that real-world data cannot facilitate.
DEJA VU: Continual Model Generalization For Unseen Domains
To overcome these limitations of DA and DG in handling the Unfamiliar Period during continual domain shift, we propose RaTP, a framework that focuses on improving models' target domain generalization (TDG) capability, while also achieving effective target domain adaptation (TDA) capability right after training on certain domains and forgetting alleviation (FA) capability on past domains.
Directed Acyclic Graph Structure Learning from Dynamic Graphs
In a dynamic graph, we propose to simultaneously estimate contemporaneous relationships and time-lagged interaction relationships between the node features.
Private Multi-Winner Voting for Machine Learning
We use our mechanisms to enable privacy-preserving multi-label learning in the central setting by extending the canonical single-label technique: PATE.
Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric
In this paper, we propose a single-stage backbone network for Color-Event Unified Tracking (CEUTrack), which achieves the above functions simultaneously.
Ranked #3 on
Object Tracking
on COESOT
Rethinking Batch Sample Relationships for Data Representation: A Batch-Graph Transformer based Approach
(1) It employs a flexible graph model, termed Batch Graph to jointly encode the visual and semantic relationships of samples within each mini-batch.
HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors
The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which are suffered from illumination, fast motion, privacy-preserving, and large energy consumption.
Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes
While reinforcement learning produces very promising results for many applications, its main disadvantage is the lack of safety guarantees, which prevents its use in safety-critical systems.
Uncovering the Structural Fairness in Graph Contrastive Learning
Recent studies show that graph convolutional network (GCN) often performs worse for low-degree nodes, exhibiting the so-called structural unfairness for graphs with long-tailed degree distributions prevalent in the real world.
Revisiting Graph Contrastive Learning from the Perspective of Graph Spectrum
Then we theoretically prove that GCL is able to learn the invariance information by contrastive invariance theorem, together with our GAME rule, for the first time, we uncover that the learned representations by GCL essentially encode the low-frequency information, which explains why GCL works.
Debiasing Graph Neural Networks via Learning Disentangled Causal Substructure
However, by presenting a graph classification investigation on the training graphs with severe bias, surprisingly, we discover that GNNs always tend to explore the spurious correlations to make decision, even if the causal correlation always exists.
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification
sampleFormer aims to capture the dependence of samples in support and query sets for image representation.
See Finer, See More: Implicit Modality Alignment for Text-based Person Retrieval
To explore the fine-grained alignment, we further propose two implicit semantic alignment paradigms: multi-level alignment (MLA) and bidirectional mask modeling (BMM).
