Search Results for author:
Found 394 papers, 151 papers with code
SOM-NCSCM : An Efficient Neural Chinese Sentence Compression Model Enhanced with Self-Organizing Map
Sentence Compression (SC), which aims to shorten sentences while retaining important words that express the essential meanings, has been studied for many years in many languages, especially in English.
More Classifiers, Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning
Since those regularization strategies are mostly associated with classifier outputs, we propose a MUlti-Classifier (MUC) incremental learning paradigm that integrates an ensemble of auxiliary classifiers to estimate more effective regularization constraints.
A Gravity-informed Spatiotemporal Transformer for Human Activity Intensity Prediction
Specifically, it (1) estimates two spatially explicit mass parameters based on inflow and outflow, (2) models the likelihood of cross-unit interaction using closed-form solutions of spatial interactions to constrain spatial modeling randomness, and (3) utilizes the learned spatial interaction to guide and mitigate the over-smoothing phenomenon in transformer attention matrices.
Domain Switching on the Pareto Front: Multi-Objective Deep Kernel Learning in Automated Piezoresponse Force Microscopy
Ferroelectric polarization switching underpins the functional performance of a wide range of materials and devices, yet its dependence on complex local microstructural features renders systematic exploration by manual or grid-based spectroscopic measurements impractical.
AliTok: Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model
Autoregressive image generation aims to predict the next token based on previous ones.
Ranked #12 on
Image Generation
on ImageNet 256x256
AliTok: Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model
Autoregressive image generation aims to predict the next token based on previous ones.
AuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers
We propose AuralNet, a novel 3D multi-source binaural sound source localization approach that localizes overlapping sources in both azimuth and elevation without prior knowledge of the number of sources.
OpenCarbon: A Contrastive Learning-based Cross-Modality Neural Approach for High-Resolution Carbon Emission Prediction Using Open Data
Our model, OpenCarbon, features two major designs that target the challenges: a cross-modality information extraction and fusion module to extract complementary functionality information from two modules and model their interactions, and a neighborhood-informed aggregation module to capture the spatial contiguity correlations.
Exploring Domain Wall Pinning in Ferroelectrics via Automated High Throughput AFM
Domain-wall dynamics in ferroelectric materials are strongly position-dependent since each polar interface is locked into a unique local microstructure.
A Tool for Generating Exceptional Behavior Tests With Large Language Models
Exceptional behavior tests (EBTs) are crucial in software development for verifying that code correctly handles unwanted events and throws appropriate exceptions.
SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation
Leveraging the diffusion transformer (DiT) architecture, models like Sora, CogVideoX and Wan have achieved remarkable progress in text-to-video, image-to-video, and video editing tasks.
Emotional Supporters often Use Multiple Strategies in a Single Turn
We formally redefine the ESC task to account for this, proposing a revised formulation that requires generating the full sequence of strategy-utterance pairs given a dialogue history.
SurvUnc: A Meta-Model Based Uncertainty Quantification Framework for Survival Analysis
To bridge this gap, in this work, we introduce SurvUnc, a novel meta-model based framework for post-hoc uncertainty quantification for survival models.
Generalizable Multispectral Land Cover Classification via Frequency-Aware Mixture of Low-Rank Token Experts
Existing methods predominantly rely on domain adaptation and generalization strategies, often utilizing small-scale models that exhibit limited performance.
M3Depth: Wavelet-Enhanced Depth Estimation on Mars via Mutual Boosting of Dual-Modal Data
Depth estimation plays a great potential role in obstacle avoidance and navigation for further Mars exploration missions.
MPMA: Preference Manipulation Attack Against Model Context Protocol
To achieve MPMA, we first design a Direct Preference Manipulation Attack ($\mathtt{DPMA}$) that achieves significant effectiveness by inserting the manipulative word and phrases into the tool name and description.
CheX-DS: Improving Chest X-ray Image Classification with Ensemble Learning Based on DenseNet and Swin Transformer
The automatic diagnosis of chest diseases is a popular and challenging task.
Seed1.5-VL Technical Report
We present Seed1. 5-VL, a vision-language foundation model designed to advance general-purpose multimodal understanding and reasoning.
Ranked #1 on
Video Question Answering
on TVBench
ICNN-enhanced 2SP: Leveraging input convex neural networks for solving two-stage stochastic programming
Existing learning-based methods like Neural Two-Stage Stochastic Programming (Neur2SP) employ neural networks (NNs) as recourse function surrogates but rely on computationally intensive mixed-integer programming (MIP) formulations.
From Word to Sentence: A Large-Scale Multi-Instance Dataset for Open-Set Aerial Detection
Using this label engine, we expand existing aerial detection datasets with rich textual annotations and construct a novel benchmark dataset, called Multi-instance Open-set Aerial Dataset (MI-OAD), addressing the limitations of current remote sensing grounding data and enabling effective open-set aerial detection.
PhysioSync: Temporal and Cross-Modal Contrastive Learning Inspired by Physiological Synchronization for EEG-Based Emotion Recognition
Besides, it introduces Long- and Short-Term Temporal Contrastive Learning (LS-TCL) to capture emotional synchronization at different temporal resolutions within modalities.
ADT: Tuning Diffusion Models with Adversarial Supervision
To address this problem, we propose an intuitive but effective fine-tuning framework, called Adversarial Diffusion Tuning (ADT), by stimulating the inference process during optimization and aligning the final outputs with training data by adversarial supervision.
APSQ: Additive Partial Sum Quantization with Algorithm-Hardware Co-Design
DNN accelerators, significantly advanced by model compression and specialized dataflow techniques, have marked considerable progress.
The Power of the Pareto Front: Balancing Uncertain Rewards for Adaptive Experimentation in scanning probe microscopy
Automated experimentation has the potential to revolutionize scientific discovery, but its effectiveness depends on well-defined optimization targets, which are often uncertain or probabilistic in real-world settings.
AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization
In this paper, we proposes AdPO, a novel adversarial defense strategy for LVLMs based on preference optimization.
High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioning
Face swapping aims to seamlessly transfer a source facial identity onto a target while preserving target attributes such as pose and expression.
Wan: Open and Advanced Large-Scale Video Generative Models
Openness: We open-source the entire series of Wan, including source code and all models, with the goal of fostering the growth of the video generation community.
Targetless 6DoF Calibration of LiDAR and 2D Scanning Radar Based on Cylindrical Occupancy
Consequently, a cost function involving extrinsic calibration parameters is formulated based on the spatial overlap of 3D grids and LiDAR points.
ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing
In this paper, we propose ICE-Bench, a unified and comprehensive benchmark designed to rigorously assess image generation models.
Federated Mixture-of-Expert for Non-Overlapped Cross-Domain Sequential Recommendation
In the real world, users always have multiple interests while surfing different services to enrich their daily lives, e. g., watching hot short videos/live streamings.
VACE: All-in-One Video Creation and Editing
Further pursuing the unification of generation and editing tasks has yielded significant progress in the domain of image content creation.
OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection
Out-of-distribution (OOD) detection is crucial for ensuring the reliability and safety of machine learning models in real-world applications.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster
Recently, Large Language Models (LLMs) and Foundation Models (FMs) have become prevalent for time series forecasting tasks.
Rethinking Video Tokenization: A Conditioned Diffusion-based Approach
Trained using only a basic MSE diffusion loss for reconstruction, along with KL term and LPIPS perceptual loss from scratch, extensive experiments demonstrate that CDT achieves state-of-the-art performance in video reconstruction tasks with just a single-step sampling.
NTR-Gaussian: Nighttime Dynamic Thermal Reconstruction with 4D Gaussian Splatting Based on Thermodynamics
Thermal infrared imaging offers the advantage of all-weather capability, enabling non-intrusive measurement of an object's surface temperature.
Molecule Generation for Target Protein Binding with Hierarchical Consistency Diffusion Model
Effective generation of molecular structures, or new chemical entities, that bind to target proteins is crucial for lead identification and optimization in drug discovery.
ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting
Existing methods often fail to effectively integrate information across different object states, limiting the accuracy of part-mesh reconstruction and part dynamics modeling, particularly for complex multi-part articulated objects.
Rewards-based image analysis in microscopy
They can function as wrappers over classical and DCNN-based methods, making them applicable to both unsupervised and supervised workflows (e. g., classification, regression for structure-property mapping) across imaging and hyperspectral data.
Transfer learning in Scalable Graph Neural Network for Improved Physical Simulation
To enable the transfer learning between differently configured SGUNETs, we propose a set of mapping functions to align the parameters between the pre-trained model and the target model.
MapFusion: A Novel BEV Feature Fusion Network for Multi-modal Map Construction
To address these issues, we propose MapFusion, a novel multi-modal Bird's-Eye View (BEV) feature fusion method for map construction.
Learning to Learn Weight Generation via Local Consistency Diffusion
Diffusion-based algorithms have emerged as promising techniques for weight generation.
RegionGCN: Spatial-Heterogeneity-Aware Graph Convolutional Networks
However, current geographically weighting approaches are ineffective on graph neural networks, yielding no significant improvement in prediction accuracy.
TranStable: Towards Robust Pixel-level Online Video Stabilization by Jointing Transformer and CNN
Video stabilization often struggles with distortion and excessive cropping.
DiffDoctor: Diagnosing Image Diffusion Models Before Treating
In spite of the recent progress, image diffusion models still produce artifacts.
Universal Actions for Enhanced Embodied Foundation Models
Training on diverse, internet-scale data is a key factor in the success of recent large foundation models.
VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization
Video colorization aims to transform grayscale videos into vivid color representations while maintaining temporal consistency and structural integrity.
MangaNinja: Line Art Colorization with Precise Reference Following
Derived from diffusion models, MangaNinjia specializes in the task of reference-guided line art colorization.
GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing
In addition, to address the absence of large-scale datasets for training pixel-level RS MLLMs, we construct the GeoPixInstruct dataset, comprising 65, 463 images and 140, 412 instances, with each instance annotated with text descriptions, bounding boxes, and masks.
Data driven discovery of human mobility models
However, for decades new mathematical formulas to model mobility phenomena have been scarce and usually discovered by analogy to physical processes, such as the gravity model and the radiation model.
ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling
There are many models in the community based on the post-training of text-to-image foundational models that meet this training paradigm of the first stage.
MSC-Bench: Benchmarking and Analyzing Multi-Sensor Corruption for Driving Perception
Multi-sensor fusion models play a crucial role in autonomous driving perception, particularly in tasks like 3D object detection and HD map construction.
Decompositional Neural Scene Reconstruction with Generative Diffusion Prior
To this end, we propose DP-Recon, which employs diffusion priors in the form of Score Distillation Sampling (SDS) to optimize the neural representation of each individual object under novel views.
INFELM: In-depth Fairness Evaluation of Large Text-To-Image Models
The rapid development of large language models (LLMs) and large vision models (LVMs) have propelled the evolution of multi-modal AI systems, which have demonstrated the remarkable potential for industrial applications by emulating human-like cognition.
Automated Materials Discovery Platform Realized: Scanning Probe Microscopy of Combinatorial Libraries
Here we demonstrate the implementation of fully automated SPM to explore the evolution of ferroelectric properties in combinatorial libraries, focusing on Sm-doped BiFeO3 and ZnxMg1-xO systems.
Color Enhancement for V-PCC Compressed Point Cloud via 2D Attribute Map Optimization
This paper introduces a framework designed to enhance the color quality in the V-PCC compressed point clouds.
ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers
Building upon this foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive visual generation framework that leverages pretrained diffusion transformers in their original form, requiring no additional tuning, adapters, or modifications.
Smartphone-based Iris Recognition through High-Quality Visible Spectrum Iris Capture
However, a thorough study of iris recognition using smartphone-captured 'High-Quality' VIS images and cross-spectral matching with previously enrolled NIR images has not been conducted.
MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
Repurposing pre-trained diffusion models has been proven to be effective for NVS.
IDEA-Bench: How Far are Generative Models from Professional Designing?
Real-world design tasks - such as picture book creation, film storyboard development using character sets, photo retouching, visual effects, and font transfer - are highly diverse and complex, requiring deep interpretation and extraction of various elements from instructions, descriptions, and reference images.
VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping
Our approach effectively mitigates key challenges in video face swapping, including temporal flickering, identity preservation, and robustness to occlusions and pose variations.
GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection
To further enhance performance, we propose sparsely coded residuals to reduce the errors caused by low-rank approximation on the first- and second-order moments of the optimizers and weight updates.
EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
To effectively exploit consistent visual elements within multiple images, we leverage the multi-image comprehension and instruction-following capabilities of the multimodal large language model (MLLM), prompting it to capture consistent visual elements based on the instruction.
See Further When Clear: Curriculum Consistency Model
Specifically, we regard the distillation process at each timestep as a curriculum and introduce a metric based on Peak Signal-to-Noise Ratio (PSNR) to quantify the learning complexity of this curriculum, then ensure that the curriculum maintains consistent learning complexity across different timesteps by having the teacher model iterate more steps when the noise intensity is low.
The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
We present The Matrix, the first foundational realistic world simulator capable of generating continuous 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives, enabling immersive exploration of richly dynamic environments.
Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
However, existing works exhibit significant variations in training schemes and RL optimization objectives, and some methods are only applicable to diffusion models.
Pretrained Reversible Generation as Unsupervised Visual Representation Learning
Recent generative models based on score matching and flow matching have significantly advanced generation tasks, but their potential in discriminative tasks remains underexplored.
Reward driven workflows for unsupervised explainable analysis of phases and ferroic variants from atomically resolved imaging data
We demonstrate that a reward-driven approach can be used to optimize these key hyperparameters across the full workflow, where rewards were designed to reflect domain wall continuity and straightness, ensuring that the analysis aligns with the material's physical behavior.
MTFusion: Reconstructing Any 3D Object from Single Image Using Multi-word Textual Inversion
The latest advances for single-image 3D reconstruction extract a textual description from the input image and further utilize it to synthesize 3D models.
Infrared-Assisted Single-Stage Framework for Joint Restoration and Fusion of Visible and Infrared Images under Hazy Conditions
To address this, we propose a joint learning framework that utilizes infrared image for the restoration and fusion of hazy IR-VIS images.
Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks
The primary value of infrared and visible image fusion technology lies in applying the fusion results to downstream tasks.
Improved Video VAE for Latent Video Diffusion Model
Specifically, the KTC architecture divides the latent space into two branches, in which one half completely inherits the compression prior of keyframes from a lower-dimension image VAE while the other half involves temporal compression through 3D group causal convolution, reducing temporal-spatial conflicts and accelerating the convergence speed of video VAE.
Generalizing Hyperedge Expansion for Hyper-relational Knowledge Graph Modeling
To tackle this issue, in this paper, we generalize the hyperedge expansion in hypergraph learning and propose an equivalent transformation for HKG modeling, referred to as TransEQ.
Not Just Object, But State: Compositional Incremental Learning without Forgetting
Most incremental learners excessively prioritize coarse classes of objects while neglecting various kinds of states (e. g. color and material) attached to the objects.
In-Context LoRA for Diffusion Transformers
While task-specific in terms of tuning data, our framework remains task-agnostic in architecture and pipeline, offering a powerful tool for the community and providing valuable insights for further research on product-level task-agnostic generation systems.
Synergizing LLM Agents and Knowledge Graph for Socioeconomic Prediction in LBSN
Motivated by the remarkable abilities of large language models (LLMs) in commonsense reasoning, embedding, and multi-agent collaboration, in this work, we synergize LLM agents and knowledge graph for socioeconomic prediction.
NT-VOT211: A Large-Scale Benchmark for Night-time Visual Object Tracking
To this end, this paper presents NT-VOT211, a new benchmark tailored for evaluating visual object tracking algorithms in the challenging night-time conditions.
Depth Attention for Robust RGB Tracking
RGB video object tracking is a fundamental task in computer vision.
STAR-RIS-Enabled Full-Duplex Integrated Sensing and Communication System
With the challenging limitations of traditional SIC approaches, this paper proposes a novel simultaneous transmitting and reflecting reconfigurable intelligent surface (STAR-RIS)-enabled FD ISAC system, where STAR-RIS enhances simultaneous communication and target sensing and reduces self-interference (SI) to a level comparable to traditional SIC approaches.
A 10.60 $μ$W 150 GOPS Mixed-Bit-Width Sparse CNN Accelerator for Life-Threatening Ventricular Arrhythmia Detection
This paper proposes an ultra-low power, mixed-bit-width sparse convolutional neural network (CNN) accelerator to accelerate ventricular arrhythmia (VA) detection.
Group Diffusion Transformers are Unsupervised Multitask Learners
In this work, we introduce Group Diffusion Transformers (GDTs), a novel framework that unifies diverse visual generation tasks by redefining them as a group generation problem.
LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment
LoD-Loc mainly achieves this goal by aligning the wireframe derived from the LoD projected model with that predicted by the neural network.
Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques
Recently, the remarkable success of ChatGPT has sparked a renewed wave of interest in artificial intelligence (AI), and the advancements in visual language models (VLMs) have pushed this enthusiasm to new heights.
SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction
Extensive experiments on multiple datasets demonstrate that SmartPretrain consistently improves the performance of state-of-the-art prediction models across datasets, data splits and main metrics.
GLRT-Based Metric Learning for Remote Sensing Object Retrieval
We reorganize datasets for CBRSOR tasks based on fine-grained ship remote sensing image slices (FGSRSI-23) and military aircraft recognition (MAR20) datasets.
Unsupervised Meta-Learning via Dynamic Head and Heterogeneous Task Construction for Few-Shot Classification
Based on the above conclusion, we argue a promising future for meta-learning in the unsupervised area, and thus propose DHM-UHT, a dynamic head meta-learning algorithm with unsupervised heterogeneous task construction.
Measurements with Noise: Bayesian Optimization for Co-optimizing Noise and Property Discovery in Automated Experiments
We have developed a Bayesian optimization (BO) workflow that integrates intra-step noise optimization into automated experimental cycles.
Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning
Multimodal task specification is essential for enhanced robotic performance, where \textit{Cross-modality Alignment} enables the robot to holistically understand complex task instructions.
ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer
To comprehensively evaluate the performance of our model, we establish a benchmark of manually annotated pairs data across a variety of visual generation tasks.
CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation
Then, we deploy contrastive learning to train the CLSP encoder to effectively represent precise state information.
ControlEdit: A MultiModal Local Clothing Image Editing Method
Multimodal clothing image editing refers to the precise adjustment and modification of clothing images using data such as textual descriptions and visual images as control conditions, which effectively improves the work efficiency of designers and reduces the threshold for user design.
MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines
We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine.
Valuation Model of Chinese Convertible Bonds Based on Monte Carlo Simulation
We tackle the problem of pricing Chinese convertible bonds(CCBs) using Monte Carlo simulation and dynamic programming.
Discovering Cyclists' Visual Preferences Through Shared Bike Trajectories and Street View Images Using Inverse Reinforcement Learning
We find that cyclists focus on specific street visual elements when making route decisions, which can be summarized as their attention to safety, street enclosure, and cycling comfort.
Explainable artificial intelligence
Explainable Artificial Intelligence (XAI)
Data-Driven Parametrization of Molecular Mechanics Force Fields for Expansive Chemical Space Coverage
A force field is a critical component in molecular dynamics simulations for computational drug discovery.
OVA-DETR: Open Vocabulary Aerial Object Detection Using Image-Text Alignment and Fusion
Then, we propose Bidirectional Vision-Language Fusion (Bi-VLF), which includes a dual-attention fusion encoder and a multi-level text-guided Fusion Decoder.
SlotLifter: Slot-guided Feature Lifting for Learning Object-centric Radiance Fields
The ability to distill object-centric abstractions from intricate visual scenes underpins human-level generalization.
Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detection
Then, we extract realistic features of seen samples and mix them with synthetic features together, allowing the model to train seen and unseen classes jointly.
Human-Object Interaction Detection
Zero-Shot Human-Object Interaction Detection
Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval
Simply applying existing cross-modal approaches to this new task fails to adequately capture latent multi-modal semantics and effectively bridge the modality gap between 2D and 3D.
Machine Learning-Based Reward-Driven Tuning of Scanning Probe Microscopy: Towards Fully Automated Microscopy
This automated workflow gives optimal scanning parameters for different probes and samples and gives high-quality SPM images consistently in the attractive mode.
Multiscale Representation Enhanced Temporal Flow Fusion Model for Long-Term Workload Forecasting
Despite recent advances with transformer-based forecasting models, challenges remain due to the non-stationary, nonlinear characteristics of workload time series and the long-term dependencies.
PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements
To address the challenge of unquantifiable psychological traits, we introduce a novel training paradigm that involves learning the ranking of proxy variables associated with these traits, culminating in a robust score model for MBTI measurements.
Fast Context-Based Low-Light Image Enhancement via Neural Implicit Representations
Current deep learning-based low-light image enhancement methods often struggle with high-resolution images, and fail to meet the practical demands of visual perception across diverse and unseen scenarios.
MergeNet: Explicit Mesh Reconstruction from Sparse Point Clouds via Edge Prediction
This paper introduces a novel method for reconstructing meshes from sparse point clouds by predicting edge connection.
QVD: Post-training Quantization for Video Diffusion Models
Furthermore, we investigate significant inter-channel disparities and asymmetries in the activation of video diffusion models, resulting in low coverage of quantization levels by individual channels and increasing the challenge of quantization.
Cross Domain Object Detection via Multi-Granularity Confidence Alignment based Mean Teacher
Cross domain object detection learns an object detector for an unlabeled target domain by transferring knowledge from an annotated source domain.
Incremental Multiview Point Cloud Registration
This process primarily constructs a coarse multiview registration and refines the model by adjusting the positions of the keypoints on the Track.
A Marginal Distributionally Robust Kalman Filter for Centralized Fusion
State estimation is a fundamental problem for multi-sensor information fusion, essential in applications such as target tracking, power systems, and control automation.
BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs
Advancements in large Vision-Language Models have brought precise, accurate image captioning, vital for advancing multi-modal image understanding and processing.
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
This paper introduces MM-Instruct, a large-scale dataset of diverse and high-quality visual instruction data designed to enhance the instruction-following capabilities of large multimodal models (LMMs).
Ranked #139 on
Visual Question Answering
on MM-Vet
CombAlign: Enhancing Model Expressiveness in Unsupervised Graph Alignment
We investigate the model expressiveness from two aspects.
GMP-AR: Granularity Message Passing and Adaptive Reconciliation for Temporal Hierarchy Forecasting
These methods merely take the temporal hierarchical structure to maintain coherence without improving the forecasting accuracy.
From Pixels to Progress: Generating Road Network from Satellite Imagery for Socioeconomic Insights in Impoverished Areas
The Sustainable Development Goals (SDGs) aim to resolve societal challenges, such as eradicating poverty and improving the lives of vulnerable populations in impoverished areas.
Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
To deal with this issue, we propose a novel framework to fully harness the capabilities of LLMs.
Technique Report of CVPR 2024 PBDL Challenges
The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies.
UniZero: Generalized and Efficient Planning with Scalable Latent World Models
To overcome these limitations, we introduce UniZero, a novel approach that employs a modular transformer-based world model to effectively learn a shared latent space.
Compressed Video Quality Enhancement with Temporal Group Alignment and Fusion
In this paper, we propose a temporal group alignment and fusion network to enhance the quality of compressed videos by using the long-short term correlations between frames.
The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models
As a result, previous debiasing methods mainly finetune or even pre-train language models on newly constructed anti-stereotypical datasets, which are high-cost.
VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models
However, sound comprehensive research on detecting program vulnerabilities, a more specific task related to code, and evaluating the performance of LLMs in this more specialized scenario is still lacking.
Zero-shot Image Editing with Reference Imitation
Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like.
Research on the Spatial Data Intelligent Foundation Model
This report focuses on spatial data intelligent large models, delving into the principles, methods, and cutting-edge applications of these models.
Instruction-Guided Visual Masking
To achieve more accurate and nuanced multimodal instruction following, we introduce Instruction-guided Visual Masking (IVM), a new versatile visual grounding model that is compatible with diverse multimodal models, such as LMM and robot model.
Ranked #1 on
Visual Question Answering
on V*bench
Enhancing Vision-Language Model with Unmasked Token Alignment
Contrastive pre-training on image-text pairs, exemplified by CLIP, becomes a standard technique for learning multi-modal visual-language representations.
Phased Consistency Models
In this paper, we identify three key flaws in the current design of Latent Consistency Models (LCMs).
Integration of Scanning Probe Microscope with High-Performance Computing: fixed-policy and reward-driven workflows implementation
The rapid development of computation power and machine learning algorithms has paved the way for automating scientific discovery with a scanning probe microscope (SPM).
Novel Class Discovery for Ultra-Fine-Grained Visual Categorization
To tackle this problem, we devise a Region-Aligned Proxy Learning (RAPL) framework, which comprises a Channel-wise Region Alignment (CRA) module and a Semi-Supervised Proxy Learning (SemiPL) strategy.
Batched Stochastic Bandit for Nondegenerate Functions
More specifically, we introduce an algorithm, called Geometric Narrowing (GN), whose regret bound is of order $\widetilde{{\mathcal{O}}} ( A_{+}^d \sqrt{T} )$.
Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models
In this study, we propose Deep Reward Tuning (DRTune), an algorithm that directly supervises the final output image of a text-to-image diffusion model and back-propagates through the iterative sampling process to the input noise.
ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze
Monte Carlo Tree Search (MCTS)-based algorithms, such as MuZero and its derivatives, have achieved widespread success in various decision-making domains.
Improving TAS Adaptability with a Variable Temperature Threshold
Thermal-Aware Scheduling (TAS) provides methods to manage the thermal dissipation of a computing chip during task execution.
MoVA: Adapting Mixture of Vision Experts to Multimodal Context
In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts.
InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior
3D Gaussians have recently emerged as an efficient representation for novel view synthesis.
GLID: Pre-training a Generalist Encoder-Decoder Vision Model
This paper proposes a GeneraLIst encoder-Decoder (GLID) pre-training method for better handling various downstream computer vision tasks.
Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance
Classifier-Free Guidance (CFG) has been widely used in text-to-image diffusion models, where the CFG scale is introduced to control the strength of text guidance on the whole image space.
CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching
We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm.
Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers
Non-linear functions are prevalent in Transformers and their lightweight variants, incurring substantial and frequently underestimated hardware costs.
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions.
FlashFace: Human Image Personalization with High-fidelity Identity Preservation
This work presents FlashFace, a practical tool with which users can easily personalize their own photos on the fly by providing one or a few reference face images and a text prompt.
Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific Adaptation
We introduce MOTIA Mastering Video Outpainting Through Input-Specific Adaptation, a diffusion-based pipeline that leverages both the intrinsic data-specific patterns of the source video and the image/video generative prior for effective outpainting.
FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis
In this study, we delve into the generation of high-resolution images from pre-trained diffusion models, addressing persistent challenges, such as repetitive patterns and structural distortions, that emerge when models are applied beyond their trained resolutions.
SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction
Context information, such as road maps and surrounding agents' states, provides crucial geometric and semantic information for motion behavior prediction.
Depth-induced Saliency Comparison Network for Diagnosis of Alzheimer's Disease via Jointly Analysis of Visual Stimuli and Eye Movements
Early diagnosis of Alzheimer's Disease (AD) is very important for following medical treatments, and eye movements under special visual stimuli may serve as a potential non-invasive biomarker for detecting cognitive abnormalities of AD patients.
CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement
Specifically, the ITA module aggregates temporal information from consecutive frames and coding priors, while the MNA module globally captures spatial information guided by residual frames.
CSCNET: Class-Specified Cascaded Network for Compositional Zero-Shot Learning
Inspired by this, we propose a novel A-O disentangled framework for CZSL, namely Class-specified Cascaded Network (CSCNet).
DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning
Multimodal pretraining is an effective strategy for the trinity of goals of representation learning in autonomous robots: 1) extracting both local and global task progressions; 2) enforcing temporal consistency of visual representation; 3) capturing trajectory-level language grounding.
Ranked #1 on
Contrastive Learning
on 10,000 People - Human Pose Recognition Data
(using extra training data)
DPBalance: Efficient and Fair Privacy Budget Scheduling for Federated Learning as a Service
We also theoretically prove the existence of a fairness-efficiency tradeoff in privacy budgeting.
Extensible Multi-Granularity Fusion Network for Aspect-based Sentiment Analysis
This paper presents the Extensible Multi-Granularity Fusion (EMGF) network, which integrates information from dependency and constituent syntactic, attention semantic , and external knowledge graphs.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+2
Estimating On-road Transportation Carbon Emissions from Open Data of Road Network and Origin-destination Flow Data
To relieve this issue by utilizing the strong pattern recognition of artificial intelligence, we incorporate two sources of open data representative of the transportation demand and capacity factors, the origin-destination (OD) flow data and the road network data, to build a hierarchical heterogeneous graph learning method for on-road carbon emission estimation (HENCE).
Space Group Constrained Crystal Generation
Crystals are the foundation of numerous scientific and industrial applications.
AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Data
This paper introduces an effective method for computation-efficient personalized style video generation without requiring access to any personalized video data.
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis
To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model's ability to capture the true semantic representation of visual scenes.
Deep-Learning Channel Estimation for IRS-Assisted Integrated Sensing and Communication System
This problem is challenging due to the lack of signal processing capacity in passive IRS, as well as the presence of mutual interference between sensing and communication (SAC) signals in ISAC systems.
Deep-Learning-Based Channel Estimation for IRS-Assisted ISAC System
A deep-learning framework is proposed to estimate the sensing and communication (S&C) channels in such a system.
Extreme Learning Machine-based Channel Estimation in IRS-Assisted Multi-User ISAC System
Multi-user integrated sensing and communication (ISAC) assisted by intelligent reflecting surface (IRS) has been recently investigated to provide a high spectral and energy efficiency transmission.
Efficient Neural Network
Integrated sensing and communication
+1
UV-SAM: Adapting Segment Anything Model for Urban Village Identification
Urban villages, defined as informal residential areas in or around urban centers, are characterized by inadequate infrastructures and poor living conditions, closely related to the Sustainable Development Goals (SDGs) on poverty, adequate housing, and sustainable cities.
Knowledge-aware Graph Transformer for Pedestrian Trajectory Prediction
To overcome this limitation, this paper proposes a graph transformer structure to improve prediction performance, capturing the differences between the various sites and scenarios contained in the datasets.
EasyDrag: Efficient Point-based Manipulation on Diffusion Models
Generative models are gaining increasing popularity and the demand for precisely generating images is on the rise.
Multi-agent Collaborative Perception via Motion-aware Robust Communication Network
Collaborative perception allows for information sharing between multiple agents such as vehicles and infrastructure to obtain a comprehensive view of the environment through communication and fusion.
Check Locate Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation
To align the generated image with layout instructions we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time.
AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Framework
To maintain modality autonomy, AllSpark uses modality-specific encoders to extract the tokens of various spatio-temporal modalities.
Mutual Information as Intrinsic Reward of Reinforcement Learning Agents for On-demand Ride Pooling
The emergence of on-demand ride pooling services allows each vehicle to serve multiple passengers at a time, thus increasing drivers' income and enabling passengers to travel at lower prices than taxi/car on-demand services (only one passenger can be assigned to a car at a time like UberX and Lyft).
TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification
The crux of learning vision-language models is to extract semantically aligned information from visual and linguistic data.
Critic-Guided Decision Transformer for Offline Reinforcement Learning
Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner.
ARBiBench: Benchmarking Adversarial Robustness of Binarized Neural Networks
2) BNNs consistently exhibit better adversarial robustness under black-box attacks.
Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches
The selection of the assumed effect size (AES) critically determines the duration of an experiment, and hence its accuracy and efficiency.
VideoLCM: Video Latent Consistency Model
Consistency models have demonstrated powerful capability in efficient image generation and allowed synthesis within a few sampling steps, alleviating the high computational cost in diffusion models.
Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptation
Building embodied agents on integrating Large Language Models (LLMs) and Reinforcement Learning (RL) have revolutionized human-AI interaction: researchers can now leverage language instructions to plan decision-making for open-ended tasks.
A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
In this paper, from a novel perspective, we systematically study the challenges that remain in O2O RL and identify that the reason behind the slow improvement of the performance and the instability of online finetuning lies in the inaccurate Q-value estimation inherited from offline pretraining.
LMDrive: Closed-Loop End-to-End Driving with Large Language Models
On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e. g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans.
CCM: Adding Conditional Controls to Text-to-Image Consistency Models
Consistency Models (CMs) have showed a promise in creating visual content efficiently and with high quality.
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern.
LivePhoto: Real Image Animation with Text-guided Motion Control
In particular, considering the facts that (1) text can only describe motions roughly (e. g., regardless of the moving speed) and (2) text may include both content and motion descriptions, we introduce a motion intensity estimation module as well as a text re-weighting module to reduce the ambiguity of text-to-motion mapping.
Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation
Experimental results show that existing subject-driven customization methods fail to learn the representative characteristics of actions and struggle in decoupling actions from context features, including appearance.
Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation
To align the generated image with layout instructions, we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time.
Devil in the Landscapes: Inferring Epidemic Exposure Risks from Street View Imagery
Built environment supports all the daily activities and shapes our health.
CHAIN: Exploring Global-Local Spatio-Temporal Information for Improved Self-Supervised Video Hashing
Compressing videos into binary codes can improve retrieval speed and reduce storage overhead.
Towards Large-scale Masked Face Recognition
During the COVID-19 coronavirus epidemic, almost everyone is wearing masks, which poses a huge challenge for deep learning-based face recognition algorithms.
Decoupled DETR: Spatially Disentangling Localization and Classification for Improved End-to-End Object Detection
We observe that different regions of interest in the visual feature map are suitable for performing query classification and box localization tasks, even for the same object.
MaskMA: Towards Zero-Shot Multi-Agent Decision Making with Mask-Based Collaborative Learning
Building a single generalist agent with strong zero-shot capability has recently sparked significant advancements.
Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
Considering that obtaining expert demonstrations can be costly, the focus of current IRL techniques is on learning a better-than-demonstrator policy using a reward function derived from sub-optimal demonstrations.
LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios
Building agents based on tree-search planning capabilities with learned models has achieved remarkable success in classic decision-making problems, such as Go and Atari.
Continuous Invariance Learning
To address this challenge, we then propose Continuous Invariance Learning (CIL), which extracts invariant features across continuously indexed domains.
Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformers
Vector-quantized image modeling has shown great potential in synthesizing high-quality images.
Magicremover: Tuning-free Text-guided Image inpainting with Diffusion Models
In this paper, we propose MagicRemover, a tuning-free method that leverages the powerful diffusion models for text-guided image inpainting.
Continuous 3D Myocardial Motion Tracking via Echocardiography
Myocardial motion tracking stands as an essential clinical tool in the prevention and detection of cardiovascular diseases (CVDs), the foremost cause of death globally.
Regulating CPU Temperature With Thermal-Aware Scheduling Using a Reduced Order Learning Thermal Model
Additionally, we compare the performance of a state of the art TAS algorithm, RT-TAS, to our proposed POD-TAS algorithm.
Liveness Detection Competition -- Noncontact-based Fingerprint Algorithms and Systems (LivDet-2023 Noncontact Fingerprint)
The competition serves as an important benchmark in noncontact-based fingerprint PAD, offering (a) independent assessment of the state-of-the-art in noncontact-based fingerprint PAD for algorithms and systems, and (b) common evaluation protocol, which includes finger photos of a variety of Presentation Attack Instruments (PAIs) and live fingers to the biometric research community (c) provides standard algorithm and system evaluation protocols, along with the comparative analysis of state-of-the-art algorithms from academia and industry with both old and new android smartphones.
Towards Generative Modeling of Urban Flow through Knowledge-enhanced Denoising Diffusion
To capture the effect of multiple factors on urban flow, such as region features and urban environment, we employ diffusion model to generate urban flow for regions under different conditions.
BigFUSE: Global Context-Aware Image Fusion in Dual-View Light-Sheet Fluorescence Microscopy with Image Formation Prior
Light-sheet fluorescence microscopy (LSFM), a planar illumination technique that enables high-resolution imaging of samples, experiences defocused image quality caused by light scattering when photons propagate through thick tissues.
Evaluation Mappings of Spatial Accelerator Based On Data Placement
Based on data placement relations, polyAcc accurately analyzes the data volume for different reuse patterns and estimate metrics, including data reuse, latency, and energy.
Snow Removal for LiDAR Point Clouds with Spatio-temporal Conditional Random Fields
The proposed approach first constructs the CRF based on k-nearest neighbors with the snow confidence derived from the physical priors of snow, such as intensity and distribution.
A Satellite Imagery Dataset for Long-Term Sustainable Development in United States Cities
Especially satellite imagery is a potential data source for studying sustainable urban development.
3D Semantic Subspace Traverser: Empowering 3D Generative Model with Shape Editing Capability
Our method utilizes implicit functions as the 3D shape representation and combines a novel latent-space GAN with a linear subspace model to discover semantic dimensions in the local latent space of 3D shapes.
Theoretically Guaranteed Policy Improvement Distilled from Model-Based Planning
Model-based reinforcement learning (RL) has demonstrated remarkable successes on a range of continuous control tasks due to its high sample efficiency.
A Physics-Informed Data-Driven Fault Location Method for Transmission Lines Using Single-Ended Measurements with Field Data Validation
This procedure merges the gap between simulation and practical power systems, and at the same time considers the uncertainty of system and fault parameters in practice.
AnyDoor: Zero-shot Object-level Image Customization
This work presents AnyDoor, a diffusion-based image generator with the power to teleport target objects to new scenes at user-specified locations in a harmonious way.
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Before sending to LLM, the reference is replaced by RoI features and interleaved with language embeddings as a sequence.
Ranked #1 on
Visual Question Answering (VQA)
on VCR (Q-AR) test
OpenSiteRec: An Open Dataset for Site Recommendation
As a representative information retrieval task, site recommendation, which aims at predicting the optimal sites for a brand or an institution to open new branches in an automatic data-driven way, is beneficial and crucial for brand development in modern business.
Lipschitz Singularities in Diffusion Models
Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models.
Learning Search-Space Specific Heuristics Using Neural Networks
We propose and evaluate a system which learns a neuralnetwork heuristic function for forward search-based, satisficing classical planning.
