Search Results for author:
Found 308 papers, 138 papers with code
ScoreAdv: Score-based Targeted Generation of Natural Adversarial Examples via Diffusion Models
In this paper, we introduce a novel approach for generating UAEs based on diffusion models, named ScoreAdv.
Fine-grained Image Retrieval via Dual-Vision Adaptation
Fine-Grained Image Retrieval~(FGIR) faces challenges in learning discriminative visual representations to retrieve images with similar fine-grained features.
Learning to Reason Across Parallel Samples for LLM Reasoning
By separating LLMs to generate answers and LLMs to analyze and aggregate sampled answers, our approach can work with the outputs from premier black box models easily and efficiently.
Learning Compact Vision Tokens for Efficient Large Multimodal Models
Specifically, we propose a Spatial Token Fusion (STF) method to learn compact vision tokens for short vision token sequence, where spatial-adjacent tokens are fused into one.
Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
In this paper, we rethink token reduction and unify the process as an explicit form of token matrix transformation, in which all existing methods are constructing special forms of matrices within the framework.
Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment
Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e. g., reducing visual quality to improve attack success).
FOLIAGE: Towards Physical Intelligence World Models Via Unbounded Surface Evolution
We propose FOLIAGE, a physics-informed multimodal world model for unbounded accretive surface growth.
Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation
To address these issues, we propose a framework that adaptively selects and aggregates knowledge from diverse LLMs to build a single, stronger model, avoiding the high memory overhead of ensemble and inflexible weight merging.
Effective Context in Neural Speech Models
Modern neural speech models benefit from having longer context, and many approaches have been proposed to increase the maximum context a model can use.
SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams
To address this gap, we propose SpikeStereoNet, a brain-inspired framework and the first to estimate stereo depth directly from raw spike streams.
Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality
We highlight its potential to drive new model architectures and learning strategies that improve robustness, increase interpretability, and better align with the objectives of generative modeling.
SAMba-UNet: Synergizing SAM2 and Mamba in UNet with Heterogeneous Aggregation for Cardiac MRI Segmentation
To address the challenge of complex pathological feature extraction in automated cardiac MRI segmentation, this study proposes an innovative dual-encoder architecture named SAMba-UNet.
Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models
Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images, leaving them ill-suited for robotics and other real-world applications that require multi-frame reasoning.
Programmatic Video Prediction Using Large Language Models
The task of estimating the world model describing the dynamics of a real world process assumes immense importance for anticipating and preparing for future outcomes.
Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Image
For those scenes that are impossible to happen in real world and anti-physics, we should spare no efforts in increasing the factual feel, which means synthesizing images that people think very likely to be happening, and concept alignment, which means all the required objects should be in the same frame.
CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation
Although autoregressive models have dominated language modeling in recent years, there has been a growing interest in exploring alternative paradigms to the conventional next-token prediction framework.
Structured Agent Distillation for Large Language Model
Large language models (LLMs) exhibit strong capabilities as decision-making agents by interleaving reasoning and actions, as seen in ReAct-style frameworks.
PoE-World: Compositional World Modeling with Products of Programmatic Experts
Learning how the world works is central to building AI agents that can adapt to complex environments.
Improved Algorithms for Differentially Private Language Model Alignment
Language model alignment is crucial for ensuring that large language models (LLMs) align with human preferences, yet it often involves sensitive user data, raising significant privacy concerns.
Semantic-Guided Diffusion Model for Single-Step Image Super-Resolution
To address this limitation, we propose SAMSR, a semantic-guided diffusion framework that incorporates semantic segmentation masks into the sampling process.
LLM-Guided Probabilistic Program Induction for POMDP Model Estimation
Partially Observable Markov Decision Processes (POMDPs) model decision making under uncertainty.
Multimodal Large Language Models for Medicine: A Comprehensive Survey
At the end of the survey, we discuss the challenges faced by MLLMs in the medical and healthcare domain and propose feasible methods to mitigate or overcome these issues.
Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models
A primary limitation is the substantial number of image tokens required for AR models, which constrains both training and inference efficiency, as well as image resolution.
EventVAD: Training-Free Event-Aware Video Anomaly Detection
Therefore, we propose EventVAD, an event-aware video anomaly detection framework that combines tailored dynamic graph architectures and multimodal LLMs through temporal-event reasoning.
3D CoCa: Contrastive Learners are 3D Captioners
3D captioning, which aims to describe the content of 3D scenes in natural language, remains highly challenging due to the inherent sparsity of point clouds and weak cross-modal alignment in existing methods.
Ranked #1 on
3D dense captioning
on Nr3D
Multi-scale Activation, Refinement, and Aggregation: Exploring Diverse Cues for Fine-Grained Bird Recognition
Specifically, we first propose a multi-scale cue activation module to ensure the discriminative cues learned at different stage are mutually different.
Ranked #3 on
Fine-Grained Image Classification
on CUB-200-2011
Follow Your Motion: A Generic Temporal Consistency Portrait Editing Framework with Trajectory Guidance
However, they often face challenges with temporal consistency, particularly in the talking head domain, where continuous changes in facial expressions intensify the level of difficulty.
PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model
As interest grows in world models that predict future states from current observations and actions, accurately modeling part-level dynamics has become increasingly relevant for various applications.
Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models
Vision-Language Models (VLMs) excel at identifying and describing objects but struggle with spatial reasoning such as accurately understanding the relative positions of objects.
MambaIC: State Space Models for High-Performance Learned Image Compression
Inspired by the effectiveness of state space models (SSMs) in capturing long-range dependencies, we leverage SSMs to address computational inefficiency in existing methods and improve image compression from multiple perspectives.
Dynamic Scene Reconstruction: Recent Advance in Real-time Rendering and Streaming
Representing and rendering dynamic scenes from 2D images is a fundamental yet challenging problem in computer vision and graphics.
OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection
Out-of-distribution (OOD) detection is crucial for ensuring the reliability and safety of machine learning models in real-world applications.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
TR-DQ: Time-Rotation Diffusion Quantization
TR-DQ achieves state-of-the-art (SOTA) performance on image generation and video generation tasks and a 1. 38-1. 89x speedup and 1. 97-2. 58x memory reduction in inference compared to existing quantization methods.
UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
Generalist models have achieved remarkable success in both language and vision-language tasks, showcasing the potential of unified modeling.
Improved YOLOv7x-Based Defect Detection Algorithm for Power Equipment
The normal operation of power equipment plays a critical role in the power system, making anomaly detection for power equipment highly significant.
Parameter Efficient Merging for Multimodal Large Language Models with Complementary Parameter Adaptation
With the expansion in data and model size, parameter efficient tuning becomes the common practice for obtaining task-specific models efficiently.
Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization
Existing approaches, such as regularizing weight changes during fine-tuning, may fail to maintain sufficiently high feature similarity with the pre-trained model, and thus could possibly lose cross-task generalization.
FE-UNet: Frequency Domain Enhanced U-Net with Segment Anything Capability for Versatile Image Segmentation
To further emulate the human visual system, we introduce the Frequency Domain Enhanced Receptive Field Block (FE-RFB), which integrates WSPM to extract enriched features from the frequency domain.
RFMedSAM 2: Automatic Prompt Refinement for Enhanced Volumetric Medical Image Segmentation with SAM 2
Segment Anything Model 2 (SAM 2), a prompt-driven foundation model extending SAM to both image and video domains, has shown superior zero-shot performance compared to its predecessor.
Self-Prompt SAM: Medical Image Segmentation via Automatic Prompt SAM Adaptation
We design a multi-scale prompt generator combined with the image encoder in SAM to generate auxiliary masks.
In-Context Meta LoRA Generation
As a result, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE.
A Training-free Synthetic Data Selection Method for Semantic Segmentation
Then we propose a class-balance Annotation Similarity Filter (ASF) by comparing the synthetic annotation with the response of CLIP to remove the samples related to low-quality annotations.
UDiTQC: U-Net-Style Diffusion Transformer for Quantum Circuit Synthesis
Quantum computing is a transformative technology with wide-ranging applications, and efficient quantum circuit generation is crucial for unlocking its full potential.
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass
Multi-view 3D reconstruction remains a core challenge in computer vision, particularly in applications requiring accurate and scalable representations across diverse perspectives.
Enhanced Multi-Scale Cross-Attention for Person Image Generation
Moreover, we propose two novel cross-attention blocks to effectively transfer and update the person's shape and appearance embeddings for mutual improvement.
RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation
Although Low-Rank Adaptation (LoRA) is widely used and effective for fine-tuning, we have observed that its scaling factor can limit or even reduce performance as the rank size increases.
End-to-End Long Document Summarization using Gradient Caching
Training transformer-based encoder-decoder models for long document summarization poses a significant challenge due to the quadratic memory consumption during training.
Ranked #1 on
Long-Form Narrative Summarization
on SummScreen
Boosting Adversarial Transferability with Spatial Adversarial Alignment
To achieve high transferability, we propose a technique termed Spatial Adversarial Alignment (SAA), which employs an alignment loss and leverages a witness model to fine-tune the surrogate model.
HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models
We introduce HOIGPT, a token-based generative method that unifies 3D hand-object interactions (HOI) perception and generation, offering the first comprehensive solution for captioning and generating high-quality 3D HOI sequences from a diverse range of conditional signals (e. g. text, objects, partial sequences).
Artificial Intelligence for Central Dogma-Centric Multi-Omics: Challenges and Breakthroughs
With the rapid development of high-throughput sequencing platforms, an increasing number of omics technologies, such as genomics, metabolomics, and transcriptomics, are being applied to disease genetics research.
PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model
PolSAR data presents unique challenges due to its rich and complex characteristics.
Divide-and-Conquer: Confluent Triple-Flow Network for RGB-T Salient Object Detection
Furthermore, a Modality-aware Dynamic Aggregation Module in the modality-complementary flow dynamically aggregates saliency-related cues from both modality-specific flows.
Network Inversion and Its Applications
Neural networks have emerged as powerful tools across various applications, yet their decision-making process often remains opaque, leading to them being perceived as "black boxes."
Multimodal Alignment and Fusion: A Survey
This survey offers a comprehensive review of recent advancements in multimodal alignment and fusion within machine learning, spurred by the growing diversity of data types such as text, images, audio, and video.
Text-to-Image Synthesis: A Decade Survey
When humans read a specific text, they often visualize the corresponding images, and we hope that computers can do the same.
ConsistentAvatar: Learning to Diffuse Fully Consistent Talking Head Avatar with Temporal Guidance
In this paper, we propose ConsistentAvatar, a novel framework for fully consistent and high-fidelity talking avatar generation.
Hierarchical Cross-Attention Network for Virtual Try-On
In this paper, we present an innovative solution for the challenges of the virtual try-on task: our novel Hierarchical Cross-Attention Network (HCANet).
AllRestorer: All-in-One Transformer for Image Restoration under Composite Degradations
Specifically, we introduce an All-in-One Transformer Block (AiOTB), which adaptively removes all degradations present in a given image by modeling the relationships between all degradations and the image embedding in latent space.
DiffFNO: Diffusion Fourier Neural Operator
We introduce DiffFNO, a novel diffusion framework for arbitrary-scale super-resolution strengthened by a Weighted Fourier Neural Operator (WFNO).
KMM: Key Frame Mask Mamba for Extended Motion Generation
The recent Mamba architecture shows promising results in efficiently modeling long and complex sequences, yet two significant challenges remain: Firstly, directly applying Mamba to extended motion generation is ineffective, as the limited capacity of the implicit memory leads to memory decay.
Layer-Wise Feature Metric of Semantic-Pixel Matching for Few-Shot Learning
In Few-Shot Learning (FSL), traditional metric-based approaches often rely on global metrics to compute similarity.
Combining Induction and Transduction for Abstract Reasoning
When learning an input-output mapping from very few examples, is it better to first infer a latent function that explains the examples, or is it better to directly predict new test outputs, e. g. using a neural network?
GWQ: Gradient-Aware Weight Quantization for Large Language Models
Compared to current quantization methods, GWQ can be applied to multiple language models and achieves lower PPL on the WikiText2 and C4 dataset.
VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning
Broadly intelligent agents should form task-specific abstractions that selectively expose the essential elements of a task, while abstracting away the complexity of the raw sensorimotor space.
MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation
We also present MobBench, a dataset designed for complex mobile interactions.
Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention
Recent studies have attempted to incorporate camera control into the generation process, but their results are often limited to simple trajectories or lack the ability to generate consistent videos from multiple distinct camera paths for the same scene.
VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers
Recent advancements in test time compute, particularly through the use of verifier models, have significantly enhanced the reasoning capabilities of Large Language Models (LLMs).
OmniPose6D: Towards Short-Term Object Pose Tracking in Dynamic Scenes from Monocular RGB
To address the challenge of short-term object pose tracking in dynamic environments with monocular RGB input, we introduce a large-scale synthetic dataset OmniPose6D, crafted to mirror the diversity of real-world conditions.
Toward Zero-Shot Learning for Visual Dehazing of Urological Surgical Robots
In addition, we organize and propose a dehaze dataset for robotic vision in urological surgery (USRobot-Dehaze dataset).
M$^{2}$M: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations need
This universal controllable approach allows the model to achieve greater accuracy.
Brain Tumor Classification on MRI in Light of Molecular Markers
As a result, the diagnostic results are unreliable by the transfer learning model.
A Simple HMM with Self-Supervised Representations for Phone Segmentation
Despite the recent advance in self-supervised representations, unsupervised phonetic segmentation remains challenging.
Estimating the Completeness of Discrete Speech Units
We find that speaker information is sufficiently present in HuBERT discrete units, and that phonetic information is sufficiently present in the residual, showing that vector quantization does not achieve disentanglement.
Property Neurons in Self-Supervised Speech Transformers
It is, however, desirable to have an approach that can pinpoint exactly a subset of neurons that is responsible for a particular property of speech, being amenable to model pruning and model editing.
Data-Free Class Incremental Gesture Recognition via Synthetic Feature Sampling
Data-Free Class Incremental Learning (DFCIL) aims to enable models to continuously learn new classes while retraining knowledge of old classes, even when the training data for old classes is unavailable.
Barbie: Text to Barbie-Style 3D Avatars
In this paper, we propose Barbie, a novel framework for generating 3D avatars that can be dressed in diverse and high-quality Barbie-like garments and accessories.
ADen: Adaptive Density Representations for Sparse-view Camera Pose Estimation
Recent data-driven approaches aim to directly output camera poses, either through regressing the 6DoF camera poses or formulating rotation as a probability distribution.
Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
This work proposes Quasar-ViT, a hardware-oriented quantization-aware architecture search framework for ViTs, to design efficient ViT models for hardware implementation while preserving the accuracy.
Stable-Hair: Real-World Hair Transfer via Diffusion Model
In the second stage, we specifically designed a Hair Extractor and a Latent IdentityNet to transfer the target hairstyle with highly detailed and high-fidelity to the bald image.
InfiniMotion: Mamba Boosts Memory in Transformer for Arbitrary Long Motion Generation
Text-to-motion generation holds potential for film, gaming, and robotics, yet current methods often prioritize short motion generation, making it challenging to produce long motion sequences effectively: (1) Current methods struggle to handle long motion sequences as a single input due to prohibitively high computational cost; (2) Breaking down the generation of long motion sequences into shorter segments can result in inconsistent transitions and requires interpolation or inpainting, which lacks entire sequence modeling.
3D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidance
In this paper, we propose 3DSS-VLG, a weakly supervised approach for 3D Semantic Segmentation with 2D Vision-Language Guidance, an alternative approach that a 3D model predicts dense-embedding for each point which is co-embedded with both the aligned image and text spaces from the 2D vision-language model.
3x2: 3D Object Part Segmentation by 2D Semantic Correspondences
While substantial progress has been made in 2D object part segmentation, the 3D counterpart has received less attention, in part due to the scarcity of annotated 3D datasets, which are expensive to collect.
GMC: A General Framework of Multi-stage Context Learning and Utilization for Visual Detection Tasks
In this paper, GMC, a general framework is proposed for multistage context learning and utilization, with various deep network architectures for various visual detection tasks.
ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling
Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims to minimize the distance between sketches and corresponding images in the embedding space.
Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations
Self-supervised speech representations can hugely benefit downstream speech technologies, yet the properties that make them useful are still poorly understood.
DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models
We introduce Data Adaptive Self-Supervised Early Exit (DAISY), an approach that decides when to exit based on the self-supervised loss, eliminating the need for multiple round of training and fine-tuning.
From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks
Large Vision Language Models (LVLMs) achieve great performance on visual-language reasoning tasks, however, the black-box nature of LVLMs hinders in-depth research on the reasoning mechanism.
Dataset Growth
To tackle this challenge, we propose InfoGrowth, an efficient online algorithm for data cleaning and selection, resulting in a growing dataset that keeps up to date with awareness of cleanliness and diversity.
Code Repair with LLMs gives an Exploration-Exploitation Tradeoff
Iteratively improving and repairing source code with large language models (LLMs), known as refinement, has emerged as a popular way of generating programs that would be too complex to construct in one shot.
A predictive learning model can simulate temporal dynamics and context effects found in neural representations of continuous speech
Speech perception involves storing and integrating sequentially presented items.
Multi-task learning for molecular electronic structure approaching coupled-cluster accuracy
Machine learning (ML) plays an important role in quantum chemistry, providing fast-to-evaluate predictive models for various properties of molecules.
DVF: Advancing Robust and Accurate Fine-Grained Image Retrieval with Retrieval Guidelines
This paper presents a meticulous analysis leading to the proposal of practical guidelines to identify subcategory-specific discrepancies and generate discriminative features to design effective FGIR models.
AccidentBlip: Agent of Accident Warning based on MA-former
In particular, AccidentBlip achieves SOTA performance in both accident detection and prediction tasks on the DeepAccident dataset.
A Survey on Multimodal Wearable Sensor-based Human Action Recognition
We begin by presenting the recent sensor modalities as well as deep learning approaches in HAR.
StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion
3) The story visualization and continuation models are trained and inferred independently, which is not user-friendly.
HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud
Extracting keypoint locations from input hand frames, known as 3D hand pose estimation, is a critical task in various human-computer interaction applications.
Towards Robust 3D Pose Transfer with Adversarial Learning
This work is driven by the intuition that the robustness of the model can be enhanced by introducing adversarial samples into the training, leading to a more invulnerable model to the noisy inputs, which even can be further extended to directly handling the real-world data like raw point clouds/scans without intermediate processing.
Versatile Navigation under Partial Observability via Value-guided Diffusion Policy
To address these deficiencies, we propose a versatile diffusion-based approach for both 2D and 3D route planning under partial observability.
On the Faithfulness of Vision Transformer Explanations
In contrast, our proposed SaCo offers a reliable faithfulness measurement, establishing a robust metric for interpretations.
Learning with Unreliability: Fast Few-shot Voxel Radiance Fields with Relative Geometric Consistency
We propose a voxel-based optimization framework, ReVoRF, for few-shot radiance fields that strategically address the unreliability in pseudo novel view synthesis.
Rotate to Scan: UNet-like Mamba with Triplet SSM Module for Medical Image Segmentation
Image segmentation holds a vital position in the realms of diagnosis and treatment within the medical domain.
Towards Online Real-Time Memory-based Video Inpainting Transformers
Video inpainting tasks have seen significant improvements in recent years with the rise of deep neural networks and, in particular, vision transformers.
Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer
To incorporate the influence of token transformation into interpretation, we propose TokenTM, a novel post-hoc explanation method that utilizes our introduced measurement of token transformation effects.
MaskSAM: Towards Auto-prompt SAM with Mask Classification for Medical Image Segmentation
Each pair of auxiliary mask and box prompts, which can solve the requirements of extra prompts, is associated with class label predictions by the sum of the auxiliary classifier token and the learnable global classifier tokens in the mask decoder of SAM to solve the predictions of semantic labels.
StableGarment: Garment-Centric Generation via Stable Diffusion
In this paper, we introduce StableGarment, a unified framework to tackle garment-centric(GC) generation tasks, including GC text-to-image, controllable GC text-to-image, stylized GC text-to-image, and robust virtual try-on.
Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment
For example, HyWIA surpasses the cutting-edge LLM-Pruner by an average margin of 2. 82% in accuracy across seven downstream tasks when pruning LLaMA-7B by 50%.
SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesis
Semantic image synthesis (SIS) shows good promises for sensor simulation.
GiT: Towards Generalist Vision Transformer through Universal Language Interface
Due to its simple design, this paradigm holds promise for narrowing the architectural gap between vision and language.
Ranked #2 on
Video Captioning
on MSVD-CTN
(using extra training data)
Motion Mamba: Efficient and Long Sequence Motion Generation
Human motion generation stands as a significant pursuit in generative computer vision, while achieving long-sequence and efficient motion generation remains challenging.
Ranked #12 on
Motion Synthesis
on KIT Motion-Language
InstructGIE: Towards Generalizable Image Editing
In response to this challenge, our study introduces a novel image editing framework with enhanced generalization robustness by boosting in-context learning capability and unifying language instruction.
Hierarchical Indexing for Retrieval-Augmented Opinion Summarization
We show that HIRO learns an encoding space that is more semantically structured than prior work, and generates summaries that are more representative of the opinions in the input reviews.
WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment
We give a model-based agent that builds a Python program representing its knowledge of the world based on its interactions with the environment.
Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challenge
The resulting FracNet+ demonstrates competitive performance in rib fracture detection, which lays a foundation for further research and development in AI-assisted rib fracture detection and diagnosis.
ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization
Neural Radiance Fields (NeRF) exhibit remarkable performance for Novel View Synthesis (NVS) given a set of 2D images.
Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generation
Finally, we propose a novel self-guided pre-training method for graph representation learning.
Generative Adversarial Network
Graph Representation Learning
+2
SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation
Recent advancements in subject-driven image generation have led to zero-shot generation, yet precise selection and focus on crucial subject representations remain challenging.
Enlighten-Your-Voice: When Multimodal Meets Zero-shot Low-light Image Enhancement
Low-light image enhancement is a crucial visual task, and many unsupervised methods tend to overlook the degradation of visible information in low-light scenes, which adversely affects the fusion of complementary information and hinders the generation of satisfactory results.
Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Models
In this work, we propose that by incorporating correspondence regularization into diffusion models, the process of 3D editing can be significantly accelerated.
Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samples
In the field of intelligent multimedia analysis, ultra-fine-grained visual categorization (Ultra-FGVC) plays a vital role in distinguishing intricate subcategories within broader categories.
Multi-view Information Integration and Propagation for Occluded Person Re-identification
Moreover, as multiple images with the same identity are not accessible in the testing stage, we devise an Information Propagation (IP) mechanism to distill knowledge from the comprehensive representation to that of a single occluded image.
Towards High-quality HDR Deghosting with Conditional Diffusion Models
To address this challenge, we formulate the HDR deghosting problem as an image generation that leverages LDR features as the diffusion model's condition, consisting of the feature condition generator and the noise predictor.
Towards Matching Phones and Speech Representations
We study two key properties that enable matching, namely, whether cluster centroids of self-supervised representations reduce the variability of phone instances and respect the relationship among phones.
Blind quantum machine learning with quantum bipartite correlator
Privacy in distributed quantum computing is critical for maintaining confidentiality and protecting the data in the presence of untrusted computing nodes.
Pedestrian Accessible Infrastructure Inventory: Assessing Zero-Shot Segmentation on Multi-Mode Geospatial Data for All Pedestrian Types
In this paper, a Segment Anything Model (SAM)-based pedestrian infrastructure segmentation workflow is designed and optimized, which is capable of efficiently processing multi-sourced geospatial data including LiDAR data and satellite imagery data.
Does Graph Distillation See Like Vision Dataset Counterpart?
We validate the proposed SGDD across 9 datasets and achieve state-of-the-art results on all of them: for example, on the YelpChi dataset, our approach maintains 98. 6% test accuracy of training on the original graph dataset with 1, 000 times saving on the scale of the graph.
Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day
Although the fidelity and generalizability are greatly improved, training such a powerful diffusion model requires a vast volume of training data and model parameters, resulting in a notoriously long time and high computational costs.
Distilling ODE Solvers of Diffusion Models into Smaller Steps
Notably, our method incurs negligible computational overhead compared to previous distillation techniques, facilitating straightforward and rapid integration with existing samplers.
Light Field Diffusion for Single-View Novel View Synthesis
Starting from the camera pose matrices, LFD transforms them into light field encoding, with the same shape as the reference image, to describe the direction of each ray.
Delving into Multimodal Prompting for Fine-grained Visual Classification
In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model.
Ranked #13 on
Fine-Grained Image Classification
on NABirds
Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation
Modern approaches have proved the huge potential of addressing semantic segmentation as a mask classification task which is widely used in instance-level segmentation.
Adapting Segment Anything Model for Change Detection in HR Remote Sensing Images
In this work, we aim to utilize the strong visual recognition capabilities of VFMs to improve the change detection of high-resolution Remote Sensing Images (RSIs).
UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representation
Jointly processing information from multiple sensors is crucial to achieving accurate and robust perception for reliable autonomous driving systems.
Ranked #8 on
3D Object Detection
on nuScenes
M$^3$Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition
Due to the scarcity of manually annotated data required for fine-grained video understanding, few-shot fine-grained (FS-FG) action recognition has gained significant attention, with the aim of classifying novel fine-grained action categories with only a few labeled instances.
Interactive Neural Painting
This paper advances the state of the art in this emerging research domain by proposing the first approach for Interactive NP.
Hybrid-CSR: Coupling Explicit and Implicit Shape Representation for Cortical Surface Reconstruction
We present Hybrid-CSR, a geometric deep-learning model that combines explicit and implicit shape representations for cortical surface reconstruction.
Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis
To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information.
Erasing, Transforming, and Noising Defense Network for Occluded Person Re-Identification
Occlusion perturbation presents a significant challenge in person re-identification (re-ID), and existing methods that rely on external visual cues require additional computational resources and only consider the issue of missing information caused by occlusion.
Inter-Instance Similarity Modeling for Contrastive Learning
In this paper, we propose a novel image mix method, PatchMix, for contrastive learning in Vision Transformer (ViT), to model inter-instance similarities among images.
Enlighten Anything: When Segment Anything Model Meets Low-Light Image Enhancement
Image restoration is a low-level visual task, and most CNN methods are designed as black boxes, lacking transparency and intrinsic aesthetics.
Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling
Acoustic word embeddings are typically created by training a pooling function using pairs of word-like units.
Edge-guided Representation Learning for Underwater Object Detection
The key to addressing these challenges is to focus the model on obtaining more discriminative information.
Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generation
This means that we approach this problem by considering the generation of a distribution of the listener's appropriate facial reactions instead of multiple different appropriate facial reactions, i. e., 'many' appropriate facial reaction labels are summarised as 'one' distribution label during training.
Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces
Self-supervised speech representations are known to encode both speaker and phonetic information, but how they are distributed in the high-dimensional space remains largely unexplored.
Attributable and Scalable Opinion Summarization
We propose a method for unsupervised opinion summarization that encodes sentences from customer reviews into a hierarchical discrete latent space, then identifies common opinions based on the frequency of their encodings.
Master: Meta Style Transformer for Controllable Zero-Shot and Few-Shot Artistic Style Transfer
On the other hand, different from the vanilla version, we adopt a learnable scaling operation on content features before content-style feature interaction, which better preserves the original similarity between a pair of content features while ensuring the stylization quality.
SMAE: Few-shot Learning for HDR Deghosting with Saturation-Aware Masked Autoencoders
In this work, we propose a novel semi-supervised approach to realize few-shot HDR imaging via two stages of training, called SSHDR.
Localized Region Contrast for Enhancing Self-Supervised Learning in Medical Image Segmentation
In this paper, we propose a novel contrastive learning framework that integrates Localized Region Contrast (LRC) to enhance existing self-supervised pre-training methods for medical image segmentation.
Unsupervised Deep Probabilistic Approach for Partial Point Cloud Registration
Deep point cloud registration methods face challenges to partial overlaps and rely on labeled data.
Graph Transformer GANs for Graph-Constrained House Generation
We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations in an end-to-end fashion for the challenging graph-constrained house generation task.
Analysis and Evaluation of Explainable Artificial Intelligence on Suicide Risk Assessment
This study investigates the effectiveness of Explainable Artificial Intelligence (XAI) techniques in predicting suicide risks and identifying the dominant causes for such behaviours.
DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network
To this end, a novel framework termed Mathematical Architecture Design for Deep CNN (DeepMAD) is proposed to design high-performance CNN models in a principled way.
Ranked #1 on
Neural Architecture Search
on ImageNet
Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased
A crucial limitation of this framework is that every policy in the pool is optimized w. r. t.
GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis
The complex scene understanding ability of CLIP enables the discriminator to accurately assess the image quality.
Ranked #4 on
Text-to-Image Generation
on CUB
Bipartite Graph Diffusion Model for Human Interaction Generation
The generation of natural human motion interactions is a hot topic in computer vision and computer animation.
Learning Concordant Attention via Target-aware Alignment for Visible-Infrared Person Re-identification
Owing to the large distribution gap between the heterogeneous data in Visible-Infrared Person Re-identification (VI Re-ID), we point out that existing paradigms often suffer from the inter-modal semantic misalignment issue and thus fail to align and compare local details properly.
Pruning Parameterization With Bi-Level Optimization for Efficient Semantic Segmentation on the Edge
With the ever-increasing popularity of edge devices, it is necessary to implement real-time segmentation on the edge for autonomous driving and many other applications.
HOTCOLD Block: Fooling Thermal Infrared Detectors with a Novel Wearable Design
Adversarial attacks on thermal infrared imaging expose the risk of related applications.
Few-shot Medical Image Segmentation with Cycle-resemblance Attention
Recently, due to the increasing requirements of medical imaging applications and the professional requirements of annotating medical images, few-shot learning has gained increasing attention in the medical image semantic segmentation field.
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization.
Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers
Transformer-based self-supervised models have achieved remarkable success in speech processing, but their large size and high inference cost present significant challenges for real-world deployment.
MelHuBERT: A simplified HuBERT on Mel spectrograms
Self-supervised models have had great success in learning speech representations that can generalize to various downstream tasks.
Deep Unsupervised Key Frame Extraction for Efficient Video Classification
The proposed TSDPC is a generic and powerful framework and it has two advantages compared with previous works, one is that it can calculate the number of key frames automatically.
Bipartite Graph Reasoning GANs for Person Pose and Facial Image Synthesis
To further capture the change in pose of each part more precisely, we propose a novel part-aware bipartite graph reasoning (PBGR) block to decompose the task of reasoning the global structure transformation with a bipartite graph into learning different local transformations for different semantic body/face parts.
Data Level Lottery Ticket Hypothesis for Vision Transformers
That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches.
Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models
While discrete latent variable models have had great success in self-supervised learning, most models assume that frames are independent.
Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models
Given the strong results of self-supervised models on various tasks, there have been surprisingly few studies exploring self-supervised representations for acoustic word embeddings (AWE), fixed-dimensional vectors representing variable-length spoken word segments.
Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution
Moreover, by coupling the proposed sampling method with an unconditional DM, i. e., a DM with no auxiliary inputs to its noise predictor, we can generalize it to a wide range of SR setups.
ADPS: Asymmetric Distillation Post-Segmentation for Image Anomaly Detection
However, existing KDAD methods suffer from two main limitations: 1) the student network can effortlessly replicate the teacher network's representations, and 2) the features of the teacher network serve solely as a ``reference standard" and are not fully leveraged.
On Compressing Sequences for Self-Supervised Speech Models
Subsampling while training self-supervised models not only improves the overall performance on downstream tasks under certain frame rates, but also brings significant speed-up in inference.
SiNeRF: Sinusoidal Neural Radiance Fields for Joint Pose Estimation and Scene Reconstruction
NeRFmm is the Neural Radiance Fields (NeRF) that deal with Joint Optimization tasks, i. e., reconstructing real-world scenes and registering camera parameters simultaneously.
Boosting Few-shot Fine-grained Recognition with Background Suppression and Foreground Alignment
To address this challenging task, we propose a two-stage background suppression and foreground alignment framework, which is composed of a background activation suppression (BAS) module, a foreground object alignment (FOA) module, and a local-to-local (L2L) similarity metric.
Physical Adversarial Attack meets Computer Vision: A Decade Survey
Building upon this foundation, we uncover the pervasive role of artifacts carrying adversarial perturbations in the physical world.
PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation
In this paper, we propose the token-Pruned Pose Transformer (PPT) for 2D human pose estimation, which can locate a rough human mask and performs self-attention only within selected tokens.
Ranked #20 on
3D Human Pose Estimation
on Human3.6M
(using extra training data)
Facial Expression Translation using Landmark Guided GANs
We propose a simple yet powerful Landmark guided Generative Adversarial Network (LandmarkGAN) for the facial expression-to-expression translation using a single image, which is an important and challenging task in computer vision since the expression-to-expression translation is a non-linear and non-aligned problem.
Facial Expression Translation
Generative Adversarial Network
+1
Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Search
Moreover, existing methods seldom consider the information inequality problem between modalities caused by image-specific information.
Training and Tuning Generative Neural Radiance Fields for Attribute-Conditional 3D-Aware Face Generation
Generative Neural Radiance Fields (GNeRF)-based 3D-aware GANs have showcased remarkable prowess in crafting high-fidelity images while upholding robust 3D consistency, particularly face generation.
Identity-Sensitive Knowledge Propagation for Cloth-Changing Person Re-identification
To mitigate the resolution degradation issue and mine identity-sensitive cues from human faces, we propose to restore the missing facial details using prior facial knowledge, which is then propagated to a smaller network.
G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model
Our approach focuses on the transformation of sign gloss sequences into their corresponding sign pose sequences (G2P).
Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0. 47% to 1. 36% higher Top-1 accuracy under the same bit-width.
Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution
Instead of measuring the speed on mobile devices at each iteration during the search process, a speed model incorporated with compiler optimizations is leveraged to predict the inference latency of the SR block with various width configurations for faster convergence.
Towards Interpretable Video Super-Resolution via Alternating Optimization
These issues can be alleviated by a cascade of three separate sub-tasks, including video deblurring, frame interpolation, and super-resolution, which, however, would fail to capture the spatial and temporal correlations among video sequences.
Ranked #7 on
Video Super-Resolution
on REDS4- 4x upscaling
Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation
The essence of video semantic segmentation (VSS) is how to leverage temporal information for prediction.
Ranked #4 on
Video Semantic Segmentation
on VSPW
MLP-GAN for Brain Vessel Image Segmentation
Brain vessel image segmentation can be used as a promising biomarker for better prevention and treatment of different diseases.
RCRN: Real-world Character Image Restoration Network via Skeleton Extraction
SENet aims to preserve the structural consistency of the character and normalize complex noise.
CharFormer: A Glyph Fusion based Attentive Framework for High-precision Character Image Denoising
Degraded images commonly exist in the general sources of character images, leading to unsatisfactory character recognition results.
RZCR: Zero-shot Character Recognition via Radical-based Reasoning
The long-tail effect is a common issue that limits the performance of deep learning models on real-world datasets.
PI-Trans: Parallel-ConvMLP and Implicit-Transformation Based GAN for Cross-View Image Translation
For semantic-guided cross-view image translation, it is crucial to learn where to sample pixels from the source view image and where to reallocate them guided by the target view semantic map, especially when there is little overlap or drastic view difference between the source and target images.
Contrastive Learning from Spatio-Temporal Mixed Skeleton Sequences for Self-Supervised Skeleton-Based Action Recognition
First, SkeleMix utilizes the topological information of skeleton data to mix two skeleton sequences by randomly combing the cropped skeleton fragments (the trimmed view) with the remaining skeleton sequences (the truncated view).
Interaction Transformer for Human Reaction Generation
To address this limitation, we propose a novel interaction Transformer (InterFormer) consisting of a Transformer network with both temporal and spatial attention.
Unsupervised High-Resolution Portrait Gaze Correction and Animation
Solving this problem using an unsupervised method remains an open problem, especially for high-resolution face images in the wild, which are not easy to annotate with gaze and head pose labels.
3D-Aware Video Generation
Generative models have emerged as an essential building block for many image synthesis and editing tasks.
GraphMLP: A Graph MLP-Like Architecture for 3D Human Pose Estimation
To the best of our knowledge, this is the first MLP-Like architecture for 3D human pose estimation in a single frame and a video sequence.
Ranked #67 on
3D Human Pose Estimation
on Human3.6M
From Perception to Programs: Regularize, Overparameterize, and Amortize
Toward combining inductive reasoning with perception abilities, we develop techniques for neurosymbolic program synthesis where perceptual input is first parsed by neural nets into a low-dimensional interpretable representation, which is then processed by a synthesized program.
Medical Image Registration via Neural Fields
Traditional methods for image registration are primarily optimization-driven, finding the optimal deformations that maximize the similarity between two images.
Real-Time Portrait Stylization on the Edge
In this work we demonstrate real-time portrait stylization, specifically, translating self-portrait into cartoon or anime style on mobile devices.
DE-Net: Dynamic Text-guided Image Editing Adversarial Networks
To solve these limitations, we propose: (i) a Dynamic Editing Block (DEBlock) which composes different editing modules dynamically for various editing requirements.
AO2-DETR: Arbitrary-Oriented Object Detection Transformer
Comprehensive experiments on several challenging datasets show that our method achieves superior performance on the AOOD task.
Supervised Attention in Sequence-to-Sequence Models for Speech Recognition
Attention mechanism in sequence-to-sequence models is designed to model the alignments between acoustic features and output tokens in speech recognition.
Deep Algebraic Fitting for Multiple Circle Primitives Extraction from Raw Point Clouds
The shape of circle is one of fundamental geometric primitives of man-made engineering objects.
Autoregressive Co-Training for Learning Discrete Speech Representations
While several self-supervised approaches for learning discrete speech representation have been proposed, it is unclear how these seemingly similar approaches relate to each other.
