Search Results for author:
Found 101 papers, 45 papers with code
Where and when to look? Spatial-temporal attention for action recognition in videos
Our model is efficient, as it proposes a separable spatio-temporal mechanism for video attention, while being able to identify important parts of the video both spatially and temporally.
Action Recognition In Videos
Temporal Action Localization
+1
CardiacMamba: A Multimodal RGB-RF Fusion Framework with State Space Models for Remote Physiological Measurement
However, traditional single-modality approaches (RGB or Radio Frequency (RF)) face challenges in balancing robustness and accuracy due to lighting variations, motion artifacts, and skin tone bias.
Enhancing Generalization via Sharpness-Aware Trajectory Matching for Dataset Condensation
These issues deteriorate when the datasets are learned via matching the trajectories of networks trained on the real and synthetic datasets with a long horizon inner-loop.
Normalizing Batch Normalization for Long-Tailed Recognition
Differently, in this paper, we identify that the bias towards the frequent class may be encoded into features, i. e., the rare-specific features which play a key role in discriminating the rare class are much weaker than the frequent-specific features.
MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
Despite the rapidly growing demand for multimodal retrieval, progress in this field remains severely constrained by a lack of training data.
Ranked #1 on
Zero-Shot Composed Image Retrieval (ZS-CIR)
on CIRCO
Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?
We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain.
Emu3: Next-Token Prediction is All You Need
While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e. g., Stable Diffusion) and compositional approaches (e. g., CLIP combined with LLMs).
Ranked #134 on
Visual Question Answering
on MM-Vet
Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding
The VST module is trained by instruction fine-tuning, where two optimizing strategies are offered.
Automated design of nonreciprocal thermal emitters via Bayesian optimization
Nonreciprocal thermal emitters that break Kirchhoff's law of thermal radiation promise exciting applications for thermal and energy applications.
Enhancing Long Video Understanding via Hierarchical Event-Based Memory
While this method excels in short video understanding, it may result in a blend of multiple event information in long videos due to coarse compression, which causes information redundancy.
TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations
Multimodal Large Language Models (MLLMs) have significantly improved performance across various image-language applications.
52B to 1T: Lessons Learned via Tele-FLM Series
Large Language Models (LLMs) represent a significant stride toward Artificial General Intelligence.
PVUW 2024 Challenge on Complex Video Understanding: Methods and Results
Moreover, we provide a new motion expression guided video segmentation dataset MeViS to study the natural language-guided video understanding in complex environments.
2nd Place Solution for MeViS Track in CVPR 2024 PVUW Workshop: Motion Expression guided Video Segmentation
Motion Expression guided Video Segmentation is a challenging task that aims at segmenting objects in the video based on natural language expressions with motion descriptions.
Instance Segmentation
Referring Video Object Segmentation
+5
SpatialBot: Precise Spatial Understanding with Vision Language Models
Vision Language Models (VLMs) have achieved impressive performance in 2D image understanding, however they are still struggling with spatial understanding which is the foundation of Embodied AI.
Ranked #4 on
Spatial Reasoning
on 6-DoF SpatialBench
Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
To this end, we manually construct a benchmark with 12 categories and design evaluation metrics that assess the degree of error in MLLM responses even when the visual content is seemingly understood.
VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data.
Ranked #7 on
Image Retrieval
on CIRR
Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking
6D Object Pose Estimation is a crucial yet challenging task in computer vision, suffering from a significant lack of large-scale datasets.
MLVU: Benchmarking Multi-task Long Video Understanding
To address the above problems, we propose a new benchmark called MLVU (Multi-task Long Video Understanding Benchmark) for the comprehensive and in-depth evaluation of LVU.
The SkatingVerse Workshop & Challenge: Methods and Results
The SkatingVerse Workshop & Challenge aims to encourage research in developing novel and accurate methods for human action understanding.
Efficient Multimodal Large Language Models: A Survey
In the past year, Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning.
Large Language Model-aided Edge Learning in Distribution System State Estimation
Distribution system state estimation (DSSE) plays a crucial role in the real-time monitoring, control, and operation of distribution networks.
Understanding the Difficulty of Solving Cauchy Problems with PINNs
Physics-Informed Neural Networks (PINNs) have gained popularity in scientific computing in recent years.
FlexiFilm: Long Video Generation with Flexible Conditions
Generating long and consistent videos has emerged as a significant yet challenging problem.
Tele-FLM Technical Report
Large language models (LLMs) have showcased profound capabilities in language understanding and generation, facilitating a wide array of applications.
Advances and Open Challenges in Federated Foundation Models
This paper provides a comprehensive survey of the emerging field of Federated Foundation Models (FedFM), elucidating their synergistic relationship and exploring novel methodologies, challenges, and future directions that the FL research field needs to focus on in order to thrive in the age of FMs.
M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models
Additionally, we propose M3D-LaMed, a versatile multi-modal large language model for 3D medical image analysis.
SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model
To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models.
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability
In this paper, we extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously.
Efficient Multimodal Learning from Data-centric Perspective
Multimodal Large Language Models (MLLMs) have demonstrated notable capabilities in general visual understanding and reasoning tasks.
RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model
Recent advancements in Multi-Modal Large Language models (MLLMs) have shown promising potential in enhancing the explainability as a driving agent by producing control predictions along with natural language explanations.
Spin: An Efficient Secure Computation Framework with GPU Acceleration
Accuracy and efficiency remain challenges for multi-party computation (MPC) frameworks.
Distributional Counterfactual Explanations With Optimal Transport
Counterfactual explanations (CE) are the de facto method for providing insights into black-box decision-making models by identifying alternative inputs that lead to different outcomes.
Learning Position-Aware Implicit Neural Network for Real-World Face Inpainting
Face inpainting requires the model to have a precise global understanding of the facial position structure.
Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections
After a GPU change, Scalai uses the PTC to transform the job state: the PTC repartitions the dataset state under data parallelism and exposes it to DL workers through a virtual file system; and the PTC obtains the model state as partitioned checkpoints and transforms them to reflect the new parallelization configuration.
Open-DDVM: A Reproduction and Extension of Diffusion Model for Optical Flow Estimation
Recently, Google proposes DDVM which for the first time demonstrates that a general diffusion model for image-to-image translation task works impressively well on optical flow estimation task without any specific designs like RAFT.
SegVol: Universal and Interactive Volumetric Medical Image Segmentation
Precise image segmentation provides clinical study with instructive information.
A global product of fine-scale urban building height based on spaceborne lidar
Our results revealed that the estimated method of building height samples based on the GEDI data was effective with 0. 78 of Pearson's r and 3. 67 m of RMSE in comparison to the reference data.
Real-Fake: Effective Training Data Synthesis Through Distribution Matching
Synthetic training data has gained prominence in numerous learning tasks and scenarios, offering advantages such as dataset augmentation, generalization evaluation, and privacy preservation.
Bayesian Cramér-Rao Bound Estimation with Score-Based Models
The Bayesian Cram\'er-Rao bound (CRB) provides a lower bound on the mean square error of any Bayesian estimator under mild regularity conditions.
Image Captions are Natural Prompts for Text-to-Image Models
With the rapid development of Artificial Intelligence Generated Content (AIGC), it has become common practice in many learning tasks to train or fine-tune large models on synthetic data due to the data-scarcity and privacy leakage problems.
SVIT: Scaling up Visual Instruction Tuning
Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc.
Federated Generative Learning with Foundation Models
Thanks to the emerging foundation generative models, we propose a novel federated learning framework, namely Federated Generative Learning.
Pushing the Limits of 3D Shape Generation at Scale
Through the adaptation of the Auto-Regressive model and the utilization of large language models, we have developed a remarkable model with an astounding 3. 6 billion trainable parameters, establishing it as the largest 3D shape generation model to date, named Argus-3D.
Large-scale Dataset Pruning with Dynamic Uncertainty
The state of the art of many learning tasks, e. g., image classification, is advanced by collecting larger datasets and then training larger models on them.
DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting
While diffusion models can successfully generate data and make predictions, they are predominantly designed for static images.
Improving Convergence and Generalization Using Parameter Symmetries
Finally, we show that integrating teleportation into a wide range of optimization algorithms and optimization-based meta-learning improves convergence.
Accelerated MR Fingerprinting with Low-Rank and Generative Subspace Modeling
To address this problem, we present a new image reconstruction method for MR Fingerprinting, integrating low-rank and subspace modeling with a deep generative prior.
Toward Moiré-Free and Detail-Preserving Demosaicking
In both applications, our model substantially alleviates artifacts such as Moir\'e and over-smoothness at similar or lower computational cost to currently top-performing models, as validated by diverse evaluations.
Accelerating Dataset Distillation via Model Augmentation
Dataset Distillation (DD), a newly emerging field, aims at generating much smaller but efficient synthetic training datasets from large ones.
The Impact of Weather on Local Government Spending
While there is a new and rapidly growing literature on the effects of climatic factors on economic and social outcomes, little research has been conducted to understand the fiscal impact of weather, especially at the sub-state level.
CR-LSO: Convex Neural Architecture Optimization in the Latent Space of Graph Variational Autoencoder with Input Convex Neural Networks
In neural architecture search (NAS) methods based on latent space optimization (LSO), a deep generative model is trained to embed discrete neural architectures into a continuous latent space.
Symmetries, flat minima, and the conserved quantities of gradient flow
Empirical studies of the loss landscape of deep networks have revealed that many local minima are connected through low-loss valleys.
Nowhere to Hide: A Lightweight Unsupervised Detector against Adversarial Examples
In this paper, we propose an AutoEncoder-based Adversarial Examples (AEAE) detector, that can guard DNN models by detecting adversarial examples with low computation in an unsupervised manner.
MSRL: Distributed Reinforcement Learning with Dataflow Fragments
Yet, current distributed RL systems tie the definition of RL algorithms to their distributed execution: they hard-code particular distribution strategies and only accelerate specific parts of the computation (e. g. policy network updates) on GPU workers.
Everyone's Preference Changes Differently: Weighted Multi-Interest Retrieval Model
In this paper, we propose the Multi-Interest Preference (MIP) model, an approach that not only produces multi-interest for users by using the user's sequential engagement more effectively but also automatically learns a set of weights to represent the preference over each embedding so that the candidates can be retrieved from each interest proportionally.
LIMO: Latent Inceptionism for Targeted Molecule Generation
We corroborate these docking-based results with more accurate molecular dynamics-based calculations of absolute binding free energy and show that one of our generated drug-like compounds has a predicted $K_D$ (a measure of binding affinity) of $6 \cdot 10^{-14}$ M against the human estrogen receptor, well beyond the affinities of typical early-stage drug candidates and most FDA-approved drugs to their respective targets.
Privacy for Free: How does Dataset Condensation Help Privacy?
In this work, we for the first time identify that dataset condensation (DC) which is originally designed for improving training efficiency is also a better solution to replace the traditional data generators for private data generation, thus providing privacy for free.
FaceMAE: Privacy-Preserving Face Recognition via Masked Autoencoders
Firstly, randomly masked face images are used to train the reconstruction module in FaceMAE.
Symmetry Teleportation for Accelerated Optimization
Experimentally, we show that teleportation improves the convergence speed of gradient descent and AdaGrad for several optimization problems including test functions, multi-layer regressions, and MNIST classification.
DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
Coreset selection, which aims to select a subset of the most informative training samples, is a long-standing learning problem that can benefit many downstream tasks such as data-efficient learning, continual learning, neural architecture search, active learning, etc.
Synthesizing Informative Training Samples with GAN
However, traditional GANs generated images are not as informative as the real training samples when being used to train deep neural networks.
CAFE: Learning to Condense Dataset by Aligning Features
Dataset condensation aims at reducing the network training effort through condensing a cumbersome training set into a compact synthetic one.
Towards Understanding and Harnessing the Effect of Image Transformation in Adversarial Detection
Experimental results show that the contribution of image transformations to adversarial detection is significantly different, the combination of them can significantly improve the generic detection ability against state-of-the-art adversarial attacks.
Makeup216: Logo Recognition with Adversarial Attention Representations
One of the challenges of logo recognition lies in the diversity of forms, such as symbols, texts or a combination of both; further, logos tend to be extremely concise in design while similar in appearance, suggesting the difficulty of learning discriminative representations.
One-dimensional Deep Low-rank and Sparse Network for Accelerated MRI
Deep learning has shown astonishing performance in accelerated magnetic resonance imaging (MRI).
Dataset Condensation with Distribution Matching
Computational cost of training state-of-the-art deep models in many learning problems is rapidly increasing due to more sophisticated models and larger datasets.
Ranked #8 on
Dataset Distillation - 1IPC
on TinyImageNet
Accelerated MRI Reconstruction with Separable and Enhanced Low-Rank Hankel Regularization
The combination of the sparse sampling and the low-rank structured matrix reconstruction has shown promising performance, enabling a significant reduction of the magnetic resonance imaging data acquisition time.
Feature-Filter: Detecting Adversarial Examples through Filtering off Recessive Features
In this paper, we reveal that imperceptible adversarial examples are the product of recessive features misleading neural networks, and an adversarial attack is essentially a kind of method to enrich these recessive features in the image.
FedCom: A Byzantine-Robust Local Model Aggregation Rule Using Data Commitment for Federated Learning
The results demonstrate its effectiveness and superior performance compared to the state-of-the-art Byzantine-robust schemes in defending against typical data poisoning and model poisoning attacks under practical Non-IID data distributions.
Concentric Spherical GNN for 3D Representation Learning
Learning 3D representations that generalize well to arbitrarily oriented inputs is a challenge of practical importance in applications varying from computer vision to physics and chemistry.
Interpretable Attention Guided Network for Fine-grained Visual Classification
We propose an Interpretable Attention Guided Network (IAGN) for fine-grained visual classification.
Dataset Condensation with Differentiable Siamese Augmentation
In many machine learning problems, large-scale datasets have become the de-facto standard to train state-of-the-art deep networks at the price of heavy computation load.
Ranked #5 on
Dataset Distillation - 1IPC
on CUB-200-2011
Quantum Adiabatic Doping for Atomic Fermi-Hubbard Quantum Simulations
Its feasibility has been demonstrated with numerical simulations of the adiabatic preparation for certain incommensurate particle-doping fractions, where the major problem to circumvent is the atomic localization in the incommensurate lattice.
Quantum Gases Strongly Correlated Electrons Quantum Physics
Multiple Plans are Better than One: Diverse Stochastic Planning
In planning problems, it is often challenging to fully model the desired specifications.
Active Sampling for Accelerated MRI with Low-Rank Tensors
In this paper, we introduce an active low-rank tensor model for fast MR imaging.
GreedyFool: Multi-Factor Imperceptibility and Its Application to Designing a Black-box Adversarial Attack
Adversarial examples are well-designed input samples, in which perturbations are imperceptible to the human eyes, but easily mislead the output of deep neural networks (DNNs).
Attribute-guided image generation from layout
Also, the generated images from our model have higher resolution, object classification accuracy and consistency, as compared to the previous state-of-the-art.
Augmented Bi-path Network for Few-shot Learning
Finally, the model learns to compare global and local features separately, i. e., in two paths, before merging the similarities.
PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest
Latent user representations are widely adopted in the tech industry for powering personalized recommender systems.
Privacy-Preserving Technology to Help Millions of People: Federated Prediction Model for Stroke Prevention
It adopts federated averaging during the model training process, without patient data being taken out of the hospitals during the whole process of model training and forecasting.
Dataset Condensation with Gradient Matching
As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive.
Ranked #4 on
Dataset Distillation - 1IPC
on CUB-200-2011
Continual Representation Learning for Biometric Identification
With the explosion of digital data in recent years, continuously learning new tasks from a stream of data without forgetting previously acquired knowledge has become increasingly important.
iDLG: Improved Deep Leakage from Gradients
Particularly, our approach can certainly extract the ground-truth labels as opposed to DLG, hence we name it Improved DLG (iDLG).
DwNet: Dense warp-based network for pose-guided human video generation
In this paper, we focus on human motion transfer - generation of a video depicting a particular subject, observed in a single image, performing a series of motions exemplified by an auxiliary (driving) video.
Image Generation from Layout
The representation of each object is disentangled into a specified/certain part (category) and an unspecified/uncertain part (appearance).
Ranked #2 on
Layout-to-Image Generation
on Visual Genome 64x64
Interpretable Spatio-temporal Attention for Video Action Recognition
Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose an interpretable and easy plug-in spatial-temporal attention mechanism for video action recognition.
Guess Me if You Can: Acronym Disambiguation for Enterprises
The framework takes the enterprise corpus as input and produces a high-quality acronym disambiguation system as output.
MSplit LBI: Realizing Feature Selection and Dense Estimation Simultaneously in Few-shot and Zero-shot Learning
To solve this task, $L_{1}$ regularization is widely used for the pursuit of feature selection and avoiding overfitting, and yet the sparse estimation of features in $L_{1}$ regularization may cause the underfitting of training data.
A Large-scale Attribute Dataset for Zero-shot Learning
In classical ZSL algorithms, attributes are introduced as the intermediate semantic representation to realize the knowledge transfer from seen classes to unseen classes.
Modular Generative Adversarial Networks
Existing methods for multi-domain image-to-image translation (or generation) attempt to directly map an input image (or a random vector) to an image in one of the output domains.
Left-Right Comparative Recurrent Model for Stereo Matching
Left-right consistency check is an effective way to enhance the disparity estimation by referring to the information from the opposite view.
Zero-shot Learning via Shared-Reconstruction-Graph Pursuit
With the learned SRG, each unseen class prototype (cluster center) in the image feature space can be synthesized by the linear combination of other class prototypes, so that testing instances can be classified based on the distance to these synthesized prototypes.
AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
Significant progress has been achieved in Computer Vision by leveraging large-scale image datasets.
Optimal Experiment Design for Magnetic Resonance Fingerprinting: Cramér-Rao Bound Meets Spin Dynamics
Magnetic resonance (MR) fingerprinting is a new quantitative imaging paradigm, which simultaneously acquires multiple MR tissue parameter maps in a single experiment.
Signal Processing
Neural Person Search Machines
We investigate the problem of person search in the wild in this work.
Ranked #4 on
Person Re-Identification
on CUHK-SYSU
Memory-Augmented Attribute Manipulation Networks for Interactive Fashion Search
We introduce a new fashion search protocol where attribute manipulation is allowed within the interaction between users and search engines, e. g. manipulating the color attribute of the clothing from red to blue.
Multi-View Image Generation from a Single-View
This paper addresses a challenging problem -- how to generate multi-view cloth images from only a single view input.
Zero-Shot Learning posed as a Missing Data Problem
This paper presents a method of zero-shot learning (ZSL) which poses ZSL as the missing data problem, rather than the missing label problem.
Diversified Visual Attention Networks for Fine-Grained Object Classification
Fine-grained object classification is a challenging task due to the subtle inter-class difference and large intra-class variation.
Fast Low-rank Representation based Spatial Pyramid Matching for Image Classification
Although the methods achieve a higher recognition rate than the traditional SPM, they consume more time to encode the local descriptors extracted from the image.
