Search Results for author:
Found 148 papers, 77 papers with code
Self-Taught Agentic Long Context Understanding
Answering complex, long-context questions remains a major challenge for large language models (LLMs) as it requires effective question clarifications and context retrieval.
Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification
In this paper, we follow the guidance of the central dogma to redesign both the data and model pipeline and offer a comprehensive framework, Life-Code, that spans different biological functions.
Agent Laboratory: Using LLM Agents as Research Assistants
Historically, scientific discovery has been a lengthy and costly process, demanding substantial time and resources from initial conception to final results.
Artificial Intelligence for Central Dogma-Centric Multi-Omics: Challenges and Breakthroughs
With the rapid development of high-throughput sequencing platforms, an increasing number of omics technologies, such as genomics, metabolomics, and transcriptomics, are being applied to disease genetics research.
SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer
With its fully-differentiable design and semantic-rich latent space, our experiment demonstrates that SoftVQ-VAE achieves efficient tokenization without compromising generation quality, paving the way for more efficient generative models.
B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
To restrict the number of visual tokens, existing VLLMs either: (1) uniformly downsample videos into a fixed number of frames or (2) reducing the number of visual tokens encoded from each frame.
Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning
This paper delves into the interplay between vision backbones and optimizers, unvealing an inter-dependent phenomenon termed \textit{\textbf{b}ackbone-\textbf{o}ptimizer \textbf{c}oupling \textbf{b}ias} (BOCB).
Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization
We show that PSO can directly adapt distilled models to human-preferred generation with both offline and online-generated pairwise preference image data.
Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs
In this paper, we present DoSSR, a Domain Shift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images.
A Survey on Mixup Augmentations and Beyond
As Deep Neural Networks have achieved thrilling breakthroughs in the past decade, data augmentations have garnered increasing attention as regularization techniques when massive labeled data are unavailable.
AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition
With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism.
MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3. 5 Sonnet is the best model with a score of 71. 8, slightly outperforming GPT-4o which scored 71. 0.
IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation
First, to enable dual-modal generation and maximize the information exchange between video and depth generation, we propose a unified dual-modal U-Net, a parameter-sharing framework for joint video and depth denoising, wherein a modality label guides the denoising target, and cross-modal attention enables the mutual information flow.
CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph
To broaden the scope, we have adapted these models to a range of tasks essential in drug design, which are considered sub-tasks within the graph fill-in-the-blank tasks.
Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences
To mitigate the computational complexity in the self-attention mechanism on long sequences, linear attention utilizes computation tricks to achieve linear complexity, while state space models (SSMs) popularize a favorable practice of using non-data-dependent memory pattern, i. e., emphasize the near and neglect the distant, to processing sequences.
Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions
Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields and have shown significant potential in the academic peer-review process.
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models
The Genomic Foundation Model (GFM) paradigm is expected to facilitate the extraction of generalizable representations from massive genomic data, thereby enabling their application across a spectrum of downstream applications.
VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling
In this paper, we introduce VQDNA, a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning.
LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory
Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences.
A robust assessment for invariant representations
The performance of machine learning models can be impacted by changes in data over time.
Advances of Deep Learning in Protein Science: A Comprehensive Survey
Protein representation learning plays a crucial role in understanding the structure and function of proteins, which are essential biomolecules involved in various biological processes.
PPFlow: Target-aware Peptide Design with Torsional Flow Matching
Therapeutic peptides have proven to have great pharmaceutical value and potential in recent decades.
Switch EMA: A Free Lunch for Better Flatness and Sharpness
Exponential Moving Average (EMA) is a widely used weight averaging (WA) regularization to learn flat optima for better generalizations without extra cost in deep neural network (DNN) optimization.
PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Prediction
Compound-Protein Interaction (CPI) prediction aims to predict the pattern and strength of compound-protein interactions for rational drug discovery.
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis
To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model's ability to capture the true semantic representation of visual scenes.
A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization
To capture the conditional dependence between branching and nested parameters, a unified Bayesian optimization framework is proposed.
Bring Metric Functions into Diffusion Models
The first module, similar to a standard DDPM, learns to predict the added noise and is unaffected by the metric function.
Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning
We also show the effectiveness of our RL fine-tuning framework on enhancing the diversity of image generation with different types of diffusion models including class-conditional models and text-conditional models e. g. StableDiffusion.
Masked Modeling for Self-supervised Representation Learning on Vision and Beyond
As the deep learning revolution marches on, self-supervised learning has garnered increasing attention in recent years thanks to its remarkable representation learning ability and the low dependence on labeled data.
Segment and Caption Anything
We propose a method to efficiently equip the Segment Anything Model (SAM) with the ability to generate regional captions.
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the generation of audio descriptions (AD).
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
We first benchmark MM-Navigator on our collected iOS screen dataset.
MM-VID: Advancing Video Understanding with GPT-4V(ision)
We present MM-VID, an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding.
Exploring Invariance in Images through One-way Wave Equations
In this paper, we empirically reveal an invariance over images-images share a set of one-way wave equations with latent speeds.
Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation
We introduce ``Idea to Image,'' a system that enables multimodal iterative self-refinement with GPT-4V(ision) for automatic image design and generation.
OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation
We hope our proposed framework, benchmark, and LMM evaluation could help establish the intriguing interleaved image-text generation task.
SemiReward: A General Reward Model for Semi-supervised Learning
The main challenge is how to distinguish high-quality pseudo labels against the confirmation bias.
Completing Visual Objects via Bridging Generation and Segmentation
This paper presents a novel approach to object completion, with the primary goal of reconstructing a complete object from its partially visible components.
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
We hope that this preliminary exploration will inspire future research on the next-generation multimodal task formulation, new ways to exploit and enhance LMMs to solve real-world problems, and gaining better understanding of multimodal foundation models.
Ranked #3 on
MMR total
on MRR-Benchmark
(using extra training data)
ORES: Open-vocabulary Responsible Visual Synthesis
In this work, we formalize a new task, Open-vocabulary Responsible Visual Synthesis (ORES), where the synthesis model is able to avoid forbidden visual concepts while allowing users to input any desired content.
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking.
An Empirical Study of Large-Scale Data-Driven Full Waveform Inversion
This paper investigates the impact of big data on deep learning models to help solve the full waveform inversion (FWI) problem.
Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models
In this paper, we study the denoising diffusion probabilistic model (DDPM) in wavelet space, instead of pixel space, for visual synthesis.
DisCo: Disentangled Control for Realistic Human Dance Generation
In this paper, we depart from the traditional paradigm of human motion transfer and emphasize two additional critical attributes for the synthesis of human dance content in social media contexts: (i) Generalizability: the model should be able to generalize beyond generic human viewpoints as well as unseen human subjects, backgrounds, and poses; (ii) Compositionality: it should allow for the seamless composition of seen/unseen subjects, backgrounds, and poses from different sources.
OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning
Spatio-temporal predictive learning is a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsupervised manner.
RefineVIS: Video Instance Segmentation with Temporal Attention Refinement
We introduce a novel framework called RefineVIS for Video Instance Segmentation (VIS) that achieves good object association between frames and accurate segmentation masks by iteratively refining the representations using sequence context.
Ranked #5 on
Video Instance Segmentation
on YouTube-VIS 2021
(using extra training data)
PaintSeg: Training-free Segmentation via Painting
The paper introduces PaintSeg, a new unsupervised method for segmenting objects without any training.
Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance
This paper introduces a novel mathematical property applicable to diverse images, referred to as FINOLA (First-Order Norm+Linear Autoregressive).
Conformal Inference for Invariant Risk Minimization
The application of machine learning models can be significantly impeded by the occurrence of distributional shifts, as the assumption of homogeneity between the population of training and testing samples in machine learning and statistics may not be feasible in practical situations.
Neural Voting Field for Camera-Space 3D Hand Pose Estimation
We present a unified framework for camera-space 3D hand pose estimation from a single RGB image based on 3D implicit representation.
Auto-Linear Phenomenon in Subsurface Imaging
Subsurface imaging involves solving full waveform inversion (FWI) to predict geophysical properties from measurements.
Adaptive Human Matting for Dynamic Videos
The most recent efforts in video matting have focused on eliminating trimap dependency since trimap annotations are expensive and trimap-based methods are less adaptable for real-time applications.
Binary Latent Diffusion
In this paper, we show that a binary latent space can be explored for compact yet expressive image representations.
Towards Reasonable Budget Allocation in Untargeted Graph Structure Attacks via Gradient Debias
It has become cognitive inertia to employ cross-entropy loss function in classification related tasks.
Equivariant Similarity for Vision-Language Foundation Models
Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes.
Ranked #7 on
Visual Reasoning
on Winoground
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation.
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
Ranked #86 on
Visual Question Answering
on MM-Vet
Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer.
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
3D photography renders a static image into a video with appealing 3D visual effects.
Ranked #1 on
Image Outpainting
on MSCOCO
Energy-Inspired Self-Supervised Pretraining for Vision Models
In the proposed framework, we model energy estimation and data restoration as the forward and backward passes of a single network without any auxiliary components, e. g., an extra decoder.
RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design
We crafted a large, well-curated benchmark dataset and designed a comprehensive structural modeling approach to represent the complex RNA tertiary structure.
Federated Learning for Inference at Anytime and Anywhere
Federated learning has been predominantly concerned with collaborative training of deep networks from scratch, and especially the many challenges that arise, such as communication cost, robustness to heterogeneous data, and support for diverse device capabilities.
GRiT: A Generative Region-to-text Transformer for Object Understanding
Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions.
Ranked #2 on
Dense Captioning
on Visual Genome
MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction
Traditional methods of reconstructing 3D human pose and mesh from single images rely on paired image-mesh datasets, which can be difficult and expensive to obtain.
Ranked #17 on
3D Human Pose Estimation
on 3DPW
Self-Supervised Learning based on Heat Equation
When transferring to object detection with frozen backbone, QB-Heat outperforms MoCo-v2 and supervised pre-training on ImageNet by 7. 9 and 4. 5 AP respectively.
ReCo: Region-Controlled Text-to-Image Generation
Human evaluation on PaintSkill shows that ReCo is +19. 28% and +17. 21% more accurate in generating images with correct object count and spatial relationship than the T2I model.
Conditional Text-to-Image Synthesis
Layout-to-Image Generation
+1
Exploring Discrete Diffusion Models for Image Captioning
The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one.
MogaNet: Multi-order Gated Aggregation Network
Notably, MogaNet hits 80. 0\% and 87. 8\% accuracy with 5. 2M and 181M parameters on ImageNet-1K, outperforming ParC-Net and ConvNeXt-L, while saving 59\% FLOPs and 17M parameters, respectively.
Ranked #1 on
Instance Segmentation
on COCO val2017
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years.
Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification
Despite their great academic success, Multi-Layer Perceptrons (MLPs) remain the primary workhorse for practical industrial applications.
Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation
Self-supervised learning on graphs has recently achieved remarkable success in graph representation learning.
OpenMixup: Open Mixup Toolbox and Benchmark for Visual Representation Learning
Mixup augmentation has emerged as a widely used technique for improving the generalization ability of deep neural networks (DNNs).
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Masked visual modeling (MVM) has been recently proven effective for visual pre-training.
Ranked #1 on
Video Question Answering
on LSMDC-MC
Are Gradients on Graph Structure Reliable in Gray-box Attacks?
These errors arise from rough gradient usage due to the discreteness of the graph structure and from the unreliability in the meta-gradient on the graph structure.
NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
In this paper, we present NUWA-Infinity, a generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos.
Ranked #1 on
Image Outpainting
on LHQC
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable attention.
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Dev
Consistent Video Instance Segmentation with Inter-Frame Recurrent Attention
We propose a consistent end-to-end video instance segmentation framework with Inter-Frame Recurrent Attention to model both the temporal instance consistency for adjacent frames and the global temporal context.
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
In this work, we explore a unified VidL framework LAVENDER, where Masked Language Modeling (MLM) is used as the common interface for all pre-training and downstream tasks.
GIT: A Generative Image-to-text Transformer for Vision and Language
In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering.
Ranked #1 on
Image Captioning
on nocaps-XD near-domain
Cross-modal Representation Learning for Zero-shot Action Recognition
The model design provides a natural mechanism for visual and semantic representations to be learned in a shared knowledge space, whereby it encourages the learned visual embedding to be discriminative and more semantically consistent.
Ranked #4 on
Zero-Shot Action Recognition
on ActivityNet
An Intriguing Property of Geophysics Inversion
In particular, when dealing with the inversion from seismic data to subsurface velocity governed by a wave equation, the integral results of velocity with Gaussian kernels are linearly correlated to the integral of seismic data with sine kernels.
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks.
Ranked #1 on
Object Detection
on ELEVATER
MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images
For this task, we propose a simple yet effective method to train a generalizable NeRF with multiview images as conditional input.
Deep Frequency Filtering for Domain Generalization
Improving the generalization ability of Deep Neural Networks (DNNs) is critical for their practical uses, which has been a longstanding challenge.
Harnessing Hard Mixed Samples with Decoupled Regularizer
However, we found that the extra optimizing step may be redundant because label-mismatched mixed samples are informative hard mixed samples for deep models to localize discriminative features.
The Overlooked Classifier in Human-Object Interaction Recognition
Human-Object Interaction (HOI) recognition is challenging due to two factors: (1) significant imbalance across classes and (2) requiring multiple labels per image.
SA-VQA: Structured Alignment of Visual and Semantic Representations for Visual Question Answering
As a multi-modality task, it is challenging since it requires not only visual and textual understanding, but also the ability to align cross-modality representations.
The Overlooked Classifier in Human-Object Interaction Recognition
Human-Object Interaction (HOI) recognition is challenging due to two factors: (1) significant imbalance across classes and (2) requiring multiple labels per image.
Ranked #1 on
Human-Object Interaction Detection
on HICO
Lifelong Unsupervised Domain Adaptive Person Re-identification with Coordinated Anti-forgetting and Adaptation
In this paper, to address more practical scenarios, we propose a new task, Lifelong Unsupervised Domain Adaptive (LUDA) person ReID.
Domain Adaptive Person Re-Identification
Knowledge Distillation
+4
Improving Vision Transformers for Incremental Learning
This paper proposes a working recipe of using Vision Transformer (ViT) in class incremental learning.
Injecting Semantic Concepts into End-to-End Image Captioning
In this paper, we are concerned with a better-performing detector-free image captioning model, and propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features.
MLP Architectures for Vision-and-Language Modeling: An Empirical Study
Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs?
MMPTRACK: Large-scale Densely Annotated Multi-camera Multiple People Tracking Benchmark
This dataset provides a more reliable benchmark of multi-camera, multi-object tracking systems in cluttered and crowded environments.
Ranked #2 on
Object Tracking
on MMPTRACK
Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup
Accordingly, we propose $\eta$-balanced mixup loss for complementary learning of the two sub-objectives.
Ranked #7 on
Image Classification
on Places205
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Based on this model architecture, we show that video captioning can benefit significantly from more densely sampled video frames as opposed to previous successes with sparsely sampled video frames for video-and-language understanding tasks (e. g., video question answering).
Scaling Up Vision-Language Pre-training for Image Captioning
In this paper, we present LEMON, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning.
Ranked #3 on
Image Captioning
on nocaps-XD entire
(using extra training data)
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
Further, unlike previous studies that found pre-training tasks on video inputs (e. g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling.
Ranked #21 on
Zero-Shot Video Retrieval
on DiDeMo
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling
On grounded captioning, UniTAB presents a simpler solution with a single output head, and significantly outperforms state of the art in both grounding and captioning evaluations.
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e. g., image or language) or multimodal inputs (e. g., the concatenation of the image and the question), for vision-language (VL) representation learning.
Physics-guided Loss Functions Improve Deep Learning Performance in Inverse Scattering
Solving electromagnetic inverse scattering problems (ISPs) is challenging due to the intrinsic nonlinearity, ill-posedness, and expensive computational cost.
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
Ranked #20 on
Cross-Modal Retrieval
on COCO 2014
(using extra training data)
GenURL: A General Framework for Unsupervised Representation Learning
For example, dimension reduction methods, t-SNE, and UMAP optimize pair-wise data relationships by preserving the global geometric structure, while self-supervised learning, SimCLR, and BYOL focus on mining the local statistics of instances under specific augmentations.
An Empirical Study: Extensive Deep Temporal Point Process
In this paper, we first review recent research emphasis and difficulties in modeling asynchronous event sequences with deep temporal point process, which can be concluded into four fields: encoding of history sequence, formulation of conditional intensity function, relational discovery of events and learning approaches for optimization.
Unsupervised Learning of Full-Waveform Inversion: Connecting CNN and Partial Differential Equation in a Loop
In particular, we use finite difference to approximate the forward modeling of PDE as a differentiable operator (from velocity map to seismic data) and model its inversion by CNN (from seismic data to velocity map).
Improving Discriminative Visual Representation Learning via Automatic Mixup
In this paper, we decompose mixup into two sub-tasks of mixup generation and classification and formulate it for discriminative representations as class- and instance-level mixup.
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA.
Ranked #21 on
Visual Question Answering (VQA)
on OK-VQA
(using extra training data)
Mobile-Former: Bridging MobileNet and Transformer
This structure leverages the advantages of MobileNet at local processing and transformer at global interaction.
MicroNet: Improving Image Recognition with Extremely Low FLOPs
This paper aims at addressing the problem of substantial performance degradation at extremely low computational cost (e. g. 5M FLOPs on ImageNet classification).
OVIS: Open-Vocabulary Visual Instance Search via Visual-Semantic Aligned Representation Learning
We introduce the task of open-vocabulary visual instance search (OVIS).
Is Object Detection Necessary for Human-Object Interaction Recognition?
This paper revisits human-object interaction (HOI) recognition at image level without using supervisions of object location and human pose.
Probabilistic Model Distillation for Semantic Correspondence
We address this problem with the use of a novel Probabilistic Model Distillation (PMD) approach which transfers knowledge learned by a probabilistic teacher model on synthetic data to a static student model with the use of unlabeled real image pairs.
A data-based comparative review and AI-driven symbolic model for longitudinal dispersion coefficient in natural streams
Whereas implicit ML-driven methods are black-boxes in nature, explicit ML-driven methods have more potential in prediction of LDC.
End-to-End Semi-Supervised Object Detection with Soft Teacher
This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods.
Ranked #6 on
Semi-Supervised Object Detection
on COCO 100% labeled data
(using extra training data)
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
Playing Lottery Tickets with Vision and Language
However, we can find "relaxed" winning tickets at 50%-70% sparsity that maintain 99% of the full accuracy.
Compressing Visual-linguistic Model via Knowledge Distillation
In this paper, we study knowledge distillation (KD) to effectively compress a transformer-based large VL model into a small VL model.
TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking
TransMOT effectively models the interactions of a large number of objects by arranging the trajectories of the tracked objects as a set of sparse weighted graphs, and constructing a spatial graph transformer encoder layer, a temporal transformer encoder layer, and a spatial graph transformer decoder layer based on the graphs.
Ranked #2 on
Multi-Object Tracking
on 2DMOT15
(using extra training data)
Mesh Graphormer
We present a graph-convolution-reinforced transformer, named Mesh Graphormer, for 3D human pose and mesh reconstruction from a single image.
Disentanglement-based Cross-Domain Feature Augmentation for Effective Unsupervised Domain Adaptive Person Re-identification
Each recomposed feature, obtained based on the domain-invariant feature (which enables a reliable inheritance of identity) and an enhancement from a domain specific feature (which enables the approximation of real distributions), is thus an "ideal" augmentation.
AutoMix: Unveiling the Power of Mixup for Stronger Classifiers
Specifically, AutoMix reformulates the mixup classification into two sub-tasks (i. e., mixed sample generation and mixup classification) with corresponding sub-networks and solves them in a bi-level optimization framework.
Ranked #8 on
Image Classification
on Places205
Revisiting Dynamic Convolution via Matrix Decomposition
It has two limitations: (a) it increases the number of convolutional weights by K-times, and (b) the joint optimization of dynamic attention and static convolution kernels is challenging.
Stronger NAS with Weaker Predictors
We propose a paradigm shift from fitting the whole architecture space using one strong predictor, to progressively fitting a search path towards the high-performance sub-space through a set of weaker predictors.
SEED: Self-supervised Distillation For Visual Representation
This paper is concerned with self-supervised learning for small models.
Weak NAS Predictor Is All You Need
Rather than expecting a single strong predictor to model the whole space, we seek a progressive line of weak predictors that can connect a path to the best architecture, thus greatly simplifying the learning task of each predictor.
3D Human motion anticipation and classification
Our model learns to predict multiple future sequences of human poses from the same input sequence.
End-to-End Human Pose and Mesh Reconstruction with Transformers
We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D human pose and mesh vertices from a single image.
MiniVLM: A Smaller and Faster Vision-Language Model
We design a Two-stage Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet network, to significantly reduce the time cost of visual feature extraction by $95\%$, compared to a baseline model.
MicroNet: Towards Image Recognition with Extremely Low FLOPs
In this paper, we present MicroNet, which is an efficient convolutional neural network using extremely low computational cost (e. g. 6 MFLOPs on ImageNet classification).
Semantic Change Detection with Asymmetric Siamese Networks
Given two multi-temporal aerial images, semantic change detection aims to locate the land-cover variations and identify their change types with pixel-wise boundaries.
Deep Clustering and Representation Learning that Preserves Geometric Structures
To overcome the problem that clusteringoriented losses may deteriorate the geometric structure of embeddings in the latent space, an isometric loss is proposed for preserving intra-manifold structure locally and a ranking loss for inter-manifold structure globally.
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps).
Ranked #3 on
Image Captioning
on nocaps-XD out-of-domain
Generalized Clustering and Multi-Manifold Learning with Geometric Structure Preservation
Though manifold-based clustering has become a popular research topic, we observe that one important factor has been omitted by these works, namely that the defined clustering loss may corrupt the local and global structure of the latent space.
Learning Nonparametric Human Mesh Reconstruction from a Single Image without Ground Truth Meshes
Experimental results on multiple public datasets show that without using 3D ground truth meshes, the proposed approach outperforms the previous state-of-the-art approaches that require ground truth meshes for training.
Dynamic Convolution: Attention over Convolution Kernels
Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability.
Ranked #979 on
Image Classification
on ImageNet
Cross-Domain Complementary Learning Using Pose for Multi-Person Part Segmentation
On the other hand, if part labels are also available in the real-images during training, our method outperforms the supervised state-of-the-art methods by a large margin.
Ranked #1 on
Human Part Segmentation
on PASCAL-Part
(using extra training data)
Large Scale Incremental Learning
We believe this is because of the combination of two factors: (a) the data imbalance between the old and new classes, and (b) the increasing number of visually similar classes.
Rethinking Classification and Localization for Object Detection
Two head structures (i. e. fully connected head and convolution head) have been widely used in R-CNN based detectors for classification and localization tasks.
Incremental Classifier Learning with Generative Adversarial Networks
To address these problems, we propose (a) a new loss function to combine the cross-entropy loss and distillation loss, (b) a simple way to estimate and remove the unbalance between the old and new classes , and (c) using Generative Adversarial Networks (GANs) to generate historical data and select representative exemplars during generation.
HP-GAN: Probabilistic 3D human motion prediction via GAN
Our model, which we call HP-GAN, learns a probability density function of future human poses conditioned on previous poses.
Ranked #7 on
Human Pose Forecasting
on Human3.6M
(APD metric)
Reinforced Temporal Attention and Split-Rate Transfer for Depth-Based Person Re-Identification
We address the problem of person re-identification from commodity depth sensors.
A Tube-and-Droplet-based Approach for Representing and Analyzing Motion Trajectories
Our approach first leverages the complete information from given trajectories to construct a thermal transfer field which provides a context-rich way to describe the global motion pattern in a scene.
Tensor-Based Human Body Modeling
In this paper, we present a novel approach to model 3D human body with variations on both human shape and pose, by exploring a tensor decomposition technique.
Semi-supervised Node Splitting for Random Forest Construction
Node splitting is an important issue in Random Forest but robust splitting requires a large number of training samples.
Probabilistic Graphlet Cut: Exploiting Spatial Structure Cue for Weakly Supervised Image Segmentation
Finally, we propose a novel image segmentation algorithm, called graphlet cut, that leverages the learned graphlet distribution in measuring the homogeneity of a set of spatially structured superpixels.
HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences
In contrast, we describe the depth sequence using a histogram capturing the distribution of the surface normal orientation in the 4D space of time, depth, and spatial coordinates.
