Search Results for author:
Found 83 papers, 38 papers with code
On the Scalability of Diffusion-based Text-to-Image Generation
On the data scaling side, we show the quality and diversity of the training set matters more than simply dataset size.
HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data
In this paper, we propose HOIDiffusion for generating realistic and diverse 3D hand-object interaction data.
Bayesian Diffusion Models for 3D Shape Reconstruction
We present Bayesian Diffusion Models (BDM), a prediction algorithm that performs effective Bayesian inference by tightly coupling the top-down (prior) information with the bottom-up (data-driven) procedure via joint diffusion processes.
Enhancing Vision-Language Pre-training with Rich Supervisions
We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel pre-training paradigm for Vision-Language Models using data from large-scale web screenshot rendering.
Non-autoregressive Sequence-to-Sequence Vision-Language Models
Sequence-to-sequence vision-language models are showing promise, but their applicability is limited by their inference latency due to their autoregressive way of generating predictions.
AffordanceLLM: Grounding Affordance from Vision Language Models
Affordance grounding refers to the task of finding the area of an object with which one can interact.
Restoration by Generation with Constrained Priors
We propose a method to adapt a pretrained diffusion model for image restoration by simply adding noise to the input image to be restored and then denoise.
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages
However, concrete evidence for the effects of multilinguality on language modeling performance in individual languages remains scarce.
Dolfin: Diffusion Layout Transformers without Autoencoder
In this paper, we introduce a novel generative model, Diffusion Layout Transformers without Autoencoder (Dolfin), which significantly improves the modeling capability with reduced complexity compared to existing methods.
SkeleTR: Towrads Skeleton-based Action Recognition in the Wild
It first models the intra-person skeleton dynamics for each skeleton sequence with graph convolutions, and then uses stacked Transformer encoders to capture person interactions that are important for action recognition in general scenarios.
Object-Centric Multiple Object Tracking
Unsupervised object-centric learning methods allow the partitioning of scenes into entities without additional localization information and are excellent candidates for reducing the annotation burden of multiple-object tracking (MOT) pipelines.
Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
We quantify the final surprisal, within-run variability, age of acquisition, forgettability, and cross-run variability of learning curves for individual tokens in context.
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence.
Patched Denoising Diffusion Models For High-Resolution Image Synthesis
Feature collage systematically crops and combines partial features of the neighboring patches to predict the features of a shifted image patch, allowing the seamless generation of the entire image due to the overlap in the patch feature space.
DocTr: Document Transformer for Structured Information Extraction in Documents
We present a new formulation for structured information extraction (SIE) from visually rich documents.
Ranked #2 on
Entity Linking
on FUNSD
Distilling Large Vision-Language Model with Out-of-Distribution Generalizability
Model distillation, the process of creating smaller, faster models that maintain the performance of larger models, is a promising direction towards the solution.
Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts
We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer.
DiffusionRig: Learning Personalized Priors for Facial Appearance Editing
On a high level, DiffusionRig learns to map simplistic renderings of 3D face models to realistic photos of a given person.
Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images.
Guided Recommendation for Model Fine-Tuning
With thousands of historical training jobs, a recommendation system can be learned to predict the model selection score given the features of the dataset and the model as input.
MasQCLIP for Open-Vocabulary Universal Image Segmentation
We present a new method for open-vocabulary universal image segmentation, which is capable of performing instance, semantic, and panoptic segmentation under a unified framework.
Uni-3D: A Universal Model for Panoptic 3D Scene Reconstruction
Performing holistic 3D scene understanding from a single-view observation, involving generating instance shapes and 3D scene segmentation, is a long-standing challenge.
SkeleTR: Towards Skeleton-based Action Recognition in the Wild
It first models the intra-person skeleton dynamics for each skeleton sequence with graph convolutions, and then uses stacked Transformer encoders to capture person interactions that are important for action recognition in the wild.
Ranked #2 on
Human Interaction Recognition
on NTU RGB+D
On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
Reinforcement Learning (RL) algorithms can solve challenging control problems directly from image observations, but they often require millions of environment interactions to do so.
Point Cloud Recognition with Position-to-Structure Attention Transformers
In this paper, we present Position-to-Structure Attention Transformers (PS-Former), a Transformer-based algorithm for 3D point cloud recognition.
An In-depth Study of Stochastic Backpropagation
In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks.
Open-Vocabulary Universal Image Segmentation with MaskCLIP
In this paper, we tackle an emerging computer vision task, open-vocabulary universal image segmentation, that aims to perform semantic/instance/panoptic segmentation (background semantic labeling + foreground instance segmentation) for arbitrary categories of text-based descriptions in inference time.
Semi-supervised Vision Transformers at Scale
We study semi-supervised learning (SSL) for vision transformers (ViT), an under-explored topic despite the wide adoption of the ViT architectures to different tasks.
The Geometry of Multilingual Language Model Representations
The subspace means differ along language-sensitive axes that are relatively stable throughout middle layers, and these axes encode information such as token vocabularies.
ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training
Based on the observation, we present a method, called Ensemble Logit Difference Inhibition (ELODI), to train a classification system that achieves paragon performance in both error rate and NFR, at the inference cost of a single model.
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks
In this paper, we study the challenging instance-wise vision-language tasks, where the free-form language is required to align with the objects instead of the whole image.
Text Spotting Transformers
In this paper, we present TExt Spotting TRansformers (TESTR), a generic end-to-end text spotting framework using Transformers for text detection and recognition in the wild.
Ranked #6 on
Text Spotting
on ICDAR 2015
MeMOT: Multi-Object Tracking with Memory
We propose an online tracking algorithm that performs the object detection and data association under a common framework, capable of linking objects after a long time span.
Contrastive Neighborhood Alignment
We present Contrastive Neighborhood Alignment (CNA), a manifold learning approach to maintain the topology of learned features whereby data points that are mapped to nearby representations by the source (teacher) model are also mapped to neighbors by the target (student) model.
Instance Segmentation With Mask-Supervised Polygonal Boundary Transformers
In this paper, we present an end-to-end instance segmentation method that regresses a polygonal boundary for each object instance.
Towards Panoptic 3D Parsing for Single Image in the Wild
Our proposed panoptic 3D parsing framework points to a promising direction in computer vision.
ViTGAN: Training GANs with Vision Transformers
Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases.
Ranked #68 on
Image Generation
on CIFAR-10
Long Short-Term Transformer for Online Action Detection
We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged sequence data.
Ranked #3 on
Online Action Detection
on TVSeries
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
In this paper, we detail the relationship between convolutions and self-attention in natural language tasks.
Compatibility-aware Heterogeneous Visual Search
Existing systems use the same embedding model to compute representations (embeddings) for the query and gallery images.
Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries
Computer vision applications such as visual relationship detection and human object interaction can be formulated as a composite (structured) set detection problem in which both the parts (subject, object, and predicate) and the sum (triplet as a whole) are to be detected in a hierarchical fashion.
Pose Recognition with Cascade Transformers
Here we utilize the encoder-decoder structure in Transformers to perform regression-based person and keypoint detection that is general-purpose and requires less heuristic design compared with the existing approaches.
Co-Scale Conv-Attentional Image Transformers
In this paper, we present Co-scale conv-attentional image Transformers (CoaT), a Transformer-based image classifier equipped with co-scale and conv-attentional mechanisms.
Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques.
Self-Supervised Learning
Semi-Supervised Image Classification
Line Segment Detection Using Transformers without Edges
In this paper, we present a joint end-to-end line segment detection algorithm using Transformers that is post-processing and heuristics-guided intermediate processing (edge/junction/region detection) free.
Ranked #1 on
Line Segment Detection
on York Urban Dataset
(FH metric)
Constellation Nets for Few-Shot Learning
The success of deep convolutional neural networks builds on top of the learning of effective convolution operations, capturing a hierarchy of structured features via filtering, activation, and pooling.
Ranked #15 on
Few-Shot Image Classification
on FC100 5-way (5-shot)
Dual Contradistinctive Generative Autoencoder
Our model, named dual contradistinctive generative autoencoder (DC-VAE), integrates an instance-level discriminative loss (maintaining the instance-level fidelity for the reconstruction/synthesis) with a set-level adversarial loss (encouraging the set-level fidelity for there construction/synthesis), both being contradistinctive.
Ranked #2 on
Image Generation
on LSUN Bedroom 128 x 128
One-pixel Signature: Characterizing CNN Models for Backdoor Detection
One-pixel signature is a general representation that can be used to characterize CNN models beyond backdoor detection.
Guided Variational Autoencoder for Disentanglement Learning
We propose an algorithm, guided variational autoencoder (Guided-VAE), that is able to learn a controllable generative model by performing latent representation disentanglement learning.
Neural Program Synthesis By Self-Learning
Neural inductive program synthesis is a task generating instructions that can produce desired outputs from given inputs.
Rethinking Exposure Bias In Language Modeling
Exposure bias describes the phenomenon that a language model trained under the teacher forcing schema may perform poorly at the inference stage when its predictions are conditioned on its previous predictions unseen from the training corpus.
Unaligned Image-to-Sequence Transformation with Loop Consistency
We tackle the problem of modeling sequential visual phenomena.
Learning Instance Occlusion for Panoptic Segmentation
Panoptic segmentation requires segments of both "things" (countable object instances) and "stuff" (uncountable and amorphous regions) within a single output.
Ranked #22 on
Panoptic Segmentation
on COCO test-dev
Local Binary Pattern Networks for Character Recognition
Memory and computation efficient deep learning architectures are crucial to the continued proliferation of machine learning capabilities to new platforms and systems.
Attentional ShapeContextNet for Point Cloud Recognition
We tackle the problem of point cloud recognition.
Local Binary Pattern Networks
In this paper, we tackle the problem us- ing a strategy different from the existing literature by proposing local binary pattern networks or LBPNet, that is able to learn and perform binary operations in an end-to-end fashion.
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification.
Ranked #27 on
Action Recognition
on UCF101
(using extra training data)
Controllable Top-down Feature Transformer
We study the intrinsic transformation of feature maps across convolutional network layers with explicit top-down control.
DeepRadiologyNet: Radiologist Level Pathology Detection in CT Head Images
Thus, DeepRadiologyNet enables significant reduction in the workload of human radiologists by automatically filtering studies and reporting on the high-confidence ones at an operating point well below the literal error rate for US Board Certified radiologists, estimated at 0. 82%.
Wasserstein Introspective Neural Networks
We present Wasserstein introspective neural networks (WINN) that are both a generator and a discriminator within a single model.
Introspective Neural Networks for Generative Modeling
We study unsupervised learning by developing a generative model built from progressively learned deep convolutional neural networks.
Binarized Convolutional Neural Networks with Separable Filters for Efficient Hardware Acceleration
State-of-the-art convolutional neural networks are enormously costly in both compute and memory, demanding massively parallel GPUs for execution.
Introspective Classification with Convolutional Nets
We propose introspective convolutional networks (ICN) that emphasize the importance of having convolutional neural networks empowered with generative capabilities.
Introspective Generative Modeling: Decide Discriminatively
We study unsupervised learning by developing introspective generative modeling (IGM) that attains a generator using progressively learned deep convolutional neural networks.
Object Detection Free Instance Segmentation With Labeling Transformations
Instance segmentation has attracted recent attention in computer vision and existing methods in this domain mostly have an object detection stage.
Deep Convolutional Neural Networks with Merge-and-Run Mappings
A deep residual network, built by stacking a sequence of residual blocks, is easy to train, because identity mappings skip residual branches and thus improve information flow.
Aggregated Residual Transformations for Deep Neural Networks
Our simple design results in a homogeneous, multi-branch architecture that has only a few hyper-parameters to set.
Ranked #3 on
Image Classification
on GasHisSDB
Deeply supervised salient object detection with short connections
Recent progress on saliency detection is substantial, benefiting mostly from the explosive development of Convolutional Neural Networks (CNNs).
Ranked #4 on
RGB Salient Object Detection
on SBU
Deep FisherNet for Object Classification
Our proposed FisherNet combines convolutional neural network training and Fisher Vector encoding in a single end-to-end structure.
Dense Volume-to-Volume Vascular Boundary Detection
In this work, we present a novel 3D-Convolutional Neural Network (CNN) architecture called I2I-3D that predicts boundary location in volumetric data.
Top-Down Learning for Structured Labeling with Convolutional Pseudoprior
Current practice in convolutional neural networks (CNN) remains largely bottom-up and the role of top-down process in CNN for pattern analysis and visual inference is not very clear.
What Happened to My Dog in That Network: Unraveling Top-down Generators in Convolutional Neural Networks
Top-down information plays a central role in human perception, but plays relatively little role in many current state-of-the-art deep networks, such as Convolutional Neural Networks (CNNs).
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures.
Ranked #17 on
Image Classification
on MNIST
Training Deeper Convolutional Networks with Deep Supervision
One of the most promising ways of improving the performance of deep convolutional neural networks is by increasing the number of convolutional layers.
Holistically-Nested Edge Detection
We develop a new edge detection algorithm that tackles two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning.
Deeply-Supervised Nets
Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent.
Ranked #23 on
Image Classification
on MNIST
MILCut: A Sweeping Line Multiple Instance Learning Paradigm for Interactive Image Segmentation
Interactive segmentation, in which a user provides a bounding box to an object of interest for image segmentation, has been applied to a variety of applications in image editing, crowdsourcing, computer vision, and medical imaging.
Layered Logic Classifiers: Exploring the `And' and `Or' Relations
Many the solutions to the problem require to perform logic operations such as `and', `or', and `not'.
Scalable $k$-NN graph construction
The $k$-NN graph has played a central role in increasingly popular data-driven techniques for various learning and vision tasks; yet, finding an efficient and effective way to construct $k$-NN graphs remains a challenge, especially for large-scale high-dimensional data.
Harvesting Mid-level Visual Concepts from Large-Scale Internet Images
Obtaining effective mid-level representations has become an increasingly important task in computer vision.
Robust Estimation of Nonrigid Transformation for Point Set Registration
In the second step, we estimate the transformation using a robust estimator called L 2 E. This is the main novelty of our approach and it enables us to deal with the noise and outliers which arise in the correspondence step.
Sparse Subspace Denoising for Image Manifolds
With the increasing availability of high dimensional data and demand in sophisticated data analysis algorithms, manifold learning becomes a critical technique to perform dimensionality reduction, unraveling the intrinsic data structure.
