Search Results for author:
Found 29 papers, 11 papers with code
Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Since hardware resources are limited, the objective of training deep learning models is typically to maximize accuracy subject to the time and memory constraints of training and inference.
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
In this work, we explore a unified VidL framework LAVENDER, where Masked Language Modeling (MLM) is used as the common interface for all pre-training and downstream tasks.
GIT: A Generative Image-to-text Transformer for Vision and Language
In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering.
Ranked #1 on
Image Captioning
on nocaps-XD out-of-domain
Cross-modal Representation Learning for Zero-shot Action Recognition
The model design provides a natural mechanism for visual and semantic representations to be learned in a shared knowledge space, whereby it encourages the learned visual embedding to be discriminative and more semantically consistent.
Ranked #1 on
Zero-Shot Action Recognition
on HMDB51
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Based on this model architecture, we show that video captioning can benefit significantly from more densely sampled video frames as opposed to previous successes with sparsely sampled video frames for video-and-language understanding tasks (e. g., video question answering).
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
Further, unlike previous studies that found pre-training tasks on video inputs (e. g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling.
OVIS: Open-Vocabulary Visual Instance Search via Visual-Semantic Aligned Representation Learning
We introduce the task of open-vocabulary visual instance search (OVIS).
Mesh Graphormer
We present a graph-convolution-reinforced transformer, named Mesh Graphormer, for 3D human pose and mesh reconstruction from a single image.
Ranked #1 on
3D Hand Pose Estimation
on FreiHAND
Do Abstractions Have Politics? Toward a More Critical Algorithm Analysis
The expansion of computer science (CS) education in K--12 and higher-education in the United States has prompted deeper engagement with equity that moves beyond inclusion toward a more critical CS education.
Computers and Society K.3.2
End-to-End Human Pose and Mesh Reconstruction with Transformers
We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D human pose and mesh vertices from a single image.
Ranked #3 on
3D Hand Pose Estimation
on FreiHAND
Constructing Taxonomies from Pretrained Language Models
The tree reconciliation module treats the task as a graph optimization problem and outputs the maximum spanning tree of this graph.
Nifty Web Apps: Build a Web App for Any Text-Based Programming Assignment
While many students now interact with web apps across a variety of smart devices, the vast majority of our Nifty Assignments still present traditional user interfaces such as console input/output and desktop GUI.
Computers and Society K.3.2
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps).
Ranked #2 on
Image Captioning
on nocaps-XD out-of-domain
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Learning Nonparametric Human Mesh Reconstruction from a Single Image without Ground Truth Meshes
Experimental results on multiple public datasets show that without using 3D ground truth meshes, the proposed approach outperforms the previous state-of-the-art approaches that require ground truth meshes for training.
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Since hardware resources are limited, the objective of training deep learning models is typically to maximize accuracy subject to the time and memory constraints of training and inference.
Learning to Generate Multiple Style Transfer Outputs for an Input Sentence
Specifically, we decompose the latent representation of the input sentence to a style code that captures the language style variation and a content code that encodes the language style-independent content.
Neural Module Networks for Reasoning over Text
Answering compositional questions that require multiple steps of reasoning against text is challenging, especially when they involve discrete, symbolic operations.
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions
QuaRTz contains general qualitative statements, e. g., "A sunscreen with a higher SPF protects the skin longer.
Reasoning Over Paragraph Effects in Situations
A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation.
Cross-Domain Complementary Learning Using Pose for Multi-Person Part Segmentation
On the other hand, if part labels are also available in the real-images during training, our method outperforms the supervised state-of-the-art methods by a large margin.
Ranked #1 on
Human Part Segmentation
on PASCAL-Part
(using extra training data)
Grammar-based Neural Text-to-SQL Generation
The sequence-to-sequence paradigm employed by neural text-to-SQL models typically performs token-level decoding and does not consider generating SQL hierarchically from a grammar.
Adversarial Learning for Fine-grained Image Search
Fine-grained image search is still a challenging problem due to the difficulty in capturing subtle differences regardless of pose variations of objects from fine-grained categories.
A Sharp Error Analysis for the Fused Lasso, with Application to Approximate Changepoint Screening
In the 1-dimensional multiple changepoint detection problem, we derive a new fast error rate for the fused lasso estimator, under the assumption that the mean vector has a sparse number of changepoints.
Adversarial Ranking for Language Generation
Rather than training the discriminator to learn and assign absolute binary predicate for individual data sample, the proposed RankGAN is able to analyze and rank a collection of human-written and machine-written sentences by giving a reference group.
Ranked #2 on
Text Generation
on Chinese Poems
Learning Compact Binary Descriptors With Unsupervised Deep Neural Networks
In this paper, we propose a new unsupervised deep learning approach called DeepBit to learn compact binary descriptor for efficient visual object matching.
Supervised Learning of Semantics-Preserving Hash via Deep Convolutional Neural Networks
SSDH is simple and can be realized by a slight enhancement of an existing deep architecture for classification; yet it is effective and outperforms other hashing approaches on several benchmarks and large datasets.
Optimization for Compressed Sensing: the Simplex Method and Kronecker Sparsification
For the first approach, we note that the zero vector can be taken as the initial basic (infeasible) solution for the linear programming problem and therefore, if the true signal is very sparse, some variants of the simplex method can be expected to take only a small number of pivots to arrive at a solution.
