Search Results for author:
Found 51 papers, 25 papers with code
VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining
Our results show that our pretrained aesthetic vision-language model outperforms prior works on image aesthetic captioning over the AVA-Captions dataset, and it has powerful zero-shot capability for aesthetic tasks such as zero-shot style classification and zero-shot IAA, surpassing many supervised baselines.
Ranked #46 on
Video Quality Assessment
on MSU SR-QA Dataset
Vector-quantized Image Modeling with Improved VQGAN
Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively.
Conformer: Convolution-augmented Transformer for Speech Recognition
Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs).
Ranked #12 on
Speech Recognition
on LibriSpeech test-other
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CoCa: Contrastive Captioners are Image-Text Foundation Models
We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively.
Ranked #1 on
Visual Question Answering
on VQA v2 test-dev
Self-supervised Learning with Random-projection Quantizer for Speech Recognition
In particular the quantizer projects speech inputs with a randomly initialized matrix, and does a nearest-neighbor lookup in a randomly-initialized codebook.
Free-Form Image Inpainting with Gated Convolution
We present a generative image inpainting system to complete images with free-form mask and guidance.
Ranked #3 on
Image Inpainting
on Places2 val
Generative Image Inpainting with Contextual Attention
Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions.
Wide Activation for Efficient and Accurate Image Super-Resolution
Keras-based implementation of WDSR, EDSR and SRGAN for single image super-resolution
Ranked #4 on
Multi-Frame Super-Resolution
on PROBA-V
AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74. 2% top-1 accuracy, 2. 4% better than default MobileNet-v2 (301M FLOPs), and even 0. 2% better than RL-searched MNasNet (317M FLOPs).
Slimmable Neural Networks
Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization.
Universally Slimmable Networks and Improved Training Techniques
We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning.
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
We demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves a word error rate (WER) of 2. 1%/4. 6% without external language model (LM), 1. 9%/4. 1% with LM and 2. 9%/7. 0% with only 10M parameters on the clean/noisy LibriSpeech test sets.
Ranked #12 on
Speech Recognition
on LibriSpeech test-clean
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge.
Ranked #1 on
Text-to-Image Generation
on LAION COCO
Pyramid Attention Networks for Image Restoration
Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales.
Improving Object Detection from Scratch via Gated Feature Reuse
In this paper, we present a simple and parameter-efficient drop-in module for one-stage object detectors like SSD when learning from scratch (i. e., without pre-trained models).
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs.
Ranked #30 on
Neural Architecture Search
on ImageNet
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks.
Ranked #4 on
Visual Entailment
on SNLI-VE val
Scale-wise Convolution for Image Restoration
In this paper, we show that properly modeling scale-invariance into neural networks can bring significant benefits to image restoration performance.
Neural Sparse Representation for Image Restoration
Inspired by the robustness and efficiency of sparse representation in sparse coding based image restoration models, we investigate the sparsity of neurons in deep networks.
Deep object detection for waterbird monitoring using aerial imagery
In this work, we present a deep learning pipeline that can be used to precisely detect, count, and monitor waterbirds using aerial imagery collected by a commercial drone.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Ranked #1 on
Question Answering
on StrategyQA
FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
FastEmit also improves streaming ASR accuracy from 4. 4%/8. 9% to 3. 1%/7. 5% WER, meanwhile reduces 90th percentile latency from 210 ms to only 30 ms on LibriSpeech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
De-Diffusion Makes Text a Strong Cross-Modal Interface
We demonstrate text as a strong cross-modal interface.
Normalization effects on shallow neural networks and related asymptotic expansions
In addition, we show that to leading order in $N$, the variance of the neural network's statistical output decays as the implied normalization by the scaling parameter approaches the mean field normalization.
Normalization effects on deep neural networks
A given layer $i$ with $N_{i}$ hidden units is allowed to be normalized by $1/N_{i}^{\gamma_{i}}$ with $\gamma_{i}\in[1/2, 1]$ and we study the effect of the choice of the $\gamma_{i}$ on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set.
UnitBox: An Advanced Object Detection Network
In present object detection systems, the deep convolutional neural networks (CNNs) are utilized to predict bounding boxes of object candidates, and have gained performance advantages over the traditional region proposal methods.
A Simple Non-i.i.d. Sampling Approach for Efficient Training and Better Generalization
While training on samples drawn from independent and identical distribution has been a de facto paradigm for optimizing image classification networks, humans learn new concepts in an easy-to-hard manner and on the selected examples progressively.
Foreground-aware Image Inpainting
We show that by such disentanglement, the contour completion model predicts reasonable contours of objects, and further substantially improves the performance of image inpainting.
An Empirical Study on Regularization of Deep Neural Networks by Local Rademacher Complexity
In this paper, we investigate the role of Rademacher complexity in improving generalization of DNNs and propose a novel regularizer rooted in Local Rademacher Complexity (LRC).
FSNet: Compression of Deep Convolutional Neural Networks by Filter Summary
FSNet has the same architecture as that of the baseline CNN to be compressed, and each convolution layer of FSNet has the same number of filters from FS as that of the basline CNN in the forward process.
Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement
In order to distill diverse knowledge from different trained (teacher) models, we propose to use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously.
Dynamic Sparsity Neural Networks for Automatic Speech Recognition
In automatic speech recognition (ASR), model pruning is a widely adopted technique that reduces model size and latency to deploy neural network models on edge devices with resource constraints.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Cross-Supervised Object Detection
After learning a new object category from image-level annotations (with no object bounding boxes), humans are remarkably good at precisely localizing those objects.
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner.
Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible, while full-context ASR waits for the completion of a full speech utterance before emitting completed hypotheses.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Cascaded encoders for unifying streaming and non-streaming ASR
The proposed model consists of streaming and non-streaming encoders.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Better and Faster End-to-End Model for Streaming ASR
To address this, we explore replacing the LSTM layers in the encoder of our E2E model with Conformer layers [4], which has shown good improvements for ASR.
Audio and Speech Processing Sound
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio.
Ranked #1 on
Speech Recognition
on Common Voice
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Combined Scaling for Zero-shot Transfer Learning
Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood.
Scaling Up Neural Architecture Search with Big Single-Stage Models
In this work, we propose BigNAS, an approach that simplifies this workflow and scales up neural architecture search to target a wide range of model sizes simultaneously.
Co-training Transformer with Videos and Images Improves Action Recognition
We term this approach as Co-training Videos and Images for Action Recognition (CoVeR).
Ranked #8 on
Action Classification
on MiT
(using extra training data)
Exploiting Category Names for Few-Shot Classification with Vision-Language Models
Vision-language foundation models pretrained on large-scale data provide a powerful tool for many visual understanding tasks.
VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
We explore an efficient approach to establish a foundational video-text model.
Ranked #1 on
Video Captioning
on ActivityNet Captions
(using extra training data)
Noise2Music: Text-conditioned Music Generation with Diffusion Models
We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts.
Ranked #2 on
Text-to-Music Generation
on MusicCaps
CoBIT: A Contrastive Bi-directional Image-Text Generation Model
The field of vision and language has witnessed a proliferation of pre-trained foundation models.
Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR
Conformer models maintain a large number of internal states, the vast majority of which are associated with self-attention layers.
AudioPaLM: A Large Language Model That Can Speak and Listen
AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2.
Chemist-X: Large Language Model-empowered Agent for Reaction Condition Recommendation in Chemical Synthesis
Recent AI research plots a promising future of automatic chemical reactions within the chemistry society.
IG Captioner: Information Gain Captioners are Strong Zero-shot Classifiers
To tackle this problem, we redesign the scoring objective for the captioner to alleviate the distributional bias and focus on measuring the gain of information brought by the visual inputs.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation
Additionally, Parrot employs a joint optimization approach for the T2I model and the prompt expansion network, facilitating the generation of quality-aware text prompts, thus further enhancing the final image quality.
