Search Results for author:
Found 28 papers, 22 papers with code
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications.
Leveraging Prompt-Based Large Language Models: Predicting Pandemic Health Decisions and Outcomes Through Social Media Language
We introduce a multi-step reasoning framework using prompt-based LLMs to examine the relationship between social media language patterns and trends in national health outcomes.
InteractiveVideo: User-Centric Controllable Video Generation with Synergistic Multimodal Instructions
We introduce $\textit{InteractiveVideo}$, a user-centric framework for video generation.
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities
We propose to improve transformers of a specific modality with irrelevant data from other modalities, e. g., improve an ImageNet model with audio or point cloud datasets.
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
In combination with the existing text tokenizer and detokenizer, this framework allows for the encoding of interleaved image-text data into a multimodal sequence, which can subsequently be fed into the transformer model.
Online Vectorized HD Map Construction using Geometry
In our work, we propose GeMap ($\textbf{Ge}$ometry $\textbf{Map}$), which end-to-end learns Euclidean shapes and relations of map instances beyond basic perception.
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition
1) We propose four architectural guidelines for designing large-kernel ConvNets, the core of which is to exploit the essential characteristics of large kernels that distinguish them from small kernels - they can see wide without going deep.
Ranked #1 on
Object Detection
on COCO 2017
(mAP metric)
Advancing Vision Transformers with Group-Mix Attention
The attention map is computed based on the mixtures of tokens and group proxies and used to re-combine the tokens and groups in Value.
Towards Unified and Effective Domain Generalization
We propose $\textbf{UniDG}$, a novel and $\textbf{Uni}$fied framework for $\textbf{D}$omain $\textbf{G}$eneralization that is capable of significantly enhancing the out-of-distribution generalization performance of foundation models regardless of their architectures.
Ranked #1 on
Domain Generalization
on TerraIncognita
RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNets
We propose Re-parameterized Refocusing Convolution (RefConv) as a replacement for regular convolutional layers, which is a plug-and-play module to improve the performance without any inference costs.
Evolving Semantic Prototype Improves Generative Zero-Shot Learning
After alignment, synthesized sample features from unseen classes are closer to the real sample features and benefit DSP to improve existing generative ZSL methods by 8. 5\%, 8. 0\%, and 9. 7\% on the standard CUB, SUN AWA2 datasets, the significant performance improvement indicates that evolving semantic prototype explores a virgin field in ZSL.
Sticker820K: Empowering Interactive Retrieval with Stickers
Therefore, we propose StickerCLIP as a benchmark model on the Sticker820K dataset.
What Makes for Good Visual Tokenizers for Large Language Models?
In our benchmark, which is curated to evaluate MLLMs visual semantic understanding and fine-grained perception capabilities, we discussed different visual tokenizers pre-trained with dominant methods (i. e., DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models capture more semantics than self-supervised models, but the gap is narrowed by scaling up the pre-training dataset.
ToxVis: Enabling Interpretability of Implicit vs. Explicit Toxicity Detection Models with Interactive Visualization
By making the classification process explainable, ToxVis provides a valuable tool for understanding the nuances of hateful content and supporting more effective content moderation.
Same Words, Different Meanings: Semantic Polarization in Broadcast Media Language Forecasts Polarization on Social Media Discourse
By analyzing a decade's worth of closed captions (2 million speaker turns) from CNN and Fox News along with topically corresponding discourse from Twitter, we provide a novel framework for measuring semantic polarization between America's two major broadcast networks to demonstrate how semantic polarization between these outlets has evolved (Study 1), peaked (Study 2) and influenced partisan discussions on Twitter (Study 3) across the last decade.
Re-parameterizing Your Optimizers rather than Architectures
For the extreme simplicity of model structure, we focus on a VGG-style plain model and showcase that such a simple model trained with a RepOptimizer, which is referred to as RepOpt-VGG, performs on par with or better than the recent well-designed models.
Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs
We revisit large kernel design in modern convolutional neural networks (CNNs).
Ranked #75 on
Image Classification
on ImageNet
A Likelihood Ratio based Domain Adaptation Method for E2E Models
In this work, we explore a contextual biasing approach using likelihood-ratio that leverages text data sources to adapt RNN-T model to new domains and entities.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality
Our results reveal that 1) Locality Injection is a general methodology for MLP models; 2) RepMLPNet has favorable accuracy-efficiency trade-off compared to the other MLPs; 3) RepMLPNet is the first MLP that seamlessly transfer to Cityscapes semantic segmentation.
Ranked #58 on
Semantic Segmentation
on Cityscapes val
Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops.
RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
We propose RepMLP, a multi-layer-perceptron-style neural network building block for image recognition, which is composed of a series of fully-connected (FC) layers.
Ranked #753 on
Image Classification
on ImageNet
Diverse Branch Block: Building a Convolution as an Inception-like Unit
We propose a universal building block of Convolutional Neural Network (ConvNet) to improve the performance without any inference-time costs.
RepVGG: Making VGG-style ConvNets Great Again
We present a simple but powerful architecture of convolutional neural network, which has a VGG-like inference-time body composed of nothing but a stack of 3x3 convolution and ReLU, while the training-time model has a multi-branch topology.
Ranked #42 on
Semantic Segmentation
on Cityscapes val
ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting
Via training with regular SGD on the former but a novel update rule with penalty gradients on the latter, we realize structured sparsity.
Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices.
ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
We propose Asymmetric Convolution Block (ACB), an architecture-neutral structure as a CNN building block, which uses 1D asymmetric convolutions to strengthen the square convolution kernels.
Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
It is not easy to design and run Convolutional Neural Networks (CNNs) due to: 1) finding the optimal number of filters (i. e., the width) at each layer is tricky, given an architecture; and 2) the computational intensity of CNNs impedes the deployment on computationally limited devices.
Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
The redundancy is widely recognized in Convolutional Neural Networks (CNNs), which enables to remove unimportant filters from convolutional layers so as to slim the network with acceptable performance drop.
