Search Results for author:
Found 36 papers, 26 papers with code
Seed1.5-VL Technical Report
We present Seed1. 5-VL, a vision-language foundation model designed to advance general-purpose multimodal understanding and reasoning.
Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines
To address this limitation, we propose Vision Search Assistant, a novel framework that facilitates collaboration between VLMs and web agents.
Scaling Up Your Kernels: Large Kernel Design in ConvNets towards Universal Representations
This paper proposes the paradigm of large convolutional kernels in designing modern Convolutional Neural Networks (ConvNets).
CounterQuill: Investigating the Potential of Human-AI Collaboration in Online Counterspeech Writing
Online hate speech has become increasingly prevalent on social media platforms, causing harm to individuals and society.
Quantized Prompt for Efficient Generalization of Vision-Language Models
During downstream adaptation, the most challenging problems are overfitting and catastrophic forgetting, which can cause the model to overly focus on the current data and lose more crucial domain-general knowledge.
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications.
Leveraging Prompt-Based Large Language Models: Predicting Pandemic Health Decisions and Outcomes Through Social Media Language
We introduce a multi-step reasoning framework using prompt-based LLMs to examine the relationship between social media language patterns and trends in national health outcomes.
InteractiveVideo: User-Centric Controllable Video Generation with Synergistic Multimodal Instructions
We introduce $\textit{InteractiveVideo}$, a user-centric framework for video generation.
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities
We propose to improve transformers of a specific modality with irrelevant data from other modalities, e. g., improve an ImageNet model with audio or point cloud datasets.
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio Video Point Cloud Time-Series and Image Recognition
1) We propose four architectural guidelines for designing large-kernel ConvNets the core of which is to exploit the essential characteristics of large kernels that distinguish them from small kernels - they can see wide without going deep.
Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs
We empirically show that sparse attention not only reduces computational demands but also enhances model performance in both NLP and multi-modal tasks.
Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability
To better understand the nature of prompt tuning, we propose the concept of ``Information Density'' (ID) to indicate whether a matrix strongly belongs to certain feature spaces rather than being evenly distributed across various feature spaces.
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
In combination with the existing text tokenizer and detokenizer, this framework allows for the encoding of interleaved image-text data into a multimodal sequence, which can subsequently be fed into the transformer model.
Online Vectorized HD Map Construction using Geometry
In our work, we propose GeMap ($\textbf{Ge}$ometry $\textbf{Map}$), which end-to-end learns Euclidean shapes and relations of map instances beyond basic perception.
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition
1) We propose four architectural guidelines for designing large-kernel ConvNets, the core of which is to exploit the essential characteristics of large kernels that distinguish them from small kernels - they can see wide without going deep.
Ranked #1 on
Object Detection
on COCO 2017
(mAP metric)
Advancing Vision Transformers with Group-Mix Attention
The attention map is computed based on the mixtures of tokens and group proxies and used to re-combine the tokens and groups in Value.
Towards Unified and Effective Domain Generalization
We propose $\textbf{UniDG}$, a novel and $\textbf{Uni}$fied framework for $\textbf{D}$omain $\textbf{G}$eneralization that is capable of significantly enhancing the out-of-distribution generalization performance of foundation models regardless of their architectures.
Ranked #1 on
Domain Generalization
on TerraIncognita
RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNets
We propose Re-parameterized Refocusing Convolution (RefConv) as a replacement for regular convolutional layers, which is a plug-and-play module to improve the performance without any inference costs.
Sticker820K: Empowering Interactive Retrieval with Stickers
Therefore, we propose StickerCLIP as a benchmark model on the Sticker820K dataset.
Evolving Semantic Prototype Improves Generative Zero-Shot Learning
After alignment, synthesized sample features from unseen classes are closer to the real sample features and benefit DSP to improve existing generative ZSL methods by 8. 5\%, 8. 0\%, and 9. 7\% on the standard CUB, SUN AWA2 datasets, the significant performance improvement indicates that evolving semantic prototype explores a virgin field in ZSL.
What Makes for Good Visual Tokenizers for Large Language Models?
In our benchmark, which is curated to evaluate MLLMs visual semantic understanding and fine-grained perception capabilities, we discussed different visual tokenizers pre-trained with dominant methods (i. e., DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models capture more semantics than self-supervised models, but the gap is narrowed by scaling up the pre-training dataset.
ToxVis: Enabling Interpretability of Implicit vs. Explicit Toxicity Detection Models with Interactive Visualization
By making the classification process explainable, ToxVis provides a valuable tool for understanding the nuances of hateful content and supporting more effective content moderation.
Same Words, Different Meanings: Semantic Polarization in Broadcast Media Language Forecasts Polarization on Social Media Discourse
By analyzing a decade's worth of closed captions (2 million speaker turns) from CNN and Fox News along with topically corresponding discourse from Twitter, we provide a novel framework for measuring semantic polarization between America's two major broadcast networks to demonstrate how semantic polarization between these outlets has evolved (Study 1), peaked (Study 2) and influenced partisan discussions on Twitter (Study 3) across the last decade.
Re-parameterizing Your Optimizers rather than Architectures
For the extreme simplicity of model structure, we focus on a VGG-style plain model and showcase that such a simple model trained with a RepOptimizer, which is referred to as RepOpt-VGG, performs on par with or better than the recent well-designed models.
Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs
We revisit large kernel design in modern convolutional neural networks (CNNs).
Ranked #68 on
Image Classification
on ImageNet
(Top 1 Accuracy metric)
A Likelihood Ratio based Domain Adaptation Method for E2E Models
In this work, we explore a contextual biasing approach using likelihood-ratio that leverages text data sources to adapt RNN-T model to new domains and entities.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality
Our results reveal that 1) Locality Injection is a general methodology for MLP models; 2) RepMLPNet has favorable accuracy-efficiency trade-off compared to the other MLPs; 3) RepMLPNet is the first MLP that seamlessly transfer to Cityscapes semantic segmentation.
Ranked #69 on
Semantic Segmentation
on Cityscapes val
Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops.
RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
We propose RepMLP, a multi-layer-perceptron-style neural network building block for image recognition, which is composed of a series of fully-connected (FC) layers.
Ranked #1 on
Image Classification
on ImageNet
(Hardware Burden metric)
Diverse Branch Block: Building a Convolution as an Inception-like Unit
We propose a universal building block of Convolutional Neural Network (ConvNet) to improve the performance without any inference-time costs.
RepVGG: Making VGG-style ConvNets Great Again
We present a simple but powerful architecture of convolutional neural network, which has a VGG-like inference-time body composed of nothing but a stack of 3x3 convolution and ReLU, while the training-time model has a multi-branch topology.
Ranked #48 on
Semantic Segmentation
on Cityscapes val
ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting
Via training with regular SGD on the former but a novel update rule with penalty gradients on the latter, we realize structured sparsity.
Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices.
ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
We propose Asymmetric Convolution Block (ACB), an architecture-neutral structure as a CNN building block, which uses 1D asymmetric convolutions to strengthen the square convolution kernels.
Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
It is not easy to design and run Convolutional Neural Networks (CNNs) due to: 1) finding the optimal number of filters (i. e., the width) at each layer is tricky, given an architecture; and 2) the computational intensity of CNNs impedes the deployment on computationally limited devices.
Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
The redundancy is widely recognized in Convolutional Neural Networks (CNNs), which enables to remove unimportant filters from convolutional layers so as to slim the network with acceptable performance drop.
