Search Results for author:
Found 464 papers, 289 papers with code
Mining Inter-Video Proposal Relations for Video Object Detection
Recent studies have shown that, context aggregating information from proposals in different frames can clearly enhance the performance of video object detection.
Ranked #13 on
Video Object Detection
on ImageNet VID
The Best of Both Worlds: Combining Engineered Features with Transformers for Improved Mental Health Prediction from Reddit Posts
In recent years, there has been increasing interest in the application of natural language processing and machine learning techniques to the detection of mental health conditions (MHC) based on social media data.
MANTIS at SMM4H’2022: Pre-Trained Language Models Meet a Suite of Psycholinguistic Features for the Detection of Self-Reported Chronic Stress
This paper describes our submission to Social Media Mining for Health (SMM4H) 2022 Shared Task 8, aimed at detecting self-reported chronic stress on Twitter.
SPADE: A Big Five-Mturk Dataset of Argumentative Speech Enriched with Socio-Demographics for Personality Detection
In recent years, there has been increasing interest in automatic personality detection based on language.
Automated Classification of Written Proficiency Levels on the CEFR-Scale through Complexity Contours and RNNs
The key to the present paper is the combined use of what we refer to as ‘complexity contours’, a series of measurements of indices of L2 proficiency obtained by a computational tool that implements a sliding window technique, and recurrent neural network (RNN) classifiers that adequately capture the sequential information in those contours.
RBF-Softmax: Learning Deep Representative Prototypes with Radial Basis Function Softmax
To address this problem, this paper introduces a novel Radial Basis Function (RBF) distances to replace the commonly used inner products in the softmax loss function, such that it can adaptively assign losses to regularize the intra-class and inter-class distances by reshaping the relative differences, and thus creating more representative prototypes of classes to improve optimization.
Language that Captivates the Audience: Predicting Affective Ratings of TED Talks in a Multi-Label Classification Task
The aim of the paper is twofold: (1) to automatically predict the ratings assigned by viewers to 14 categories available for TED talks in a multi-label classification task and (2) to determine what types of features drive classification accuracy for each of the categories.
A Language-Based Approach to Fake News Detection Through Interpretable Features and BRNN
We demonstrate that our approach is promising as it achieves similar results on these two datasets as the best performing black box models reported in the literature.
FANG-COVID: A New Large-Scale Benchmark Dataset for Fake News Detection in German
As the world continues to fight the COVID-19 pandemic, it is simultaneously fighting an ‘infodemic’ – a flood of disinformation and spread of conspiracy theories leading to health threats and the division of society.
GigaGS: Scaling up Planar-Based 3D Gaussians for Large Scene Surface Reconstruction
To this end, we propose GigaGS, the first work for high-quality surface reconstruction for large-scale scenes using 3DGS.
A Preliminary Exploration Towards General Image Restoration
Existing deep models, tailored for specific individual image restoration tasks, often fall short in effectively addressing these challenges.
MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration
Despite recent advancements in text-to-image generation, most existing methods struggle to create images with multiple objects and complex spatial relationships in 3D world.
Learning A Low-Level Vision Generalist via Visual Task Prompt
In addition, these methods are sensitive to prompt image content and often struggle with low-frequency information processing.
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields.
VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge
The need for improved diagnostic methods in ophthalmology is acute, especially in the less developed regions with limited access to specialists and advanced equipment.
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions.
MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
To fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks.
Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model
To tackle this challenge, we introduce Diffree, a Text-to-Image (T2I) model that facilitates text-guided object addition with only text control.
Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models
Diffusion models have achieved great progress in image animation due to powerful generative capabilities.
MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3. 5, and manual correction.
ViLLa: Video Reasoning Segmentation with Large Language Model
To bridge the gap between image and video, in this work, we propose a new video segmentation task - video reasoning segmentation.
The Better Angels of Machine Personality: How Personality Relates to LLM Safety
In this paper, we discover that LLMs' personality traits are closely related to their safety abilities, i. e., toxicity, privacy, and fairness, based on the reliable MBTI-M scale.
GRIDS: Grouped Multiple-Degradation Restoration with Image Degradation Similarity
Based on the degradation similarity, GRIDS divides restoration tasks into one of the optimal groups, where tasks within the same group are highly correlated.
Building Intelligence Identification System via Large Language Model Watermarking: A Survey and Beyond
Large Language Models (LLMs) are increasingly integrated into diverse industries, posing substantial security risks due to unauthorized replication and misuse.
GRUtopia: Dream General Robots in a City at Scale
Recent works have been exploring the scaling laws in the field of Embodied AI.
Data Adaptive Traceback for Vision-Language Foundation Models in Image Classification
Due to the limitations of these weak-paired samples, the pre-training model are unable to mine all the knowledge from pre-training data.
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
Extensive experiments demonstrate that EfficientQAT outperforms previous quantization methods across a range of models, including base LLMs, instruction-tuned LLMs, and multimodal LLMs, with scales from 7B to 70B parameters at various quantization bits.
VEnhancer: Generative Space-Time Enhancement for Video Generation
Given a generated low-quality video, our approach can increase its spatial and temporal resolution simultaneously with arbitrary up-sampling space and time scales through a unified video diffusion model.
LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages
To address this, we dedicate 35, 000 A100-SXM4-80GB GPU hours in conducting extensive multilingual continual pre-training on the LLaMA series models, enabling translation support across more than 100 languages.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
This long-context capability allows IXC-2. 5 to excel in tasks requiring extensive input and output contexts.
Ranked #25 on
Visual Question Answering
on MM-Vet
EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation
In this report, we present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge.
Ranked #1 on
Long Term Action Anticipation
on Ego4D
(using extra training data)
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions.
Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Models
One of the primary driving forces contributing to the superior performance of Large Language Models (LLMs) is the extensive availability of human-annotated natural language data, which is used for alignment fine-tuning.
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
Generating natural and meaningful responses to communicate with multi-modal human inputs is a fundamental capability of Large Vision-Language Models(LVLMs).
Ranked #60 on
Visual Question Answering
on MM-Vet
PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models
Our findings reveal that: (1) even advanced models frequently err in various physical scenarios, except for optics; (2) GPT-4o, with item-specific scoring instructions, effectively evaluates the models' understanding of physical commonsense, closely aligning with human assessments; and (3) current T2I models are primarily focused on text-to-image translation, lacking profound reasoning regarding physical commonsense.
DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models
Scientific documents record research findings and valuable human knowledge, comprising a vast corpus of high-quality data.
SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model
To address these limitations, we propose a Safety Preference Alignment dataset for Vision Language Models named SPA-VL.
Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicality
To address these challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a comprehensive and standardized protocol for T2V models.
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
It not only allows flexible transmission of task information and gradient feedback between the MLLM and multiple downstream decoders but also effectively resolves training conflicts in multi-tasking scenarios.
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Smartphone users often navigate across multiple applications (apps) to complete tasks such as sharing content between social media platforms.
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
In this paper, we introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset.
MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
In this paper, we present MLLMGuard, a multidimensional safety evaluation suite for MLLMs, including a bilingual image-text evaluation dataset, inference utilities, and a lightweight evaluator.
Needle In A Multimodal Haystack
In this work, we present Needle In A Multimodal Haystack (MM-NIAH), the first benchmark specifically designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.
Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data.
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length.
Parameter-Inverted Image Pyramid Networks
Our core idea is to use models with different parameter sizes to process different resolution levels of the image pyramid, thereby balancing computational efficiency and performance.
Learning 1D Causal Visual Representation with De-focus Attention Networks
The issue of "over-focus" hinders the model's ability to extract diverse visual features and to receive effective gradients for optimization.
Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT
Lumina-T2X is a nascent family of Flow-based Large Diffusion Transformers that establishes a unified framework for transforming noise into various modalities, such as images and videos, conditioned on text instructions.
Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
However, training these two stages separately leads to significant data bias in the inference phase, thus affecting the quality of reconstructed results.
Learning Manipulation by Predicting Interaction
To this end, we propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction (MPI) and enhances the visual representation. Given a pair of keyframes representing the initial and final states, along with language instructions, our algorithm predicts the transition frame and detects the interaction object, respectively.
4Diffusion: Multi-view Video Diffusion Model for 4D Generation
We first design a unified diffusion model tailored for multi-view video generation by incorporating a learnable motion module into a frozen 3D-aware diffusion model to capture multi-view spatial-temporal correlations.
Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
In this work, we introduce $\textit{weak-to-strong search}$, framing the alignment of a large language model as a test-time greedy search to maximize the log-likelihood difference between small tuned and untuned models while sampling from the frozen large model.
Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving
Experiments also demonstrate that as the memory bank expands, the Heuristic Process with only 1. 8B parameters can inherit the knowledge from a GPT-4 powered Analytic Process and achieve continuous performance improvement.
SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge
In this paper, we propose a plug-and-play framework, for augmenting existing LVLMs in handling visual question answering (VQA) about up-to-date knowledge, dubbed SearchLVLMs.
FLoRA: Low-Rank Core Space for N-dimension
To tackle the diversity of dimensional spaces across different foundation models and provide a more precise representation of the changes within these spaces, this paper introduces a generalized parameter-efficient fine-tuning framework, FLoRA, designed for various dimensional parameter space.
Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
Sora unveils the potential of scaling Diffusion Transformer for generating photorealistic images and videos at arbitrary resolutions, aspect ratios, and durations, yet it still lacks sufficient implementation details.
Causal Evaluation of Language Models
Recent advances in language models have expanded the horizons of artificial intelligence across various domains, sparking inquiries into their potential for causal reasoning.
Towards Real-world Video Face Restoration: A New Benchmark
Blind face restoration (BFR) on images has significantly progressed over the last several years, while real-world video face restoration (VFR), which is more challenging for more complex face motions such as moving gaze directions and facial orientations involved, remains unsolved.
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
Compared to both open-source and proprietary models, InternVL 1. 5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks.
Ranked #10 on
Visual Question Answering
on MM-Vet v2
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation.
FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneity
First, we employ clustering on the local representations of each client, aiming to capture intra-class information based on these local clusters at a high level of granularity.
Logit Calibration and Feature Contrast for Robust Federated Learning on Non-IID Data
Federated learning (FL) is a privacy-preserving distributed framework for collaborative model training on devices in edge networks.
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution.
Ranked #19 on
Visual Question Answering
on MM-Vet
DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement
We present Dive Into the BoundarieS (DIBS), a novel pretraining framework for dense video captioning (DVC), that elaborates on improving the quality of the generated event captions and their associated pseudo event boundaries from unlabeled videos.
Linear Attention Sequence Parallelism
In this paper, we introduce Linear Attention Sequence Parallel (LASP), an efficient SP method tailored to linear attention-based language models.
LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction
LLaMA-Excitor ensures a self-adaptive allocation of additional attention to input instructions, thus effectively preserving LLMs' pre-trained knowledge when fine-tuning LLMs on low-quality instruction-following datasets.
VideoDistill: Language-aware Vision Distillation for Video Question Answering
In this paper, we are inspired by the human recognition and learning pattern and propose VideoDistill, a framework with language-aware (i. e., goal-driven) behavior in both vision perception and answer generation process.
DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model
Text-to-image (T2I) generative models have attracted significant attention and found extensive applications within and beyond academic research.
Are We on the Right Way for Evaluating Large Vision-Language Models?
We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.
Within the Dynamic Context: Inertia-aware 3D Human Modeling with Pose Sequence
Neural rendering techniques have significantly advanced 3D human body modeling.
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
The ultimate goals of robotic learning is to acquire a comprehensive and generalizable robotic system capable of performing both seen skills within the training distribution and unseen skills in novel environments.
InternLM2 Technical Report
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI).
Ranked #5 on
Long-Context Understanding
on Ada-LEval (BestAnswer)
Assessment of Multimodal Large Language Models in Alignment with Human Values
Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh).
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
Along with the videos we record high-quality gaze data and provide detailed multimodal annotations, formulating a playground for modeling the human ability to bridge asynchronous procedural actions from different viewpoints.
Ranked #1 on
Action Anticipation
on EgoExoLearn
(using extra training data)
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue.
Ranked #1 on
Video Question Answering
on Perception Test
DreamDA: Generative Data Augmentation with Diffusion Models
The acquisition of large-scale, high-quality data is a resource-intensive and time-consuming endeavor.
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
It is a long-lasting goal to design a generalist-embodied agent that can follow diverse instructions in human-like ways.
AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Adversarial Visual-Instructions
To bridge this gap, we introduce AVIBench, a framework designed to analyze the robustness of LVLMs when facing various adversarial visual-instructions (AVIs), including four types of image-based AVIs, ten types of text-based AVIs, and nine types of content bias AVIs (such as gender, violence, cultural, and racial biases, among others).
Desigen: A Pipeline for Controllable Design Template Generation
In this paper, we present Desigen, an automatic template creation pipeline which generates background images as well as harmonious layout elements over the background.
GenAD: Generalized Predictive Model for Autonomous Driving
In this paper, we introduce the first large-scale video prediction model in the autonomous driving discipline.
VideoMamba: State Space Model for Efficient Video Understanding
Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain.
Embodied Understanding of Driving Scenarios
Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans.
Towards Robust Federated Learning via Logits Calibration on Non-IID Data
Meanwhile, the non-independent and identically distributed (non-IID) challenge of data distribution between edge devices can further degrade the performance of models.
Position: Towards Implicit Prompt For Text-To-Image Models
We call for increased attention to the potential and risks of implicit prompts in the T2I community and further investigation into the capabilities and impacts of implicit prompts, advocating for a balanced approach that harnesses their benefits while mitigating their risks.
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures
Our evaluations demonstrate that VRWKV surpasses ViT's performance in image classification and has significantly faster speeds and lower memory usage processing high-resolution inputs.
Efficient Action Counting with Dynamic Queries
Temporal repetition counting aims to quantify the repeated action cycles within a video.
Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition
Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks.
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
In addition, we design a new benchmark, termed Circular-based Relation Probing Evaluation (CRPE) for comprehensively evaluating the relation comprehension capabilities of MLLMs.
Ranked #49 on
Visual Question Answering
on MM-Vet
WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset
To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb.
Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
This research provides an initial exploration of trustworthiness modeling during LLM pre-training, seeking to unveil new insights and spur further developments in the field.
Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
Language-conditioned robot behavior plays a vital role in executing complex tasks by associating human commands or instructions with perception and actions.
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI.
Ranked #112 on
Visual Question Answering
on MM-Vet
RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
To bridge this ``ideal-to-real'' gap, this paper presents \textbf{RobotScript}, a platform for 1) a deployable robot manipulation pipeline powered by code generation; and 2) a code generation benchmark for robot manipulation tasks in free-form natural language.
ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
Recently, many versatile Multi-modal Large Language Models (MLLMs) have emerged continuously.
Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
Large language models (LLMs) undergo safety alignment to ensure safe conversations with humans.
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
Large language models (LLMs) have demonstrated outstanding performance in various tasks, such as text summarization, text question-answering, and etc.
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey
Large Language Models (LLMs) are now commonplace in conversation applications.
OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs.
Real-time Holistic Robot Pose Estimation with Unknown States
Estimating robot pose from RGB images is a crucial problem in computer vision and robotics.
Ranked #1 on
Robot Pose Estimation
on DREAM-dataset
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX.
Ranked #9 on
Video Question Answering
on MVBench
SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
In the rapidly evolving landscape of Large Language Models (LLMs), ensuring robust safety measures is paramount.
Safety of Multimodal Large Language Models on Images and Texts
In this paper, we systematically survey current efforts on the evaluation, attack, and defense of MLLMs' safety on images and text.
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension.
Ranked #14 on
Visual Question Answering
on MM-Vet v2
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
CO2 is able to attain a high scalability even on extensive multi-node clusters constrained by very limited communication bandwidth.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality
In this paper, we propose Multimodal Federated Cross Prototype Learning (MFCPL), a novel approach for MFL under severely missing modalities by conducting the complete prototypes to provide diverse modality knowledge in modality-shared level with the cross-modal regularization and modality-specific level with cross-modal contrastive mechanism.
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild
We introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up.
SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning
Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning.
PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety
In this paper, we explore these concerns through the innovative lens of agent psychology, revealing that the dark psychological states of agents constitute a significant threat to safety.
MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer
Developing generative models for interleaved image-text data has both research and practical value.
Vlogger: Make Your Dream A Vlog
More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor.
Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications
The advancements in speed and efficiency of DCNv4, combined with its robust performance across diverse vision tasks, show its potential as a foundational building block for future vision models.
Latte: Latent Diffusion Transformer for Video Generation
We propose a novel Latent Diffusion Transformer, namely Latte, for video generation.
ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning
Charts play a vital role in data visualization, understanding data patterns, and informed decision-making.
ScoreHypo: Probabilistic Human Mesh Estimation with Hypothesis Scoring
Experimental results demonstrate that HypoNet outperforms existing state-of-the-art probabilistic methods as a multi-hypothesis mesh estimator.
Language-aware Visual Semantic Distillation for Video Question Answering
In this paper we are inspired by the human recognition and learning pattern and propose VideoDistill a framework with language-aware (i. e. goal-driven) behavior in both vision perception and answer generation process.
Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models
Based on this observation we generalize diffusion methods to model font generative process by separating the reverse diffusion process into three stages with different functions: The structure construction stage first generates the structure information for the target character based on the source image and the font transfer stage subsequently transforms the source font to the target font.
Point Transformer V3: Simpler Faster Stronger
This paper is not motivated to seek innovation within the attention mechanism.
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT-full
(using extra training data)
Critic-Guided Decision Transformer for Offline Reinforcement Learning
Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner.
M-BEV: Masked BEV Perception for Robust Autonomous Driving
It mimics various missing cases by randomly masking features of different camera views, then leverages the original features of these views as self-supervision, and reconstructs the masked ones with the distinct spatio-temporal context across views.
Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model
To tackle this problem, we propose Diff-Text, which is a training-free scene text generation framework for any language.
Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey
The advent of foundation models, which are pre-trained on vast datasets, has ushered in a new era of computer vision, characterized by their robustness and remarkable zero-shot generalization capabilities.
Point Transformer V3: Simpler, Faster, Stronger
This paper is not motivated to seek innovation within the attention mechanism.
Ranked #1 on
3D Semantic Segmentation
on ScanNet++
(using extra training data)
DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD).