Search Results for author:
Found 413 papers, 248 papers with code
Knowledge-based Fully Convolutional Network and Its Application in Segmentation of Lung CT Images
A variety of deep neural networks have been applied in medical image segmentation and achieve good performance.
Boosting up Scene Text Detectors with Guided CNN
Deep CNNs have achieved great success in text detection.
Deep Embedding Convolutional Neural Network for Synthesizing CT Image from T1-Weighted MR Image
After repeat-ing this embedding procedure for several times in the network, we can eventually synthesize a final CT image in the end of the DECNN.
Locally-Supervised Deep Hybrid Model for Scene Recognition
Secondly, we propose a new Local Convolutional Supervision (LCS) layer to enhance the local structure of the image by directly propagating the label information to the convolutional layers.
Transferring Object-Scene Convolutional Neural Networks for Event Recognition in Still Images
These newly designed transferring techniques exploit multi-task learning frameworks to incorporate extra knowledge from other networks and additional datasets into the training procedure of event CNNs.
DeepWriter: A Multi-Stream Deep CNN for Text-independent Writer Identification
The main contributions are: 1) we design and optimize multi-stream structure for writer identification task; 2) we introduce data augmentation learning to enhance the performance of DeepWriter; 3) we introduce a patch scanning strategy to handle text image with different lengths.
Actionness Estimation Using Hybrid Fully Convolutional Networks
Actionness was introduced to quantify the likelihood of containing a generic action instance at a specific location.
Ranked #11 on
Action Detection
on J-HMDB
Text-Attentional Convolutional Neural Networks for Scene Text Detection
The rich supervision information enables the Text-CNN with a strong capability for discriminating ambiguous texts, and also increases its robustness against complicated background components.
Reading Scene Text in Deep Convolutional Sequences
We develop a Deep-Text Recurrent Network (DTRN) that regards scene text reading as a sequence labelling problem.
Better Exploiting OS-CNNs for Better Event Recognition in Images
Event recognition from still images is one of the most important problems for image understanding.
Local Multi-Grouped Binary Descriptor with Ring-based Pooling Configuration and Optimization
The performance of RMGD was evaluated on a number of publicly available benchmarks, where the RMGD outperforms the state-of-the-art binary descriptors significantly.
Local Color Contrastive Descriptor for Image Classification
Our descriptor enriches local image representation with both color and contrast information.
Boosting Optical Character Recognition: A Super-Resolution Approach
Text image super-resolution is a challenging yet open research problem in the computer vision community.
Object-Scene Convolutional Neural Networks for Event Recognition in Images
Meanwhile, we investigate different network architectures for OS-CNN design, and adapt the deep (AlexNet) and very-deep (GoogLeNet) networks to the task of event recognition.
Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice
Many efforts have been made in each step independently in different scenarios and their effect on action recognition is still unknown.
A Study on Unsupervised Dictionary Learning and Feature Encoding for Action Classification
Many efforts have been devoted to develop alternative methods to traditional vector quantization in image domain such as sparse coding and soft-assignment.
Prostate Segmentation using 2D Bridged U-net
To address the aforementioned three problems, we propose and validate a deeper network that can fit medical image datasets that are usually small in the sample size.
PIRM Challenge on Perceptual Image Enhancement on Smartphones: Report
This paper reviews the first challenge on efficient perceptual image enhancement with the focus on deploying deep learning models on smartphones.
Super-Identity Convolutional Neural Network for Face Hallucination
Face hallucination is a generative task to super-resolve the facial image with low resolution while human perception of face heavily relies on identity information.
Temporal Hallucinating for Action Recognition With Few Still Images
To mimic this capacity, we propose a novel Hybrid Video Memory (HVM) machine, which can hallucinate temporal features of still images from video memory, in order to boost action recognition with few still images.
Find and Focus: Retrieve and Localize Video Events with Natural Language Queries
The thriving of video sharing services brings new challenges to video retrieval, e. g. the rapid growth in video duration and content diversity.
Motionlets: Mid-level 3D Parts for Human Motion Recognition
We postulate three key properties of motionlet for action recognition: high motion saliency, multiple scale representation, and representative-discriminative ability.
Multi-View Super Vector for Action Recognition
Kernel average is then applied on these components to produce recognition result.
A Key Volume Mining Deep Framework for Action Recognition
Training with a large proportion of irrelevant volumes will hurt performance.
Latent Factor Guided Convolutional Neural Networks for Age-Invariant Face Recognition
In order to address this problem, we propose a novel deep face recognition framework to learn the age-invariant deep face features through a carefully designed CNN model.
Ranked #7 on
Age-Invariant Face Recognition
on CACDVS
Detecting Faces Using Inside Cascaded Contextual CNN
Deep Convolutional Neural Networks (CNNs) achieve substantial improvements in face detection in the wild.
Range Loss for Deep Face Recognition With Long-Tailed Training Data
Unlike these work, this paper investigated how long-tailed data impact the training of face CNNs and develop a novel loss function, called range loss, to effectively utilize the tailed data in training process.
P2SGrad: Refined Gradients for Optimizing Deep Face Models
Cosine-based softmax losses significantly improve the performance of deep face recognition networks.
Suppressing Model Overfitting for Image Super-Resolution Networks
Large deep networks have demonstrated competitive performance in single image super-resolution (SISR), with a huge volume of data involved.
Bootstrap Model Ensemble and Rank Loss for Engagement Intensity Regression
Based on our winner solution last year, we mainly explore head features and body features with a bootstrap strategy and two novel loss functions in this paper.
Product Image Recognition with Guidance Learning and Noisy Supervision
Instead of collecting product images by labor-and time-intensive image capturing, we take advantage of the web and download images from the reviews of several e-commerce websites where the images are casually captured by consumers.
Learning Category Correlations for Multi-label Image Recognition with Graph Networks
In this paper, instead of using a pre-defined graph which is inflexible and may be sub-optimal for multi-label classification, we propose the A-GCN, which leverages the popular Graph Convolutional Networks with an Adaptive label correlation graph to model label dependencies.
Pose-Assisted Multi-Camera Collaboration for Active Object Tracking
In the system, each camera is equipped with two controllers and a switcher: The vision-based controller tracks targets based on observed images.
FD-GAN: Generative Adversarial Networks with Fusion-discriminator for Single Image Dehazing
With the proposed Fusion-discriminator which takes frequency information as additional priors, our model can generator more natural and realistic dehazed images with less color distortion and fewer artifacts.
Progressive Object Transfer Detection
Inspired by this procedure of learning to detect, we propose a novel Progressive Object Transfer Detection (POTD) framework.
Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units
Recurrent neural network (RNN) has been widely studied in sequence learning tasks, while the mainstream models (e. g., LSTM and GRU) rely on the gating mechanism (in control of how information flows between hidden states).
TTPP: Temporal Transformer with Progressive Prediction for Efficient Action Anticipation
Video action anticipation aims to predict future action categories from observed frames.
Text-attentional convolutional neural network for scene text detection
Recent deep learning models have demonstrated strong capabilities for classifying text and non-text components in natural images.
Learning to Predict Context-adaptive Convolution for Semantic Segmentation
Long-range contextual information is essential for achieving high-performance semantic segmentation.
COCAS: A Large-Scale Clothes Changing Person Dataset for Re-identification
To address the clothes changing person re-id problem, we construct a novel large-scale re-id benchmark named ClOthes ChAnging Person Set (COCAS), which provides multiple images of the same identity with different clothes.
Understanding the Dynamics of Second Language Writing through Keystroke Logging and Complexity Contours
The purpose of this paper is twofold: [1] to introduce, to our knowledge, the largest available resource of keystroke logging (KSL) data generated by Etherpad (https://etherpad. org/), an open-source, web-based collaborative real-time editor, that captures the dynamics of second language (L2) production and [2] to relate the behavioral data from KSL to indices of syntactic and lexical complexity of the texts produced obtained from a tool that implements a sliding window approach capturing the progression of complexity within a text.
Becoming Linguistically Mature: Modeling English and German Children's Writing Development Across School Grades
In this paper we employ a novel approach to advancing our understanding of the development of writing in English and German children across school grades using classification tasks.
Exploring Multi-Scale Feature Propagation and Communication for Image Super Resolution
Multi-scale techniques have achieved great success in a wide range of computer vision tasks.
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition
Second, the parameter frequency distribution is further adopted to guide the student network to learn the appearance modeling process from the teacher.
Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition
The audio-video based emotion recognition aims to classify a given video into basic emotions.
Facial Expression Recognition (FER)
Video Emotion Recognition
Multi-scale Information Assembly for Image Matting
Image matting is a long-standing problem in computer graphics and vision, mostly identified as the accurate estimation of the foreground in input images.
Unsupervised Person Re-Identification with Multi-Label Learning Guided Self-Paced Clustering
The multi-label learning module leverages a memory feature bank and assigns each image with a multi-label vector based on the similarities between the image and feature bank.
PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos
By coupling advanced 3D pose estimators and HMR in a serial or parallel manner, these two frameworks can effectively correct human mesh with guidance of a concise pose calibration module.
Ranked #4 on
3D Human Pose Estimation
on Surreal
Smart Scribbles for Image Mating
Image matting is an ill-posed problem that usually requires additional user input, such as trimaps or scribbles.
Very Lightweight Photo Retouching Network with Conditional Sequential Modulation
In practice, photo retouching can be accomplished by a series of image processing operations.
The Impact of ASR on the Automatic Analysis of Linguistic Complexity and Sophistication in Spontaneous L2 Speech
In recent years, automated approaches to assessing linguistic complexity in second language (L2) writing have made significant progress in gauging learner performance, predicting human ratings of the quality of learner productions, and benchmarking L2 development.
NTIRE 2021 Challenge on Perceptual Image Quality Assessment
This paper reports on the NTIRE 2021 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2021.
Neighbourhood-guided Feature Reconstruction for Occluded Person Re-Identification
Person images captured by surveillance cameras are often occluded by various obstacles, which lead to defective feature representation and harm person re-identification (Re-ID) performance.
Multiple Domain Experts Collaborative Learning: Multi-Source Domain Generalization For Person Re-Identification
In UDCL, a universal expert supervises the learning of domain experts and continuously gathers knowledge from all domain experts.
TSI: Temporal Saliency Integration for Video Action Recognition
Specifically, SME aims to highlight the motion-sensitive area through local-global motion modeling, where the saliency alignment and pyramidal feature difference are conducted successively between neighboring frames to capture motion dynamics with less noises caused by misaligned background.
Scalable Transformers for Neural Machine Translation
In this paper, we propose a novel Scalable Transformers, which naturally contains sub-Transformers of different scales and have shared parameters.
Alzheimer's Disease Detection from Spontaneous Speech through Combining Linguistic Complexity and (Dis)Fluency Features with Pretrained Language Models
In this paper, we combined linguistic complexity and (dis)fluency features with pretrained language models for the task of Alzheimer's disease detection of the 2021 ADReSSo (Alzheimer's Dementia Recognition through Spontaneous Speech) challenge.
Prior-Induced Information Alignment for Image Matting
Image matting is an ill-posed problem that aims to estimate the opacity of foreground pixels in an image.
Blind Image Super-Resolution: A Survey and Beyond
This paper serves as a systematic review on recent progress in blind image SR, and proposes a taxonomy to categorize existing methods into three different classes according to their ways of degradation modelling and the data used for solving the SR model.
RankSRGAN: Super Resolution Generative Adversarial Networks with Learning to Rank
To address the problem, we propose Super-Resolution Generative Adversarial Networks with Ranker (RankSRGAN) to optimize generator in the direction of different perceptual metrics.
Transferable Knowledge-Based Multi-Granularity Aggregation Network for Temporal Action Localization: Submission to ActivityNet Challenge 2021
This technical report presents an overview of our solution used in the submission to 2021 HACS Temporal Action Localization Challenge on both Supervised Learning Track and Weakly-Supervised Learning Track.
Learning Dynamical Human-Joint Affinity for 3D Pose Estimation in Videos
Graph Convolution Network (GCN) has been successfully used for 3D human pose estimation in videos.
Ranked #10 on
3D Human Pose Estimation
on HumanEva-I
A Novel Hybrid Convolutional Neural Network for Accurate Organ Segmentation in 3D Head and Neck CT Images
An essential step of RT planning is the accurate segmentation of various organs-at-risks (OARs) in HaN CT images.
Group Shift Pointwise Convolution for Volumetric Medical Image Segmentation
Recent studies have witnessed the effectiveness of 3D convolutions on segmenting volumetric medical images.
Self-Slimming Vision Transformer
Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs by dynamic token aggregation.
Estimating IRI based on pavement distress type, density, and severity: Insights from machine learning techniques
However, it is costly to measure IRI, and for this reason, certain road classes are excluded from IRI measurements at a network level.
Automated Classification of Written Proficiency Levels on the CEFR-Scale through Complexity Contours and RNNs
The key to the present paper is the combined use of what we refer to as ‘complexity contours’, a series of measurements of indices of L2 proficiency obtained by a computational tool that implements a sliding window technique, and recurrent neural network (RNN) classifiers that adequately capture the sequential information in those contours.
A Language-Based Approach to Fake News Detection Through Interpretable Features and BRNN
We demonstrate that our approach is promising as it achieves similar results on these two datasets as the best performing black box models reported in the literature.
Language that Captivates the Audience: Predicting Affective Ratings of TED Talks in a Multi-Label Classification Task
The aim of the paper is twofold: (1) to automatically predict the ratings assigned by viewers to 14 categories available for TED talks in a multi-label classification task and (2) to determine what types of features drive classification accuracy for each of the categories.
Prediction of Listener Perception of Argumentative Speech in a Crowdsourced Dataset Using (Psycho-)Linguistic and Fluency Features
One of the key communicative competencies is the ability to maintain fluency in monologic speech and the ability to produce sophisticated language to argue a position convincingly.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
INTERN: A New Learning Paradigm Towards General Vision
Enormous waves of technological innovations over the past several years, marked by the advances in AI technologies, are profoundly reshaping the industry and the society.
CPRAL: Collaborative Panoptic-Regional Active Learning for Semantic Segmentation
Acquiring the most representative examples via active learning (AL) can benefit many data-dependent computer vision tasks by minimizing efforts of image-level or pixel-wise annotations.
CP-Net: Contour-Perturbed Reconstruction Network for Self-Supervised Point Cloud Learning
To fill this gap, we propose a generic Contour-Perturbed Reconstruction Network (CP-Net), which can effectively guide self-supervised reconstruction to learn semantic content in the point cloud, and thus promote discriminative power of point cloud representation.
Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning
Especially, we develop a variant of ViT for 3D point cloud feature extraction, which also achieves comparable results with existing backbones when combined with our framework, and visualization of the attention maps show that our model does understand the point cloud by combining the global shape information and multiple local structural information, which is consistent with the inspiration of our representation learning method.
Hilbert Flattening: a Locality-Preserving Matrix Unfolding Method for Visual Discrimination
Flattening is essential in computer vision by converting multi-dimensional feature maps or images into one-dimensional vectors.
Measuring the Impact of (Psycho-)Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns
There is a growing interest in the combined use of NLP and machine learning methods to predict gaze patterns during naturalistic reading.
X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation
2) Squeeze Stage: X-Learner condenses the model to a reasonable size and learns the universal and generalizable representation for various tasks transferring.
Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition
Learning spatial-temporal relation among multiple actors is crucial for group activity recognition.
Pushing on Personality Detection from Verbal Behavior: A Transformer Meets Text Contours of Psycholinguistic Features
Research at the intersection of personality psychology, computer science, and linguistics has recently focused increasingly on modeling and predicting personality from language use.
Evaluating the Generalization Ability of Super-Resolution Networks
However, research on the generalization ability of Super-Resolution (SR) networks is currently absent.
Level 2 Autonomous Driving on a Single Device: Diving into the Devils of Openpilot
Equipped with a wide span of sensors, predominant autonomous driving solutions are becoming more modular-oriented for safe system design.
CP3: Unifying Point Cloud Completion by Pretrain-Prompt-Predict Paradigm
Inspired by prompting approaches from NLP, we creatively reinterpret point cloud generation and refinement as the prompting and predicting stages, respectively.
HQANN: Efficient and Robust Similarity Search for Hybrid Queries with Structured and Unstructured Constraints
The in-memory approximate nearest neighbor search (ANNS) algorithms have achieved great success for fast high-recall query processing, but are extremely inefficient when handling hybrid queries with unstructured (i. e., feature vectors) and structured (i. e., related attributes) constraints.
GenText: Unsupervised Artistic Text Generation via Decoupled Font and Texture Manipulation
Specifically, our current work incorporates three different stages, stylization, destylization, and font transfer, respectively, into a unified platform with a single powerful encoder network and two separate style generator networks, one for font transfer, the other for stylization and destylization.
Collaboration of Pre-trained Models Makes Better Few-shot Learner
In this paper, we propose CoMo, a Collaboration of pre-trained Models that incorporates diverse prior knowledge from various pre-training paradigms for better few-shot learning.
Low-Resolution Action Recognition for Tiny Actions Challenge
Second, these activities are naturally distributed in a long-tailed way.
SPADE: A Big Five-Mturk Dataset of Argumentative Speech Enriched with Socio-Demographics for Personality Detection
In recent years, there has been increasing interest in automatic personality detection based on language.
MANTIS at SMM4H’2022: Pre-Trained Language Models Meet a Suite of Psycholinguistic Features for the Detection of Self-Reported Chronic Stress
This paper describes our submission to Social Media Mining for Health (SMM4H) 2022 Shared Task 8, aimed at detecting self-reported chronic stress on Twitter.
The Best of Both Worlds: Combining Engineered Features with Transformers for Improved Mental Health Prediction from Reddit Posts
In recent years, there has been increasing interest in the application of natural language processing and machine learning techniques to the detection of mental health conditions (MHC) based on social media data.
Wider and Higher: Intensive Integration and Global Foreground Perception for Image Matting
This paper reviews recent deep-learning-based matting research and conceives our wider and higher motivation for image matting.
Hierarchical and Progressive Image Matting
In this paper, we propose an end-to-end Hierarchical and Progressive Attention Matting Network (HAttMatting++), which can better predict the opacity of the foreground from single RGB images without additional input.
VideoPipe 2022 Challenge: Real-World Video Understanding for Urban Pipe Inspection
For this reason, we propose to advance research areas of video understanding, with a shift from traditional action recognition to industrial anomaly analysis.
Stare at What You See: Masked Image Modeling without Reconstruction
However, unlike the low-level features such as pixel values, we argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact image. This raises one question: is reconstruction necessary in Masked Image Modeling (MIM) with a teacher model?
Improving Training and Inference of Face Recognition Models via Random Temperature Scaling
Intuitively, FR algorithms can benefit from both the estimation of uncertainty and the detection of out-of-distribution (OOD) samples.
Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE
This technical report briefly describes our JDExplore d-team's Vega v2 submission on the SuperGLUE leaderboard.
Ranked #1 on
Common Sense Reasoning
on ReCoRD
Improving the Generalizability of Text-Based Emotion Detection by Leveraging Transformers with Psycholinguistic Features
In recent years, there has been increased interest in building predictive models that harness natural language processing and machine learning techniques to detect emotions from various text sources, including social media posts, micro-blogs or news articles.
MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency
Besides, such scenes with progressive masking ratios can also serve to self-distill their intrinsic spatial consistency, requiring to learn the consistent representations from unmasked areas.
MANTIS at TSAR-2022 Shared Task: Improved Unsupervised Lexical Simplification with Pretrained Encoders
In this paper we present our contribution to the TSAR-2022 Shared Task on Lexical Simplification of the EMNLP 2022 Workshop on Text Simplification, Accessibility, and Readability.
(Psycho-)Linguistic Features Meet Transformer Models for Improved Explainable and Controllable Text Simplification
State-of-the-art text simplification (TS) systems adopt end-to-end neural network models to directly generate the simplified version of the input text, and usually function as a blackbox.
Exploring Hybrid and Ensemble Models for Multiclass Prediction of Mental Health Status on Social Media
In recent years, there has been a surge of interest in research on automatic mental health detection (MHD) from social media data leveraging advances in natural language processing and machine learning techniques.
Content Rating Classification for Fan Fiction
The problem is to take fan fiction text and determine the appropriate content rating.
Uncertainty-Estimation with Normalized Logits for Out-of-Distribution Detection
However, neural networks often suffer from the overconfidence issue, making high confidence for OOD data which are never seen during training process and may be irrelevant to training data, namely in-distribution (ID) data.
FCN+: Global Receptive Convolution Makes FCN Great Again
The motivation of GRC is that different channels of a convolutional filter can have different grid sampling locations across the whole input feature map.
Rethinking Range View Representation for LiDAR Segmentation
We show that, for the first time, a range view method is able to surpass the point, voxel, and multi-view fusion counterparts in the competing LiDAR semantic and panoptic segmentation benchmarks, i. e., SemanticKITTI, nuScenes, and ScribbleKITTI.
Ranked #4 on
3D Semantic Segmentation
on SemanticKITTI
A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?
As ChatGPT goes viral, generative AI (AIGC, a. k. a AI-generated content) has made headlines everywhere because of its ability to analyze and create text, images, and beyond.
Prototype Helps Federated Learning: Towards Faster Convergence
In this paper, a prototype-based federated learning framework is proposed, which can achieve better inference performance with only a few changes to the last global iteration of the typical federated learning process.
MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligence
The existing single prototype-based strategy represents a class by using the mean of the feature space.
STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training
However, the state-of-the-art models for medical image segmentation are still small-scale, with their parameters only in the tens of millions.
One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era
Overall, this work is the first to survey ChatGPT with a comprehensive review of its underlying technology, applications, and challenges.
Perception Imitation: Towards Synthesis-free Simulator for Autonomous Vehicles
We propose a perception imitation method to simulate results of a certain perception model, and discuss a new heuristic route of autonomous driving simulator without data synthesis.
Hierarchical Diffusion Autoencoders and Disentangled Image Manipulation
To mitigate those limitations, we propose Hierarchical Diffusion Autoencoders (HDAE) that exploit the fine-grained-to-abstract and lowlevel-to-high-level feature hierarchy for the latent space of diffusion models.
Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects Cannot Be Easily Detected
Meta AI Research has recently released SAM (Segment Anything Model) which is trained on a large segmentation dataset of over 1 billion masks.
Distilling Focal Knowledge From Imperfect Expert for 3D Object Detection
Specifically, a set of queries are leveraged to locate the instance-level areas for masked feature generation, to intensify feature representation ability in these areas.
Learning Weather-General and Weather-Specific Features for Image Restoration Under Multiple Adverse Weather Conditions
Inspired by this observation, we design an efficient unified framework with a two-stage training strategy to explore the weather-general and weather-specific features.
Denoising Diffusion Semantic Segmentation with Mask Prior Modeling
In this paper, we propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a recently-developed denoising diffusion generative model.
A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering
This is an ongoing project and we intend to update the manuscript on a regular basis.
Robustness of SAM: Segment Anything Under Corruptions and Beyond
Following by interpreting the effects of synthetic corruption as style changes, we proceed to conduct a comprehensive evaluation for its robustness against 15 types of common corruption.
Align, Adapt and Inject: Sound-guided Unified Image Generation
Then, we propose the audio adapter to adapt audio representation into an audio token enriched with specific semantics, which can be injected into a frozen T2I model flexibly.
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Video Question Answering (VideoQA) has been significantly advanced from the scaling of recent Large Language Models (LLMs).
Ranked #3 on
Temporal/Casual QA
on NExT-QA
(using extra training data)
Boosting Federated Learning Convergence with Prototype Regularization
As a distributed machine learning technique, federated learning (FL) requires clients to collaboratively train a shared model with an edge server without leaking their local data.
FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning
Bringing this concept into a system, we develop a distillation-based multimodal embedding knowledge transfer mechanism, namely FedMEKT, which allows the server and clients to exchange the joint knowledge of their learning models extracted from a small multimodal proxy dataset.
HTML: Hybrid Temporal-scale Multimodal Learning Framework for Referring Video Object Segmentation
To tackle this problem, we propose a concise Hybrid Temporal-scale Multimodal Learning (HTML) framework, which can effectively align lingual and visual features to discover core object semantics in the video, by learning multimodal interaction hierarchically from different temporal scales.
Ranked #6 on
Referring Video Object Segmentation
on Refer-YouTube-VOS
(using extra training data)
Multi-view Spectral Polarization Propagation for Video Glass Segmentation
In this paper, we present the first polarization-guided video glass segmentation propagation solution (PGVS-Net) that can robustly and coherently propagate glass segmentation in RGB-P video sequences.
UniFormerV2: Unlocking the Potential of Image ViTs for Video Understanding
The prolific performances of Vision Transformers (ViTs) in image tasks have prompted research into adapting the image ViTs for video tasks.
Exploring Counterfactual Alignment Loss towards Human-centered AI
In particular, we utilize the counterfactual generation's ability for causal attribution to introduce a novel loss called the CounterFactual Alignment (CF-Align) loss.
Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching
Current image-based keypoint detection methods for animal (including human) bodies and faces are generally divided into full-supervised and few-shot class-agnostic approaches.
On the Evaluation and Refinement of Vision-Language Instruction Tuning Datasets
Instead of evaluating the models directly, in this paper, we try to evaluate the Vision-Language Instruction-Tuning (VLIT) datasets.
Tree-Planner: Efficient Close-loop Task Planning with Large Language Models
This paper studies close-loop task planning, which refers to the process of generating a sequence of skills (a plan) to accomplish a specific goal while adapting the plan based on real-time observations.
Unifying Image Processing as Visual Prompting Question Answering
To address this issue, we propose a universal model for general image processing that covers image restoration, image enhancement, image feature extraction tasks, etc.
SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction
The goal is to generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos.
Asymmetric Masked Distillation for Pre-Training Small Foundation Models
And AMD achieves 73. 3% classification accuracy using the ViT-B model on the Something-in-Something V2 dataset, a 3. 7% improvement over the original ViT-B model from VideoMAE.
Ranked #20 on
Action Recognition
on Something-Something V2
Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models
Although Large Language Models (LLMs) demonstrate remarkable ability in processing and generating human-like text, they do have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language(e. g., chemical molecular formula).
Understanding Segment Anything Model: SAM is Biased Towards Texture Rather than Shape
Different from label-oriented recognition tasks, the SAM is trained to predict a mask for covering the object shape based on a promt.
DiffusionMat: Alpha Matting as Sequential Refinement Learning
In this paper, we introduce DiffusionMat, a novel image matting framework that employs a diffusion model for the transition from coarse to refined alpha mattes.
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities.
VideoBooth: Diffusion-based Video Generation with Image Prompts
In this paper, we study the task of video generation with image prompts, which provide more accurate and direct content control beyond the text prompts.
Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies
OpenAI's latest large vision-language model (LVLM), GPT-4V(ision), has piqued considerable interest for its potential in medical applications.
MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding
While several long-form VideoQA datasets have been introduced, the length of both videos used to curate questions and sub-clips of clues leveraged to answer those questions have not yet reached the criteria for genuine long-form video understanding.
EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
Generating multiview images from a single view facilitates the rapid generation of a 3D mesh conditioned on a single image.
Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft
Many reinforcement learning environments (e. g., Minecraft) provide only sparse rewards that indicate task completion or failure with binary values.
Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey
The advent of foundation models, which are pre-trained on vast datasets, has ushered in a new era of computer vision, characterized by their robustness and remarkable zero-shot generalization capabilities.
Critic-Guided Decision Transformer for Offline Reinforcement Learning
Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner.
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild
We introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up.
SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning
Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning.
Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality
In this paper, we propose Multimodal Federated Cross Prototype Learning (MFCPL), a novel approach for MFL under severely missing modalities by conducting the complete prototypes to provide diverse modality knowledge in modality-shared level with the cross-modal regularization and modality-specific level with cross-modal contrastive mechanism.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptation
Building embodied agents on integrating Large Language Models (LLMs) and Reinforcement Learning (RL) have revolutionized human-AI interaction: researchers can now leverage language instructions to plan decision-making for open-ended tasks.
RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
To bridge this ``ideal-to-real'' gap, this paper presents \textbf{RobotScript}, a platform for 1) a deployable robot manipulation pipeline powered by code generation; and 2) a code generation benchmark for robot manipulation tasks in free-form natural language.
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI.
Ranked #76 on
Visual Question Answering
on MM-Vet
Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
Language-conditioned robot behavior plays a vital role in executing complex tasks by associating human commands or instructions with perception and actions.
Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition
Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks.
WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset
To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb.
Towards Implicit Prompt For Text-To-Image Models
We call for increased attention to the potential and risks of implicit prompts in the T2I community and further investigation into the capabilities and impacts of implicit prompts, advocating for a balanced approach that harnesses their benefits while mitigating their risks.
Towards Robust Federated Learning via Logits Calibration on Non-IID Data
Meanwhile, the non-independent and identically distributed (non-IID) challenge of data distribution between edge devices can further degrade the performance of models.
Exploring Safety Generalization Challenges of Large Language Models via Code
The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse.
AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Adversarial Visual-Instructions
To bridge this gap, we introduce AVIBench, a framework designed to analyze the robustness of LVLMs when facing various adversarial visual-instructions (AVIs), including four types of image-based AVIs, ten types of text-based AVIs, and nine types of content bias AVIs (such as gender, violence, cultural, and racial biases, among others).
Desigen: A Pipeline for Controllable Design Template Generation
In this paper, we present Desigen, an automatic template creation pipeline which generates background images as well as harmonious layout elements over the background.
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
The ultimate goals of robotic learning is to acquire a comprehensive and generalizable robotic system capable of performing both seen skills within the training distribution and unseen skills in novel environments.
Within the Dynamic Context: Inertia-aware 3D Human Modeling with Pose Sequence
Neural rendering techniques have significantly advanced 3D human body modeling.
LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction
LLaMA-Excitor ensures a self-adaptive allocation of additional attention to input instructions, thus effectively preserving LLMs' pre-trained knowledge when fine-tuning LLMs on low-quality instruction-following datasets.
VideoDistill: Language-aware Vision Distillation for Video Question Answering
In this paper, we are inspired by the human recognition and learning pattern and propose VideoDistill, a framework with language-aware (i. e., goal-driven) behavior in both vision perception and answer generation process.
DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement
We present Dive Into the BoundarieS (DIBS), a novel pretraining framework for dense video captioning (DVC), that elaborates on improving the quality of the generated event captions and their associated pseudo event boundaries from unlabeled videos.
Logit Calibration and Feature Contrast for Robust Federated Learning on Non-IID Data
Federated learning (FL) is a privacy-preserving distributed framework for collaborative model training on devices in edge networks.
FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneity
First, we employ clustering on the local representations of each client, aiming to capture intra-class information based on these local clusters at a high level of granularity.
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation.
Towards Real-world Video Face Restoration: A New Benchmark
Blind face restoration (BFR) on images has significantly progressed over the last several years, while real-world video face restoration (VFR), which is more challenging for more complex face motions such as moving gaze directions and facial orientations involved, remains unsolved.
A Comprehensive Study on Temporal Modeling for Online Action Detection
Online action detection (OAD) is a practical yet challenging task, which has attracted increasing attention in recent years.
Fake Alignment: Are LLMs Really Aligned Well?
The growing awareness of safety concerns in large language models (LLMs) has sparked considerable interest in the evaluation of safety.
Causal Evaluation of Language Models
Recent advances in language models have expanded the horizons of artificial intelligence across various domains, sparking inquiries into their potential for causal reasoning.
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
CO2 is able to attain a high scalability even on extensive multi-node clusters constrained by very limited communication bandwidth.
Range Loss for Deep Face Recognition with Long-tail
Convolutional neural networks have achieved great improvement on face recognition in recent years because of its extraordinary ability in learning discriminative features of people with different identities.
FANG-COVID: A New Large-Scale Benchmark Dataset for Fake News Detection in German
As the world continues to fight the COVID-19 pandemic, it is simultaneously fighting an ‘infodemic’ – a flood of disinformation and spread of conspiracy theories leading to health threats and the division of society.
DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation
In addition to the scene generation, the final part of DiffInDScene can be used as a post-processing module to refine the 3D reconstruction results from multi-view stereo.
Vlogger: Make Your Dream A Vlog
More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor.
Accurate Text Localization in Natural Image with Cascaded Convolutional Text Network
We propose a novel Cascaded Convolutional Text Network (CCTN) that joints two customized convolutional networks for coarse-to-fine text localization.
LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion
Notably, LoGoNet ranks 1st on Waymo 3D object detection leaderboard and obtains 81. 02 mAPH (L2) detection performance.
Causal Discovery via Conditional Independence Testing with Proxy Variables
Distinguishing causal connections from correlations is important in many scenarios.
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
The two branches are connected so that the control branch receives corresponding guidance from the trajectory branch at each time step.
Ranked #3 on
Autonomous Driving
on CARLA Leaderboard
Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation
To address the needs of learning representations for all languages in a unified space, we propose a novel efficient training recipe, upon which we build an effective detachable model, Lego-MT.
Real-time Holistic Robot Pose Estimation with Unknown States
We propose an end-to-end pipeline for real-time, holistic robot pose estimation from a single RGB image, even in the absence of known robot states.
Ranked #1 on
Robot Pose Estimation
on DREAM-dataset
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
Large language models (LLMs) have demonstrated outstanding performance in various tasks, such as text summarization, text question-answering, and etc.
Efficient Action Counting with Dynamic Queries
Temporal repetition counting aims to quantify the repeated action cycles within a video.
Investigate Indistinguishable Points in Semantic Segmentation of 3D Point Cloud
This paper investigates the indistinguishable points (difficult to predict label) in semantic segmentation for large-scale 3D point clouds.
Refining Pseudo Labels with Clustering Consensus over Generations for Unsupervised Object Re-identification
Unsupervised object re-identification targets at learning discriminative representations for object retrieval without any annotations.
TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation
What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity.
Mask as Supervision: Leveraging Unified Mask Information for Unsupervised 3D Pose Estimation
Automatic estimation of 3D human pose from monocular RGB images is a challenging and unsolved problem in computer vision.
DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model
Text-to-image (T2I) generative models have attracted significant attention and found extensive applications within and beyond academic research.
BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation
In this paper, we present BSN++, a new framework which exploits complementary boundary regressor and relation modeling for temporal proposal generation.
FineAction: A Fine-Grained Video Dataset for Temporal Action Localization
Temporal action localization (TAL) is an important and challenging problem in video understanding.
POS-BERT: Point Cloud One-Stage BERT Pre-Training
We propose to use the dynamically updated momentum encoder as the tokenizer, which is updated and outputs the dynamic supervision signal along with the training process.
Recurrent Bilinear Optimization for Binary Neural Networks
To address this issue, Recurrent Bilinear Optimization is proposed to improve the learning process of BNNs (RBONNs) by associating the intrinsic bilinear variables in the back propagation process.
DreamDA: Generative Data Augmentation with Diffusion Models
The acquisition of large-scale, high-quality data is a resource-intensive and time-consuming endeavor.
MGMAE: Motion Guided Masking for Video Masked Autoencoding
Based on this masking volume, we can track the unmasked tokens in time and sample a set of temporal consistent cubes from videos.
PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety
In this paper, we explore these concerns through the innovative lens of agent psychology, revealing that the dark psychological states of agents constitute a significant threat to safety.
StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding
Charts are common in literature across different scientific fields, conveying rich information easily accessible to readers.
Ranked #19 on
Chart Question Answering
on ChartQA
(using extra training data)
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
A single language model (LM), despite aligning well with an average labeler through reinforcement learning from human feedback (RLHF), may not universally suit diverse human preferences.
Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
Large language models (LLMs) need to undergo safety alignment to ensure safe conversations with humans.
Structured Triplet Learning with POS-tag Guided Attention for Visual Question Answering
Visual question answering (VQA) is of significant interest due to its potential to be a strong test of image understanding systems and to probe the connection between language and vision.
PalGAN: Image Colorization with Palette Generative Adversarial Networks
Multimodal ambiguity and color bleeding remain challenging in colorization.
