Search Results for author:
Found 65 papers, 32 papers with code
Language Resource Efficient Learning for Captioning
Our analysis of caption models with SC loss shows that the performance degradation is caused by the increasingly noisy estimation of reward and baseline with fewer language resources.
DialogueEIN: Emotion Interaction Network for Dialogue Affective Analysis
Emotion Recognition in Conversation (ERC) has attracted increasing attention in the affective computing research field.
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs.
POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View World
We humans are good at translating third-person observations of hand-object interactions (HOI) into an egocentric view.
SPAFormer: Sequential 3D Part Assembly with Transformers
We introduce SPAFormer, an innovative model designed to overcome the combinatorial explosion challenge in the 3D Part Assembly (3D-PA) task.
Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective
In this paper, we explore a new angle of this issue: overly detailed training data hinders the model's ability to timely terminate generation, leading to continued outputs beyond visual perception limits.
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs.
Explore and Tell: Embodied Visual Captioning in 3D Environments
To overcome this limitation, we propose a novel task called Embodied Captioning, which equips visual captioning models with navigation capabilities, enabling them to actively explore the scene and reduce visual ambiguity from suboptimal viewpoints.
Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences
To address these limitations, we explore the problem of Few-Shot Stylized Visual Captioning, which aims to generate captions in any desired style, using only a few examples as guidance during inference, without requiring further training.
No-frills Temporal Video Grounding: Multi-Scale Neighboring Attention and Zoom-in Boundary Detection
Temporal video grounding (TVG) aims to retrieve the time interval of a language query from an untrimmed video.
Movie101: A New Movie Understanding Benchmark
Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking.
InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation
Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative.
Knowledge Enhanced Model for Live Video Comment Generation
Live video commenting is popular on video media platforms, as it can create a chatting atmosphere and provide supplementary information for users while watching videos.
Rethinking Benchmarks for Cross-modal Image-text Retrieval
The reason is that a large amount of images and texts in the benchmarks are coarse-grained.
MPMQA: Multimodal Question Answering on Product Manuals
Taking into account the length of product manuals and the fact that a question is always related to a small number of pages, MPMQA can be naturally split into two subtasks: retrieving most related pages and then generating multimodal answers.
Accommodating Audio Modality in CLIP for Multimodal Processing
In this paper, we extend the stateof-the-art Vision-Language model CLIP to accommodate the audio modality for Vision-Language-Audio multimodal processing.
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Experimental results indicate that the models incorporating large language models (LLM) can generate more diverse responses, while the model utilizing knowledge graphs to introduce external knowledge performs the best overall.
Open-Category Human-Object Interaction Pre-Training via Language Modeling Framework
Human-object interaction (HOI) has long been plagued by the conflict between limited supervised data and a vast number of possible interaction combinations in real life.
CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge
Automatically generating textual descriptions for massive unlabeled images on the web can greatly benefit realistic web applications, e. g. multimodal retrieval and recommendation.
Exploring Anchor-based Detection for Ego4D Natural Language Query
In this paper we provide the technique report of Ego4D natural language query challenge in CVPR 2022.
Multi-Task Learning Framework for Emotion Recognition in-the-wild
This paper presents our system for the Multi-Task Learning (MTL) Challenge in the 4th Affective Behavior Analysis in-the-wild (ABAW) competition.
Unifying Event Detection and Captioning as Sequence Generation via Pre-Training
Dense video captioning aims to generate corresponding text descriptions for a series of events in the untrimmed video, which can be divided into two sub-tasks, event detection and event captioning.
TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval
In this paper, we propose Token Shift and Selection Network (TS2-Net), a novel token shift and selection transformer architecture, which dynamically adjusts the token sequence and selects informative tokens in both temporal and spatial dimensions from input video samples.
Ranked #8 on
Video Retrieval
on VATEX
Generalizing Multimodal Pre-training into Multilingual via Language Acquisition
Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models.
M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database
In this work, we propose a Multi-modal Multi-scene Multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9, 082 turns and 24, 449 utterances.
Cultural Vocal Bursts Intensity Prediction
Emotion Recognition
Progressive Learning for Image Retrieval with Hybrid-Modality Queries
Image retrieval with hybrid-modality queries, also known as composing text and image for image retrieval (CTI-IR), is a retrieval task where the search intention is expressed in a more complex query format, involving both vision and text modalities.
SingAug: Data Augmentation for Singing Voice Synthesis with Cycle-consistent Training Strategy
Deep learning based singing voice synthesis (SVS) systems have been demonstrated to flexibly generate singing with better qualities, compared to conventional statistical parametric based methods.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Multi-modal Emotion Estimation for in-the-wild Videos
In this paper, we briefly introduce our submission to the Valence-Arousal Estimation Challenge of the 3rd Affective Behavior Analysis in-the-wild (ABAW) competition.
Image Difference Captioning with Pre-training and Contrastive Learning
The Image Difference Captioning (IDC) task aims to describe the visual differences between two similar images with natural language.
VRDFormer: End-to-End Video Visual Relation Detection With Transformers
Most previous works adopt a multi-stage framework for video visual relation detection (VidVRD), which cannot capture long-term spatiotemporal contexts in different stages and also suffers from inefficiency.
MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition
Multimodal emotion recognition study is hindered by the lack of labelled corpora in terms of scale and diversity, due to the high annotation cost and label ambiguity.
Survey: Transformer based Video-Language Pre-training
Inspired by the success of transformer-based pre-training methods on natural language tasks and further computer vision tasks, researchers have begun to apply transformer to video processing.
Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training
Firstly, there are many specialized jargons in the product description, which are ambiguous to translate without the product image.
ICECAP: Information Concentrated Entity-aware Image Captioning
In this work, we focus on the entity-aware news image captioning task which aims to generate informative captions by leveraging the associated news articles to provide background knowledge about the target image.
Question-controlled Text-aware Image Captioning
To explore how to generate personalized text-aware captions, we define a new challenging task, namely Question-controlled Text-aware Image Captioning (Qc-TextCap).
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities
However, in real-world applications, we often encounter the problem of missing modality, and which modalities will be missing is uncertain.
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation
Emotion recognition in conversation (ERC) is a crucial component in affective dialogue systems, which helps the system understand users' emotions and generate empathetic responses.
Pre-Trained Models: Past, Present and Future
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI).
Team RUC_AIM3 Technical Report at ActivityNet 2021: Entities Object Localization
For the object grounding, we fine-tune the state-of-the-art detection model MDETR and design a post processing method to make the grounding results more faithful.
Towards Diverse Paragraph Captioning for Untrimmed Videos
Video paragraph captioning aims to describe multiple events in untrimmed videos with descriptive paragraphs.
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model.
Ranked #1 on
Image Retrieval
on RUC-CAS-WenLan
Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Loss
The neural network (NN) based singing voice synthesis (SVS) systems require sufficient data to train well and are prone to over-fitting due to data scarcity.
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
Team RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioning
Detecting meaningful events in an untrimmed video is essential for dense video captioning.
Ranked #3 on
Dense Video Captioning
on ActivityNet Captions
YouMakeup VQA Challenge: Towards Fine-grained Action Understanding in Domain-Specific Videos
The goal of the YouMakeup VQA Challenge 2020 is to provide a common benchmark for fine-grained action understanding in domain-specific videos e. g. makeup instructional videos.
Better Captioning with Sequence-Level Exploration
In this work, we show the limitation of the current sequence-level learning objective for captioning tasks from both theory and empirical result.
Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs
From the ASG, we propose a novel ASG2Caption model, which is able to recognise user intentions and semantics in the graph, and therefore generate desired captions according to the graph structure.
Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning
To improve fine-grained video-text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video-text matching into global-to-local levels.
Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products.
YouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension
Multimodal semantic comprehension has attracted increasing research interests recently such as visual question answering and caption generation.
Integrating Temporal and Spatial Attentions for VATEX Video Captioning Challenge 2019
This notebook paper presents our model in the VATEX video captioning challenge.
Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards
We employ self-supervision from mono-lingual corpus in the target language to provide fluency reward, and propose a multi-level visual semantic matching model to provide both sentence-level and concept-level visual relevancy rewards.
Activitynet 2019 Task 3: Exploring Contexts for Dense Captioning Events in Videos
The overall system achieves the state-of-the-art performance on the dense-captioning events in video task with 9. 91 METEOR score on the challenge testing set.
From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots
However, a picture tells a thousand words, which makes multi-lingual sentences pivoted by the same image noisy as mutual translations and thus hinders the translation model learning.
Unsupervised Bilingual Lexicon Induction from Mono-lingual Multimodal Data
The linguistic feature is learned from the sentence contexts with visual semantic constraints, which is beneficial to learn translation for words that are less visual-relevant.
RUC+CMU: System Report for Dense Captioning Events in Videos
This notebook paper presents our system in the ActivityNet Dense Captioning in Video task (task 3).
Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
Continuous dimensional emotion prediction is a challenging task where the fusion of various modalities usually achieves state-of-the-art performance such as early fusion or late fusion.
Video Captioning with Guidance of Multimodal Latent Topics
For the topic prediction task, we use the mined topics as the teacher to train a student topic prediction model, which learns to predict the latent topics from multimodal contents of videos.
Generating Video Descriptions with Topic Guidance
In addition to predefined topics, i. e., category tags crawled from the web, we also mine topics in a data-driven way based on training captions by an unsupervised topic mining model.
Improving Image Captioning by Concept-based Sentence Reranking
This paper describes our winning entry in the ImageCLEF 2015 image sentence generation task.
Detecting Violence in Video using Subclasses
We enrich the MediaEval 2015 violence dataset by \emph{manually} labeling violence videos with respect to the subclasses.
Adaptive Tag Selection for Image Annotation
The key insight is to divide the vocabulary into two disjoint subsets, namely a seen set consisting of tags having ground truth available for optimizing their thresholds and a novel set consisting of tags without any ground truth.
