Search Results for author:
Found 72 papers, 18 papers with code
A Transformer-based Threshold-Free Framework for Multi-Intent NLU
Furthermore, we propose a threshold-free intent multi-intent classifier that utilizes the output of IND task and detects the multiple intents without depending on the threshold.
PoseRAC: Pose Saliency Transformer for Repetitive Action Counting
Moreover, we introduce a pose-level method, PoseRAC, which is based on this representation and achieves state-of-the-art performance on two new version datasets by using Pose Saliency Annotation to annotate salient poses for training.
Improve Retrieval-based Dialogue System via Syntax-Informed Attention
Since incorporating syntactic information like dependency structures into neural models can promote a better understanding of the sentences, such a method has been widely used in NLP tasks.
ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and believable outputs and significantly outperforms existing zero-shot methods.
Improving Weakly Supervised Sound Event Detection with Causal Intervention
Existing weakly supervised sound event detection (WSSED) work has not explored both types of co-occurrences simultaneously, i. e., some sound events often co-occur, and their occurrences are usually accompanied by specific background sounds, so they would be inevitably entangled, causing misclassification and biased localization results with only clip-level supervision.
FTM: A Frame-level Timeline Modeling Method for Temporal Graph Representation Learning
Existing neighborhood aggregation strategies fail to capture either the short-term features or the long-term features of temporal graph attributes, leading to unsatisfactory model performance and even poor robustness and domain generality of the representation learning method.
FiTs: Fine-grained Two-stage Training for Knowledge-aware Question Answering
Despite the promising result of recent KAQA systems which tend to integrate linguistic knowledge from pre-trained language models (PLM) and factual knowledge from knowledge graphs (KG) to answer complex questions, a bottleneck exists in effectively fusing the representations from PLMs and KGs because of (i) the semantic and distributional gaps between them, and (ii) the difficulties in joint reasoning over the provided knowledge from both modalities.
Generating Templated Caption for Video Grounding
Video grounding aims to locate a moment of interest matching the given query sentence from an untrimmed video.
M3ST: Mix at Three Levels for Speech Translation
How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)?
Aligning Source Visual and Target Language Domains for Unpaired Video Captioning
To this end, we introduce the unpaired video captioning task aiming to train models without coupled video-caption pairs in target language.
A Dynamic Graph Interactive Framework with Label-Semantic Injection for Spoken Language Understanding
Multi-intent detection and slot filling joint models are gaining increasing traction since they are closer to complicated real-world scenarios.
DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention
To enhance the correlation between vision and language in disentangled spaces, we introduce the visual concepts to DiMBERT which represent visual information in textual format.
Prophet Attention: Predicting Attention with Future Attention for Improved Image Captioning
Especially for image captioning, the attention based models are expected to ground correct image regions with proper generated words.
Video Referring Expression Comprehension via Transformer with Content-aware Query
To this end, we introduce two new datasets (i. e., VID-Entity and VidSTG-Entity) by augmenting the VIDSentence and VidSTG datasets with the explicitly referred words in the whole sentence, respectively.
LocVTP: Video-Text Pre-training for Temporal Localization
To further enhance the temporal reasoning ability of the learned feature, we propose a context projection head and a temporal aware contrastive loss to perceive the contextual relationships.
Correspondence Matters for Video Referring Expression Comprehension
Extensive experiments demonstrate that our DCNet achieves state-of-the-art performance on both video and image REC benchmarks.
Diffsound: Discrete Diffusion Model for Text-to-sound Generation
In this study, we investigate generating sound conditioned on a text prompt and propose a novel text-to-sound generation framework that consists of a text encoder, a Vector Quantized Variational Autoencoder (VQ-VAE), a decoder, and a vocoder.
Ranked #3 on
Audio Generation
on AudioCaps
LAE: Language-Aware Encoder for Monolingual and Multilingual ASR
Experiments conducted on Mandarin-English code-switched speech suggest that the proposed LAE is capable of discriminating different languages in frame-level and shows superior performance on both monolingual and multilingual ASR tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Improving Dual-Microphone Speech Enhancement by Learning Cross-Channel Features with Multi-Head Attention
Hand-crafted spatial features, such as inter-channel intensity difference (IID) and inter-channel phase difference (IPD), play a fundamental role in recent deep learning based dual-microphone speech enhancement (DMSE) systems.
End-to-end Spoken Conversational Question Answering: Task, Dataset and Model
To evaluate the capacity of SCQA systems in a dialogue-style interaction, we assemble a Spoken Conversational Question Answering (Spoken-CoQA) dataset with more than 40k question-answer pairs from 4k conversations.
Ranked #1 on
Spoken Language Understanding
on Spoken-SQuAD
Conversational Question Answering
Spoken Language Understanding
+1
Speaker-Aware Mixture of Mixtures Training for Weakly Supervised Speaker Extraction
Dominant researches adopt supervised training for speaker extraction, while the scarcity of ideally clean corpus and channel mismatch problem are rarely considered.
Target Confusion in End-to-end Speaker Extraction: Analysis and Approaches
However, its performance is often inferior to that of a blind source separation (BSS) counterpart with a similar network architecture, due to the auxiliary speaker encoder may sometimes generate ambiguous speaker embeddings.
Learning Decoupling Features Through Orthogonality Regularization
Bearing this in mind, a two-branch deep network (KWS branch and SV branch) with the same network structure is developed and a novel decoupling feature learning method is proposed to push up the performance of KWS and SV simultaneously where speaker-invariant keyword representations and keyword-invariant speaker representations are expected respectively.
SpatioTemporal Focus for Skeleton-based Action Recognition
Inspired by the recent attention mechanism, we propose a multi-grain contextual focus module, termed MCF, to capture the action associated relation information from the body joints and parts.
Integrating Lattice-Free MMI into End-to-End Speech Recognition
However, the effectiveness and efficiency of the MBR-based methods are compromised: the MBR criterion is only used in system training, which creates a mismatch between training and decoding; the on-the-fly decoding process in MBR-based methods results in the need for pre-trained models and slow training speeds.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Unsupervised Pre-training for Temporal Action Localization Tasks
These pre-trained models can be sub-optimal for temporal localization tasks due to the inherent discrepancy between video-level classification and clip-level localization.
Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model
Then, the LM score of the hypothesis is obtained by intersecting the generated lattice with an external word N-gram LM.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Consistent Training and Decoding For End-to-end Speech Recognition Using Lattice-free MMI
Recently, End-to-End (E2E) frameworks have achieved remarkable results on various Automatic Speech Recognition (ASR) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CLIP Meets Video Captioning: Concept-Aware Representation Learning Does Matter
DCD is an auxiliary task that requires a caption model to learn the correspondence between video content and concepts and the co-occurrence relations between concepts.
Ranked #7 on
Video Captioning
on MSR-VTT
Towards Joint Intent Detection and Slot Filling via Higher-order Attention
Intent detection (ID) and Slot filling (SF) are two major tasks in spoken language understanding (SLU).
On Pursuit of Designing Multi-modal Transformer for Video Grounding
Almost all existing video grounding methods fall into two frameworks: 1) Top-down model: It predefines a set of segment candidates and then conducts segment classification and regression.
Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering
In this paper, we propose novel training schemes for spoken question answering with a self-supervised training stage and a contrastive representation learning stage.
HAN: Higher-order Attention Network for Spoken Language Understanding
Spoken Language Understanding (SLU), including intent detection and slot filling, is a core component in human-computer interaction.
Fully Non-Homogeneous Atmospheric Scattering Modeling with Convolutional Neural Networks for Single Image Dehazing
However, it is noted that ASM-based SIDM degrades its performance in dehazing real world hazy images due to the limited modelling ability of ASM where the atmospheric light factor (ALF) and the angular scattering coefficient (ASC) are assumed as constants for one image.
Joint Multiple Intent Detection and Slot Filling via Self-distillation
With the auxiliary knowledge provided by the MIL Intent Decoder, we set Final Slot Decoder as the teacher model that imparts knowledge back to Initial Slot Decoder to complete the loop.
Deep Motion Prior for Weakly-Supervised Temporal Action Localization
In this paper, we analyze that the motion cues behind the optical flow features are complementary informative.
Optical Flow Estimation
Weakly-supervised Temporal Action Localization
+1
Text Anchor Based Metric Learning for Small-footprint Keyword Spotting
Recently proposed metric learning approaches improved the generalizability of models for the KWS task, and 1D-CNN based KWS models have achieved the state-of-the-arts (SOTA) in terms of model size.
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Video captioning combines video understanding and language generation.
Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model
As a result, the proposed approach can handle various tasks including: Audio-Oriented Multimodal Machine Comprehension, Machine Reading Comprehension and Machine Listening Comprehension, in a single model, making fair comparisons possible between our model and the existing unimodal MC models.
Long-Short Temporal Modeling for Efficient Action Recognition
Efficient long-short temporal modeling is key for enhancing the performance of action recognition task.
SRF-Net: Selective Receptive Field Network for Anchor-Free Temporal Action Detection
Current mainstream one-stage TAD approaches localize and classify action proposals relying on pre-defined anchors, where the location and scale for action instances are set by designers.
All You Need is a Second Look: Towards Arbitrary-Shaped Text Detection
Compared to the traditional single-stage segmentation network, our NASK conducts the detection in a coarse-to-fine manner with the first stage segmentation spotting the rectangle text proposals and the second one retrieving compact representations.
Exploring Semantic Relationships for Unpaired Image Captioning
To further improve the quality of captions generated by the model, we propose the Semantic Relationship Explorer, which explores the relationships between semantic concepts for better understanding of the image.
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
In detail, PoKE explores the posterior knowledge, which provides explicit abnormal visual regions to alleviate visual data bias; PrKE explores the prior knowledge from the prior medical knowledge graph (medical knowledge) and prior radiology reports (working experience) to alleviate textual data bias.
Contrastive Attention for Automatic Chest X-ray Report Generation
In addition, according to the analysis, the CA model can help existing models better attend to the abnormal regions and provide more accurate descriptions which are crucial for an interpretable diagnosis.
Self-supervised Dialogue Learning for Spoken Conversational Question Answering
We also utilize the proposed self-supervised learning tasks to capture intra-sentence coherence.
Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could be addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection.
RR-Net: Injecting Interactive Semantics in Human-Object Interaction Detection
Upon the frame, an Interaction Intensifier Module and a Correlation Parsing Module are carefully designed, where: a) interactive semantics from humans can be exploited and passed to objects to intensify interactions, b) interactive correlations among humans, objects and interactions are integrated to promote predictions.
Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency
Transformer-based self-supervised models are trained as feature extractors and have empowered many downstream speech tasks to achieve state-of-the-art performance.
SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification
In this paper, we present SpecAugment++, a novel data augmentation method for deep neural networks based acoustic scene classification (ASC).
CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning
In this paper, we argue that learning by comparing helps identify these hard snippets and we propose to utilize snippet Contrastive learning to Localize Actions, CoLA for short.
FWB-Net:Front White Balance Network for Color Shift Correction in Single Image Dehazing via Atmospheric Light Estimation
However, for images taken in real-world, the illumination is not uniformly distributed over whole image which brings model mismatch and possibly results in color shift of the deep models using ASM.
Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension
To predict the answer, it is common practice to employ a predictor to draw information only from the final encoder layer which generates the \textit{coarse-grained} representations of the source sequences, i. e., passage and question.
Federated Learning for Spoken Language Understanding
To this end, we propose a federated learning framework, which could unify various types of datasets as well as tasks to learn and fuse various types of knowledge, i. e., text representations, from different datasets and tasks, without the sharing of downstream task data.
Rethinking Skip Connection with Layer Normalization
In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could by addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection.
Prophet Attention: Predicting Attention with Future Attention
Especially for image captioning, the attention based models are expected to ground correct image regions with proper generated words.
Contextualized Attention-based Knowledge Transfer for Spoken Conversational Question Answering
Spoken conversational question answering (SCQA) requires machines to model complex dialogue flow given the speech utterances and text corpora.
Audio Signal Processing
Conversational Question Answering
+2
Knowledge Distillation for Improved Accuracy in Spoken Question Answering
However, the recent work shows that ASR systems generate highly noisy transcripts, which critically limit the capability of machine comprehension on the SQA task.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Towards Data Distillation for End-to-end Spoken Conversational Question Answering
In spoken question answering, QA systems are designed to answer questions from contiguous text spans within the related speech transcripts.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
PIN: A Novel Parallel Interactive Network for Spoken Language Understanding
However, we noted that, so far, the efforts to obtain better performance by supporting bidirectional and explicit information exchange between ID and SF are not well studied. In addition, few studies attempt to capture the local context information to enhance the performance of SF.
PAN: Towards Fast Action Recognition via Learning Persistence of Appearance
In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries.
Ranked #2 on
Action Recognition
on Jester (Gesture Recognition)
A Graph-based Interactive Reasoning for Human-Object Interaction Detection
However, recent HOI detection methods mostly rely on additional annotations (e. g., human pose) and neglect powerful interactive reasoning beyond convolutions.
All you need is a second look: Towards Tighter Arbitrary shape text detection
Specifically, \textit{NASK} consists of a Text Instance Segmentation network namely \textit{TIS} (\(1^{st}\) stage), a Text RoI Pooling module and a Fiducial pOint eXpression module termed as \textit{FOX} (\(2^{nd}\) stage).
Multi-modal Multi-channel Target Speech Separation
Target speech separation refers to extracting a target speaker's voice from an overlapped audio of simultaneous talkers.
GID-Net: Detecting Human-Object Interaction with Global and Instance Dependency
GID block breaks through the local neighborhoods and captures long-range dependency of pixels both in global-level and instance-level from the scene to help detecting interactions between instances.
Enhancing End-to-End Multi-channel Speech Separation via Spatial Feature Learning
Hand-crafted spatial features (e. g., inter-channel phase difference, IPD) play a fundamental role in recent deep learning based multi-channel speech separation (MCSS) methods.
Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation
To address these challenges, we propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture in reverberant environments, assisted with directional information of the speaker(s).
Environmental Sound Classification with Parallel Temporal-spectral Attention
Convolutional neural networks (CNN) are one of the best-performing neural network architectures for environmental sound classification (ESC).
Non-Autoregressive Coarse-to-Fine Video Captioning
However, mainstream video captioning methods suffer from slow inference speed due to the sequential manner of autoregressive decoding, and prefer generating generic descriptions due to the insufficient training of visual words (e. g., nouns and verbs) and inadequate decoding paradigm.
C-RPNs: Promoting Object Detection in real world via a Cascade Structure of Region Proposal Networks
Although two-stage detectors like Faster R-CNN achieved big successes in object detection due to the strategy of extracting region proposals by region proposal network, they show their poor adaption in real-world object detection as a result of without considering mining hard samples during extracting region proposals.
End-to-End Multi-Channel Speech Separation
This paper extended the previous approach and proposed a new end-to-end model for multi-channel speech separation.
KCRC-LCD: Discriminative Kernel Collaborative Representation with Locality Constrained Dictionary for Visual Categorization
The LCD similarity measure can be kernelized under KCRC, which theoretically links CRC and LCD under the kernel method.
