Search Results for author:
Found 36 papers, 6 papers with code
ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and believable outputs and significantly outperforms existing zero-shot methods.
Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective
For medical image segmentation, contrastive learning is the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples.
Aligning Source Visual and Target Language Domains for Unpaired Video Captioning
To this end, we introduce the unpaired video captioning task aiming to train models without coupled video-caption pairs in target language.
Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations
Most video-and-language representation learning approaches employ contrastive learning, e. g., CLIP, to project the video and text features into a common latent space according to the semantic similarities of text-video pairs.
Ranked #1 on
Video Retrieval
on LSMDC
(text-to-video Mean Rank metric)
DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention
To enhance the correlation between vision and language in disentangled spaces, we introduce the visual concepts to DiMBERT which represent visual information in textual format.
Generating Accurate and Faithful Discharge Instructions: Task, Dataset, and Model
We build a benchmark clinical dataset and propose the Re3Writer, which imitates the working patterns of physicians to first retrieve related working experience from historical PIs written by physicians, then reason related medical knowledge.
Prophet Attention: Predicting Attention with Future Attention for Improved Image Captioning
Especially for image captioning, the attention based models are expected to ground correct image regions with proper generated words.
Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels
Blindly leveraging all pixels in training hence can lead to the data imbalance issues, and cause deteriorated performance; (2) consistency: it remains unclear whether a segmentation model has learned meaningful and yet consistent anatomical features due to the intra-class variations between different anatomical features; and (3) diversity: the intra-slice correlations within the entire dataset have received significantly less attention.
Competence-based Multimodal Curriculum Learning for Medical Report Generation
Medical report generation task, which targets to produce long and coherent descriptions of medical images, has attracted growing research interests recently.
Graph-in-Graph Network for Automatic Gene Ontology Description Generation
This task aims to automatically generate a sentence that describes the function of a GO term belonging to one of the three categories, i. e., molecular function, biological process, and cellular component.
End-to-end Spoken Conversational Question Answering: Task, Dataset and Model
To evaluate the capacity of SCQA systems in a dialogue-style interaction, we assemble a Spoken Conversational Question Answering (Spoken-CoQA) dataset with more than 40k question-answer pairs from 4k conversations.
Ranked #1 on
Spoken Language Understanding
on Spoken-SQuAD
Conversational Question Answering
Spoken Language Understanding
+1
AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation
The acquired disease-grounded visual features can better represent the abnormal regions of the input image, which could alleviate data bias problem; 2) MGT module effectively uses the multi-grained features and Transformer framework to generate the long medical report.
Class-Aware Adversarial Transformers for Medical Image Segmentation
In this work, we present CASTformer, a novel type of adversarial transformers, for 2D medical image segmentation.
Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation
KGAE consists of a pre-constructed knowledge graph, a knowledge-driven encoder and a knowledge-driven decoder.
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Video captioning combines video understanding and language generation.
Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model
As a result, the proposed approach can handle various tasks including: Audio-Oriented Multimodal Machine Comprehension, Machine Reading Comprehension and Machine Listening Comprehension, in a single model, making fair comparisons possible between our model and the existing unimodal MC models.
Exploring Semantic Relationships for Unpaired Image Captioning
To further improve the quality of captions generated by the model, we propose the Semantic Relationship Explorer, which explores the relationships between semantic concepts for better understanding of the image.
Contrastive Attention for Automatic Chest X-ray Report Generation
In addition, according to the analysis, the CA model can help existing models better attend to the abnormal regions and provide more accurate descriptions which are crucial for an interpretable diagnosis.
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
In detail, PoKE explores the posterior knowledge, which provides explicit abnormal visual regions to alleviate visual data bias; PrKE explores the prior knowledge from the prior medical knowledge graph (medical knowledge) and prior radiology reports (working experience) to alleviate textual data bias.
Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could be addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection.
Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension
To predict the answer, it is common practice to employ a predictor to draw information only from the final encoder layer which generates the \textit{coarse-grained} representations of the source sequences, i. e., passage and question.
Federated Learning for Spoken Language Understanding
To this end, we propose a federated learning framework, which could unify various types of datasets as well as tasks to learn and fuse various types of knowledge, i. e., text representations, from different datasets and tasks, without the sharing of downstream task data.
Rethinking Skip Connection with Layer Normalization
In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could by addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection.
Prophet Attention: Predicting Attention with Future Attention
Especially for image captioning, the attention based models are expected to ground correct image regions with proper generated words.
Towards Data Distillation for End-to-end Spoken Conversational Question Answering
In spoken question answering, QA systems are designed to answer questions from contiguous text spans within the related speech transcripts.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
PIN: A Novel Parallel Interactive Network for Spoken Language Understanding
However, we noted that, so far, the efforts to obtain better performance by supporting bidirectional and explicit information exchange between ID and SF are not well studied. In addition, few studies attempt to capture the local context information to enhance the performance of SF.
Rethinking and Improving Natural Language Generation with Layer-Wise Multi-View Decoding
While it is common practice to draw information from only the last encoder layer, recent work has proposed to use representations from different encoder layers for diversified levels of information.
Exploring and Distilling Cross-Modal Information for Image Captioning
Recently, attention-based encoder-decoder models have been used extensively in image captioning.
Non-Autoregressive Coarse-to-Fine Video Captioning
However, mainstream video captioning methods suffer from slow inference speed due to the sequential manner of autoregressive decoding, and prefer generating generic descriptions due to the insufficient training of visual words (e. g., nouns and verbs) and inadequate decoding paradigm.
Self-Adaptive Scaling for Learnable Residual Structure
Residual has been widely applied to build deep neural networks with enhanced feature propagation and improved accuracy.
Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations
In vision-and-language grounding problems, fine-grained representations of the image are considered to be of paramount importance.
DLIMD: Dictionary Learning based Image-domain Material Decomposition for spectral CT
Second, we employ the direct inversion (DI) method to obtain initial material decomposition results, and a set of image patches are extracted from the mode-1 unfolding of normalized material image tensor to train a united dictionary by the K-SVD technique.
Block Matching Frame based Material Reconstruction for Spectral CT
Spectral computed tomography (CT) has a great potential in material identification and decomposition.
simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
The encode-decoder framework has shown recent success in image captioning.
Non-local Low-rank Cube-based Tensor Factorization for Spectral CT Reconstruction
Then, as a new regularizer, Kronecker-Basis-Representation (KBR) tensor factorization is employed into a basic spectral CT reconstruction model to enhance the ability of extracting image features and protecting spatial edges, generating the non-local low-rank cube-based tensor factorization (NLCTF) method.
Low-dose spectral CT reconstruction using L0 image gradient and tensor dictionary
The L0TDL method inherits the advantages of tensor dictionary learning (TDL) by employing the similarity of spectral CT images.
