Search Results for author:
Found 111 papers, 55 papers with code
MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
Previous models for voice interactions are categorized as native and aligned.
RMTransformer: Accurate Radio Map Construction and Coverage Prediction
Radio map, or pathloss map prediction, is a crucial method for wireless network modeling and management.
LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis
The intuitive nature of drag-based interaction has led to its growing adoption for controlling object trajectories in image-to-video synthesis.
AniDoc: Animation Creation Made Easier
The production of 2D animation follows an industry-standard workflow, encompassing four essential stages: character design, keyframe animation, in-betweening, and coloring.
CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
By training on a large-scale multilingual dataset, CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in the streaming mode.
Multi-Functional RIS Integrated Sensing and Communications for 6G Networks
We propose a MF-RIS-enabled multi-user and multi-target ISAC system, and formulate an optimization problem to maximize the signal-to-interference-plus-noise ratio (SINR) of sensing targets.
MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation
Recent advancements in video generation models, like Stable Video Diffusion, show promising results, but primarily focus on short, single-scene videos.
SEAGULL: No-reference Image Quality Assessment for Regions of Interest via Vision-Language Instruction Tuning
The quality analysis of ROIs can provide fine-grained guidance for image quality improvement and is crucial for scenarios focusing on region-level quality.
MagicQuill: An Intelligent Interactive Image Editing System
Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques.
HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems
To alleviate this problem, we propose HtmlRAG, which uses HTML instead of plain text as the format of retrieved knowledge in RAG.
Framer: Interactive Frame Interpolation
We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech.
Two Birds With One Stone: Enhancing Communication and Sensing via Multi-Functional RIS
Furthermore, we present two schemes that utilize MF-RISs to enhance the performance of integrated sensing and communication (ISAC).
Unified Audio Event Detection
Sound Event Detection (SED) detects regions of sound events, while Speaker Diarization (SD) segments speech conversations attributed to individual speakers.
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information.
Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts
To improve readability, we propose a Contextualized Spoken-to-Written conversion (CoS2W) task to address ASR and grammar errors and also transfer the informal text into the formal style with content preserved, utilizing contexts and auxiliary information.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Multimodal Fusion and Coherence Modeling for Video Topic Segmentation
In this work, we improve supervised VTS by thoroughly exploring multimodal fusion and multimodal coherence modeling.
MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering.
FreeCompose: Generic Zero-Shot Image Composition with Diffusion Prior
We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image.
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs).
ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World
Existing clinical diagnostic evaluation benchmarks for evaluating medical agents powered by LLMs have severe limitations.
Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
SDPN assigns the representation of the augmented views of an utterance to the same prototypes as the representation of the original view, thereby enabling effective knowledge transfer between the views.
Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformers
The Transformer architecture has significantly advanced deep learning, particularly in natural language processing, by effectively managing long-range dependencies.
ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec
In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style, merely based on a few seconds of audio prompt and a simple textual style description prompt.
From Single to Multi-Functional RIS: Architecture, Key Technologies, Challenges, and Applications
Although reconfigurable intelligent surfaces (RISs) have demonstrated the potential to boost network capacity and expand coverage by adjusting their electromagnetic properties, existing RIS architectures have certain limitations, such as double-fading attenuation and restricted half-space coverage.
FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition
Experiments show that our method's produced images are consistent with the given concepts and better aligned with the input text.
CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
By evaluating 17 popular LLMs using this benchmark, we reveal significant differences in their accuracy and reliability in code generation, offering detailed insights for further improving the code generation capabilities of LLMs.
Privacy-Enhanced Training-as-a-Service for On-Device Intelligence: Concept, Architectural Scheme, and Open Problems
On-device intelligence (ODI) enables artificial intelligence (AI) applications to run on end devices, providing real-time and customized AI inference without relying on remote servers.
3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization
With 3D-Speaker-Toolkit, we establish a new benchmark for multimodal speaker analysis.
LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models
Customization generation techniques have significantly advanced the synthesis of specific concepts across varied contexts.
Aligning Knowledge Graph with Visual Perception for Object-goal Navigation
Object-goal navigation is a challenging task that requires guiding an agent to specific objects based on first-person visual observations.
GenDeF: Learning Generative Deformation Field for Video Generation
We offer a new perspective on approaching the task of video generation.
Improving Communication Efficiency of Federated Distillation via Accumulating Local Updates
ALU drastically decreases the frequency of communication in federated distillation, thereby significantly reducing the communication overhead during the training process.
AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort
We empirically find that sparse control conditions, such as bounding boxes, are suitable for layout planning, while dense control conditions, e. g., sketches and keypoints, are suitable for generating high-quality image content.
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
Finally, we systematically evaluate and analyze eight mainstream LLMs and demonstrate the superior breadth and challenges of CodeScope for evaluating LLMs on code understanding and generation tasks compared to other benchmarks.
Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASR
We find that applying the conventional cross-entropy loss on input speech tokens does not consistently improve the ASR performance over the Loss Masking approach.
Towards Plastic and Stable Exemplar-Free Incremental Learning: A Dual-Learner Framework with Cumulative Parameter Averaging
The knowledge from the plastic learner is transferred to the stable learner via cumulative parameter averaging.
Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling
Our approach improve $F_1$ of old SOTA by 3. 42 (73. 74 -> 77. 16) and reduces $P_k$ by 1. 11 points (15. 0 -> 13. 89) on WIKI-727K and achieves an average relative reduction of 4. 3% on $P_k$ on WikiSection.
Object-aware Inversion and Reassembly for Image Editing
Subsequently, we propose an additional reassembly step to seamlessly integrate the respective editing results and the non-editing region to obtain the final edited image.
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation.
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT
Previous mainstream audio-and-text LLMs use discrete audio tokens to represent both input and output audio; however, they suffer from performance degradation on tasks such as automatic speech recognition, speech-to-text translation, and speech enhancement over models using continuous speech features.
Multi-Functional Reconfigurable Intelligent Surface: System Modeling and Performance Optimization
In this paper, we propose and study a multi-functional reconfigurable intelligent surface (MF-RIS) architecture.
Performance Analysis and Optimization of Reconfigurable Multi-Functional Surface Assisted Wireless Communications
To realize a self-sustainable communication system, we investigate the use of MF-RIS in improving the sum-rate of multi-user wireless networks.
B2C-AFM: Bi-Directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition
Human Action Recognition plays a driving engine of many human-computer interaction applications.
Ranked #20 on
Action Recognition
on NTU RGB+D
Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
To mitigate this problem, we introduce a diversity regularization term to embeddings in SDPN.
Improving BERT with Hybrid Pooling Network and Drop Mask
Transformer-based pre-trained language models, such as BERT, achieve great success in various natural language understanding tasks.
Exploiting Correlations Between Contexts and Definitions with Multiple Definition Modeling
In this paper, we carefully design a new task called Multiple Definition Modeling (MDM) that pool together all contexts and definition of target words.
Advancing Precise Outline-Conditioned Text Generation with Task Duality and Explicit Outline Control
Furthermore, we identify an issue of imbalanced utilization of the outline information in the precise outline-conditioned generation, which is ubiquitously observed across fine-tuned models and zero-shot inference models.
Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings
Prior studies diagnose the anisotropy problem in sentence representations from pre-trained language models, e. g., BERT, without fine-tuning.
SegGPT: Segmenting Everything In Context
We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images.
Ranked #1 on
Few-Shot Semantic Segmentation
on PASCAL-5i (5-Shot)
(using extra training data)
Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models
Our vid2vid-zero leverages off-the-shelf image diffusion models, and doesn't require training on any video.
Meeting Action Item Detection with Regularized Context Modeling
We construct and release the first Chinese meeting corpus with manual action item annotations.
Overview of the ICASSP 2023 General Meeting Understanding and Generation Challenge (MUG)
ICASSP2023 General Meeting Understanding and Generation Challenge (MUG) focuses on prompting a wide range of spoken language processing (SLP) research on meeting transcripts, as SLP applications are critical to improve users' efficiency in grasping important information in meetings.
MUG: A General Meeting Understanding and Generation Benchmark
To prompt SLP advancement, we establish a large-scale general Meeting Understanding and Generation Benchmark (MUG) to benchmark the performance of a wide range of SLP tasks, including topic segmentation, topic-level and session-level extractive summarization and topic title generation, keyphrase extraction, and action item detection.
Adaptive Knowledge Distillation between Text and Speech Pre-trained Models
Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models.
Weighted Sampling for Masked Language Modeling
Experiments on the Semantic Textual Similarity benchmark (STS) show that WSBERT significantly improves sentence embeddings over BERT.
SegGPT: Towards Segmenting Everything in Context
We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images.
Fast Contextual Scene Graph Generation With Unbiased Context Augmentation
Therefore, at the context level, we can produce diverse context descriptions by using a context augmentation method based on the original dataset.
Decoupling Learning and Remembering: A Bilevel Memory Framework With Knowledge Projection for Task-Incremental Learning
BMKP decouples the functions of learning and knowledge remembering via a bilevel-memory design: a working memory responsible for adaptively model learning, to ensure plasticity; a long-term memory in charge of enduringly storing the knowledge incorporated within the learned model, to guarantee stability.
Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation
Moreover, the pipelined approaches of retrieval and generation might result in poor generation performance when retrieval performance is low.
DopplerBAS: Binaural Audio Synthesis Addressing Doppler Effect
Recently, binaural audio synthesis (BAS) has emerged as a promising research field for its applications in augmented and virtual realities.
Images Speak in Images: A Generalist Painter for In-Context Visual Learning
In this work, we present Painter, a generalist model which addresses these obstacles with an "image"-centric solution, that is, to redefine the output of core vision tasks as images, and specify task prompts as also images.
Ranked #6 on
Personalized Segmentation
on PerSeg
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data.
Ranked #1 on
Object Detection
on COCO-O
RL-MD: A Novel Reinforcement Learning Approach for DNA Motif Discovery
The extraction of sequence patterns from a collection of functionally linked unlabeled DNA sequences is known as DNA motif discovery, and it is a key task in computational biology.
Diffsound: Discrete Diffusion Model for Text-to-sound Generation
In this study, we investigate generating sound conditioned on a text prompt and propose a novel text-to-sound generation framework that consists of a text encoder, a Vector Quantized Variational Autoencoder (VQ-VAE), a decoder, and a vocoder.
Ranked #15 on
Audio Generation
on AudioCaps
(FD metric)
TGRMPT: A Head-Shoulder Aided Multi-Person Tracker and a New Large-Scale Dataset for Tour-Guide Robot
A service robot serving safely and politely needs to track the surrounding people robustly, especially for Tour-Guide Robot (TGR).
CLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose
Motivated by the progress of visual-language research, we propose that pre-trained language models (e. g., CLIP) can facilitate animal pose estimation by providing rich prior knowledge for describing animal keypoints in text.
Safeguarding NOMA Networks via Reconfigurable Dual-Functional Surface under Imperfect CSI
This paper investigates the use of the reconfigurable dual-functional surface to guarantee the full-space secure transmission in non-orthogonal multiple access (NOMA) networks.
A Simple yet Effective Framework for Active Learning to Rank
To handle the diverse query requests from users at web-scale, Baidu has done tremendous efforts in understanding users' queries, retrieve relevant contents from a pool of trillions of webpages, and rank the most relevant webpages on the top of results.
DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware Decoder
We propose a novel and general Dependency-Aware Decoder (DePA) to enhance target dependency modeling in the decoder of fully NAT models from two perspectives: decoder self-attention and decoder input.
Towards Data-Efficient Detection Transformers
Besides, we introduce a simple yet effective label augmentation method to provide richer supervision and improve data efficiency.
Exemplar-free Class Incremental Learning via Discriminative and Comparable One-class Classifiers
DisCOIL follows the basic principle of POC, but it adopts variational auto-encoders (VAE) instead of other well-established one-class classifiers (e. g. deep SVDD), because a trained VAE can not only identify the probability of an input sample belonging to a class but also generate pseudo samples of the class to assist in learning new tasks.
Ranked #7 on
Exemplar-Free Counting
on FSC147
Supervised Homogeneity Fusion: a Combinatorial Approach
Fusing regression coefficients into homogenous groups can unveil those coefficients that share a common value within each group.
MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction
In this work, we propose a novel unsupervised embedding-based KPE approach, Masked Document Embedding Rank (MDERank), to address this problem by leveraging a mask strategy and ranking candidates by the similarity between embeddings of the source document and the masked document.
PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
We propose a novel Pooling Network (PoNet) for token mixing in long sequences with linear complexity.
FP-DETR: Detection Transformer Advanced by Fully Pre-training
To this end, we propose the task adapter which leverages self-attention to model the contextual relation between object query embedding.
Parsing Table Structures in the Wild
In contrast to existing studies that mainly focus on parsing well-aligned tabular images with simple layouts from scanned PDF documents, we aim to establish a practical table structure parsing system for real-world scenarios where tabular input images are taken or scanned with severe deformation, bending or occlusions.
Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers
In DQFA, a novel domain query is used to aggregate and align global context from the token sequence of both domains.
Sequence Model with Self-Adaptive Sliding Window for Efficient Spoken Document Segmentation
Evaluations are conducted on the English Wiki-727K document segmentation benchmark, a Chinese Wikipedia-based document segmentation dataset we created, and an in-house Chinese spoken document dataset.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
IntraLoss: Further Margin via Gradient-Enhancing Term for Deep Face Recognition
This method, named IntraLoss, explicitly performs gradient enhancement in the anisotropic region so that the intra-class distribution continues to shrink, resulting in isotropic and more compact intra-class distribution and further margin between identities.
TGRNet: A Table Graph Reconstruction Network for Table Structure Recognition
A table arranging data in rows and columns is a very effective data structure, which has been widely used in business and scientific research.
Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition
Although these methods have the innate ability to handle nested NER, they suffer from high computational cost, ignorance of boundary information, under-utilization of the spans that partially match with entities, and difficulties in long entity recognition.
Ranked #6 on
Nested Named Entity Recognition
on GENIA
Chinese Named Entity Recognition
named-entity-recognition
+3
Reranking Machine Translation Hypotheses with Structured and Web-based Language Models
In this paper, we investigate the use of linguistically motivated and computationally efficient structured language models for reranking N-best hypotheses in a statistical machine translation system.
Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning
In this paper, we propose a novel joint textual-phonetic pre-training approach for learning spoken language representations, aiming at exploring the full potentials of phonetic information to improve SLU robustness to ASR errors.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
Discriminative Self-training for Punctuation Prediction
Punctuation prediction for automatic speech recognition (ASR) output transcripts plays a crucial role for improving the readability of the ASR transcripts and for improving the performance of downstream natural language processing applications.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Graph-Based Tri-Attention Network for Answer Ranking in CQA
However, they encounter two main limitations: (1) Correlations between answers in the same question are often overlooked.
What Makes a Star Teacher? A Hierarchical BERT Model for Evaluating Teacher's Performance in Online Education
Based on these insights, we then propose a hierarchical course BERT model to predict teachers' performance in online education.
SUOD: Accelerating Large-Scale Unsupervised Heterogeneous Outlier Detection
Outlier detection (OD) is a key machine learning (ML) task for identifying abnormal objects from general samples with numerous high-stake applications including fraud detection and intrusion detection.
TTPP: Temporal Transformer with Progressive Prediction for Efficient Action Anticipation
Video action anticipation aims to predict future action categories from observed frames.
Transfer Learning for Context-Aware Spoken Language Understanding
We explore different transfer learning approaches to reduce dependency on data collection and annotation.
Spoken Language Understanding
Task-Oriented Dialogue Systems
+2
Sequential Neural Networks for Noetic End-to-End Response Selection
The noetic end-to-end response selection challenge as one track in the 7th Dialog System Technology Challenges (DSTC7) aims to push the state of the art of utterance classification for real world goal-oriented dialog systems, for which participants need to select the correct next utterances from a set of candidates for the multi-turn context.
Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection
With the increased applications of automatic speech recognition (ASR) in recent years, it is essential to automatically insert punctuation marks and remove disfluencies in transcripts, to improve the readability of the transcripts as well as the performance of subsequent applications, such as machine translation, dialogue systems, and so forth.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Beyond Clicks: Modeling Multi-Relational Item Graph for Session-Based Target Behavior Prediction
Specifically, we build a Multi-Relational Item Graph (MRIG) based on all behavior sequences from all sessions, involving target and auxiliary behavior types.
A Comprehensive Study on Temporal Modeling for Online Action Detection
Online action detection (OAD) is a practical yet challenging task, which has attracted increasing attention in recent years.
A Discriminative Learned CNN Embedding for Remote Sensing Image Scene Classification
Concretely, for remote sensing image scene classification, we would like to map images from the same scene to feature vectors that are close, and map images from different scenes to feature vectors that are widely separated.
Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
We also observe that fine-tuned models after the proposed pre-training approach maintain comparable performance on other NLP tasks, such as sentence classification and natural language inference tasks, compared to the original BERT models.
Ranked #27 on
Common Sense Reasoning
on CommonsenseQA
Automated Curriculum Learning for Turn-level Spoken Language Understanding with Weak Supervision
The RBSMA algorithm improves the test set accuracy by 7. 8% relative compared to the standard beam search.
Attention: A Big Surprise for Cross-Domain Person Re-Identification
Unlike existing cross-domain Re-ID methods, leveraging the auxiliary information of those unlabeled target-domain data, we aim at enhancing the model generalization and adaptation by discriminative feature learning, and directly exploiting a pre-trained model to new domains (datasets) without any utilization of the information from target domains.
The General Pair-based Weighting Loss for Deep Metric Learning
A large amount of loss functions based on pair distances have been presented in the literature for guiding the training of deep metric learning.
BERT for Joint Intent Classification and Slot Filling
Intent classification and slot filling are two essential tasks for natural language understanding.
Ranked #3 on
Slot Filling
on ATIS
Sequential Attention-based Network for Noetic End-to-End Response Selection
The noetic end-to-end response selection challenge as one track in Dialog System Technology Challenges 7 (DSTC7) aims to push the state of the art of utterance classification for real world goal-oriented dialog systems, for which participants need to select the correct next utterances from a set of candidates for the multi-turn context.
Ranked #1 on
Conversational Response Selection
on Advising Corpus
Early Stratification of Patients at Risk for Postoperative Complications after Elective Colectomy
Stratifying patients at risk for postoperative complications may facilitate timely and accurate workups and reduce the burden of adverse events on patients and the health system.
Temporal Action Detection by Joint Identification-Verification
Temporal action detection aims at not only recognizing action category but also detecting start time and end time for each action instance in an untrimmed video.
Articulatory information and Multiview Features for Large Vocabulary Continuous Speech Recognition
This paper explores the use of multi-view features and their discriminative transforms in a convolutional deep neural network (CNN) architecture for a continuous large vocabulary speech recognition task.
Discriminative Covariance Oriented Representation Learning for Face Recognition With Image Sets
For face recognition with image sets, while most existing works mainly focus on building robust set models with hand-crafted feature, it remains a research gap to learn better image representations which can closely match the subsequent image set modeling and classification.
Exploiting Out-of-Domain Data Sources for Dialectal Arabic Statistical Machine Translation
Statistical machine translation for dialectal Arabic is characterized by a lack of data since data acquisition involves the transcription and translation of spoken language.
Discriminant Analysis on Riemannian Manifold of Gaussian Distributions for Face Recognition With Image Sets
This paper presents a method named Discriminant Analysis on Riemannian manifold of Gaussian distributions (DARG) to solve the problem of face recognition with image sets.
Deeply Coupled Auto-encoder Networks for Cross-view Classification
In this paper, we propose a simple but effective coupled neural network, called Deeply Coupled Autoencoder Networks (DCAN), which seeks to build two deep neural networks, coupled with each other in every corresponding layers.
