Search Results for author:
Found 107 papers, 47 papers with code
PALM: Pre-training an Autoencoding\&Autoregressive Language Model for Context-conditioned Generation
An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
Abstractive Text Summarization
Conversational Response Generation
+8
TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning
Charts are important for presenting and explaining complex data relationships.
Adaptive Feature Fusion Neural Network for Glaucoma Segmentation on Unseen Fundus Images
Fundus image segmentation on unseen domains is challenging, especially for the over-parameterized deep models trained on the small medical datasets.
Shortcuts Arising from Contrast: Effective and Covert Clean-Label Attacks in Prompt-Based Learning
In addressing this issue, we are inspired by the notion that a backdoor acts as a shortcut and posit that this shortcut stems from the contrast between the trigger and the data utilized for poisoning.
RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method
Comprehensive capturing of human motions requires both accurate captures of complex poses and precise localization of the human within scenes.
Collaborative Knowledge Infusion for Low-resource Stance Detection
Specifically, our stance detection approach leverages target background knowledge collaboratively from different knowledge sources with the help of knowledge alignment.
ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy
In WebShop, the 1-shot performance of the A$^3$T agent matches human average, and 4 rounds of iterative refinement lead to the performance approaching human experts.
RoleInteract: Evaluating the Social Interaction of Role-Playing Agents
In this paper, we introduce RoleInteract, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions.
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs.
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training
In vision-language pre-training (VLP), masked image modeling (MIM) has recently been introduced for fine-grained cross-modal alignment.
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval
In this work, we propose the UNIFY framework, which learns lexicon representations to capture fine-grained semantics and combines the strengths of latent and lexicon representations for video-text retrieval.
Budget-Constrained Tool Learning with Planning
Despite intensive efforts devoted to tool learning, the problem of budget-constrained tool learning, which focuses on resolving user queries within a specific budget constraint, has been widely overlooked.
Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models
Large Vision Language Models exhibit remarkable capabilities but struggle with hallucinations inconsistencies between images and their descriptions.
Model Composition for Multimodal Large Language Models
In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model.
PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs
While Large language models (LLMs) have demonstrated considerable capabilities across various natural language tasks, they often fall short of the performance achieved by domain-specific state-of-the-art models.
Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion
With the bloom of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs) that incorporate LLMs with pre-trained vision models have recently demonstrated impressive performance across diverse vision-language tasks.
Meta Ranking: Less Capable Language Models are Capable for Single Response Judgement
Although Large Language Models (LLMs) have demonstrated strong performance on a wide range of tasks, they still face reliability challenges such as hallucination.
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent
Each component is implemented by a single LLM that focuses on a specific capability and collaborates with others to accomplish the task.
Knowledge Distillation for Closed-Source Language Models
The prior estimation aims to derive a prior distribution by utilizing the corpus generated by closed-source language models, while the posterior estimation employs a proxy model to update the prior distribution and derive a posterior distribution.
Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection
Vision Transformers (ViTs) have become increasingly popular in large-scale Vision and Language Pre-training (VLP) models.
LARP: Language-Agent Role Play for Open-World Games
Language agents have shown impressive problem-solving skills within defined settings and brief timelines.
TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training
Self-supervised Multi-modal Contrastive Learning (SMCL) remarkably advances modern Vision-Language Pre-training (VLP) models by aligning visual and linguistic modalities.
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
We first analyzed the representation distribution of textual and visual tokens in MLLM, revealing two important findings: 1) there is a significant gap between textual and visual representations, indicating unsatisfactory cross-modal representation alignment; 2) representations of texts that contain and do not contain hallucinations are entangled, making it challenging to distinguish them.
Ranked #74 on
Visual Question Answering
on MM-Vet
mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model
In this work, towards a more versatile copilot for academic paper writing, we mainly focus on strengthening the multi-modal diagram analysis ability of Multimodal LLMs.
AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
Despite making significant progress in multi-modal tasks, current Multi-modal Large Language Models (MLLMs) encounter the significant challenge of hallucinations, which may lead to harmful consequences.
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks.
Ranked #11 on
Visual Question Answering (VQA)
on InfiMM-Eval
MCC-KD: Multi-CoT Consistent Knowledge Distillation
Large language models (LLMs) have showcased remarkable capabilities in complex reasoning through chain of thought (CoT) prompting.
Physical Information Neural Networks for Solving High-index Differential-algebraic Equation Systems Based on Radau Methods
In this paper, we propose a PINN computational framework, combined Radau IIA numerical method with a neural network structure via the attention mechanisms, to directly solve high-index DAEs.
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs.
ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models
Large language models (LLMs) have recently demonstrated remarkable capabilities to comprehend human intentions, engage in reasoning, and design planning-like behavior.
Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detection
In response to these identified gaps, we introduce the Ladder-of-Thought (LoT) for the stance detection task.
Evaluation and Analysis of Hallucination in Large Vision-Language Models
In this paper, we propose Hallucination Evaluation based on Large Language Models (HaELM), an LLM-based hallucination evaluation framework.
CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility
In this paper, we present CValues, the first Chinese human values evaluation benchmark to measure the alignment ability of LLMs in terms of both safety and responsibility criteria.
BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization
Specifically, We incorporate a Text-Semantics-Aware Patch Selector (TSPS) into the ViT backbone to perform a coarse-grained visual token extraction and then attach a flexible Transformer-based Patch Abstraction Decoder (PAD) upon the backbone for top-level visual abstraction.
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding.
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks
In addition, to facilitate a comprehensive evaluation of video-language models, we carefully build the largest human-annotated Chinese benchmarks covering three popular video-language tasks of cross-modal retrieval, video captioning, and video category classification.
CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding
In order to utilize vision and language pre-trained models to address the grounding problem, and reasonably take advantage of pseudo-labels, we propose CLIP-VG, a novel method that can conduct self-paced curriculum adapting of CLIP with pseudo-language labels.
Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answering
Existing knowledge-enhanced methods have achieved remarkable results in certain QA tasks via obtaining diverse knowledge from different knowledge bases.
AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inference
To support cost-effective language inference in multilingual settings, we propose AMTSS, an adaptive multi-teacher single-student distillation framework, which allows distilling knowledge from multiple teachers to a single student.
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github. com/X-PLUG/mPLUG-Owl.
Ranked #3 on
Visual Question Answering (VQA)
on HallusionBench
Visual Question Answering (VQA)
Zero-Shot Video Question Answer
From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping
Although previous works have created generation capacity for CLIP through additional language models, a modality gap between the CLIP representations of different modalities and the inability of CLIP to model the offset of this gap, which fails the concept to transfer across modalities.
ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human
In this paper, we present ChatPLUG, a Chinese open-domain dialogue system for digital human applications that instruction finetunes on a wide range of dialogue tasks in a unified internet-augmented format.
Improved Visual Fine-tuning with Natural Language Supervision
Fine-tuning a visual pre-trained model can leverage the semantic information from large-scale pre-training data and mitigate the over-fitting problem on downstream vision tasks with limited training examples.
CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions
The core of this dataset is a blending optimization process, which corrects for the pose as it drifts and is affected by the magnetic conditions.
Correspondence-Free Domain Alignment for Unsupervised Cross-Domain Image Retrieval
Thanks to the ISS and CCA, our method could encode the discrimination into the domain-invariant embedding space for unsupervised cross-domain image retrieval.
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement.
Ranked #1 on
Video Captioning
on MSR-VTT
Adaptively Clustering Neighbor Elements for Image Captioning
We design a novel global-local Transformer named \textbf{Ada-ClustFormer} (\textbf{ACF}) to generate captions.
Learning Trajectory-Word Alignments for Video-Language Tasks
To amend this, we propose a novel TW-BERT to learn Trajectory-Word alignment by a newly designed trajectory-to-word (T2W) attention for solving video-language tasks.
BUS: Efficient and Effective Vision-Language Pre-Training with Bottom-Up Patch Summarization.
In this paper, we propose a Bottom-Up Patch Summarization approach named BUS which is inspired by the Document Summarization Task in NLP to learn a concise visual summary of lengthy visual token sequences, guided by textual semantics.
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training
We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e. g., SSv2-Template and SSv2-Label) with 8. 6% and 11. 1% improvement respectively.
Ranked #1 on
Visual Question Answering (VQA)
on TGIF-QA
FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction
Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical.
Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment
We further propose Anchor Augment to guide the generative model's attention to the fine-grained information in the representation of CLIP.
Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence models
Eye movements are known to reflect cognitive processes in reading, and psychological reading research has shown that eye gaze patterns differ between readers with and without dyslexia.
Communication-Efficient Topologies for Decentralized Learning with $O(1)$ Consensus Rate
In the proposed family, EquiStatic has a degree of $\Theta(\ln(n))$, where $n$ is the network size, and a series of time-dependent one-peer topologies, EquiDyn, has a constant degree of 1.
Construction and Applications of Billion-Scale Pre-Trained Multimodal Business Knowledge Graph
We release all the open resources (OpenBG benchmarks) derived from it for the community and report experimental results of KG-centric tasks.
DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning
We evaluate our approach on a variety of knowledge driven and language understanding tasks, including NER, relation extraction, CommonsenseQA, OpenBookQA and GLUE.
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval
However, cross-grained contrast, which is the contrast between coarse-grained representations and fine-grained representations, has rarely been explored in prior research.
Ranked #12 on
Video Retrieval
on MSVD
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
Large-scale pretrained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks.
Ranked #1 on
Image Captioning
on COCO Captions
WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types
In this paper, we present WikiDiverse, a high-quality human-annotated MEL dataset with diversified contextual topics and entity types from Wikinews, which uses Wikipedia as the corresponding knowledge base.
Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Grounding
Moreover, since the backbones are query-agnostic, it is difficult to completely avoid the inconsistency issue by training the visual backbone end-to-end in the visual grounding framework.
Learning Deep Semantic Model for Code Search using CodeSearchNet Corpus
Different from typical information retrieval tasks, code search requires to bridge the semantic gap between the programming language and natural language, for better describing intrinsic concepts and semantics.
Achieving Human Parity on Visual Question Answering
The Visual Question Answering (VQA) task utilizes both visual image and language analysis to answer a textual question with respect to an image.
Ranked #7 on
Visual Question Answering (VQA)
on VQA v2 test-dev
Improving Prosody for Unseen Texts in Speech Synthesis by Utilizing Linguistic Information and Noisy Data
Recent advancements in end-to-end speech synthesis have made it possible to generate highly natural speech.
Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
Existing approaches to vision-language pre-training (VLP) heavily rely on an object detector based on bounding boxes (regions), where salient objects are first detected from images and then a Transformer-based model is used for cross-modal fusion.
Decentralized Composite Optimization with Compression
Decentralized optimization and communication compression have exhibited their great potential in accelerating distributed machine learning by mitigating the communication bottleneck in practice.
MinD at SemEval-2021 Task 6: Propaganda Detection using Transfer Learning and Multimodal Fusion
We describe our systems of subtask1 and subtask3 for SemEval-2021 Task 6 on Detection of Persuasion Techniques in Texts and Images.
Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction
This work focuses on generative QA which aims to generate an abstractive answer to a given question instead of extracting an answer span from a provided passage.
Provably Accelerated Decentralized Gradient Method Over Unbalanced Directed Graphs
We consider the decentralized optimization problem, where a network of $n$ agents aims to collaboratively minimize the average of their individual smooth and convex objective functions through peer-to-peer communication in a directed graph.
Elastic Graph Neural Networks
While many existing graph neural networks (GNNs) have been proven to perform $\ell_2$-based graph smoothing that enforces smoothness globally, in this work we aim to further enhance the local smoothness adaptivity of GNNs via $\ell_1$-based graph smoothing.
Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networks
The second algorithm is a broadcast-like version of CPP (B-CPP), and it also achieves linear convergence rate under the same conditions on the objective functions.
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Vision-language pre-training (VLP) on large-scale image-text pairs has achieved huge success for the cross-modal downstream tasks.
StructuralLM: Structural Pre-training for Form Understanding
Large pre-trained language models achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
RCT: Resource Constrained Training for Edge AI
For example, quantisation-aware training (QAT) method involves two copies of model parameters, which is usually beyond the capacity of on-chip memory in edge devices.
SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
Vision-language pre-training (VLP) on large-scale image-text pairs has recently witnessed rapid progress for learning cross-modal representations.
CoRe: An Efficient Coarse-refined Training Framework for BERT
In the second phase, we transform the trained relaxed BERT model into the original BERT and further retrain the model.
Deep Neural Networks with Short Circuits for Improved Gradient Learning
Deep neural networks have achieved great success both in computer vision and natural language processing tasks.
Fast algorithms for robust principal component analysis with an upper bound on the rank
The first type of algorithm applies regularization terms on the singular values of a matrix to obtain a low-rank matrix.
Linear Convergent Decentralized Optimization with Compression
Communication compression has become a key strategy to speed up distributed optimization.
Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering
Multiple-choice question answering (MCQA) is one of the most challenging tasks in machine reading comprehension since it requires more advanced reading comprehension skills such as logical reasoning, summarization, and arithmetic operations.
Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm
However, traditional genetic algorithms with fixed-length chromosomes may not be a good fit for optimizing deep learning hyperparameters, because deep learning models have variable number of hyperparameters depending on the model depth.
A Multi-Agent Primal-Dual Strategy for Composite Optimization over Distributed Features
This work studies multi-agent sharing optimization problems with the objective function being the sum of smooth local functions plus a convex (possibly non-smooth) function coupling all agents.
PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
Ranked #1 on
Text Generation
on CNN/Daily Mail
Abstractive Text Summarization
Conversational Response Generation
+8
Manifold Denoising by Nonlinear Robust Principal Component Analysis
This paper extends robust principal component analysis (RPCA) to nonlinear manifolds.
A Double Residual Compression Algorithm for Efficient Distributed Learning
Large-scale machine learning models are often trained by parallel stochastic gradient descent algorithms.
Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
To remedy this, we propose to augment the NSP task to a 3-class categorization task, which includes a category for previous sentence prediction (PSP).
Incorporating External Knowledge into Machine Reading for Generative Question Answering
Different from existing work on knowledge-aware QA, we focus on a more challenging task of leveraging external knowledge to generate answers in natural language for a given question with context.
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering.
Ranked #1 on
Natural Language Inference
on QNLI
Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning
This paper focuses on how to take advantage of external relational knowledge to improve machine reading comprehension (MRC) with multi-task learning.
On linear convergence of two decentralized algorithms
In addition, we relax the requirement for the objective functions and the mixing matrices.
Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering
Extensive experiments on the large-scale SQuAD and TriviaQA datasets validate the effectiveness of the proposed method.
A Deep Cascade Model for Multi-Document Reading Comprehension
To address this problem, we develop a novel deep cascade learning model, which progressively evolves from the document-level and paragraph-level ranking of candidate texts to more precise answer extraction with machine reading comprehension.
Ranked #2 on
Question Answering
on MS MARCO
$D^2$: Decentralized Training over Decentralized Data
While training a machine learning model using multiple workers, each of which collects data from its own data source, it would be useful when the data collected from different workers are unique and different.
Ranked #3 on
Multi-view Subspace Clustering
on ORL
D$^2$: Decentralized Training over Decentralized Data
While training a machine learning model using multiple workers, each of which collects data from their own data sources, it would be most useful when the data collected from different workers can be {\em unique} and {\em different}.
Exploring Outliers in Crowdsourced Ranking for QoE
Outlier detection is a crucial part of robust evaluation for crowdsourceable assessment of Quality of Experience (QoE) and has attracted much attention in recent years.
Nonconvex penalties with analytical solutions for one-bit compressive sensing
For several nonconvex penalties, including minimax concave penalty (MCP), $\ell_0$ norm, and sorted $\ell_1$ penalty, we provide fast algorithms for finding the analytical solutions by solving the dual problem.
A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates
This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms.
Mixed one-bit compressive sensing with applications to overexposure correction for CT reconstruction
Aiming at overexposure correction for computed tomography (CT) reconstruction, we in this paper propose a mixed one-bit compressive sensing (M1bit-CS) to acquire information from both regular and saturated measurements.
On the Convergence of Asynchronous Parallel Iteration with Unbounded Delays
Recent years have witnessed the surge of asynchronous parallel (async-parallel) iterative algorithms due to problems involving very large-scale data and a large number of decision variables.
A new primal-dual algorithm for minimizing the sum of three functions with a linear operator
For the general convex case, we prove the convergence of this new algorithm in terms of the distance to a fixed point by showing that the iteration is a nonexpansive operator.
Asynchronous Multi-Task Learning
The models for each hospital may be different because of the inherent differences in the distributions of the patient populations.
Coordinate Friendly Structures, Algorithms and Applications
To derive simple subproblems for several new classes of applications, this paper systematically studies coordinate-friendly operators that perform low-cost coordinate updates.
ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates
The agents share $x$ through either global memory or communication.
Pinball Loss Minimization for One-bit Compressive Sensing: Convex Models and Algorithms
The one-sided $\ell_1$ loss and the linear loss are two popular loss functions for 1bit-CS.
A Multiphase Image Segmentation Based on Fuzzy Membership Functions and L1-norm Fidelity
In this paper, we propose a variational multiphase image segmentation model based on fuzzy membership functions and L1-norm fidelity.
The Continuity of Images by Transmission Imaging Revisited
In transmission imaging, it was shown very recently in [49] that almost all images are continuous functions.
Restoration of Images Corrupted by Impulse Noise and Mixed Gaussian Impulse Noise using Blind Inpainting
In addition, we provide convergence analysis for these methods, these algorithms will converge to coordinatewise minimum points.
