Search Results for author:
Found 61 papers, 19 papers with code
EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing
Diffusion transformers have been widely adopted for text-to-image synthesis.
Apple Intelligence Foundation Language Models
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute.
Deep State-Space Generative Model For Correlated Time-to-Event Predictions
Capturing the inter-dependencies among multiple types of clinically-critical events is critical not only to accurate future event prediction, but also to better treatment planning.
Learning to Select the Best Forecasting Tasks for Clinical Outcome Prediction
We propose to meta-learn an a self-supervised patient trajectory forecast learning rule by meta-training on a meta-objective that directly optimizes the utility of the patient representation over the subsequent clinical outcome prediction.
Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
In this work, we revisit the settings by adopting step time as a more accurate measure of model complexity, and by determining the total compute budget under the Chinchilla compute-optimal settings.
Knowledge Graph Reasoning with Self-supervised Reinforcement Learning
To alleviate the distributional mismatch issue in general self-supervised RL (SSRL), in our supervised learning (SL) stage, the agent selects actions based on the policy network and learns from generated labels; this self-generation of labels is the intuition behind the name self-supervised.
Self-playing Adversarial Language Game Enhances LLM Reasoning
In this game, an attacker and a defender communicate around a target word only visible to the attacker.
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons.
Ranked #52 on
Visual Question Answering
on MM-Vet
Look Before You Leap: Towards Decision-Aware and Generalizable Tool-Usage for Large Language Models
Meanwhile, we propose a novel tool sampling strategy to enhance the generalizability of LLMs over unseen tools.
Are Large Language Models Good Prompt Optimizers?
Despite the success, the underlying mechanism of this approach remains unexplored, and the true effectiveness of LLMs as Prompt Optimizers requires further validation.
On Diversified Preferences of Large Language Model Alignment
Through experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance.
Learning to Skip for Language Modeling
Overparameterized large-scale language models have impressive generalization performance of in-context few-shot learning.
Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game
Targeting more efficient human preference optimization, we propose an Adversarial Preference Optimization (APO) framework, in which the LLM and the reward model update alternatively via a min-max game.
Everyone Deserves A Reward: Learning Customized Human Preferences
Besides, from the perspective of data efficiency, we propose a three-stage customized RM learning scheme, then empirically verify its effectiveness on both general preference datasets and our DSP set.
Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers
Although dominant in natural language processing, transformer-based models remain challenged by the task of long-sequence processing, because the computational cost of self-attention operations in transformers swells quadratically with the input sequence length.
Brainformers: Trading Simplicity for Efficiency
Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions.
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
Lifelong Language Pretraining with Distribution-Specialized Experts
Compared to existing lifelong learning approaches, Lifelong-MoE achieves better few-shot performance on 19 downstream NLP tasks.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Ranked #1 on
Question Answering
on StrategyQA
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
The mixture proportions of pretraining data domains (e. g., Wikipedia, books, web text) greatly affect language model (LM) performance.
Massively Multilingual Shallow Fusion with Large Language Models
In this work, we propose to train a single multilingual language model (LM) for shallow fusion in multiple languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Model Based Reinforcement Learning with Non-Gaussian Environment Dynamics and its Application to Portfolio Optimization
With the fast development of quantitative portfolio optimization in financial engineering, lots of AI-based algorithmic trading strategies have demonstrated promising results, among which reinforcement learning begins to manifest competitive advantages.
ReAct: Synergizing Reasoning and Acting in Language Models
While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e. g. chain-of-thought prompting) and acting (e. g. action plan generation) have primarily been studied as separate topics.
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Coreference Resolution
on Winograd Schema Challenge
Mixture-of-Experts with Expert Choice Routing
Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens.
ST-MoE: Designing Stable and Transferable Sparse Expert Models
But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning.
Ranked #1 on
Common Sense Reasoning
on ARC (Easy)
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing.
Ranked #10 on
Language Modelling
on LAMBADA
Finetuned Language Models Are Zero-Shot Learners
We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks.
Ranked #1 on
Question Answering
on OBQA
R2D2: Relational Text Decoding with Transformers
The other side ignores the sequential nature of the text by representing them as fixed-dimensional vectors and apply graph neural networks.
Learning to Select Best Forecast Tasks for Clinical Outcome Prediction
The paradigm of pretraining' from a set of relevant auxiliary tasks and thenfinetuning' on a target task has been successfully applied in many different domains.
The Medical Scribe: Corpus Development and Model Performance Analyses
We used this annotation scheme to label a corpus of about 6k clinical encounters.
Deep Physiological State Space Model for Clinical Forecasting
Clinical forecasting based on electronic medical records (EMR) can uncover the temporal correlations between patients' conditions and outcomes from sequences of longitudinal clinical measurements.
Learning to Infer Entities, Properties and their Relations from Clinical Conversations
Recently we proposed the Span Attribute Tagging (SAT) Model (Du et al., 2019) to infer clinical entities (e. g., symptoms) and their properties (e. g., duration).
Multi-Grained Named Entity Recognition
This paper presents a novel framework, MGNER, for Multi-Grained Named Entity Recognition where multiple entities or entity mentions in a sentence could be non-overlapping or totally nested.
Ranked #5 on
Nested Mention Recognition
on ACE 2005
Multi-Grained Named Entity Recognition
named-entity-recognition
+5
Extracting Symptoms and their Status from Clinical Conversations
This paper describes novel models tailored for a new application, that of extracting the symptoms mentioned in clinical conversations along with their status.
Entity Synonym Discovery via Multipiece Bilateral Context Matching
Being able to automatically discover synonymous entities in an open-world setting benefits various tasks such as entity disambiguation or knowledge graph canonicalization.
Joint Slot Filling and Intent Detection via Capsule Neural Networks
Being able to recognize words as slots and detect the intent of an utterance has been a keen issue in natural language understanding.
Ranked #9 on
Intent Detection
on SNIPS
Multi-Task Learning with Multi-View Attention for Answer Selection and Knowledge Base Question Answering
Second, these two tasks can benefit each other: answer selection can incorporate the external knowledge from knowledge base (KB), while KBQA can be improved by learning contextual information from answer selection.
Statistical Robust Chinese Remainder Theorem for Multiple Numbers: Wrapped Gaussian Mixture Model
Generalized Chinese Remainder Theorem (CRT) has been shown to be a powerful approach to solve the ambiguity resolution problem.
Learning Temporal Point Processes via Reinforcement Learning
Social goods, such as healthcare, smart city, and information networks, often produce ordered event data in continuous time.
Finding Similar Medical Questions from Question Answering Websites
Patients who have medical information demands tend to post questions about their health conditions on these crowdsourced Q&A websites and get answers from other users.
MedTruth: A Semi-supervised Approach to Discovering Knowledge Condition Information from Multi-Source Medical Data
In the light of these challenges, we propose a new truth discovery method, MedTruth, for medical knowledge condition discovery, which incorporates prior source quality information into the source reliability estimation procedure, and also utilizes the knowledge triple information for trustworthy information computation.
Databases
SynonymNet: Multi-context Bilateral Matching for Entity Synonyms
Being able to automatically discover synonymous entities from a large free-text corpus has transformative effects on structured knowledge discovery.
Multi-Grained Entity Proposal Network for Named Entity Recognition
In this paper, we focus on a new Named Entity Recognition (NER) task, i. e., the Multi-grained NER task.
AnatomyNet: Deep Learning for Fast and Fully Automated Whole-volume Segmentation of Head and Neck Anatomy
Methods: Our deep learning model, called AnatomyNet, segments OARs from head and neck CT images in an end-to-end fashion, receiving whole-volume HaN CT images as input and generating masks of all OARs of interest in one shot.
Cooperative Denoising for Distantly Supervised Relation Extraction
Distantly supervised relation extraction greatly reduces human efforts in extracting relational facts from unstructured texts.
Knowledge as A Bridge: Improving Cross-domain Answer Selection with External Knowledge
In this paper, we propose Knowledge-aware Attentive Network (KAN), a transfer learning framework for cross-domain answer selection, which uses the knowledge base as a bridge to enable knowledge transfer from the source domain to the target domains.
Generative Discovery of Relational Medical Entity Pairs
Online healthcare services can provide the general public with ubiquitous access to medical knowledge and reduce the information access cost for both individuals and societies.
Bringing Semantic Structures to User Intent Detection in Online Medical Queries
The healthcare status, complex medical information needs of patients are expressed diversely and implicitly in their medical text queries.
Time-Dependent Representation for Neural Event Sequence Prediction
Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization.
Scalable Influence Maximization for Multiple Products in Continuous-Time Diffusion Networks
A typical viral marketing model identifies influential users in a social network to maximize a single product adoption assuming unlimited user attention, campaign budgets, and time.
Coevolutionary Latent Feature Processes for Continuous-Time User-Item Interactions
Matching users to the right items at the right time is a fundamental task in recommendation systems.
Variational hybridization and transformation for large inaccurate noisy-or networks
In addition, we propose a transformation ranking algorithm that is very stable to large variances in network prior probabilities, a common issue that arises in medical applications of Bayesian networks.
Time-Sensitive Recommendation From Recurrent User Activities
By making personalized suggestions, a recommender system is playing a crucial role in improving the engagement of users in modern web-services.
Shaping Social Activity by Incentivizing Users
Events in an online social network can be categorized roughly into endogenous events, where users just respond to the actions of their neighbors within the network, or exogenous events, where users take actions due to drives external to the network.
Learning Time-Varying Coverage Functions
Coverage functions are an important class of discrete functions that capture laws of diminishing returns.
Budgeted Influence Maximization for Multiple Products
The typical algorithmic problem in viral marketing aims to identify a set of influential users in a social network, who, when convinced to adopt a product, shall influence other users in the network and trigger a large cascade of adoptions.
Scalable Influence Estimation in Continuous-Time Diffusion Networks
If a piece of information is released from a media site, can it spread, in 1 month, to a million web pages?
Waveform Driven Plasticity in BiFeO3 Memristive Devices: Model and Implementation
To the best of our knowledge, this is the first implementations of triplet plasticity on any physical memristive device.
Learning Networks of Heterogeneous Influence
However, the underlying transmission networks are often hidden and incomplete, and we observe only the time stamps when cascades of events happen.
