Search Results for author:
Found 24 papers, 17 papers with code
Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models
We observe proposed methods typically start with a base LM and data that has been annotated with entity metadata, then change the model, by modifying the architecture or introducing auxiliary loss terms to better capture entity knowledge.
KernelBench: Can LLMs Write Efficient GPU Kernels?
Efficient GPU kernels are crucial for building performant machine learning architectures, but writing them is a time-consuming challenge that requires significant expertise; therefore, we explore using language models (LMs) to automate kernel generation.
ThunderKittens: Simple, Fast, and Adorable AI Kernels
We match CuBLAS and FlashAttention-3 on GEMM and attention inference performance and outperform the strongest baselines by $10-40\%$ on attention backwards, $8\times$ on state space models, and $14\times$ on linear attention.
LoLCATs: On Low-Rank Linearizing of Large Language Models
When compared with prior approaches under the same compute budgets, LoLCATs significantly improves linearizing quality, closing the gap between linearized and original Llama 3. 1 70B and 405B LLMs by 77. 8% and 78. 1% on 5-shot MMLU.
Just read twice: closing the recall gap for recurrent language models
Recurrent large language models that compete with Transformers in language modeling perplexity are emerging at a rapid rate (e. g., Mamba, RWKV).
Optimistic Verifiable Training by Controlling Hardware Nondeterminism
The increasing compute demands of AI systems have led to the emergence of services that train models on behalf of clients lacking necessary resources.
Simple linear attention language models balance the recall-throughput tradeoff
In this work, we explore whether we can improve language model efficiency (e. g. by reducing memory consumption) without compromising on recall.
Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT
Retrieval pipelines-an integral component of many machine learning systems-perform poorly in domains where documents are long (e. g., 10K tokens or more) and where identifying the relevant document requires synthesizing information across the entire text.
Zoology: Measuring and Improving Recall in Efficient Language Models
To close the gap between synthetics and real language, we develop a new formalization of the task called multi-query associative recall (MQAR) that better reflects actual language.
RELIC: Investigating Large Language Model Responses using Self-Consistency
To address this challenge, we propose an interactive system that helps users gain insight into the reliability of the generated text.
Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension?
Resources and Evaluations for Multi-Distribution Dense Information Retrieval
We introduce and define the novel problem of multi-distribution information retrieval (IR) where given a query, systems need to retrieve passages from within multiple collections, each drawn from a different distribution.
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly.
Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes
Code synthesis is cheap, but far less accurate than directly processing each document with the LLM.
Ask Me Anything: A simple strategy for prompting language models
Prompting is a brittle process wherein small modifications to the prompt can cause large variations in the model predictions, and therefore significant effort is dedicated towards designing a painstakingly "perfect prompt" for a task.
Ranked #1 on
Question Answering
on Story Cloze
Can Foundation Models Help Us Achieve Perfect Secrecy?
However, privacy and quality appear to be in tension in existing systems for personal tasks.
Can Foundation Models Wrangle Your Data?
Foundation Models (FMs) are models trained on large corpora of data that, at very large scale, can generalize to new tasks without any task-specific finetuning.
Ranked #11 on
Entity Resolution
on Amazon-Google
Reasoning over Public and Private Data in Retrieval-Based Systems
We first define the PUBLIC-PRIVATE AUTOREGRESSIVE INFORMATION RETRIEVAL (PAIR) privacy framework for the novel retrieval setting over multiple privacy scopes.
Ranked #1 on
Multi-hop Question Answering
on ConcurrentQA
Metadata Shaping: Natural Language Annotations for the Tail
Since rare entities and facts are prevalent in the queries users submit to popular applications such as search and personal assistant systems, improving the ability of LMs to reliably capture knowledge over rare entities is a pressing challenge studied in significant prior work.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
A challenge for named entity disambiguation (NED), the task of mapping textual mentions to entities in a knowledge base, is how to disambiguate entities that appear rarely in the training data, termed tail entities.
Ranked #1 on
Entity Disambiguation
on AIDA-CoNLL
(Micro-F1 metric)
Energy conditions in $f(Q,T)$ gravity
The gravitational action $L$ is given by an arbitrary function $f$ of the non-metricity $Q$ and the trace of the matter-energy momentum tensor $T$.
General Relativity and Quantum Cosmology High Energy Physics - Theory
Constraining $f(Q,T)$ gravity from energy conditions
We are living in a golden age for experimental cosmology.
General Relativity and Quantum Cosmology High Energy Physics - Theory
Contextual Embeddings: When Are They Worth It?
We study the settings for which deep contextual embeddings (e. g., BERT) give large improvements in performance relative to classic pretrained embeddings (e. g., GloVe), and an even simpler baseline---random word embeddings---focusing on the impact of the training set size and the linguistic properties of the task.
