Search Results for author:
Found 160 papers, 68 papers with code
An Encoder Attribution Analysis for Dense Passage Retriever in Open-Domain Question Answering
The bi-encoder design of dense passage retriever (DPR) is a key factor to its success in open-domain question answering (QA), yet it is unclear how DPR’s question encoder and passage encoder individually contributes to overall performance, which we refer to as the encoder attribution problem.
How Does BERT Rerank Passages? An Attribution Analysis with Information Bottlenecks
Fine-tuned pre-trained transformers achieve the state of the art in passage reranking.
Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering
Our method reaches state-of-the-art performance on 5 benchmark QA datasets, with up to 10% improvement in top-100 accuracy compared to a joint-training multi-task DPR on SQuAD.
A Little Bit Is Worse Than None: Ranking with Limited Training Data
Researchers have proposed simple yet effective techniques for the retrieval problem based on using BERT as a relevance classifier to rerank initial candidates from keyword search.
Early Exiting BERT for Efficient Document Ranking
Pre-trained language models such as BERT have shown their effectiveness in various tasks.
Bag-of-Words Baselines for Semantic Code Search
The task of semantic code search is to retrieve code snippets from a source code corpus based on an information need expressed in natural language.
Learning to Rank in the Age of Muppets: Effectiveness–Efficiency Tradeoffs in Multi-Stage Ranking
It is well known that rerankers built on pretrained transformer models such as BERT have dramatically improved retrieval effectiveness in many tasks.
Unsupervised Chunking as Syntactic Structure Induction with a Knowledge-Transfer Approach
In this paper, we address unsupervised chunking as a new task of syntactic structure induction, which is helpful for understanding the linguistic structures of human languages as well as processing low-resource languages.
Voice Query Auto Completion
Query auto completion (QAC) is the task of predicting a search engine user’s final query from their intermediate, incomplete query.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval
Recent work has shown that dense passage retrieval techniques achieve better ranking accuracy in open-domain question answering compared to sparse retrieval techniques such as BM25, but at the cost of large space and memory requirements.
Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages
In this work, we challenge this assumption and present the first attempt at training a multilingual language model on only low-resource languages.
Cross-Lingual Training of Dense Retrievers for Document Retrieval
Dense retrieval has shown great success for passage ranking in English.
Cydex: Neural Search Infrastructure for the Scholarly Literature
Cydex is a platform that provides neural search infrastructure for domain-specific scholarly literature.
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval
We present an efficient training approach to text retrieval with dense representations that applies knowledge distillation using the ColBERT late-interaction ranking model.
GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration
We discuss how Pyserini - a widely used toolkit for reproducible IR research can be integrated with the Hugging Face ecosystem of open-source AI libraries and artifacts.
Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain
Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks as compared to fine tuning only on domain specific text.
How Does Generative Retrieval Scale to Millions of Passages?
Popularized by the Differentiable Search Index, the emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and encoding an entire document corpus within a single Transformer.
$SmartProbe$: A Virtual Moderator for Market Research Surveys
Market research surveys are a powerful methodology for understanding consumer perspectives at scale, but are limited by depth of understanding and insights.
Evaluating Embedding APIs for Information Retrieval
The ever-increasing size of language models curtails their widespread access to the community, thereby galvanizing many companies and startups into offering access to large language models through APIs.
Zero-Shot Listwise Document Reranking with a Large Language Model
Supervised ranking methods based on bi-encoder or cross-encoder architectures have shown success in multi-stage text ranking tasks, but they require large amounts of relevance judgments as training data.
Anserini Gets Dense Retrieval: Integration of Lucene's HNSW Indexes
With Pyserini, which provides a Python interface to Anserini, users gain access to both sparse and dense retrieval models, as Pyserini implements bindings to the Faiss vector search library alongside Lucene inverted indexes in a uniform, consistent interface.
AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation
This paper presents the AToMiC (Authoring Tools for Multimedia Content) dataset, designed to advance research in image/text cross-modal retrieval.
Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval
The advent of multilingual language models has generated a resurgence of interest in cross-lingual information retrieval (CLIR), which is the task of searching documents in one language with queries from another.
Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face
We present Spacerini, a modular framework for seamless building and deployment of interactive search applications, designed to facilitate the qualitative analysis of large scale research datasets.
How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval
We hence propose a new DA approach with diverse queries and sources of supervision to progressively train a generalizable DR. As a result, DRAGON, our dense retriever trained with diverse augmentation, is the first BERT-base-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations and even competes with models using more complex late interaction (ColBERTv2 and SPLADE++).
SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes
Multi-vector retrieval methods have demonstrated their effectiveness on various retrieval datasets, and among them, ColBERT is the most established method based on the late interaction of contextualized token embeddings of pre-trained language models.
Improving Out-of-Distribution Generalization of Neural Rerankers with Contextualized Late Interaction
Recent progress in information retrieval finds that embedding query and document representation into multi-vector yields a robust bi-encoder retriever on out-of-distribution datasets.
Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks
Industry practitioners always face the problem of choosing the appropriate model for deployment under different considerations, such as to maximize a metric that is crucial for production, or to reduce the total cost given financial concerns.
Building a Culture of Reproducibility in Academic Research
Reproducibility is an ideal that no researcher would dispute "in the abstract", but when aspirations meet the cold hard reality of the academic grind, reproducibility often "loses out".
Precise Zero-Shot Dense Retrieval without Relevance Labels
Given a query, HyDE first zero-shot instructs an instruction-following language model (e. g. InstructGPT) to generate a hypothetical document.
Less is More: Parameter-Free Text Classification with Gzip
Our method also performs particularly well in few-shot settings where labeled data are too scarce for DNNs to achieve a satisfying accuracy.
Improving Precancerous Case Characterization via Transformer-based Ensemble Learning
The application of natural language processing (NLP) to cancer pathology reports has been focused on detecting cancer cases, largely ignoring precancerous cases.
SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale
In this paper, we explore training and deploying an ASR system in the label-scarce, compute-limited setting.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval
In this paper, we unify different multi-vector retrieval models from a token routing viewpoint and propose conditional token interaction via dynamic lexical routing, namely CITADEL, for efficient and effective multi-vector retrieval.
On the Interaction Between Differential Privacy and Gradient Compression in Deep Learning
We evaluate this proposal and find that it can reduce the negative impact of noise added by differential privacy mechanisms on test accuracy by up to 24. 6%, and reduce the negative impact of gradient sparsification on test accuracy by up to 15. 1%.
XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing
We also include global translation exemplars for a target language to facilitate the translation process for large language models.
Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages
MIRACL (Multilingual Information Retrieval Across a Continuum of Languages) is a multilingual dataset we have built for the WSDM 2023 Cup challenge that focuses on ad hoc retrieval across 18 different languages, which collectively encompass over three billion native speakers around the world.
Query Expansion Using Contextual Clue Sampling with Language Models
To balance these two considerations, we propose a combination of an effective filtering strategy and fusion of the retrieved documents based on the generation probability of each context.
Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers
However, only a handful of the 7000+ languages on the planet benefit from specialized, custom-built tokenization algorithms, while the other languages are stuck with a "default" whitespace tokenizer, which cannot capture the intricacies of different languages.
What the DAAM: Interpreting Stable Diffusion Using Cross Attention
Large-scale diffusion neural networks represent a substantial milestone in text-to-image generation, but they remain poorly understood, lacking interpretability analyses.
Building an Efficiency Pipeline: Commutativity and Cumulativeness of Efficiency Operators for Transformers
There exists a wide variety of efficiency methods for natural language processing (NLP) tasks, such as pruning, distillation, dynamic inference, quantization, etc.
Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval
Pre-trained language models have been successful in many knowledge-intensive NLP tasks.
Few-Shot Non-Parametric Learning with Deep Latent Variable Model
Most real-world problems that machine learning algorithms are expected to solve face the situation with 1) unknown data distribution; 2) little domain-specific knowledge; and 3) datasets with limited annotation.
A Dense Representation Framework for Lexical and Semantic Matching
In contrast, our work integrates lexical representations with dense semantic representations by densifying high-dimensional lexical representations into what we call low-dimensional dense lexical representations (DLRs).
Domain Adaptation for Memory-Efficient Dense Retrieval
In this work, we show that binary embedding models like BPR and JPQ can perform significantly worse than baselines once there is a domain-shift involved.
Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking
For example, on MS MARCO Passage v1, our method yields an average candidate set size of 27 out of 1, 000 which increases the reranking speed by about 37 times, while the MRR@10 is greater than a pre-specified value of 0. 38 with about 90% empirical coverage and the empirical baselines fail to provide such guarantee.
To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers
In this paper we consider the problem of combining the relevance signals from sparse and dense retrievers in the context of Pseudo Relevance Feedback (PRF).
Towards Best Practices for Training Multilingual Dense Retrieval Models
Dense retrieval models using a transformer-based bi-encoder design have emerged as an active area of research.
Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval
With the recent success of dense retrieval methods based on bi-encoders, studies have applied this approach to various interesting downstream retrieval tasks with good efficiency and in-domain effectiveness.
Ranked #1 on
Math Information Retrieval
on ARQMath2 - Task 1
(using extra training data)
Tevatron: An Efficient and Flexible Toolkit for Dense Retrieval
In this paper, we present Tevatron, a dense retrieval toolkit optimized for efficiency, flexibility, and code simplicity.
Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?
Neural retrieval models are generally regarded as fundamentally different from the retrieval techniques used in the late 1990's when the TREC ad hoc test collections were constructed.
Sparsifying Sparse Representations for Passage Retrieval by Top-$k$ Masking
Sparse lexical representation learning has demonstrated much progress in improving passage retrieval effectiveness in recent models such as DeepImpact, uniCOIL, and SPLADE.
Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback: A Reproducibility Study
Finally, we contribute a study of the generalisability of the ANCE-PRF method when dense retrievers other than ANCE are used for the first round of retrieval and for encoding the PRF signal.
Densifying Sparse Representations for Passage Retrieval by Representational Slicing
Learned sparse and dense representations capture different successful approaches to text retrieval and the fusion of their results has proven to be more effective and robust.
Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation
Recent advances in retrieval models based on learned sparse representations generated by transformers have led us to, once again, consider score-at-a-time query evaluation techniques for the top-k retrieval problem.
A Proposed Conceptual Framework for a Representational Approach to Information Retrieval
This paper outlines a conceptual framework for understanding recent developments in information retrieval and natural language processing that attempts to integrate dense and sparse retrieval methods.
Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answering
Previous work on generalization of DPR mainly focus on testing both encoders in tandem on out-of-distribution (OOD) question-answering (QA) tasks, which is also known as domain adaptation.
Cross-Lingual Training with Dense Retrieval for Document Retrieval
Dense retrieval has shown great success in passage ranking in English.
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations.
Exploring Listwise Evidence Reasoning with T5 for Fact Verification
This work explores a framework for fact verification that leverages pretrained sequence-to-sequence transformer models for sentence selection and label prediction, two key sub-tasks in fact verification.
The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing
To fill this void in the literature, we study in this paper selective prediction for NLP, comparing different models and confidence estimators.
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
Recent developments in representational learning for information retrieval can be organized in a conceptual framework that establishes two pairs of contrasts: sparse vs. dense representations and unsupervised vs. learned representations.
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field.
Contextualized Query Embeddings for Conversational Search
This paper describes a compact and effective model for low-latency passage retrieval in conversational search based on learned dense representations.
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling
A vital step towards the widespread adoption of neural retrieval models is their resource efficiency throughout the training, indexing and query workflows.
Ranked #15 on
Zero-shot Text Search
on BEIR
A Replication Study of Dense Passage Retriever
Text retrieval using learned dense representations has recently emerged as a promising alternative to "traditional" text retrieval using sparse bag-of-words representations.
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression
The slow speed of BERT has motivated much research on accelerating its inference, and the early exiting idea has been proposed to make trade-offs between model quality and efficiency.
Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
Leaderboards are a ubiquitous part of modern research in applied machine learning.
Investigating the Limitations of Transformers with Simple Arithmetic Tasks
In this work, we investigate if the surface form of a number has any influence on how sequence-to-sequence language models learn simple arithmetic tasks such as addition and subtraction across a wide range of values.
Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations
Pyserini is an easy-to-use Python toolkit that supports replicable IR research by providing effective first-stage retrieval in a multi-stage ranking architecture.
Cultural Vocal Bursts Intensity Prediction
Information Retrieval
+1
The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
We propose a design pattern for tackling text ranking problems, dubbed "Expando-Mono-Duo", that has been empirically validated for a number of ad hoc retrieval tasks in different domains.
Inserting Information Bottlenecks for Attribution in Transformers
We show the effectiveness of our method in terms of attribution and the ability to provide insight into how information flows through layers.
Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models
While internalized {``}implicit knowledge{''} in pretrained transformers has led to fruitful progress in many natural language understanding tasks, how to most effectively elicit such knowledge remains an open question.
Cross-Lingual Training of Neural Models for Document Ranking
We tackle the challenge of cross-lingual training of neural document ranking models for mono-lingual retrieval, specifically leveraging relevance judgments in English to improve search in non-English languages.
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model.
Scientific Claim Verification with VERT5ERINI
This work describes the adaptation of a pretrained sequence-to-sequence model to the task of scientific claim verification in the biomedical domain.
Rainfall-Runoff Prediction at Multiple Timescales with a Single Long Short-Term Memory Network
Compared to naive prediction with a distinct LSTM per timescale, the multi-timescale architectures are computationally more efficient with no loss in accuracy.
Pretrained Transformers for Text Ranking: BERT and Beyond
There are two themes that pervade our survey: techniques for handling long documents, beyond typical sentence-by-sentence processing in NLP, and techniques for addressing the tradeoff between effectiveness (i. e., result quality) and efficiency (e. g., query latency, model and index size).
Howl: A Deployed, Open-Source Wake Word Detection System
We describe Howl, an open-source wake word detection toolkit with native support for open speech datasets, like Mozilla Common Voice and Google Speech Commands.
Ranked #4 on
Keyword Spotting
on Google Speech Commands
Don't Change Me! User-Controllable Selective Paraphrase Generation
The contribution of this work is a novel data generation technique using distant supervision that allows us to start with a pretrained sequence-to-sequence model and fine-tune a paraphrase generator that exhibits this behavior, allowing user-controllable paraphrase generation.
Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset
We present Covidex, a search engine that exploits the latest neural ranking models to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI.
Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBERT
Fine-tuned variants of BERT are able to achieve state-of-the-art accuracy on many natural language processing tasks, although at significant computational costs.
Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset
The Neural Covidex is a search engine that exploits the latest neural ranking architectures to provide information access to the COVID-19 Open Research Dataset (CORD-19) curated by the Allen Institute for AI.
Generalized and Scalable Optimal Sparse Decision Trees
Decision tree optimization is notoriously difficult from a computational perspective but essential for the field of interpretable machine learning.
A Data Scientist's Guide to Streamflow Prediction
In recent years, the paradigms of data-driven science have become essential components of physical sciences, particularly in geophysical disciplines such as climatology.
Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
Conversational search plays a vital role in conversational information seeking.
Segatron: Segment-Aware Transformer for Language Modeling and Understanding
To verify this, we propose a segment-aware Transformer (Segatron), by replacing the original token position encoding with a combined position encoding of paragraph, sentence, and token.
Ranked #13 on
Language Modelling
on WikiText-103
Showing Your Work Doesn't Always Work
In natural language processing, a recently popular line of work explores how to best report the experimental results of neural networks.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications.
Rapidly Bootstrapping a Question Answering Dataset for COVID-19
We present CovidQA, the beginnings of a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle's COVID-19 Open Research Dataset Challenge.
Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned
We present the Neural Covidex, a search engine that exploits the latest neural ranking architectures to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI.
Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2
Experimental results show that the Chinese GPT2 can generate better essay endings with \eop.
Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models
This paper presents an empirical study of conversational question reformulation (CQR) with sequence-to-sequence architectures and pretrained language models (PLMs).
TTTTTackling WinoGrande Schemas
We applied the T5 sequence-to-sequence model to tackle the AI2 WinoGrande Challenge by decomposing each example into two input text strings, each containing a hypothesis, and using the probabilities assigned to the "entailment" token as a score of the hypothesis.
Supporting Interoperability Between Open-Source Search Engines with the Common Index File Format
There exists a natural tension between encouraging a diverse ecosystem of open-source search engines and supporting fair, replicable comparisons across those systems.
Document Ranking with a Pretrained Sequence-to-Sequence Model
We investigate this observation further by varying target words to probe the model's use of latent knowledge.
Ranked #1 on
Ad-Hoc Information Retrieval
on TREC Robust04
Rapid Adaptation of BERT for Information Extraction on Domain-Specific Business Documents
Techniques for automatically extracting important content elements from business documents such as contracts, statements, and filings have the potential to make business operations more efficient.
A Prototype of Serverless Lucene
This paper describes a working prototype that adapts Lucene, the world's most popular and most widely deployed open-source search library, to operate within a serverless environment in the cloud.
Navigation-Based Candidate Expansion and Pretrained Language Models for Citation Recommendation
Citation recommendation systems for the scientific literature, to help authors find papers that should be cited, have the potential to speed up discoveries and uncover new routes for scientific exploration.
The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives
The Archives Unleashed project aims to improve scholarly access to web archives through a multi-pronged strategy involving tool creation, process modeling, and community building - all proceeding concurrently in mutually-reinforcing efforts.
The Proper Care and Feeding of CAMELS: How Limited Training Data Affects Streamflow Prediction
Accurate streamflow prediction largely relies on historical meteorological records and streamflow measurements.
Exploiting Token and Path-based Representations of Code for Identifying Security-Relevant Commits
Public vulnerability databases such as CVE and NVD account for only 60% of security vulnerabilities present in open-source projects, and are known to suffer from inconsistent quality.
MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
Our approach is model agnostic and can be easily applied on different future teacher model architectures.
What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality.
Cross-Lingual Relevance Transfer for Document Retrieval
Recent work has shown the surprising ability of multi-lingual BERT to serve as a zero-shot cross-lingual transfer model for a number of language processing tasks.
Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks
In this paper, we hypothesize that introducing an explicit, constrained pairwise word interaction mechanism to pretrained language models improves their effectiveness on semantic similarity tasks.
Scalable Knowledge Graph Construction from Text Collections
We present a scalable, open-source platform that {``}distills{''} a potentially large text collection into a knowledge graph.
Bridging the Gap between Relevance Matching and Semantic Matching for Short Text Similarity Modeling
A core problem of information retrieval (IR) is relevance matching, which is to rank documents by relevance to a user{'}s query.
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval
This paper applies BERT to ad hoc document retrieval on news articles, which requires addressing two challenges: relevance judgments in existing test collections are typically provided only at the document level, and documents often exceed the length that BERT was designed to handle.
Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling
Semantic similarity modeling is central to many NLP problems such as natural language inference and question answering.
Natural Language Generation for Effective Knowledge Distillation
Knowledge distillation can effectively transfer knowledge from BERT, a deep language representation model, to traditional, shallow word embedding-based neural networks, helping them approach or exceed the quality of other heavyweight language representation models.
Honkling: In-Browser Personalization for Ubiquitous Keyword Spotting
Used for simple commands recognition on devices from smart speakers to mobile phones, keyword spotting systems are everywhere.
Applying BERT to Document Retrieval with Birch
We present Birch, a system that applies BERT to document retrieval via integration with the open-source Anserini information retrieval toolkit to demonstrate end-to-end search over large document collections.
What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons
Memory neurons of long short-term memory (LSTM) networks encode and process information in powerful yet mysterious ways.
Multi-Stage Document Ranking with BERT
The advent of deep neural networks pre-trained via language modeling tasks has spurred a number of successful applications in natural language processing.
The Performance Envelope of Inverted Indexing on Modern Hardware
Experiments show that the largest determinants of performance are the physical characteristics of the source and target media, and that physically isolating the two yields the highest indexing throughput.
Lucene for Approximate Nearest-Neighbors Search on Arbitrary Dense Vectors
We demonstrate three approaches for adapting the open-source Lucene search library to perform approximate nearest-neighbor search on arbitrary dense vectors, using similarity search on word embeddings as a case study.
Aligning Cross-Lingual Entities with Multi-Aspect Information
Multilingual knowledge graphs (KGs), such as YAGO and DBpedia, represent entities in different languages.
Two Birds, One Stone: A Simple, Unified Model for Text Generation from Structured and Unstructured Data
We consider neural table-to-text generation and neural question generation (NQG) tasks for text generation from structured and unstructured data, respectively.
Rethinking Complex Neural Network Architectures for Document Classification
Neural network models for many NLP tasks have grown increasingly complex in recent years, making training and deployment more difficult.
Ranked #2 on
Document Classification
on IMDb-M
Detecting Customer Complaint Escalation with Recurrent Neural Networks and Manually-Engineered Features
Consumers dissatisfied with the normal dispute resolution process provided by an e-commerce company{'}s customer service agents have the option of escalating their complaints by filing grievances with a government authority.
The Simplest Thing That Can Possibly Work: Pseudo-Relevance Feedback Using Text Classification
Motivated by recent commentary that has questioned today's pursuit of ever-more complex models and mathematical formalisms in applied machine learning and whether meaningful empirical progress is actually being made, this paper tries to tackle the decades-old problem of pseudo-relevance feedback with "the simplest thing that can possibly work".
Document Expansion by Query Prediction
One technique to improve the retrieval effectiveness of a search engine is to expand documents with terms that are related or representative of the documents' content. From the perspective of a question answering system, this might comprise questions the document can potentially answer.
Ranked #1 on
Passage Re-Ranking
on TREC-PM
DocBERT: BERT for Document Classification
We present, to our knowledge, the first application of BERT to document classification.
Ranked #1 on
Document Classification
on Yelp-14
Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering
Recently, a simple combination of passage retrieval using off-the-shelf IR techniques and a BERT reader was found to be very effective for question answering directly on Wikipedia, yielding a large improvement over the previous state of the art on a standard benchmark dataset.
Ranked #2 on
Open-Domain Question Answering
on SQuAD1.1 dev
Simple BERT Models for Relation Extraction and Semantic Role Labeling
We present simple BERT-based models for relation extraction and semantic role labeling.
Ranked #29 on
Relation Extraction
on TACRED
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
In the natural language processing literature, neural networks are becoming increasingly deeper and complex.
Ranked #56 on
Sentiment Analysis
on SST-2 Binary classification
Simple Applications of BERT for Ad Hoc Document Retrieval
Following recent successes in applying BERT to question answering, we explore simple applications to ad hoc document retrieval.
Ranked #2 on
Ad-Hoc Information Retrieval
on TREC Robust04
(MAP metric)
Matching Entities Across Different Knowledge Graphs with Graph Embeddings
This paper explores the problem of matching entities across different knowledge graphs.
End-to-End Open-Domain Question Answering with BERTserini
We demonstrate an end-to-end question answering system that integrates BERT with the open-source Anserini information retrieval toolkit.
Ranked #3 on
Open-Domain Question Answering
on SQuAD1.1 dev
Streaming Voice Query Recognition using Causal Convolutional Recurrent Neural Networks
Voice-enabled commercial products are ubiquitous, typically enabled by lightweight on-device keyword spotting (KWS) and full automatic speech recognition (ASR) in the cloud.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
The Neural Hype and Comparisons Against Weak Baselines
Sculley et al. remind us that "the goal of science is not wins, but knowledge".
Ranked #3 on
Ad-Hoc Information Retrieval
on TREC Robust04
(MAP metric)
Ad-Hoc Information Retrieval
Cultural Vocal Bursts Intensity Prediction
+1
FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference.
Simple Attention-Based Representation Learning for Ranking Short Social Media Posts
This paper explores the problem of ranking short social media posts with respect to user queries using neural networks.
Progress and Tradeoffs in Neural Language Models
In recent years, we have witnessed a dramatic shift towards techniques driven by neural networks for a variety of NLP tasks.
JavaScript Convolutional Neural Networks for Keyword Spotting in the Browser: An Experimental Analysis
Overall, our robust, cross-device implementation for keyword spotting realizes a new paradigm for serving neural network applications, and one of our slim models reduces latency by 66% with a minimal decrease in accuracy of 4% from 94% to 90%.
Adaptive Pruning of Neural Language Models for Mobile Devices
Neural language models (NLMs) exist in an accuracy-efficiency tradeoff space where better perplexity typically comes at the cost of greater computation complexity.
Farewell Freebase: Migrating the SimpleQuestions Dataset to DBpedia
To address this problem, we present SimpleDBpediaQA, a new benchmark dataset for simple question answering over knowledge graphs that was created by mapping SimpleQuestions entities and predicates from Freebase to DBpedia.
Repeatability Corner Cases in Document Ranking: The Impact of Score Ties
Due to multi-threaded indexing, which makes experimentation with large modern document collections practical, internal document ids are not assigned consistently between different index instances of the same collection, and thus score ties are broken unpredictably.
Pay-Per-Request Deployment of Neural Network Models Using Serverless Architectures
We demonstrate the serverless deployment of neural networks for model inferencing in NLP applications using Amazon{'}s Lambda service for feedforward evaluation and DynamoDB for storing word embeddings.
CNNs for NLP in the Browser: Client-Side Deployment and Visualization Opportunities
We demonstrate a JavaScript implementation of a convolutional neural network that performs feedforward inference completely in the browser.
Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search
To our best knowledge, this paper presents the first substantial work tackling search over social media posts using neural ranking models.
Strong Baselines for Simple Question Answering over Knowledge Graphs with and without Neural Networks
We examine the problem of question answering over knowledge graphs, focusing on simple questions that can be answered by the lookup of a single fact.
Deep Residual Learning for Small-Footprint Keyword Spotting
We explore the application of deep residual learning and dilated convolutions to the keyword spotting task, using the recently-released Google Speech Commands Dataset as our benchmark.
Honk: A PyTorch Reimplementation of Convolutional Neural Networks for Keyword Spotting
We describe Honk, an open-source PyTorch reimplementation of convolutional neural networks for keyword spotting that are included as examples in TensorFlow.
An Insight Extraction System on BioMedical Literature with Deep Neural Networks
Mining biomedical text offers an opportunity to automatically discover important facts and infer associations among them.
Integrating Lexical and Temporal Signals in Neural Ranking Models for Searching Social Media Streams
To our knowledge, we are the first to integrate lexical and temporal signals in an end-to-end neural network architecture, in which existing neural ranking models are used to generate query-document similarity vectors that feed into a bidirectional LSTM layer for temporal modeling.
Exploring the Effectiveness of Convolutional Neural Networks for Answer Selection in End-to-End Question Answering
Most work on natural language question answering today focuses on answer selection: given a candidate list of sentences, determine which contains the answer.
Gappy Pattern Matching on GPUs for On-Demand Extraction of Hierarchical Translation Grammars
We believe that GPU-based extraction of hierarchical grammars is an attractive proposition, particularly for MT applications that demand high throughput.
Identifying Duplicate and Contradictory Information in Wikipedia
Our study identifies sentences in Wikipedia articles that are either identical or highly similar by applying techniques for near-duplicate detection of web pages.
Runtime Optimizations for Prediction with Tree-Based Models
Tree-based models have proven to be an effective solution for web ranking as well as other problems in diverse domains.
