Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks

no code implementations • 10 Oct 2022 • Charith Peris, Lizhen Tan, Thomas Gueudre, Turan Gojayev, Pan Wei, Gokmen Oz

Yet, the generic corpora used to pretrain the teacher and the corpora associated with the downstream target domain are often significantly different, which raises a natural question: should the student be distilled over the generic corpora, so as to learn from high-quality teacher predictions, or over the downstream task corpora to align with finetuning?

domain classification intent-classification +5

Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems

no code implementations • 15 Jun 2022 • Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas, Davide Bernardi, Abhishek Bhagia, Claudio Delli Bovi, Jin Cao, Rakesh Chada, Amit Chauhan, Luoxin Chen, Anurag Dwarakanath, Satyam Dwivedi, Turan Gojayev, Karthik Gopalakrishnan, Thomas Gueudre, Dilek Hakkani-Tur, Wael Hamza, Jonathan Hueser, Kevin Martin Jose, Haidar Khan, Beiye Liu, Jianhua Lu, Alessandro Manzotti, Pradeep Natarajan, Karolina Owczarzak, Gokmen Oz, Enrico Palumbo, Charith Peris, Chandana Satya Prakash, Stephen Rawls, Andy Rosenbaum, Anjali Shenoy, Saleh Soltan, Mukund Harakere Sridhar, Liz Tan, Fabian Triefenbach, Pan Wei, Haiyang Yu, Shuai Zheng, Gokhan Tur, Prem Natarajan

We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9. 3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.

Cross-Lingual Natural Language Inference intent-classification +5

Fusion of an Ensemble of Augmented Image Detectors for Robust Object Detection

no code implementations • 17 Mar 2018 • Pan Wei, John E. Ball, Derek T. Anderson

A significant challenge in object detection is accurate identification of an object's position in image space, whereas one algorithm with one set of parameters is usually not enough, and the fusion of multiple algorithms and/or parameters can lead to more robust results.

Image Augmentation Object +3

Onion-Peeling Outlier Detection in 2-D data Sets

no code implementations • 12 Mar 2018 • Archit Harsh, John E. Ball, Pan Wei

Outlier Detection is a critical and cardinal research task due its array of applications in variety of domains ranging from data mining, clustering, statistical analysis, fraud detection, network intrusion detection and diagnosis of diseases etc.

Clustering Fraud Detection +2

Measuring Conflict in a Multi-Source Environment as a Normal Measure

no code implementations • 12 Mar 2018 • Pan Wei, John E. Ball, Derek T. Anderson, Archit Harsh, Christopher Archibald

The results demonstrate that the proposed measure can represent conflict in a meaningful way similar to what a human might expect and from it we can identify conflict within our sources.

Multi-Sensor Conflict Measurement and Information Fusion

no code implementations • 12 Mar 2018 • Pan Wei, John E. Ball, Derek T. Anderson

In this work, conflict is defined in terms of how little the output from multiple sensors overlap.

Sensor Fusion