TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Slot Filling	MASSIVE	XLM-R Base	Slot F1 Score	83.6	# 1
Zero-Shot Intent Classification and Slot Filling	MASSIVE	mT5 Base (text-to-text)	Exact Match	42.8	# 2
Zero-Shot Intent Classification and Slot Filling	MASSIVE	mT5 Base (encoder-only)	Exact Match	42.8	# 2
Zero-Shot Intent Classification and Slot Filling	MASSIVE	XLM-R Base	Exact Match	52.9	# 1
Intent Classification and Slot Filling	MASSIVE	mT5 Base (text-to-text)	Exact Match	73.8	# 3
Intent Classification and Slot Filling	MASSIVE	mT5 Base (encoder-only)	Exact Match	74.7	# 2
Intent Classification and Slot Filling	MASSIVE	XLM-R Base	Exact Match	75	# 1
Zero-shot Slot Filling	MASSIVE	mT5 Base (text-to-text)	Slot F1 Score	50.6	# 3
Zero-Shot Intent Classification	MASSIVE	mT5 Base (text-to-text)	Intent Accuracy	62.9	# 2
Zero-shot Slot Filling	MASSIVE	mT5 Base (encoder-only)	Slot F1 Score	56.9	# 2
Zero-Shot Intent Classification	MASSIVE	mT5 Base (encoder-only)	Intent Accuracy	61.2	# 3
Zero-shot Slot Filling	MASSIVE	XLM-R Base	Slot F1 Score	64.2	# 1
Zero-Shot Intent Classification	MASSIVE	XLM-R Base	Intent Accuracy	70.6	# 1
Slot Filling	MASSIVE	mT5 Base (text-to-text)	Slot F1 Score	81.3	# 3
Intent Classification	MASSIVE	mT5 Base (text-to-text)	Intent Accuracy	85.3	# 2
Slot Filling	MASSIVE	mT5 Base (encoder-only)	Slot F1 Score	82.2	# 2
Intent Classification	MASSIVE	mT5 Base (encoder-only)	Intent Accuracy	86.1	# 1
Intent Classification	MASSIVE	XLM-R Base	Intent Accuracy	85.1	# 3

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/massive-a-1m-example-multilingual-natural/slot-filling-on-massive)](https://paperswithcode.com/sota/slot-filling-on-massive?p=massive-a-1m-example-multilingual-natural)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/massive-a-1m-example-multilingual-natural/zero-shot-intent-classification-and-slot)](https://paperswithcode.com/sota/zero-shot-intent-classification-and-slot?p=massive-a-1m-example-multilingual-natural)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/massive-a-1m-example-multilingual-natural/intent-classification-and-slot-filling-on)](https://paperswithcode.com/sota/intent-classification-and-slot-filling-on?p=massive-a-1m-example-multilingual-natural)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/massive-a-1m-example-multilingual-natural/zero-shot-slot-filling-on-massive)](https://paperswithcode.com/sota/zero-shot-slot-filling-on-massive?p=massive-a-1m-example-multilingual-natural)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/massive-a-1m-example-multilingual-natural/zero-shot-intent-classification-on-massive)](https://paperswithcode.com/sota/zero-shot-intent-classification-on-massive?p=massive-a-1m-example-multilingual-natural)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/massive-a-1m-example-multilingual-natural/intent-classification-on-massive)](https://paperswithcode.com/sota/intent-classification-on-massive?p=massive-a-1m-example-multilingual-natural)`

MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

18 Apr 2022 · Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, Prem Natarajan ·

We present the MASSIVE dataset--Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly.

PDF Abstract

Code

Add Remove Mark official

alexa/massive official

531

pswietojanski/slurp

ai4bharat/indicbert

rita-nlp/italic

robvanderg/sid4lr

Tasks

Add Remove

intent-classification

Intent Classification

Natural Language Understanding

Slot Filling

XLM-R

Zero-Shot Intent Classification

Zero-shot Slot Filling

Datasets

Introduced in the Paper:

MASSIVE

Used in the Paper:

SLURP Fluent Speech Commands

Results from the Paper

Edit

Ranked #1 on Slot Filling on MASSIVE

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Slot Filling	MASSIVE	XLM-R Base	Slot F1 Score	83.6	# 1	Compare
Zero-Shot Intent Classification and Slot Filling	MASSIVE	mT5 Base (text-to-text)	Exact Match	42.8	# 2	Compare
Zero-Shot Intent Classification and Slot Filling	MASSIVE	mT5 Base (encoder-only)	Exact Match	42.8	# 2	Compare
Zero-Shot Intent Classification and Slot Filling	MASSIVE	XLM-R Base	Exact Match	52.9	# 1	Compare
Intent Classification and Slot Filling	MASSIVE	mT5 Base (text-to-text)	Exact Match	73.8	# 3	Compare
Intent Classification and Slot Filling	MASSIVE	mT5 Base (encoder-only)	Exact Match	74.7	# 2	Compare
Intent Classification and Slot Filling	MASSIVE	XLM-R Base	Exact Match	75	# 1	Compare
Zero-shot Slot Filling	MASSIVE	mT5 Base (text-to-text)	Slot F1 Score	50.6	# 3	Compare
Zero-Shot Intent Classification	MASSIVE	mT5 Base (text-to-text)	Intent Accuracy	62.9	# 2	Compare
Zero-shot Slot Filling	MASSIVE	mT5 Base (encoder-only)	Slot F1 Score	56.9	# 2	Compare
Zero-Shot Intent Classification	MASSIVE	mT5 Base (encoder-only)	Intent Accuracy	61.2	# 3	Compare
Zero-shot Slot Filling	MASSIVE	XLM-R Base	Slot F1 Score	64.2	# 1	Compare
Zero-Shot Intent Classification	MASSIVE	XLM-R Base	Intent Accuracy	70.6	# 1	Compare
Slot Filling	MASSIVE	mT5 Base (text-to-text)	Slot F1 Score	81.3	# 3	Compare
Intent Classification	MASSIVE	mT5 Base (text-to-text)	Intent Accuracy	85.3	# 2	Compare
Slot Filling	MASSIVE	mT5 Base (encoder-only)	Slot F1 Score	82.2	# 2	Compare
Intent Classification	MASSIVE	mT5 Base (encoder-only)	Intent Accuracy	86.1	# 1	Compare
Intent Classification	MASSIVE	XLM-R Base	Intent Accuracy	85.1	# 3	Compare

Methods

Add Remove

Adafactor • Attention Dropout • BPE • Dense Connections • Dropout • GELU • GLU • Inverse Square Root Schedule • Layer Normalization • Linear Layer • mT5 • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • SentencePiece • Softmax • T5 • XLM-R

Edit Social Preview

MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove