IndicCorp
Introduced by Kakwani et al. in IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages
IndicCorp is a large monolingual corpora with around 9 billion tokens covering 12 of the major Indian languages. It has been developed by discovering and scraping thousands of web sources - primarily news, magazines and books, over a duration of several months.
Languages covered: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu
Corpus Format: The corpus is a single large text file containing one sentence per line. The publicly released version is randomly shuffled, untokenized and deduplicated.
Downloads
| Language | # News Articles* | Sentences | Tokens | Link |
|---|---|---|---|---|
| as | 0.60M | 1.39M | 32.6M | link |
| bn | 3.83M | 39.9M | 836M | link |
| en | 3.49M | 54.3M | 1.22B | link |
| gu | 2.63M | 41.1M | 719M | link |
| hi | 4.95M | 63.1M | 1.86B | link |
| kn | 3.76M | 53.3M | 713M | link |
| ml | 4.75M | 50.2M | 721M | link |
| mr | 2.31M | 34.0M | 551M | link |
| or | 0.69M | 6.94M | 107M | link |
| pa | 2.64M | 29.2M | 773M | link |
| ta | 4.41M | 31.5M | 582M | link |
| te | 3.98M | 47.9M | 674M | link |
* Excluding articles obtained from the OSCAR corpus
Papers
| Paper | Code | Results | Date | Stars |
|---|
Dataset Loaders
No data loaders found. You can
submit your data loader here.
No data loaders found. You can
submit your data loader here.
