Search Results for author:
Found 7 papers, 4 papers with code
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer
While offering (1) improved fine-tuning efficiency (by a factor of around 50 in our experiments), (2) a smaller parameter budget, and (3) increased language coverage, MAD-G remains competitive with more expensive methods for language-specific adapter training across the board.
Scaling Sparse Fine-Tuning to Large Language Models
We experiment with instruction-tuning of LLMs on standard dataset mixtures, finding that SpIEL is often superior to popular parameter-efficient fine-tuning methods like LoRA (low-rank adaptation) in terms of performance and comparable in terms of run time.
Cross-Lingual Transfer with Target Language-Ready Task Adapters
We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters.
Distilling Efficient Language-Specific Models for Cross-Lingual Transfer
Specifically, we use a two-phase distillation approach, termed BiStil: (i) the first phase distils a general bilingual model from the MMT, while (ii) the second, task-specific phase sparsely fine-tunes the bilingual "student" model using a task-tuned variant of the original MMT as its "teacher".
Composable Sparse Fine-Tuning for Cross-Lingual Transfer
Both these masks can then be composed with the pretrained model.
PolyLM: Learning about Polysemy through Language Modeling
To avoid the "meaning conflation deficiency" of word embeddings, a number of models have aimed to embed individual word senses.
