HEMM: Holistic Evaluation of Multimodal Foundation Models

1 code implementation • 3 Jul 2024 • Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdinov, Louis-Philippe Morency

Through comprehensive experiments across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e. g., basic skills, information flows, and use cases) that pose challenges to today's models, and (2) distill performance trends regarding how different modeling dimensions (e. g., scale, pre-training data, multimodal alignment, pre-training, and instruction tuning objectives) influence performance.

PreCogIIITH at HinglishEval : Leveraging Code-Mixing Metrics & Language Model Embeddings To Estimate Code-Mix Quality

1 code implementation • 16 Jun 2022 • Prashant Kodali, Tanmay Sachan, Akshay Goindani, Anmol Goel, Naman Ahuja, Manish Shrivastava, Ponnurangam Kumaraguru

Code-Mixing is a phenomenon of mixing two or more languages in a speech event and is prevalent in multilingual societies.

Data Augmentation Language Modeling +1

Reappraising Domain Generalization in Neural Networks

no code implementations • 15 Oct 2021 • Sarath Sivaprasad, Akshay Goindani, Vaibhav Garg, Ritam Basu, Saiteja Kosgi, Vineet Gandhi

We find that the presence of multiple domains incentivizes domain agnostic learning and is the primary reason for generalization in Tradition DG.

Data Augmentation Domain Generalization

A Dynamic Head Importance Computation Mechanism for Neural Machine Translation

no code implementations • RANLP 2021 • Akshay Goindani, Manish Shrivastava

In multi-head attention mechanism, different heads attend to different parts of the input.

Machine Translation NMT +3