Search Results for author:
Found 23 papers, 13 papers with code
Provence: efficient and robust context pruning for retrieval-augmented generation
In this work, we close this gap and introduce Provence (Pruning and Reranking Of retrieVEd relevaNt ContExts), an efficient and robust context pruner for Question Answering, which dynamically detects the needed amount of pruning for a given context and can be used out-of-the-box for various domains.
BERGEN: A Benchmarking Library for Retrieval-Augmented Generation
In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of different configurations such as evaluation datasets, collections, metrics, retrievers, and LLMs.
Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation
We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training.
Retrieval-augmented generation in multilingual settings
Retrieval-augmented generation (RAG) has recently emerged as a promising solution for incorporating up-to-date or domain-specific knowledge into large language models (LLMs) and improving LLM factuality, but is predominantly studied in English-only settings.
Zero-shot cross-lingual transfer in instruction tuning of large language models
Instruction tuning (IT) is widely used to teach pretrained large language models (LLMs) to follow arbitrary instructions, but is under-studied in multilingual settings.
Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks
Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model.
Empirical study of pretrained multilingual language models for zero-shot cross-lingual knowledge transfer in generation
Zero-shot cross-lingual knowledge transfer enables the multilingual pretrained language model (mPLM), finetuned on a task in one language, make predictions for this task in other languages.
CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code
Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code.
Should you marginalize over possible tokenizations?
Autoregressive language models (LMs) map token sequences to probabilities.
Parameter-Efficient Finetuning of Transformers for Source Code
Pretrained Transformers achieve state-of-the-art performance in various code-processing tasks but may be too large to be deployed.
Probing Pretrained Models of Source Code
Deep learning models are widely used for solving challenging code processing tasks, such as code generation or code summarization.
Machine Learning Methods for Spectral Efficiency Prediction in Massive MIMO Systems
Channel decoding, channel detection, channel assessment, and resource management for wireless multiple-input multiple-output (MIMO) systems are all examples of problems where machine learning (ML) can be successfully applied.
On the Memorization Properties of Contrastive Learning
Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches.
On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
Training neural networks with batch normalization and weight decay has become a common practice in recent years.
A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code
There is an emerging interest in the application of natural language processing models to source code processing tasks.
On the Embeddings of Variables in Recurrent Neural Networks for Source Code
In this work, we develop dynamic embeddings, a recurrent mechanism that adjusts the learned semantics of the variable when it obtains more information about the variable's role in the program.
Empirical Study of Transformers for Source Code
In this work, we conduct a thorough empirical study of the capabilities of Transformers to utilize syntactic information in different tasks.
On Power Laws in Deep Ensembles
Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement.
Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?
In this work, we consider a fixed memory budget setting, and investigate, what is more effective: to train a single wide network, or to perform a memory split -- to train an ensemble of several thinner networks, with the same total number of parameters?
Structured Sparsification of Gated Recurrent Neural Networks
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e. g. neurons.
Bayesian Sparsification of Gated Recurrent Neural Networks
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons.
Bayesian Compression for Natural Language Processing
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters.
Bayesian Sparsification of Recurrent Neural Networks
Recurrent neural networks show state-of-the-art results in many text analysis tasks but often require a lot of memory to store their weights.
