Search Results for author: Ahmad Beirami

Found 48 papers, 18 papers with code

An Analysis of State-of-the-Art Models for Situated Interactive MultiModal Conversations (SIMMC)

no code implementations SIGDIAL (ACL) 2021 Satwik Kottur, Paul Crook, Seungwhan Moon, Ahmad Beirami, Eunjoon Cho, Rajen Subba, Alborz Geramifard

There is a growing interest in virtual assistants with multimodal capabilities, e. g., inferring the context of a conversation through scene understanding.

Scene Understanding

Optimal Block-Level Draft Verification for Accelerating Speculative Decoding

no code implementations15 Mar 2024 Ziteng Sun, Jae Hun Ro, Ahmad Beirami, Ananda Theertha Suresh

To the best of our knowledge, our work is the first to establish improvement over speculative decoding through a better draft verification algorithm.

Gradient-Based Language Model Red Teaming

1 code implementation30 Jan 2024 Nevan Wichers, Carson Denison, Ahmad Beirami

Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses.

Language Modelling

Theoretical guarantees on the best-of-n alignment policy

no code implementations3 Jan 2024 Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, Ananda Theertha Suresh

A commonly used analytical expression in the literature claims that the KL divergence between the best-of-$n$ policy and the base policy is equal to $\log (n) - (n-1)/n.$ We disprove the validity of this claim, and show that it is an upper bound on the actual KL divergence.

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

no code implementations14 Dec 2023 Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, Jonathan Berant

However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.

Language Modelling

Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing

no code implementations6 Dec 2023 Lucas Monteiro Paes, Ananda Theertha Suresh, Alex Beutel, Flavio P. Calmon, Ahmad Beirami

Here, the sample complexity for estimating the worst-case performance gap across groups (e. g., the largest difference in error rates) increases exponentially with the number of group-denoting sensitive attributes.

Fairness

FRAPPÉ: A Group Fairness Framework for Post-Processing Everything

1 code implementation5 Dec 2023 Alexandru Ţifrea, Preethi Lahoti, Ben Packer, Yoni Halpern, Ahmad Beirami, Flavien Prost

Despite achieving promising fairness-error trade-offs, in-processing mitigation techniques for group fairness cannot be employed in numerous practical applications with limited computation resources or no access to the training pipeline of the prediction model.

Fairness

Improving Robustness via Tilted Exponential Layer: A Communication-Theoretic Perspective

1 code implementation2 Nov 2023 Bhagyashree Puranik, Ahmad Beirami, Yao Qin, Upamanyu Madhow

State-of-the-art techniques for enhancing robustness of deep networks mostly rely on empirical risk minimization with suitable data augmentation.

Data Augmentation

Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting

no code implementations25 Oct 2023 Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, Jilin Chen

A crucial challenge for generative large language models (LLMs) is diversity: when a user's prompt is under-specified, models may follow implicit assumptions while generating a response, which may result in homogenization of the responses, as well as certain demographic groups being under-represented or even erased from the generated responses.

Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuning

no code implementations25 Oct 2023 Ananth Balashankar, Xiao Ma, Aradhana Sinha, Ahmad Beirami, Yao Qin, Jilin Chen, Alex Beutel

As large language models (LLMs) are widely adopted, new safety issues and policies emerge, to which existing safety classifiers do not generalize well.

Data Augmentation Few-Shot Learning +1

Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks

no code implementations25 Oct 2023 Aradhana Sinha, Ananth Balashankar, Ahmad Beirami, Thi Avrahami, Jilin Chen, Alex Beutel

We demonstrate the advantages of this system on the ANLI and hate speech detection benchmark datasets - both collected via an iterative, adversarial human-and-model-in-the-loop procedure.

Hate Speech Detection

SpecTr: Fast Speculative Decoding via Optimal Transport

no code implementations NeurIPS 2023 Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, Felix Yu

We show that the optimal draft selection algorithm (transport plan) can be computed via linear programming, whose best-known runtime is exponential in $k$.

Language Modelling Large Language Model

Enhancing Group Fairness in Online Settings Using Oblique Decision Forests

1 code implementation17 Oct 2023 Somnath Basu Roy Chowdhury, Nicholas Monath, Ahmad Beirami, Rahul Kidambi, Avinava Dubey, Amr Ahmed, Snigdha Chaturvedi

In the online setting, where the algorithm has access to a single instance at a time, estimating the group fairness objective requires additional storage and significantly more computation (e. g., forward/backward passes) than the task-specific objective at every time step.

Fairness

A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

1 code implementation24 Jul 2023 Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr

This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e. g. Flamingo), image-text matching models (e. g.

Image-text matching Language Modelling +4

Towards A Scalable Solution for Improving Multi-Group Fairness in Compositional Classification

no code implementations11 Jul 2023 James Atwood, Tina Tian, Ben Packer, Meghana Deodhar, Jilin Chen, Alex Beutel, Flavien Prost, Ahmad Beirami

Despite the rich literature on machine learning fairness, relatively little attention has been paid to remediating complex systems, where the final prediction is the combination of multiple classifiers and where multiple groups are present.

Fairness

Let's Do a Thought Experiment: Using Counterfactuals to Improve Moral Reasoning

no code implementations25 Jun 2023 Xiao Ma, Swaroop Mishra, Ahmad Beirami, Alex Beutel, Jilin Chen

Language models still struggle on moral reasoning, despite their impressive performance in many other tasks.

counterfactual Math +2

Towards Robust Prompts on Vision-Language Models

no code implementations17 Apr 2023 Jindong Gu, Ahmad Beirami, Xuezhi Wang, Alex Beutel, Philip Torr, Yao Qin

With the advent of vision-language models (VLMs) that can perform in-context and prompt-based learning, how can we design prompting approaches that robustly generalize to distribution shift and can be used on novel classes outside the support set of the prompts?

In-Context Learning

Robust Conversational Agents against Imperceptible Toxicity Triggers

1 code implementation NAACL 2022 Ninareh Mehrabi, Ahmad Beirami, Fred Morstatter, Aram Galstyan

Existing work to generate such attacks is either based on human-generated attacks which is costly and not scalable or, in case of automatic attacks, the attack vector does not conform to human-like language, which can be detected using a language model loss.

Language Modelling Text Generation

Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics

2 code implementations15 Dec 2021 Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Ram Sadagopan, Shahin Shayandeh, Asli Celikyilmaz, Jonathan May, Ahmad Beirami

Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance.

Ranked #4 on Multi-domain Dialogue State Tracking on MULTIWOZ 2.1 (using extra training data)

Dialogue State Tracking Multi-domain Dialogue State Tracking +1

Database Search Results Disambiguation for Task-Oriented Dialog Systems

no code implementations NAACL 2022 Kun Qian, Ahmad Beirami, Satwik Kottur, Shahin Shayandeh, Paul Crook, Alborz Geramifard, Zhou Yu, Chinnadhurai Sankar

We find that training on our augmented dialog data improves the model's ability to deal with ambiguous scenarios, without sacrificing performance on unmodified turns.

Multi-Task Learning

Information-Theoretic Bayes Risk Lower Bounds for Realizable Models

no code implementations8 Nov 2021 Matthew Nokleby, Ahmad Beirami

For models that are (roughly) lower Lipschitz in their parameters, we bound the rate distortion function from below, whereas for VC classes, the mutual information is bounded above by $d_\mathrm{vc}\log(n)$.

Federated Learning with Heterogeneous Differential Privacy

no code implementations28 Oct 2021 Nasser Aldaghri, Hessam Mahdavifar, Ahmad Beirami

We propose a new algorithm for FL with heterogeneous DP, named FedHDP, which employs personalization and weighted averaging at the server using the privacy choices of clients, to achieve better performance on clients' local models.

Federated Learning Privacy Preserving

Robustness through Data Augmentation Loss Consistency

1 code implementation21 Oct 2021 Tianjian Huang, Shaunak Halbe, Chinnadhurai Sankar, Pooyan Amini, Satwik Kottur, Alborz Geramifard, Meisam Razaviyayn, Ahmad Beirami

Our experiments show that DAIR consistently outperforms ERM and DA-ERM with little marginal computational cost and sets new state-of-the-art results in several benchmarks involving covariant data augmentation.

Multi-domain Dialogue State Tracking Visual Question Answering

On Tilted Losses in Machine Learning: Theory and Applications

1 code implementation13 Sep 2021 Tian Li, Ahmad Beirami, Maziar Sanjabi, Virginia Smith

Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance.

BIG-bench Machine Learning Fairness +1

Annotation Inconsistency and Entity Bias in MultiWOZ

no code implementations SIGDIAL (ACL) 2021 Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, Chinnadhurai Sankar

In this work, we identify an overlooked issue with dialog state annotation inconsistencies in the dataset, where a slot type is tagged inconsistently across similar dialogs leading to confusion for DST modeling.

dialog state tracking Memorization +1

A Stochastic Optimization Framework for Fair Risk Minimization

1 code implementation NeurIPS 2021 Andrew Lowy, Sina Baharlouei, Rakesh Pavan, Meisam Razaviyayn, Ahmad Beirami

We consider the problem of fair classification with discrete sensitive attributes and potentially large models and data sets, requiring stochastic solvers.

Binary Classification Fairness +1

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

1 code implementation ACL 2021 Hung Le, Chinnadhurai Sankar, Seungwhan Moon, Ahmad Beirami, Alborz Geramifard, Satwik Kottur

A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations.

Object Tracking Visual Reasoning

Fair Empirical Risk Minimization via Exponential Rényi Mutual Information

no code implementations1 Jan 2021 Rakesh Pavan, Andrew Lowy, Sina Baharlouei, Meisam Razaviyayn, Ahmad Beirami

In this paper, we propose another notion of fairness violation, called Exponential Rényi Mutual Information (ERMI) between sensitive attributes and the predicted target.

Attribute Fairness +1

Coded Machine Unlearning

no code implementations31 Dec 2020 Nasser Aldaghri, Hessam Mahdavifar, Ahmad Beirami

We also present the corresponding unlearning protocol and show that it satisfies the perfect unlearning criterion.

Ensemble Learning Machine Unlearning

Tilted Empirical Risk Minimization

2 code implementations ICLR 2021 Tian Li, Ahmad Beirami, Maziar Sanjabi, Virginia Smith

Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly.

Fairness

Competitive Balance in Team Sports Games

no code implementations24 Jun 2020 Sofia M Nikolakaki, Ogheneovo Dibie, Ahmad Beirami, Nicholas Peterson, Navid Aghdaie, Kazi Zaman

Competition is a primary driver of player satisfaction and engagement in multiplayer online games.

Situated and Interactive Multimodal Conversations

2 code implementations COLING 2020 Seungwhan Moon, Satwik Kottur, Paul A. Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, Rajen Subba, Alborz Geramifard

Next generation virtual assistants are envisioned to handle multimodal inputs (e. g., vision, memories of previous interactions, in addition to the user's utterances), and perform multimodal actions (e. g., displaying a route in addition to generating the system's utterance).

Response Generation

Rényi Fair Inference

no code implementations ICLR 2020 Sina Baharlouei, Maher Nouiehed, Ahmad Beirami, Meisam Razaviyayn

In this paper, we use R\'enyi correlation as a measure of fairness of machine learning models and develop a general training framework to impose fairness.

BIG-bench Machine Learning Clustering +2

On Multi-Agent Learning in Team Sports Games

no code implementations25 Jun 2019 Yunqi Zhao, Igor Borovikov, Jason Rupert, Caedmon Somers, Ahmad Beirami

In recent years, reinforcement learning has been successful in solving video games from Atari to Star Craft II.

reinforcement-learning Reinforcement Learning (RL)

Towards Interactive Training of Non-Player Characters in Video Games

2 code implementations3 Jun 2019 Igor Borovikov, Jesse Harder, Michael Sadovsky, Ahmad Beirami

We propose to create such NPC behaviors interactively by training an agent in the target environment using imitation learning with a human in the loop.

Imitation Learning OpenAI Gym

Winning Isn't Everything: Enhancing Game Development with Intelligent Agents

no code implementations25 Mar 2019 Yunqi Zhao, Igor Borovikov, Fernando De Mesentier Silva, Ahmad Beirami, Jason Rupert, Caedmon Somers, Jesse Harder, John Kolen, Jervis Pinto, Reza Pourabolghasem, James Pestrak, Harold Chaput, Mohsen Sardari, Long Lin, Sundeep Narravula, Navid Aghdaie, Kazi Zaman

We discuss two fundamental metrics based on which we measure the human-likeness of agents, namely skill and style, which are multi-faceted concepts with practical implications outlined in this paper.

Information Bottleneck Methods for Distributed Learning

no code implementations26 Oct 2018 Parinaz Farajiparvar, Ahmad Beirami, Matthew Nokleby

We consider this problem for unsupervised learning for batch and sequential data.

A Characterization of Guesswork on Swiftly Tilting Curves

1 code implementation27 Jan 2018 Ahmad Beirami, Robert Calderbank, Mark Christiansen, Ken Duffy, Muriel Médard

We show that the tilt operation on a memoryless string-source parametrizes an exponential family of memoryless string-sources, which we refer to as the tilted family.

Information Theory Information Theory

On Data-Dependent Random Features for Improved Generalization in Supervised Learning

no code implementations19 Dec 2017 Shahin Shahrampour, Ahmad Beirami, Vahid Tarokh

The randomized-feature approach has been successfully employed in large-scale kernel approximation and supervised learning.

On Optimal Generalizability in Parametric Learning

no code implementations NeurIPS 2017 Ahmad Beirami, Meisam Razaviyayn, Shahin Shahrampour, Vahid Tarokh

Such bias is measured by the cross validation procedure in practice where the data set is partitioned into a training set used for training and a validation set, which is not used in training and is left to measure the out-of-sample performance.

Rate-Distortion Bounds on Bayes Risk in Supervised Learning

no code implementations8 May 2016 Matthew Nokleby, Ahmad Beirami, Robert Calderbank

We provide lower and upper bounds on the rate-distortion function, using $L_p$ loss as the distortion measure, of a maximum a priori classifier in terms of the differential entropy of the posterior distribution and a quantity called the interpolation dimension, which characterizes the complexity of the parametric distribution family.

Performance Trade-Offs in Multi-Processor Approximate Message Passing

no code implementations10 Apr 2016 Junan Zhu, Ahmad Beirami, Dror Baron

We prove that the achievable region of this MOP is convex, and conjecture how the combined cost of computation and communication scales with the desired mean squared error.

Cannot find the paper you are looking for? You can Submit a new open access paper.