Search Results for author:
Found 79 papers, 18 papers with code
Corruption-Robust Offline Two-Player Zero-Sum Markov Games
We note that we are the first to provide such a characterization of the problem of learning approximate Nash Equilibrium policies in offline two-player zero-sum Markov games under data corruption.
Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
Moreover, we extend our analysis to the approximate optimization setting and derive exponentially decaying convergence rates for both RLHF and DPO.
Informativeness of Reward Functions in Reinforcement Learning
Reward functions are central in specifying the task we want a reinforcement learning agent to perform.
Corruption Robust Offline Reinforcement Learning with Human Feedback
We aim to design algorithms that identify a near-optimal policy from the corrupted data, with provable guarantees.
Generative AI for Education (GAIED): Advances, Opportunities, and Challenges
This survey article has grown out of the GAIED (pronounced "guide") workshop organized by the authors at the NeurIPS 2023 conference.
Active Third-Person Imitation Learning
We consider the problem of third-person imitation learning with the additional challenge that the learner must select the perspective from which they observe the expert.
Optimally Teaching a Linear Behavior Cloning Agent
The goal of the teacher is to teach a realizable target policy to the learner using minimum number of state demonstrations.
Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming
We instantiate several methods based on LLM-SS framework and evaluate them using an existing benchmark, StudentSyn, for student attempt synthesis in a visual programming domain.
Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation
We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs.
Evaluating ChatGPT and GPT-4 for Visual Programming
In our work, we evaluate two models, ChatGPT (based on GPT-3. 5) and GPT-4, in visual programming domains for various scenarios and assess performance using expert-based annotations.
Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
In our work, we systematically evaluate two models, ChatGPT (based on GPT-3. 5) and GPT-4, and compare their performance with human tutors for a variety of scenarios.
Learning Embeddings for Sequential Tasks Using Population of Agents
We present an information-theoretic framework to learn fixed-dimensional embeddings for tasks in reinforcement learning.
Synthesizing a Progression of Subtasks for Block-Based Visual Programming Tasks
We propose a novel synthesis algorithm that generates a progression of subtasks that are high-quality, well-spaced in terms of their complexity, and solving this progression leads to solving the reference task.
Neural Task Synthesis for Visual Programming
Generative neural models hold great promise in enhancing programming education by synthesizing new content.
Proximal Curriculum for Reinforcement Learning Agents
We consider the problem of curriculum design for reinforcement learning (RL) agents in contextual multi-task settings.
Adaptive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes
We introduce a scaffolding framework based on pop quizzes presented as multi-choice programming tasks.
Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attacks
In this paper, we study targeted poisoning attacks in a two-agent setting where an attacker implicitly poisons the effective environment of one of the agents by modifying the policy of its peer.
Online Reinforcement Learning with Uncertain Episode Lengths
We show that minimizing regret with this new general discounting is equivalent to minimizing regret with uncertain episode lengths.
Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models
We investigate using LLMs to generate feedback for fixing syntax errors in Python programs, a key scenario in introductory programming.
Provable Defense against Backdoor Policies in Reinforcement Learning
Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a 'safe subspace', estimated from a small number of interactions with a clean (non-triggered) environment.
Specifying and Testing $k$-Safety Properties for Machine-Learning Models
Here, we show the wide applicability of $k$-safety properties for machine-learning models and present the first specification language for expressing them.
Equity and Fairness of Bayesian Knowledge Tracing
We consider the equity and fairness of curricula derived from Knowledge Tracing models.
From {Solution Synthesis} to {Student Attempt Synthesis} for Block-Based Visual Programming Tasks
We introduce a novel benchmark, StudentSyn, centered around the following challenge: For a given student, synthesize the student's attempt on a new target task after observing the student's attempt on a fixed reference task.
Actual Causality and Responsibility Attribution in Decentralized Partially Observable Markov Decision Processes
Responsibility attribution is complementary and aims to identify the extent to which decision makers (agents) are responsible for this outcome.
Admissible Policy Teaching through Reward Design
This problem can be viewed as a dual to the problem of optimal reward poisoning attacks: instead of forcing an agent to adopt a specific policy, the reward designer incentivizes an agent to avoid taking actions that are inadmissible in certain states.
Teaching via Best-Case Counterexamples in the Learning-with-Equivalence-Queries Paradigm
This learning paradigm has been extensively studied when the learner receives worst-case or random counterexamples; in this paper, we consider the optimal teacher who picks best-case counterexamples to teach the target hypothesis within a hypothesis class.
Explicable Reward Design for Reinforcement Learning Agents
By being explicable, we seek to capture two properties: (a) informativeness so that the rewards speed up the agent's convergence, and (b) sparseness as a proxy for ease of interpretability of the rewards.
Teaching an Active Learner with Contrastive Examples
Our focus is to design a teaching algorithm that can provide an informative sequence of contrastive examples to the learner to speed up the learning process.
Fairness Degrading Adversarial Attacks Against Clustering Algorithms
As a first step, we propose a fairness degrading attack algorithm for k-median clustering that operates under a whitebox threat model -- where the clustering algorithm, fairness notion, and the input dataset are known to the adversary.
Reinforcement Learning Under Algorithmic Triage
Methods to learn under algorithmic triage have predominantly focused on supervised learning settings where each decision, or prediction, is independent of each other.
On Blame Attribution for Accountable Multi-Agent Sequential Decision Making
We formalize desirable properties of blame attribution in the setting of interest, and we analyze the relationship between these properties and the studied blame attribution methods.
Reinforcement Learning for Education: Opportunities and Challenges
This survey article has grown out of the RL4ED workshop organized by the authors at the Educational Data Mining (EDM) 2021 conference.
Curriculum Design for Teaching via Demonstrations: Theory and Applications
In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner's convergence.
Fair Clustering Using Antidote Data
Extensive experiments on different clustering algorithms and fairness notions show that our algorithms can achieve desired levels of fairness on many real-world datasets with a very small percentage of antidote data added.
Loss-Aversively Fair Classification
Motivated by extensive literature in behavioral economics and behavioral psychology (prospect theory), we propose a notion of fair updates that we refer to as loss-averse updates.
Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
We study black-box reward poisoning attacks against reinforcement learning (RL), in which an adversary aims to manipulate the rewards to mislead a sequence of RL agents with unknown algorithms to learn a nefarious policy in an environment unknown to the adversary a priori.
Defense Against Reward Poisoning Attacks in Reinforcement Learning
As a threat model, we consider attacks that minimally alter rewards to make the attacker's target policy uniquely optimal under the poisoned rewards, with the optimality gap specified by an attack parameter.
Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
We provide lower/upper bounds on the attack cost, and instantiate our attacks in two settings: (i) an offline setting where the agent is doing planning in the poisoned environment, and (ii) an online setting where the agent is learning a policy with poisoned feedback.
The Teaching Dimension of Kernel Perceptron
As a warm-up, we show that the teaching complexity is $\Theta(d)$ for the exact teaching of linear perceptrons in $\mathbb{R}^d$, and $\Theta(d^k)$ for kernel perceptron with a polynomial kernel of order $k$.
Preference-Based Batch and Sequential Teaching
We analyze several properties of the teaching complexity parameter $TD(\sigma)$ associated with different families of the preference functions, e. g., comparison to the VC dimension of the hypothesis class and additivity/sub-additivity of $TD(\sigma)$ over disjoint domains.
Average-case Complexity of Teaching Convex Polytopes via Halfspace Queries
We investigate the average teaching complexity of the task, i. e., the minimal number of samples (halfspace queries) required by a teacher to help a version-space learner in locating a randomly selected target.
Environment Shaping in Reinforcement Learning using State Abstraction
However, the applicability of potential-based reward shaping is limited in settings where (i) the state space is very large, and it is challenging to compute an appropriate potential function, (ii) the feedback signals are noisy, and even with shaped rewards the agent could be trapped in local optima, and (iii) changing the rewards alone is not sufficient, and effective shaping requires changing the dynamics.
Synthesizing Tasks for Block-based Programming
Our task synthesis algorithm operates by first mutating code $\rm C^{in}$ to obtain a set of codes $\{\rm C^{out}\}$.
The Sample Complexity of Teaching-by-Reinforcement on Q-Learning
Our TDim results provide the minimum number of samples needed for reinforcement learning, and we discuss their connections to standard PAC-style RL sample complexity and teaching-by-demonstration sample complexity results.
Task-agnostic Exploration in Reinforcement Learning
To address these challenges, we propose the \textit{task-agnostic RL} framework: In the exploration phase, the agent first collects trajectories by exploring the MDP without the guidance of a reward function.
Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker.
Adaptive Reward-Poisoning Attacks against Reinforcement Learning
In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward $r_t$ into $r_t+\delta_t$ at each step, with the goal of forcing the RL agent to learn a nefarious policy.
Understanding the Power and Limitations of Teaching with Imperfect Knowledge
Machine teaching studies the interaction between a teacher and a student/learner where the teacher selects training examples for the learner to learn a specific task.
Learning to Switch Among Agents in a Team via 2-Layer Markov Decision Processes
Reinforcement learning agents have been mostly developed and evaluated under the assumption that they will operate in a fully autonomous manner -- they will take all actions.
Preference-Based Batch and Sequential Teaching: Towards a Unified View of Models
In our framework, each function $\sigma \in \Sigma$ induces a teacher-learner pair with teaching complexity as $\TD(\sigma)$.
Towards Deployment of Robust AI Agents for Human-Machine Partnerships
In the test phase, the AI agent has to interact with a user of unknown type.
Can A User Anticipate What Her Followers Want?
As a result, she may feel compelled to use the feedback she receives to (re-)estimate her followers' preferences and decides which stories to share next to receive more (positive) feedback.
Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints
We study two teaching approaches: learner-agnostic teaching, where the teacher provides demonstrations from an optimal policy ignoring the learner's preferences, and learner-aware teaching, where the teacher accounts for the learner's preferences.
Interactive Teaching Algorithms for Inverse Reinforcement Learning
We study the problem of inverse reinforcement learning (IRL) with the added twist that the learner is assisted by a helpful teacher.
Optimal Decision Making Under Strategic Behavior
Using this characterization, we first show that, in general, we cannot expect to find optimal decision policies in polynomial time and there are cases in which deterministic policies are suboptimal.
On the Fairness of Time-Critical Influence Maximization in Social Networks
As we show in this paper, the time-criticality of the information could further exacerbate the disparity of influence across groups.
Social and Information Networks Computers and Society
Unifying Ensemble Methods for Q-learning via Social Choice Theory
These methods typically work by employing an aggregation mechanism over actions of different RL algorithms.
Learning to Collaborate in Markov Decision Processes
We consider a two-agent MDP framework where agents repeatedly solve a task in a collaborative setting.
Efficient learning of smooth probability functions from Bernoulli tests with guarantees
We study the fundamental problem of learning an unknown, smooth probability function via pointwise Bernoulli tests.
Iterative Classroom Teaching
We consider the machine teaching problem in a classroom-like setting wherein the teacher has to deliver the same examples to a diverse group of students.
Teaching Inverse Reinforcement Learners via Features and Demonstrations
In this paper, we study the problem of learning from demonstrations in the setting where this is not the case, i. e., where there is a mismatch between the worldviews of the learner and the expert.
A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices
Further, our work reveals overlooked tradeoffs between different fairness notions: using our proposed measures, the overall individual-level unfairness of an algorithm can be decomposed into a between-group and a within-group component.
Enhancing the Accuracy and Fairness of Human Decision Making
Societies often rely on human experts to take a wide variety of decisions affecting their members, from jail-or-release decisions taken by judges and stop-and-frisk decisions taken by police officers to accept-or-reject decisions taken by academics.
Teaching Multiple Concepts to a Forgetful Learner
Our framework is both generic, allowing the design of teaching schedules for different memory models, and also interactive, allowing the teacher to adapt the schedule to the underlying forgetting mechanisms of the learner.
Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners
We highlight that adaptivity does not speed up the teaching process when considering existing models of version space learners, such as "worst-case" (the learner picks the next hypothesis randomly from the version space) and "preference-based" (the learner picks hypothesis according to some global preference).
An Overview of Machine Teaching
In this paper we try to organize machine teaching as a coherent set of ideas.
Fake News Detection in Social Networks via Crowd Signals
The main objective of our work is to minimize the spread of misinformation by stopping the propagation of fake news in the network.
Social and Information Networks
Learning User Preferences to Incentivize Exploration in the Sharing Economy
We provide formal guarantees on the performance of our algorithm and test the viability of our approach in a user study with data of apartments on Airbnb.
Learning to Use Learners' Advice
In our setting, the feedback at any time $t$ is limited in a sense that it is only available to the expert $i^t$ that has been selected by the central algorithm (forecaster), \emph{i. e.}, only the expert $i^t$ receives feedback from the environment and gets to learn at time $t$.
Coordinated Online Learning With Applications to Learning User Preferences
We study an online multi-task learning setting, in which instances of related tasks arrive sequentially, and are handled by task-specific online learners.
Actively Learning Hemimetrics with Applications to Eliciting User Preferences
We propose an active learning algorithm that substantially reduces this sample complexity by exploiting the structural constraints on the version space of hemimetrics.
Noisy Submodular Maximization via Adaptive Sampling with Applications to Crowdsourced Image Collection Summarization
When the underlying submodular function is unknown, users' feedback can provide noisy evaluations of the function that we seek to maximize.
Learning to Hire Teams
Furthermore, we consider an embedding of the tasks and workers in an underlying graph that may arise from task similarities or social ties, and that can provide additional side-observations for faster learning.
Crowd Access Path Optimization: Diversity Matters
Based on this intuitive idea, we introduce the Access Path Model (APM), a novel crowd model that leverages the notion of access paths as an alternative way of retrieving information.
Building Hierarchies of Concepts via Crowdsourcing
Hierarchies of concepts are useful in many applications from navigation to organization of objects.
Information Gathering in Networks via Active Exploration
How should we gather information in a network, where each node's visibility is limited to its local neighborhood?
Stochastic Privacy
Users may be willing to share private information in return for better quality of service or for incentives, or in return for assurances about the nature and extend of the logging of data.
Near-Optimally Teaching the Crowd to Classify
How should we present training examples to learners to teach them classification rules?
Incentives for Privacy Tradeoff in Community Sensing
Community sensing, fusing information from populations of privately-held sensors, presents a great opportunity to create efficient and cost-effective sensing applications.
