Search Results for author:
Found 36 papers, 5 papers with code
Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
Magentic-One uses a multi-agent architecture where a lead agent, the Orchestrator, plans, tracks progress, and re-plans to recover from errors.
Challenging the Machine: Contestability in Government AI Systems
In an October 2023 executive order (EO), President Biden issued a detailed but largely aspirational road map for the safe and responsible development and use of artificial intelligence (AI).
Recommendations for Government Development and Use of Advanced Automated Systems to Make Decisions about Individuals
Contestability -- the ability to effectively challenge a decision -- is critical to the implementation of fairness.
Teaching Language Models to Hallucinate Less with Synthetic Tasks
We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively increase hallucination.
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text.
Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions
However, creating high-quality datasets with LLMs can be challenging.
Sparks of Artificial General Intelligence: Early experiments with GPT-4
We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models.
Ranked #33 on
Arithmetic Reasoning
on GSM8K
Benchmarking Spatial Relationships in Text-to-Image Generation
We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image.
Artificial Intelligence and Life in 2030: The One Hundred Year Study on Artificial Intelligence
In September 2016, Stanford's "One Hundred Year Study on Artificial Intelligence" project (AI100) issued the first report of its planned long-term periodic assessment of artificial intelligence (AI) and its impact on society.
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups.
Investigations of Performance and Bias in Human-AI Teamwork in Hiring
In AI-assisted decision-making, effective hybrid (human-AI) teamwork is not solely dependent on AI performance alone, but also on its impact on human decision-making.
A Bayesian Approach to Identifying Representational Errors
Trained AI systems and expert decision makers can make errors that are often difficult to identify and understand.
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple.
Algorithmic Recourse in the Wild: Understanding the Impact of Data and Model Shifts
Our theoretical results establish a lower bound on the probability of recourse invalidation due to model shifts, and show the existence of a tradeoff between this invalidation probability and typical notions of "cost" minimized by modern recourse generation algorithms.
Understanding Failures of Deep Networks via Robust Feature Extraction
Traditional evaluation metrics for learned models that report aggregate scores over a test set are insufficient for surfacing important and informative patterns of failure over features and instances.
Avoiding Negative Side Effects due to Incomplete Knowledge of AI Systems
Learning to recognize and avoid such negative side effects of an agent's actions is critical to improve the safety and reliability of autonomous systems.
An Empirical Analysis of Backward Compatibility in Machine Learning Systems
In many applications of machine learning (ML), updates are performed with the goal of enhancing model performance.
Security and Machine Learning in the Real World
Machine learning (ML) models deployed in many safety- and business-critical systems are vulnerable to exploitation through adversarial examples.
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team.
Learning to Complement Humans
A rising vision for AI in the open world centers on the development of systems that can complement humans for perceptual, diagnostic, and reasoning tasks.
Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork
To optimize the team performance for this setting we maximize the team's expected utility, expressed in terms of the quality of the final decision, cost of verifying, and individual accuracies of people and machines.
Personalization in Human-AI Teams: Improving the Compatibility-Accuracy Tradeoff
AI systems that model and interact with users can update their models over time to reflect new information and changes in the environment.
SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions
We quantify the extent to which this phenomenon occurs by creating a new Reasoning split of the VQA dataset and collecting VQA-introspect, a new dataset1 which consists of 238K new perception questions which serve as sub questions corresponding to the set of perceptual tasks needed to effectively answer the complex reasoning questions in the Reasoning split.
What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring
Although systematic biases in decision-making are widely documented, the ways in which they emerge from different sources is less understood.
Toward Fairness in AI for People with Disabilities: A Research Roadmap
AI technologies have the potential to dramatically impact the lives of people with disabilities (PWD).
A Case for Backward Compatibility for Human-AI Teams
We introduce the notion of the compatibility of an AI update with prior user experience and present methods for studying the role of compatibility in human-AI teams.
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
We present Pandora, a set of hybrid human-machine methods and tools for describing and explaining system failures.
Investigating Human + Machine Complementarity for Recidivism Predictions
Dressel and Farid (2018) asked Mechanical Turk workers to evaluate a subset of defendants in the ProPublica COMPAS data for risk of recidivism, and concluded that COMPAS predictions were no more accurate or fair than predictions made by humans.
Discovering Blind Spots in Reinforcement Learning
Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments.
Interpretable & Explorable Approximations of Black Box Models
To the best of our knowledge, this is the first approach which can produce global explanations of the behavior of any given black box model through joint optimization of unambiguity, fidelity, and interpretability, while also allowing users to explore model behavior based on their preferences.
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems
We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components.
Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration
Predictive models deployed in the real world may assign incorrect labels to instances with high confidence.
Metareasoning for Planning Under Uncertainty
Our work subsumes previously studied special cases of metareasoning and shows that in the general case, metareasoning is at most polynomially harder than solving MDPs with any given algorithm that disregards the cost of thinking.
Stochastic Privacy
Users may be willing to share private information in return for better quality of service or for incentives, or in return for assurances about the nature and extend of the logging of data.
