Search Results for author:
Found 40 papers, 11 papers with code
Using LLMs to Advance the Cognitive Science of Collectives
LLMs are already transforming the study of individual cognition, but their application to studying collective cognition has been underexplored.
Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships
Artificial Intelligence (AI) systems have historically been used as tools that execute narrowly defined tasks.
When Should We Orchestrate Multiple Agents?
Strategies for orchestrating the interactions between multiple agents, both human and artificial, can wildly overestimate performance and underestimate the cost of orchestration.
Humanity's Last Exam
However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities.
Ranked #4 on
Humanity's Last Exam
on Humanity's Last Exam
Revisiting Rogers' Paradox in the Context of Human-AI Interaction
We consider strategies that can be undertaken by various stakeholders involved in a single human-AI interaction: human, AI model builder, and society or regulators around the interaction.
Quantifying Knowledge Distillation Using Partial Information Decomposition
We theoretically demonstrate that the task-relevant transferred knowledge is succinctly captured by the measure of redundant information about the task between the teacher and student.
Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
In this paper, we seek to identify the characteristics of tasks where CoT reduces performance by drawing inspiration from cognitive psychology, focusing on six representative tasks from the psychological literature where deliberation hurts performance in humans.
Adaptive Language-Guided Abstraction from Contrastive Explanations
Many approaches to robot learning begin by inferring a reward function from a set of human demonstrations.
Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience
As deep learning systems are scaled up to many billions of parameters, relating their internal structure to external behaviors becomes very challenging.
Quantifying Spuriousness of Biased Datasets Using Partial Information Decomposition
Spurious patterns refer to a mathematical association between two or more variables in a dataset that are not causally related.
Modulating Language Model Experiences through Frictions
Through a user study with real humans, we observe shifts in user behavior from the imposition of a friction over LLMs in the context of a multi-topic question-answering task as a representative task that people may use LLMs for, e. g., in education and information retrieval.
Large Language Models Assume People are More Rational than We Really are
However, by comparing LLM behavior and predictions to a large dataset of human decisions, we find that this is actually not the case: when both simulating and predicting people's choices, a suite of cutting-edge LLMs (GPT-4o & 4-Turbo, Llama-3-8B & 70B, Claude 3 Opus) assume that people are more rational than we really are.
Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People
We run 100 iterations of this process with human participants and GPT-4, then obtain a dataset of sentences and frequent conversational tones.
Representational Alignment Supports Effective Machine Teaching
A good teacher should not only be knowledgeable; but should be able to communicate in a way that the student understands -- to share the student's representation of the world.
Large language models surpass human experts in predicting neuroscience results
LLMs trained on the vast scientific literature could potentially integrate noisy yet interrelated findings to forecast novel results better than human experts.
Learning with Language-Guided State Abstractions
We describe a framework for using natural language to design state abstractions for imitation learning.
Analyzing the Roles of Language and Vision in Learning from Limited Data
Does language help make sense of the visual world?
A Rational Analysis of the Speech-to-Song Illusion
Here we provide a formal account of this phenomenon, by recasting it as a statistical inference whereby a rational agent attempts to decide whether a sequence of utterances is more likely to have been produced in a song or speech.
Measuring Implicit Bias in Explicitly Unbiased Large Language Models
Large language models (LLMs) can pass explicit social bias tests but still harbor implicit biases, similar to humans who endorse egalitarian beliefs yet exhibit subtle biases.
Preference-Conditioned Language-Guided Abstraction
We observe that how humans behave reveals how they see the world.
Concept Alignment
Discussion of AI alignment (alignment between humans and AI systems) has focused on value alignment, broadly referring to creating AI systems that share human values.
Learning Human-like Representations to Enable Learning Human Values
Using a set of textual action descriptions, we collect value judgments from humans, as well as similarity judgments from both humans and multiple language models, and demonstrate that representational alignment enables both safe exploration and improved generalization when learning human values.
Concept Alignment as a Prerequisite for Value Alignment
Value alignment is essential for building AI systems that can safely and reliably interact with people.
Getting aligned on representational alignment
These questions pertaining to the study of representational alignment are at the heart of some of the most promising research areas in contemporary cognitive science, neuroscience, and machine learning.
Dimensions of Disagreement: Unpacking Divergence and Misalignment in Cognitive Science and Artificial Intelligence
The increasing prevalence of artificial agents creates a correspondingly increasing need to manage disagreements between humans and artificial agents, as well as between artificial agents themselves.
Human Uncertainty in Concept-Based AI Systems
We study how existing concept-based models deal with uncertain interventions from humans using two novel datasets: UMNIST, a visual dataset with controlled simulated uncertainty based on the MNIST dataset, and CUB-S, a relabeling of the popular CUB concept dataset with rich, densely-annotated soft labels from humans.
Around the world in 60 words: A generative vocabulary test for online research
Six behavioral experiments (N=236) in six countries and eight languages show that (a) our test can distinguish between native speakers of closely related languages, (b) the test is reliable ($r=0. 82$), and (c) performance strongly correlates with existing tests (LexTale) and self-reports.
Large language models predict human sensory judgments across six modalities
Determining the extent to which the perceptual world can be recovered from language is a longstanding problem in philosophy and cognitive science.
On the Informativeness of Supervision Signals
Supervised learning typically focuses on learning transferable representations from training examples annotated by humans.
Human-in-the-Loop Mixup
We focus on the synthetic data used in mixup: a powerful regularizer shown to improve model robustness, generalization, and calibration.
Analyzing Diffusion as Serial Reproduction
Diffusion models are a class of generative models that learn to synthesize samples by inverting a diffusion process that gradually maps data into noise.
Words are all you need? Language as an approximation for human similarity judgments
Based on the results of this comprehensive study, we provide a concise guide for researchers interested in collecting or approximating human similarity data.
Predicting Human Similarity Judgments Using Large Language Models
Similarity judgments provide a well-established method for accessing mental representations, with applications in psychology, neuroscience and machine learning.
Can Humans Do Less-Than-One-Shot Learning?
Being able to learn from small amounts of data is a key characteristic of human intelligence, but exactly {\em how} small?
One Line To Rule Them All: Generating LO-Shot Soft-Label Prototypes
We propose a novel, modular method for generating soft-label prototypical lines that still maintains representational accuracy even when there are fewer prototypes than the number of classes in the data.
Optimal 1-NN Prototypes for Pathological Geometries
Using prototype methods to reduce the size of training datasets can drastically reduce the computational cost of classification with instance-based learning algorithms like the k-Nearest Neighbour classifier.
SecDD: Efficient and Secure Method for Remotely Training Neural Networks
We leverage what are typically considered the worst qualities of deep learning algorithms - high computational cost, requirement for large data, no explainability, high dependence on hyper-parameter choice, overfitting, and vulnerability to adversarial perturbations - in order to create a method for the secure and efficient training of remotely deployed neural networks over unsecured channels.
'Less Than One'-Shot Learning: Learning N Classes From M<N Samples
We propose the `less than one'-shot learning task where models must learn $N$ new classes given only $M<N$ examples and we show that this is achievable with the help of soft labels.
Soft-Label Dataset Distillation and Text Dataset Distillation
We propose to simultaneously distill both images and their labels, thus assigning each synthetic sample a `soft' label (a distribution of labels).
Deep Learning for System Trace Restoration
The output of the model will be a close reconstruction of the true data, and can be fed to algorithms that rely on clean data.
