Search Results for author:
Found 31 papers, 11 papers with code
Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli
We present the Brain Treebank, a large-scale dataset of electrophysiological neural responses, recorded from intracranial probes while 10 subjects watched one or more Hollywood movies.
BrainBits: How Much of the Brain are Generative Reconstruction Methods Using?
When evaluating stimuli reconstruction results it is tempting to assume that higher fidelity text and image generation is due to an improved understanding of the brain or more powerful signal extraction from neural recordings.
Using Multimodal Deep Neural Networks to Disentangle Language from Visual Aesthetics
We show that unimodal vision models (e. g. SimCLR) account for the vast majority of explainable variance in these ratings.
Training the Untrainable: Introducing Inductive Bias via Representational Alignment
If the guide is trained, this transfers over part of the architectural prior and knowledge of the guide to the target.
Baba Is AI: Break the Rules to Beat the Benchmark
Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives.
Revealing Vision-Language Integration in the Brain with Multimodal Networks
We operationalize sites of multimodal integration as regions where a multimodal vision-language model predicts recordings better than unimodal language, unimodal vision, or linearly-integrated language-vision models.
Population Transformer: Learning Population-level Representations of Neural Activity
We present a self-supervised framework that learns population-level codes for arbitrary ensembles of neural recordings at scale.
SecureLLM: Using Compositionality to Build Provably Secure Language Models for Private, Sensitive, and Secret Data
We reflect the compositional nature of such security mechanisms back into the structure of LLMs to build a provably secure LLM; that we term SecureLLM.
BrainBERT: Self-supervised representation learning for intracranial recordings
We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience.
Can Machines Imitate Humans? Integrative Turing Tests for Vision and Language Demonstrate a Narrowing Gap
To address this question, we turn to the Turing test and systematically benchmark current AIs in their abilities to imitate humans in three language tasks (Image captioning, Word association, and Conversation) and three vision tasks (Object detection, Color estimation, and Attention prediction).
Query The Agent: Improving sample efficiency through epistemic uncertainty estimation
Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sample efficiency.
Developing a Series of AI Challenges for the United States Department of the Air Force
Through a series of federal initiatives and orders, the U. S. Government has been making a concerted effort to ensure American leadership in AI.
On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation
Critically, we find that random cropping can be substituted by cortical magnification, and saccade-like sampling of the image could also assist the representation learning.
Neural Regression, Representational Similarity, Model Zoology & Neural Taskonomy at Scale in Rodent Visual Cortex
Using our benchmark as an atlas, we offer preliminary answers to overarching questions about levels of analysis (e. g. do models that better predict the representations of individual neurons also predict representational similarity across neural populations?
Trajectory Prediction with Linguistic Representations
Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions.
Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset
These results show that models trained on other datasets and then evaluated on Spoken ObjectNet tend to perform poorly due to biases in other datasets that the models have learned.
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation.
Multi-resolution modeling of a discrete stochastic process identifies cusses of cancer
Detection of cancer-causing mutations within the vast and mostly unexploredhuman genome is a major challenge.
Learning a natural-language to LTL executable semantic parser for grounded robotics
We take a step toward robots that can do the same by training a grounded semantic parser, which discovers latent linguistic representations that can be used for the execution of natural-language commands.
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding
Recent work has shown that while deep networks can mimic some human language abilities when presented with novel sentences, systematic variation uncovers the limitations in the language-understanding abilities of networks.
Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas
We demonstrate a reinforcement learning agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions.
Measuring Social Biases in Grounded Vision and Language Embeddings
We generalize the notion of social biases from language embeddings to grounded vision and language embeddings.
Deep compositional robotic planners that follow natural language commands
We demonstrate how a sampling-based robotic planner can be augmented to learn to understand a sequence of natural language commands in a continuous configuration space to move and manipulate objects.
ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models
Although we focus on object recognition here, data with controls can be gathered at scale using automated tools throughout machine learning to generate datasets that exercise models in new ways thus providing valuable feedback to researchers.
Ranked #51 on
Image Classification
on ObjectNet
(using extra training data)
Grounding language acquisition by training semantic parsers using captioned videos
We develop a semantic parser that is trained in a grounded setting using pairs of videos captioned with sentences.
Anchoring and Agreement in Syntactic Annotations
Our agreement results control for parser bias, and are consequential in that they are on par with state of the art parsing performance for English newswire.
Do You See What I Mean? Visual Resolution of Linguistic Ambiguities
Understanding language goes hand in hand with the ability to integrate complex contextual information obtained via perception.
Video In Sentences Out
We present a system that produces sentential descriptions of video: who did what to whom, and where and how they did it.
Saying What You're Looking For: Linguistics Meets Video Search
We present an approach to searching large video corpora for video clips which depict a natural-language query in the form of a sentence.
Seeing What You're Told: Sentence-Guided Activity Recognition In Video
We present a system that demonstrates how the compositional structure of events, in concert with the compositional structure of language, can interplay with the underlying focusing mechanisms in video action recognition, thereby providing a medium, not only for top-down and bottom-up integration, but also for multi-modal integration between vision and language.
Recognize Human Activities from Partially Observed Videos
In this paper, we propose a new method that can recognize human activities from partially observed videos in the general case.
