Search Results for author:
Found 53 papers, 21 papers with code
NICE: Neural Image Commenting with Empathy
Based on these annotations, we define two different tasks for the NICE dataset.
ModelCitizens:Representing Community Voices in Online Safety
Automatic toxic language detection is critical for creating safe, inclusive online spaces.
RADAR: Benchmarking Language Models on Imperfect Tabular Data
Language models (LMs) are increasingly being deployed to perform autonomous data analyses.
X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents
Multi-turn interactions with language models (LMs) pose critical safety risks, as harmful intent can be strategically spread across exchanges.
Review, Refine, Repeat: Understanding Iterative Decoding of AI Agents with Dynamic Evaluation and Selection
Hence, we propose Iterative Agent Decoding (IAD) which combines iterative refinement with dynamic candidate evaluation and selection guided by a verifier.
ShieldGemma 2: Robust and Tractable Image Content Moderation
We introduce ShieldGemma 2, a 4B parameter image content moderation model built on Gemma 3.
In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Large Language Models (LLMs) have made significant progress in open-ended dialogue, yet their inability to retain and retrieve relevant information from long-term interactions limits their effectiveness in applications requiring sustained personalization.
Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation
To address this, we propose Magnet, a principled framework for synthesizing high-quality training trajectories to enhance the function calling capability of large language model agents in multi-turn conversations with humans.
Language Models' Factuality Depends on the Language of Inquiry
Multilingual language models (LMs) are expected to recall factual knowledge consistently across languages, yet they often fail to transfer knowledge between languages even when they possess the correct information in one of the languages.
PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving
Recent agent frameworks and inference-time algorithms often struggle with complex planning problems due to limitations in verifying generated plans or reasoning and varying complexity of instances within a single task.
Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights.
Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies
Large language models, employed as multiple agents that interact and collaborate with each other, have excelled at solving complex tasks.
Reverse Thinking Makes LLMs Stronger Reasoners
Reverse thinking plays a crucial role in human reasoning.
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Extensive experiments demonstrate that Model Swarms could flexibly adapt LLM experts to a single task, multi-task domains, reward models, as well as diverse human interests, improving over 12 model composition baselines by up to 21. 0% across tasks and contexts.
MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmark
When all the instructions are also appended to the end of the model input context, the $\operatorname{PIF}$ metric improves by 22. 3 points on average, showing that the challenge with the task lies not only in following the instructions, but also in retrieving the instructions spread out in the model context.
Eureka: Evaluating and Understanding Large Foundation Models
Second, we introduce Eureka-Bench as an extensible collection of benchmarks testing capabilities that (i) are still challenging for state-of-the-art models and (ii) represent fundamental but overlooked language and multimodal capabilities.
Improving Black-box Robustness with In-Context Rewriting
Machine learning models for text classification often excel on in-distribution (ID) data but struggle with unseen out-of-distribution (OOD) inputs.
Exploring Group and Symmetry Principles in Large Language Models
Large Language Models (LLMs) have demonstrated impressive performance across a wide range of applications; however, assessing their reasoning capabilities remains a significant challenge.
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
We present a novel method to study grounding abilities using Fakepedia, a novel dataset of counterfactual texts constructed to clash with a model's internal parametric knowledge.
Orca 2: Teaching Small Language Models How to Reason
Research on training small LMs has often relied on imitation learning to replicate the output of more capable models.
Ranked #1 on
Crass AI
on BIG-bench
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services.
Diversity of Thought Improves Reasoning Abilities of LLMs
Large language models (LLMs) are documented to struggle in settings that require complex reasoning.
Teaching Language Models to Hallucinate Less with Synthetic Tasks
We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively increase hallucination.
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text.
Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models
Recent studies have revealed that the widely-used Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora.
Improving the Reusability of Pre-trained Language Models in Real-world Applications
The reusability of state-of-the-art Pre-trained Language Models (PLMs) is often limited by their generalization problem, where their performance drastically decreases when evaluated on examples that differ from the training dataset, known as Out-of-Distribution (OOD)/unseen examples.
Orca: Progressive Learning from Complex Explanation Traces of GPT-4
To address these challenges, we develop Orca (We are working with our legal team to publicly release a diff of the model weights in accordance with LLaMA's release policy to be published at https://aka. ms/orca-lm), a 13-billion parameter model that learns to imitate the reasoning process of LFMs.
Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning
Spurious correlations that degrade model generalization or lead the model to be right for the wrong reasons are one of the main robustness concerns for real-world deployments.
Sparks of Artificial General Intelligence: Early experiments with GPT-4
We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models.
Ranked #36 on
Arithmetic Reasoning
on GSM8K
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models
Large-scale Pre-Trained Language Models (PTLMs) capture knowledge from massive human-written data which contains latent societal biases and toxic contents.
A Large-Scale Robustness Analysis of Video Action Recognition Models
In this work, we perform a large-scale robustness analysis of these existing models for video action recognition.
Benchmarking Spatial Relationships in Text-to-Image Generation
We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image.
Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
We propose GRACE, a lifelong model editing method, which implements spot-fixes on streaming errors of a deployed model, ensuring minimal impact on unrelated inputs.
Deep Learning on a Healthy Data Diet: Finding Important Examples for Fairness
The fairness achieved by our method surpasses that of data augmentation on three text classification datasets, using no more than half of the examples in the augmented dataset.
NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?
While a substantial body of prior work has explored adversarial example generation for natural language understanding tasks, these examples are often unrealistic and diverge from the real-world data distributions.
Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
Machine translation has seen rapid progress with the advent of Transformer-based models.
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations
Joint visual and language modeling on large-scale datasets has recently shown good progress in multi-modal tasks when compared to single modal learning.
Large-scale Robustness Analysis of Video Action Recognition Models
In this work, we perform a large-scale robustness analysis of these existing models for video action recognition.
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups.
Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and improved syntactic interpretability in the TPR layer outputs.
Compositional Processing Emerges in Neural Networks Solving Math Problems
A longstanding question in cognitive science concerns the learning mechanisms underlying compositionality in human cognition.
Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language
In this paper, we propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning.
Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"
To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception.
Novel Human-Object Interaction Detection via Adversarial Domain Generalization
We study in this paper the problem of novel human-object interaction (HOI) detection, aiming at improving the generalization ability of the model to unseen scenarios.
HUBERT Untangles BERT to Improve Transfer across NLP Tasks
We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model.
Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations
The encoder of TP-N2F employs TPR `binding' to encode natural-language symbolic structure in vector space and the decoder uses TPR `unbinding' to generate, in symbolic space, a sequential program represented by relational tuples, each consisting of a relation (or operation) and a number of arguments.
Natural- to formal-language generation using Tensor Product Representations
Generating formal-language represented by relational tuples, such as Lisp programs or mathematical expressions, from a natural-language input is an extremely challenging task because it requires to explicitly capture discrete symbolic structural information from the input to generate the output.
Unified Vision-Language Pre-Training for Image Captioning and VQA
The model is unified in that (1) it can be fine-tuned for either vision-language generation (e. g., image captioning) or understanding (e. g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models.
Ranked #1 on
Image Captioning
on Flickr30k Captions test
Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
In this work, we tackle two fundamental language-and-vision tasks: image-text matching and image captioning, and demonstrate that neural scene graph generators can learn effective visual relation features to facilitate grounding language to visual relations and subsequently improve the two end applications.
Question-Answering with Grammatically-Interpretable Representations
In our application of TPRN, internal representations learned by end-to-end optimization in a deep neural network performing a textual question-answering (QA) task can be interpreted using basic concepts from linguistic theory.
Distributed Compressive Sensing: A Deep Learning Approach
As the proposed method is a data driven method, it is only applicable when training data is available.
Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval
The results show that the proposed method in this paper significantly outperforms it for web document retrieval task.
Learning Input and Recurrent Weight Matrices in Echo State Networks
In this paper, we devise a special technique that take advantage of this linearity in the output units of an ESN, to learn the input and recurrent matrices.
