Search Results for author:
Found 27 papers, 14 papers with code
REOrdering Patches Improves Vision Models
REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to 3. 01% and Functional Map of the World by 13. 35%.
Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint
Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs).
LISAT: Language-Instructed Segmentation Assistant for Satellite Imagery
Segmentation models can recognize a pre-defined set of objects in images.
Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations, where they generate descriptions of nonexistent objects, actions, or concepts, posing significant risks in safety-critical applications.
Higher-Order Binding of Language Model Virtual Personas: a Study on Approximating Political Partisan Misperceptions
While previous studies have examined whether models can reflect individual opinions or attitudes, we argue that a \emph{higher-order} binding of virtual personas requires successfully approximating not only the opinions of a user as an identified member of a group, but also the nuanced ways in which that user perceives and evaluates those outside the group.
TULIP: Towards Unified Language-Image Pretraining
These models, by performing language alignment, tend to prioritize high-level semantics over visual understanding, weakening their image understanding.
Analyzing The Language of Visual Tokens
Through these experiments, we demonstrate how understanding the statistical properties of discrete visual languages can inform the design of more effective computer vision models.
CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
The Automated Audio Captioning (AAC) task asks models to generate natural language descriptions of an audio input.
Rediscovering the Latent Dimensions of Personality with Large Language Models as Trait Descriptors
Assessing personality traits using large language models (LLMs) has emerged as an interesting and challenging area of research.
An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems
Dialog systems, such as voice assistants, are expected to engage with users in complex, evolving conversations.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
MIRAGE demonstrates up to 13% performance improvement over existing open-source LMMs on VHs, sets a new state-of-the-art on the RetVQA multi-image QA benchmark, and achieves competitive performance on single-image QA with state-of-the-art LMMs.
Virtual Personas for Language Models via an Anthology of Backstories
Large language models (LLMs) are trained from vast repositories of text authored by millions of distinct authors, reflecting an enormous diversity of human traits.
ALOHa: A New Measure for Hallucination in Captioning Models
Despite recent advances in multimodal pre-training for visual description, state-of-the-art models still produce captions containing errors, such as hallucinating objects not present in a scene.
ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video
The Internet's wealth of content, with up to 60% published in English, starkly contrasts the global population, where only 18. 8% are English speakers, and just 5. 1% consider it their native language, leading to disparities in online information access.
Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognition
We demonstrate that our CLC family of approaches can improve the performance of ASR models on OD3, a new public large-scale semi-synthetic meta-dataset of audio task-oriented dialogues, by up to 19. 2%.
Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification
In cases where some data/compute is available, we present Learnable-MAM, a data-driven approach to merging attention matrices, resulting in a further 2. 90% relative reduction in WER for ASR and 18. 42% relative reduction in AEC compared to fine-tuning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
IC3: Image Captioning by Committee Consensus
If you ask a human to describe an image, they might do so in a thousand different ways.
Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition
Despite improvements to the generalization performance of automated speech recognition (ASR) models, specializing ASR models for downstream tasks remains a challenging task, primarily due to reduced data availability (necessitating increased data collection), and rapidly shifting data distributions (requiring more frequent model fine-tuning).
Towards Understanding How Machines Can Learn Causal Overhypotheses
Recent work in machine learning and cognitive science has suggested that understanding causal information is essential to the development of intelligence.
Content-Context Factorized Representations for Automated Speech Recognition
Deep neural networks have largely demonstrated their ability to perform automated speech recognition (ASR) by extracting meaningful features from input audio frames.
What's in a Caption? Dataset-Specific Linguistic Diversity and Its Effect on Visual Description Models and Metrics
While there have been significant gains in the field of automated video description, the generalization performance of automated description models to novel domains remains a major barrier to using these systems in the real world.
Multi-Modal Pre-Training for Automated Speech Recognition
Traditionally, research in automated speech recognition has focused on local-first encoding of audio representations to predict the spoken phonemes in an utterance.
A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery
Urban areas consume over two-thirds of the world's energy and account for more than 70 percent of global CO2 emissions.
Active Learning for Video Description With Cluster-Regularized Ensemble Ranking
Automatic video captioning aims to train models to generate text descriptions for all segments in a video, however, the most effective approaches require large amounts of manual annotation which is slow and expensive.
Exploring Exploration: Comparing Children with RL Agents in Unified Environments
Research in developmental psychology consistently shows that children explore the world thoroughly and efficiently and that this exploration allows them to learn.
Diagnostic Visualization for Deep Neural Networks Using Stochastic Gradient Langevin Dynamics
Finally we show that diagnostic visualization using LDAM leads to a novel insight into the parameter averaging method for deep net training.
t-SNE-CUDA: GPU-Accelerated t-SNE and its Applications to Modern Data
Modern datasets and models are notoriously difficult to explore and analyze due to their inherent high dimensionality and massive numbers of samples.
