Search Results for author:
Found 22 papers, 13 papers with code
High Confidence Level Inference is Almost Free using Parallel Stochastic Optimization
We provide a rigorous theoretical guarantee for the confidence interval, demonstrating that the coverage is approximately exact with an explicit convergence rate and allowing for high confidence level inference.
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
We first benchmark MM-Navigator on our collected iOS screen dataset.
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use
These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment.
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters.
Ranked #14 on
Visual Question Answering (VQA)
on InfiMM-Eval
Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality
Stochastic Gradient Descent (SGD) is one of the simplest and most popular algorithms in modern statistical and machine learning due to its computational and memory efficiency.
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View
In this work, we propose VELMA, an embodied LLM agent that uses a verbalization of the trajectory and of visual environment observations as contextual prompt for the next action.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness.
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation
We conduct a series of experiments to compare the common edits made by humans and GPT-k, evaluate the performance of GPT-k in prompting T2I, and examine factors that may influence this process.
Multimodal Procedural Planning via Dual Text-Image Prompting
The key challenges of MPP are to ensure the informativeness, temporal coherence, and accuracy of plans across modalities.
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
We release Multimodal C4, an augmentation of the popular text-only C4 corpus with images interleaved.
Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models.
CLIP also Understands Text: Prompting CLIP for Phrase Understanding
However, the application of its text encoder solely for text understanding has been less explored.
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation
Recent advances in text-to-image synthesis make it possible to visualize machine imaginations for a given context.
Neuro-Symbolic Procedural Planning with Commonsense Prompting
Procedural planning aims to implement complex high-level goals by decomposition into sequential simpler low-level steps.
Imagination-Augmented Natural Language Understanding
Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations.
End-to-end Dense Video Captioning as Sequence Generation
Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event.
Ranked #3 on
Dense Video Captioning
on ViTT
(CIDEr metric, using extra
training data)
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references.
Diagnosing Vision-and-Language Navigation: What Really Matters
Results show that indoor navigation agents refer to both object and direction tokens when making decisions.
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations
A major challenge in visually grounded language generation is to build robust benchmark datasets and models that can generalize well in real-world settings.
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natural language instructions and navigates a real-life urban environment.
Ranked #4 on
Vision and Language Navigation
on Touchdown Dataset
(using extra training data)
Online Covariance Matrix Estimation in Stochastic Gradient Descent
This approach fits in an online setting and takes full advantage of SGD: efficiency in computation and memory.
