Search Results for author:
Found 18 papers, 8 papers with code
Language-Guided Image Tokenization for Generation
Compared to the conventional tokenizer without text conditioning, TexTok achieves average reconstruction FID improvements of 29. 2% and 48. 1% on ImageNet-256 and -512 benchmarks respectively, across varying numbers of tokens.
SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code
SceneCraft first models a scene graph as a blueprint, detailing the spatial relationships among assets in the scene.
VideoPrism: A Foundational Visual Encoder for Video Understanding
We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model.
VideoPoet: A Large Language Model for Zero-Shot Video Generation
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals.
Ranked #4 on
Text-to-Video Generation
on MSR-VTT
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation.
Ranked #2 on
Video Generation
on Kinetics-600 12 frames, 64x64
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos.
IC3: Image Captioning by Committee Consensus
If you ask a human to describe an image, they might do so in a thousand different ways.
Open-Vocabulary Temporal Action Detection with Off-the-Shelf Image-Text Features
Detecting actions in untrimmed videos should not be limited to a small, closed set of classes.
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
REVEAL consists of four key components: the memory, the encoder, the retriever and the generator.
Ranked #9 on
Visual Question Answering (VQA)
on OK-VQA
What's in a Caption? Dataset-Specific Linguistic Diversity and Its Effect on Visual Description Models and Metrics
While there have been significant gains in the field of automated video description, the generalization performance of automated description models to novel domains remains a major barrier to using these systems in the real world.
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music.
Ranked #2 on
Motion Synthesis
on BRACE
Learning Video Representations from Textual Web Supervision
Based on this observation, we propose to use text as a method for learning video representations.
Active Learning for Video Description With Cluster-Regularized Ensemble Ranking
Automatic video captioning aims to train models to generate text descriptions for all segments in a video, however, the most effective approaches require large amounts of manual annotation which is slow and expensive.
An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
We use a U-Net style 3D sparse convolution network to extract features for each frame's LiDAR point-cloud.
The AVA-Kinetics Localized Human Actions Video Dataset
The dataset is collected by annotating videos from the Kinetics-700 dataset using the AVA annotation protocol, and extending the original AVA dataset with these new AVA annotated Kinetics clips.
D3D: Distilled 3D Networks for Video Action Recognition
State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input.
Ranked #11 on
Action Recognition
on AVA v2.1
Rethinking the Faster R-CNN Architecture for Temporal Action Localization
We propose TAL-Net, an improved approach to temporal action localization in video that is inspired by the Faster R-CNN object detection framework.
Ranked #29 on
Temporal Action Localization
on THUMOS’14
AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1. 58M action labels with multiple labels per person occurring frequently.
Ranked #7 on
Action Detection
on UCF101-24
