Search Results for author:
Found 79 papers, 39 papers with code
SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
We approach this problem through the real-data-free setting in which the model is trained only on 3D simulation data and applied out-of-the-box to a wide variety of real cameras.
Ranked #1 on
Trajectory Forecasting
on ActEV
Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model
This technical report highlights the key design decisions that enhance the performance of the medium-sized diffusion model.
Synthetic Video Enhances Physical Fidelity in Video Synthesis
We investigate how to enhance the physical fidelity of video generation models by leveraging synthetic videos derived from computer graphics pipelines.
Long Context Tuning for Video Generation
Recent advances in video generation can produce realistic, minute-long single-shot videos with scalable diffusion transformers.
UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?
To evaluate the performance of automatic metrics in unified AIGV evaluation, we introduce a benchmark called UVE-Bench.
CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models
This paper introduces CameraCtrl II, a framework that enables large-scale dynamic scene exploration through a camera-controlled video diffusion model.
Diffusion Adversarial Post-Training for One-Step Video Generation
The diffusion models are widely used for image and video generation, but their iterative generation process is slow and expansive.
VideoAuteur: Towards Long Narrative Video Generation
We further introduce a Long Narrative Video Director to enhance both visual and semantic coherence in generated videos and emphasize the role of aligning visual embeddings to achieve improved overall video quality.
SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration
Video restoration poses non-trivial challenges in maintaining fidelity while recovering temporally consistent details from unknown degradations in the wild.
FullTransNet: Full Transformer with Local-Global Attention for Video Summarization
Based on this, we propose a transformer-like architecture, named FullTransNet, which has a full encoder-decoder structure with local-global sparse attention for video summarization.
Is Your Text-to-Image Model Robust to Caption Noise?
In text-to-image (T2I) generation, a prevalent training technique involves utilizing Vision Language Models (VLMs) for image re-captioning.
Is Precise Recovery Necessary? A Task-Oriented Imputation Approach for Time Series Forecasting on Variable Subset
Variable Subset Forecasting (VSF) refers to a unique scenario in multivariate time series forecasting, where available variables in the inference phase are only a subset of the variables in the training phase.
Pubic Symphysis-Fetal Head Segmentation Network Using BiFormer Attention Mechanism and Multipath Dilated Convolution
To address this issue, we introduce a dynamic, query-aware sparse attention mechanism for ultrasound image segmentation.
PSFHS Challenge Report: Pubic Symphysis and Fetal Head Segmentation from Intrapartum Ultrasound Images
This challenge aimed to enhance the development of automatic segmentation algorithms at an international scale, providing the largest dataset to date with 5, 101 intrapartum ultrasound images collected from two ultrasound machines across three hospitals from two institutions.
RATT: A Thought Structure for Coherent and Correct LLM Reasoning
To address these limitations, we introduce the Retrieval Augmented Thought Tree (RATT), a novel thought structure that considers both overall logical soundness and factual correctness at each step of the thinking process.
A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation
Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process.
TFWT: Tabular Feature Weighting with Transformer
In this paper, we propose a novel feature weighting method to address the limitation of existing feature processing methods for tabular data.
An Adaptive Cost-Sensitive Learning and Recursive Denoising Framework for Imbalanced SVM Classification
The traditional machine learning method tends to favor the majority class, which leads to the lack of minority class information in the model.
BRAU-Net++: U-Shaped Hybrid CNN-Transformer Network for Medical Image Segmentation
Specifically, BRAU-Net++ uses bi-level routing attention as the core building block to design our u-shaped encoder-decoder structure, in which both encoder and decoder are hierarchically constructed, so as to learn global semantic information while reducing computational complexity.
Spatial-Temporal Interplay in Human Mobility: A Hierarchical Reinforcement Learning Approach with Hypergraph Representation
In the realm of human mobility, the decision-making process for selecting the next-visit location is intricately influenced by a trade-off between spatial and temporal constraints, which are reflective of individual needs and preferences.
VideoPoet: A Large Language Model for Zero-Shot Video Generation
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals.
Ranked #4 on
Text-to-Video Generation
on MSR-VTT
Photorealistic Video Generation with Diffusion Models
We present W. A. L. T, a transformer-based approach for photorealistic video generation via diffusion modeling.
Ranked #1 on
Video Generation
on UCF-101
(using extra training data)
Fine-grained Controllable Video Generation via Object Appearance and Context
To achieve detailed control, we propose a unified framework to jointly inject control signals into the existing text-to-video model.
Text-Driven Image Editing via Learnable Regions
Language has emerged as a natural interface for image editing.
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation.
Ranked #2 on
Video Generation
on Kinetics-600 12 frames, 64x64
Pubic Symphysis-Fetal Head Segmentation Using Pure Transformer with Bi-level Routing Attention
In this paper, we propose a method, named BRAU-Net, to solve the pubic symphysis-fetal head segmentation task.
VideoGLUE: Video General Understanding Evaluation of Foundation Models
We evaluate the video understanding capabilities of existing foundation models (FMs) using a carefully designed experiment protocol consisting of three hallmark tasks (action recognition, temporal localization, and spatiotemporal localization), eight datasets well received by the community, and four adaptation methods tailoring an FM for downstream tasks.
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos.
StyleDrop: Text-to-Image Generation in Any Style
Pre-trained large text-to-image models synthesize impressive images with an appropriate use of text prompts.
Learning Disentangled Prompts for Compositional Image Synthesis
We study domain-adaptive image synthesis, the problem of teaching pretrained image generative models a new style or concept from as few as one image to synthesize novel images, to better understand the compositional image synthesis.
Auditing Gender Presentation Differences in Text-to-Image Models
Text-to-image models, which can generate high-quality images based on textual input, have recently enabled various content-creation tools.
Muse: Text-To-Image Generation via Masked Generative Transformers
Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding.
Ranked #1 on
Text-to-Image Generation
on MS-COCO
(FID metric)
A Multi-Source Information Learning Framework for Airbnb Price Prediction
Thirdly, we uses the points of interest(POI) around the rental house information generates a variety of spatial network graphs, and learns the embedding of the network to obtain the spatial feature embedding.
Multi-View MOOC Quality Evaluation via Information-Aware Graph Representation Learning
To tackle the challenges, we formulate the problem as a course representation learning task-based and develop an Information-aware Graph Representation Learning(IaGRL) for multi-view MOOC quality evaluation.
Streaming Traffic Flow Prediction Based on Continuous Reinforcement Learning
We propose to formulate the problem as a continuous reinforcement learning task, where the agent is the next flow value predictor, the action is the next time-series flow value in the sensor, and the environment state is a dynamically fused representation of the sensor and transportation network.
MAGVIT: Masked Generative Video Transformer
We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model.
Ranked #1 on
Video Prediction
on Something-Something V2
Visual Prompt Tuning for Generative Transfer Learning
We base our framework on state-of-the-art generative vision transformers that represent an image as a sequence of visual tokens to the autoregressive or non-autoregressive transformers.
Improved Masked Image Generation with Token-Critic
Given a masked-and-reconstructed real image, the Token-Critic model is trained to distinguish which visual tokens belong to the original image and which were sampled by the generative transformer.
MaskGIT: Masked Generative Image Transformer
At inference time, the model begins with generating all tokens of an image simultaneously, and then refines the image iteratively conditioned on the previous generation.
Ranked #2 on
Text-to-Image Generation
on LHQC
BLT: Bidirectional Layout Transformer for Controllable Layout Generation
During inference, BLT first generates a draft layout from the input and then iteratively refines it into a high-quality layout by masking out low-confident attributes.
Pyramid Adversarial Training Improves ViT Performance
In this work, we present pyramid adversarial training (PyramidAT), a simple and effective technique to improve ViT's overall performance.
Ranked #9 on
Domain Generalization
on ImageNet-C
(using extra training data)
Discrete Representations Strengthen Vision Transformer Robustness
Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition.
Ranked #3 on
Domain Generalization
on Stylized-ImageNet
ViTGAN: Training GANs with Vision Transformers
Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases.
Ranked #32 on
Image Generation
on CIFAR-10
Self-supervised and Supervised Joint Training for Resource-rich Machine Translation
Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT).
Faster Meta Update Strategy for Noise-Robust Deep Learning
It has been shown that deep neural networks are prone to overfitting on biased training data.
Regularizing Generative Adversarial Networks under Limited Data
Recent years have witnessed the rapid progress of generative adversarial networks (GANs).
Ranked #1 on
Image Generation
on CIFAR-100
Simplifying Reinforced Feature Selection via Restructured Choice Strategy of Single Agent
To address the question, we develop a single-agent reinforced feature selection approach integrated with restructured choice strategy.
Text as Neural Operator: Image Manipulation by Text Instruction
In recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community.
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval
Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing.
AdvAug: Robust Adversarial Augmentation for Neural Machine Translation
In this paper, we propose a new adversarial augmentation method for Neural Machine Translation (NMT).
Ranked #22 on
Machine Translation
on WMT2014 English-German
SimAug: Learning Robust Representations from 3D Simulation for Pedestrian Trajectory Prediction in Unseen Cameras
We refer to our method as SimAug.
Ranked #2 on
Trajectory Prediction
on ActEV
Controllable and Progressive Image Extrapolation
Image extrapolation aims at expanding the narrow field of view of a given image patch.
Neural Design Network: Graphic Layout Generation with Constraints
The first module predicts a graph with complete relations from a graph with user-specified relations.
The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
The first contribution is a new dataset, created in a realistic 3D simulator, which is based on real world trajectory data, and then extrapolated by human annotators to achieve different latent goals.
Ranked #1 on
Multi-future Trajectory Prediction
on ForkingPaths
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
Due to the lack of suitable datasets, previous research has only examined deep learning on controlled synthetic label noise, and real-world label noise has never been studied in a controlled setting.
Ranked #12 on
Image Classification
on WebVision-1000
Confident Learning: Estimating Uncertainty in Dataset Labels
Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence.
Synthetic vs Real: Deep Learning on Controlled Noise
Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels.
Feature Partitioning for Efficient Multi-Task Architectures
Multi-task learning holds the promise of less data, parameters, and time than training of separate models.
Robust Neural Machine Translation with Doubly Adversarial Inputs
Neural machine translation (NMT) often suffers from the vulnerability to noisy perturbations in the input.
State-aware Re-identification Feature for Multi-target Multi-camera Tracking
Multi-target Multi-camera Tracking (MTMCT) aims to extract the trajectories from videos captured by a set of cameras.
Eidetic 3D LSTM: A Model for Video Prediction and Beyond
We first evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Revisiting EmbodiedQA: A Simple Baseline and Beyond
In this paper, we empirically study this problem and introduce 1) a simple yet effective baseline that achieves promising performance; 2) an easier and practical setting for EmbodiedQA where an agent has a chance to adapt the trained model to a new environment before it actually answers users questions.
Let's Transfer Transformations of Shared Semantic Representations
In this work we show how one can learn transformations with no training examples by learning them on another domain and then transfer to the target domain.
Peeking into the Future: Predicting Future Person Activities and Locations in Videos
To facilitate the training, the network is learned with an auxiliary task of predicting future location in which the activity will happen.
Ranked #1 on
Activity Prediction
on ActEV
Contrastive Adaptation Network for Unsupervised Domain Adaptation
Unsupervised Domain Adaptation (UDA) makes predictions for the target domain data while manual annotations are only available in the source domain.
Ranked #9 on
Domain Adaptation
on Office-31
Composing Text and Image for Image Retrieval - An Empirical Odyssey
In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image.
Ranked #2 on
Image Retrieval with Multi-Modal Query
on MIT-States
Focal Visual-Text Attention for Memex Question Answering
In addition to a text answer, a few grounding photos are also given to justify the answer.
Ranked #1 on
Memex Question Answering
on MemexQA
Focal Visual-Text Attention for Visual Question Answering
Recent insights on language and vision with neural networks have been successfully applied to simple single-image visual question answering.
Ranked #1 on
Memex Question Answering
on MemexQA
Decoupled Novel Object Captioner
Thus, the sequence model can be decoupled from the novel object descriptions.
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
Recent deep networks are capable of memorizing the entire data even when the labels are completely random.
Ranked #16 on
Image Classification
on WebVision-1000
Graph Distillation for Action Detection with Privileged Modalities
We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available.
MemexQA: Visual Memex Question Answering
This paper proposes a new task, MemexQA: given a collection of photos or videos from a user, the goal is to automatically answer questions that help users recover their memory about events captured in the collection.
Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
We report on CMU Informedia Lab's system used in Google's YouTube 8 Million Video Understanding Challenge.
Self-paced Learning for Weakly Supervised Evidence Discovery in Multimedia Event Search
Multimedia event detection has been receiving increasing attention in recent years.
Exploiting Multi-modal Curriculum in Noisy Web Data for Large-scale Concept Learning
Learning video concept detectors automatically from the big but noisy web data with no additional manual annotations is a novel but challenging area in the multimedia and the machine learning community.
Strategies for Searching Video Content with Text Queries or Video Examples
The large number of user-generated videos uploaded on to the Internet everyday has led to many commercial video search engines, which mainly rely on text metadata for search.
A Self-Paced Multiple-Instance Learning Framework for Co-Saliency Detection
As an interesting and emerging topic, co-saliency detection aims at simultaneously extracting common salient objects in a group of images.
What Objective Does Self-paced Learning Indeed Optimize?
Self-paced learning (SPL) is a recently raised methodology designed through simulating the learning principle of humans/animals.
Self-Paced Learning with Diversity
Self-paced learning (SPL) is a recently proposed learning regime inspired by the learning process of humans and animals that gradually incorporates easy to more complex samples into training.
