multimodal generation
19 papers with code • 1 benchmarks • 1 datasets
Multimodal generation refers to the process of generating outputs that incorporate multiple modalities, such as images, text, and sound. This can be done using deep learning models that are trained on data that includes multiple modalities, allowing the models to generate output that is informed by more than one type of data.
For example, a multimodal generation model could be trained to generate captions for images that incorporate both text and visual information. The model could learn to identify objects in the image and generate descriptions of them in natural language, while also taking into account contextual information and the relationships between the objects in the image.
Multimodal generation can also be used in other applications, such as generating realistic images from textual descriptions or generating audio descriptions of video content. By combining multiple modalities in this way, multimodal generation models can produce more accurate and comprehensive output, making them useful for a wide range of applications.
Benchmarks
These leaderboards are used to track progress in multimodal generation
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
Multi-Modal CelebA-HQ
Most implemented papers
GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation (works for videos too!)
This adversarial loss guarantees the map is diverse -- a very wide range of anime can be produced from a single content code.
Finite Scalar Quantization: VQ-VAE Made Simple
google-research/google-research
•
•
Each dimension is quantized to a small set of fixed values, leading to an (implicit) codebook given by the product of these sets.
Continual and Multi-Task Architecture Search
ramakanth-pasunuru/CAS-MAS
•
•
ACL 2019
Architecture search is the process of automatically learning the neural model or cell structure that best suits the given task.
Unconditional Image-Text Pair Generation with Multimodal Cross Quantizer
ttumyche/mxq-vae
•
•
To learn a multimodal semantic correlation in a quantized space, we combine VQ-VAE with a Transformer encoder and apply an input masking strategy.
Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living
We tackle this challenge and introduce an activity domain generation framework which creates novel ADL appearances (novel domains) from different existing activity modalities (source domains) inferred from video training data.
Multimedia Generative Script Learning for Task Planning
EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning
•
•
Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities.
Unified Discrete Diffusion for Simultaneous Vision-Language Generation
mhh0318/unid3
•
•
The recently developed discrete diffusion models perform extraordinarily well in the text-to-image task, showing significant promise for handling the multi-modality signals.
Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models
Nithin-GK/UniteandConquer
•
•
CVPR 2023
We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints.
Text2Poster: Laying out Stylized Texts on Retrieved Images
chuhaojin/text2poster-icassp-22
•
•
Poster generation is a significant task for a wide range of applications, which is often time-consuming and requires lots of manual editing and artistic experience.
Grounding Language Models to Images for Multimodal Inputs and Outputs
We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images.
