Search Results for author:
Found 24 papers, 14 papers with code
From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs
While MLLMs perform well on perceptual tasks, they lack precise multimodal alignment, limiting performance.
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
Existing preference optimization objectives for language model alignment require additional hyperparameters that must be extensively tuned to achieve optimal performance, increasing both the complexity and time required for fine-tuning large language models.
Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding
Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence, yet the irrational encoding of visual positions persists in inhibiting the models' comprehensive perception performance across different levels of granularity.
DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space
The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space.
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy.
Action-based image editing guided by human instructions
We show substantial improvements in image editing using action-based text instructions and high reasoning capabilities that allow our model to use the input image as a starting scene for an action while generating a new image that shows the final scene of the action.
TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
For video understanding tasks, training-based video LLMs are difficult to build due to the scarcity of high-quality, curated video-text paired data.
Ranked #1 on
Zero-Shot Video Question Answer
on MVBench
MVBench
Video-based Generative Performance Benchmarking (Consistency)
+6
How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective
This paper introduces a novel generalized self-imitation learning ($\textbf{GSIL}$) framework, which effectively and efficiently aligns large language models with offline demonstration data.
SePPO: Semi-Policy Preference Optimization for Diffusion Alignment
To address the limitations of both on- and off-policy RLHF, we propose a preference optimization method that aligns DMs with preferences without relying on reward models or paired human-annotated data.
DMON: A Simple yet Effective Approach for Argument Structure Learning
Argument structure learning~(ASL) entails predicting relations between arguments.
Animate Your Motion: Turning Still Images into Dynamic Videos
For the first time, we integrate both semantic and motion cues within a diffusion model for video generation, as demonstrated in Fig 1.
NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties
In the pursuit to understand the intricacies of human brain's visual processing, reconstructing dynamic visual experiences from brain activities emerges as a challenging yet fascinating endeavor.
Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidence
Medical Question Answering~(medical QA) systems play an essential role in assisting healthcare workers in finding answers to their questions.
Decoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion
Reconstructing visual stimuli from human brain activities provides a promising opportunity to advance our understanding of the brain's visual system and its connection with computer vision models.
Elucidating the Exposure Bias in Diffusion Models
In this paper, we systematically investigate the exposure bias problem in diffusion models by first analytically modelling the sampling distribution, based on which we then attribute the prediction error at each sampling step as the root cause of the exposure bias issue.
Ranked #10 on
Image Generation
on CIFAR-10
Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder.
Ranked #1 on
Brain Visual Reconstruction from fMRI
on GOD
Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps
In this work, we conduct a systematic study of exposure bias in DPM and, intriguingly, we find that the exposure bias could be alleviated with a novel sampling method that we propose, without retraining the model.
Crossword: A Semantic Approach to Data Compression via Masking
The traditional methods for data compression are typically based on the symbol-level statistics, with the information source modeled as a long sequence of i. i. d.
Layout-aware Dreamer for Embodied Referring Expression Grounding
In this work, we study the problem of Embodied Referring Expression Grounding, where an agent needs to navigate in a previously unseen environment and localize a remote object described by a concise high-level natural language instruction.
Find a Way Forward: a Language-Guided Semantic Map Navigator
In this paper, we introduce the map-language navigation task where an agent executes natural language instructions and moves to the target position based only on a given 3D semantic map.
Dynamic Key-value Memory Enhanced Multi-step Graph Reasoning for Knowledge-based Visual Question Answering
Knowledge-based visual question answering (VQA) is a vision-language task that requires an agent to correctly answer image-related questions using knowledge that is not presented in the given image.
Modeling Coreference Relations in Visual Dialog
Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image.
Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win
The lottery ticket hypothesis states that sparse subnetworks exist in randomly initialized dense networks that can be trained to the same accuracy as the dense network they reside in.
Multiscale Dynamic Human Mobility Flow Dataset in the U.S. during the COVID-19 Epidemic
Understanding dynamic human mobility changes and spatial interaction patterns at different geographic scales is crucial for monitoring and measuring the impacts of non-pharmaceutical interventions (such as stay-at-home orders) during the pandemic.
Social and Information Networks Physics and Society
