Search Results for author:
Found 35 papers, 21 papers with code
Improving Personalized Explanation Generation through Visualization
In modern recommender systems, there are usually comments or reviews from users that justify their ratings for different items.
FullStack Bench: Evaluating LLMs as Full Stack Coders
As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing.
Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
Sora unveils the potential of scaling Diffusion Transformer for generating photorealistic images and videos at arbitrary resolutions, aspect ratios, and durations, yet it still lacks sufficient implementation details.
InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
We conduct a systematic evaluation for over 100 latest code LLMs on InfiBench, leading to a series of novel and insightful findings.
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX.
Ranked #14 on
Video Question Answering
on MVBench
VIP5: Towards Multimodal Foundation Models for Recommendation
In light of this, we propose the development of a multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks.
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset.
Ranked #6 on
Visual Question Answering (VQA)
on InfiMM-Eval
Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens
However, direct aligning cross-modal information using such representations is challenging, as visual patches and text tokens differ in semantic levels and granularities.
HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding.
Mono-STAR: Mono-camera Scene-level Tracking and Reconstruction
We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework.
Frozen CLIP Models are Efficient Video Learners
Video recognition has been dominated by the end-to-end learning paradigm -- first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos.
Ranked #31 on
Action Classification
on Kinetics-400
Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning
Despite the success of fully-supervised human skeleton sequence modeling, utilizing self-supervised pre-training for skeleton sequence representation learning has been an active field because acquiring task-specific skeleton annotations at large scales is difficult.
Explainable Fairness in Recommendation
In this paper, we study the problem of explainable fairness, which helps to gain insights about why a system is fair or unfair, and guides the design of fair recommender systems with a more informed and unified methodology.
Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)
For a long time, different recommendation tasks typically require designing task-specific architectures and training objectives.
Learning and Evaluating Graph Neural Network Explanations based on Counterfactual and Factual Reasoning
For quantitatively evaluating the generated explanations without the requirement of ground-truth, we design metrics based on Counterfactual and Factual reasoning to evaluate the necessity and sufficiency of the explanations.
COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality
Group Activity Recognition detects the activity collectively performed by a group of actors, which requires compositional reasoning of actors and objects.
Ranked #2 on
Group Activity Recognition
on Collective Activity
A Simple Long-Tailed Recognition Baseline via Vision-Language Model
Recent advances in large-scale contrastive visual-language pretraining shed light on a new pathway for visual recognition.
Ranked #5 on
Long-tail Learning
on Places-LT
(using extra training data)
Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning
In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at both the 7th and 8th Dialog System Technology Challenges (DSTC7, DSTC8).
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning.
Dense Contrastive Visual-Linguistic Pretraining
To overcome these issues, we propose unbiased Dense Contrastive Visual-Linguistic Pretraining (DCVLP), which replaces the region regression and classification with cross-modality region contrastive learning that requires no annotations.
Counterfactual Evaluation for Explainable AI
While recent years have witnessed the emergence of various explainable methods in machine learning, to what degree the explanations really represent the reasoning process behind the model prediction -- namely, the faithfulness of explanation -- is still an open problem.
Scalable Transformers for Neural Machine Translation
In this paper, we propose a novel Scalable Transformers, which naturally contains sub-Transformers of different scales and have shared parameters.
RomeBERT: Robust Training of Multi-Exit BERT
In this paper, we leverage gradient regularized self-distillation for RObust training of Multi-Exit BERT (RomeBERT), which can effectively solve the performance imbalance problem between early and late exits.
CAFE: Coarse-to-Fine Neural Symbolic Reasoning for Explainable Recommendation
User profiles can capture prominent user behaviors from the history, and provide valuable signals about which kinds of path patterns are more likely to lead to potential items of interest for the user.
Multi-Pass Transformer for Machine Translation
In contrast with previous approaches where information flows only towards deeper layers of a stack, we consider a multi-pass transformer (MPT) architecture in which earlier layers are allowed to process information in light of the output of later layers.
Contrastive Visual-Linguistic Pretraining
We evaluate CVLP on several down-stream tasks, including VQA, GQA and NLVR2 to validate the superiority of contrastive learning on multi-modality representation learning.
Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers
Given an input video, its associated audio, and a brief caption, the audio-visual scene aware dialog (AVSD) task requires an agent to indulge in a question-answer dialog with a human about the audio-visual content.
Fairness-Aware Explainable Recommendation over Knowledge Graphs
There has been growing attention on fairness considerations recently, especially in the context of intelligent decision making systems.
Character Matters: Video Story Understanding with Character-Aware Relations
Without identifying the connection between appearing people and character names, a model is not able to obtain a genuine understanding of the plots.
ABSent: Cross-Lingual Sentence Representation Mapping with Bidirectional GANs
A number of cross-lingual transfer learning approaches based on neural networks have been proposed for the case when large amounts of parallel text are at our disposal.
Multi-Layer Content Interaction Through Quaternion Product For Visual Question Answering
To solve the issue for the intermediate layers, we propose an efficient Quaternion Block Network (QBN) to learn interaction not only for the last layer but also for all intermediate layers simultaneously.
2nd Place Solution to the GQA Challenge 2019
We present a simple method that achieves unexpectedly superior performance for Complex Reasoning involved Visual Question Answering.
CU-Net: Coupled U-Nets
We design a new connectivity pattern for the U-Net architecture.
Ranked #30 on
Pose Estimation
on MPII Human Pose
Quantized Densely Connected U-Nets for Efficient Landmark Localization
Finally, to reduce the memory consumption and high precision operations both in training and testing, we further quantize weights, inputs, and gradients of our localization network to low bit-width numbers.
Ranked #19 on
Pose Estimation
on MPII Human Pose
