Search Results for author:
Found 64 papers, 28 papers with code
Latent Plan Transformer: Planning as Latent Variable Inference
We introduce the Latent Plan Transformer (LPT), a novel model that leverages a latent space to connect a Transformer-based trajectory generator and the final return.
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks.
Ranked #1 on
Visual Question Answering (VQA)
on A-OKVQA
(using extra training data)
Emergence of Abstract State Representations in Embodied Sequence Modeling
Decision making via sequence modeling aims to mimic the success of language models, where actions taken by an embodied agent are modeled as tokens to predict.
Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling
Intrusive PEFT techniques directly change a model's internal architecture.
Learning the Geodesic Embedding with Graph Neural Networks
Then, we can compute the geodesic distance between a pair of points using our decoding function, which requires only several matrix multiplications and can be massively parallelized on GPUs.
XGen-7B Technical Report
Most open-source LLMs, on the other hand, are limited in their ability to support longer sequence lengths, which is a key requirement for many tasks that require inference over an input context.
Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting
To search for molecules with desired properties, we propose a sampling with gradual distribution shifting (SGDS) algorithm, so that after learning the model initially on the training data of existing molecules and their properties, the proposed algorithm gradually shifts the model distribution towards the region supported by molecules with desired values of properties.
Diverse and Faithful Knowledge-Grounded Dialogue Generation via Sequential Posterior Inference
The capability to generate responses with diversity and faithfulness using factual knowledge is paramount for creating a human-like, trustworthy dialogue system.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
Few-shot Unified Question Answering: Tuning Models or Prompts?
Question-answering (QA) tasks often investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks.
Unsupervised 3D Point Cloud Representation Learning by Triangle Constrained Contrast for Autonomous Driving
In this paper, we design the Triangle Constrained Contrast (TriCC) framework tailored for autonomous driving scenes which learns 3D unsupervised representations through both the multimodal information and dynamic of temporal sequences.
Learning Probabilistic Models from Generator Latent Spaces with Hat EBM
This work proposes a method for using any generator network as the foundation of an Energy-Based Model (EBM).
Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach
This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs).
BigIssue: A Realistic Bug Localization Benchmark
As machine learning tools progress, the inevitable question arises: How can machine learning help us write better code?
Unsupervised Visual Representation Learning by Synchronous Momentum Grouping
In this paper, we propose a genuine group-level contrastive visual representation learning method whose linear evaluation performance on ImageNet surpasses the vanilla supervised learning.
Ranked #38 on
Self-Supervised Image Classification
on ImageNet
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling.
End-to-end Dense Video Captioning as Sequence Generation
Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event.
Ranked #3 on
Dense Video Captioning
on ViTT
(CIDEr metric, using extra
training data)
Semantic Segmentation by Early Region Proxy
Typical vision backbones manipulate structured features.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
To democratize this, we train and release a family of large language models up to 16. 1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER.
Ranked #79 on
Code Generation
on HumanEval
Long Document Summarization with Top-down and Bottom-up Inference
Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.
Ranked #1 on
Text Summarization
on Pubmed
CGNN: Traffic Classification with Graph Neural Network
Extensive evaluation over real-world traffic data sets, including normal, encrypted and malicious labels, show that, CGNN improves the prediction accuracy by 23\% to 29\% for application classification, by 2\% to 37\% for malicious traffic classification, and reaches the same accuracy level for encrypted traffic classification.
Unsupervised Meta-Learning via Latent Space Energy-based Model of Symbol Vector Coupling
We propose to learn an energy-based model (EBM) in the latent space of a top-down generative model such that the EBM in the low dimensional latent space is able to be learned efficiently and adapt to each task rapidly.
Long Document Summarization with Top-Down and Bottom-Up Representation Inference
Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.
MCMC Should Mix: Learning Energy-Based Model with Flow-Based Backbone
However, MCMC sampling of EBMs in high-dimensional data space is generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space.
Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classification
The energy term of the prior model couples a continuous latent vector and a symbolic one-hot vector, so that discrete category can be inferred from the observed example based on the continuous latent vector.
Robust Transfer Learning with Pretrained Language Models through Adapters
Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks.
Human Pose Regression with Residual Log-likelihood Estimation
In light of this, we propose a novel regression paradigm with Residual Log-likelihood Estimation (RLE) to capture the underlying output distribution.
Ranked #59 on
3D Human Pose Estimation
on Human3.6M
Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems
This paper studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques.
SCRIPT: Self-Critic PreTraining of Transformers
We introduce Self-CRItic Pretraining Transformers (SCRIPT) for representation learning of text.
Generative Text Modeling through Short Run Inference
It is initialized from the prior distribution of the latent variable and then runs a small number (e. g., 20) of Langevin dynamics steps guided by its posterior distribution.
Trajectory Prediction with Latent Belief Energy-Based Model
Sampling from or optimizing the learned LB-EBM yields a belief vector which is used to make a path plan, which then in turn helps to predict a long-range trajectory.
PGT: A Progressive Method for Training Models on Long Videos
This progressive training (PGT) method is able to train long videos end-to-end with limited resources and ensures the effective transmission of information.
TDAF: Top-Down Attention Framework for Vision Tasks
Such spatial and attention features are nested deeply, therefore, the proposed framework works in a mixed top-down and bottom-up manner.
Understanding Guided Image Captioning Performance across Domains
Image captioning models generally lack the capability to take into account user interest, and usually default to global descriptions that try to balance readability, informativeness, and information overload.
Multimodal Pretraining for Dense Video Captioning
First, we construct and release a new dense video captioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos together with time-stamped annotations.
Ranked #1 on
Dense Video Captioning
on YouCook2
(ROUGE-L metric, using extra
training data)
Learning Latent Space Energy-Based Prior Model for Molecule Generation
Deep generative models have recently been applied to molecule design.
Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
This paper proposes a latent space energy-based prior model for semi-supervised learning.
Fully Unsupervised Person Re-identification viaSelective Contrastive Learning
In this work, we propose a novel selective contrastive learning framework for unsupervised feature learning.
Robust Reinforcement Learning: A Case Study in Linear Quadratic Regulation
This paper studies the robustness of reinforcement learning algorithms to errors in the learning process.
ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation
In this paper, we further improve spatio-temporal point cloud feature learning with a flexible module called ASAP considering both attention and structure information across frames, which we find as two important factors for successful segmentation in dynamic point clouds.
Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation
Open-domain dialogue generation has gained increasing attention in Natural Language Processing.
Learning Latent Space Energy-Based Prior Model
Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well.
MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC
Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm.
TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model
As deep learning brings excellent performances to object detection algorithms, Tracking by Detection (TBD) has become the mainstream tracking framework.
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM).
Single Image Deraining via Scale-space Invariant Attention Neural Network
Image enhancement from degradation of rainy artifacts plays a critical role in outdoor visual computing systems.
Complex Sequential Understanding through the Awareness of Spatial and Temporal Concepts
Understanding sequential information is a fundamental task for artificial intelligence.
Robust Policy Iteration for Continuous-time Linear Quadratic Regulation
This paper studies the robustness of policy iteration in the context of continuous-time infinite-horizon linear quadratic regulation (LQR) problem.
Systems and Control Numerical Analysis Systems and Control Numerical Analysis Optimization and Control
NTIRE 2020 Challenge on Video Quality Mapping: Methods and Results
For track 2, some existing methods are evaluated, showing promising solutions to the weakly-supervised video quality mapping problem.
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube
Pretraining from unlabelled web videos has quickly become the de-facto means of achieving high performance on many video understanding tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Asynchronous Interaction Aggregation for Action Detection
We propose the Asynchronous Interaction Aggregation network (AIA) that leverages different interactions to boost action detection.
Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference
Learning such a generative model requires inferring the latent variables for each training example based on the posterior distribution of these latent variables.
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions
Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annotations (e. g., "heat the oil in the pan") improves user experiences.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems.
Ranked #8 on
Dialogue State Tracking
on CoSQL
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering.
Ranked #4 on
Visual Question Answering (VQA)
on VizWiz 2018
Neural-based Chinese Idiom Recommendation for Enhancing Elegance in Essay Writing
Although the proper use of idioms can enhance the elegance of writing, the active use of various expressions is a challenge because remembering idioms is difficult.
SParC: Cross-Domain Semantic Parsing in Context
The best model obtains an exact match accuracy of 20. 2% over all questions and less than10% over all interaction sequences, indicating that the cross-domain setting and the con-textual phenomena of the dataset present significant challenges for future research.
Deep RNN Framework for Visual Sequential Applications
There are mainly two novel designs in our deep RNN framework: one is a new RNN module called Context Bridge Module (CBM) which splits the information flowing along the sequence (temporal direction) and along depth (spatial representation direction), making it easier to train when building deep by balancing these two directions; the other is the Overlap Coherence Training Scheme that reduces the training complexity for long visual sequential tasks on account of the limitation of computing resources.
Human Action Adverb Recognition: ADHA Dataset and A Three-Stream Hybrid Model
We introduce the first benchmark for a new problem --- recognizing human action adverbs (HAA): "Adverbs Describing Human Actions" (ADHA).
The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter
Consider a person trying to spread an important message on a social network.
Thumbs up? Sentiment Classification using Machine Learning Techniques
We consider the problem of classifying documents not by topic, but by overall sentiment, e. g., determining whether a review is positive or negative.
