Search Results for author: Bo Pang

Found 66 papers, 29 papers with code

RLHF Workflow: From Reward Modeling to Online RLHF

3 code implementations13 May 2024 Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang

We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature.

Chatbot Language Modelling +1

Latent Plan Transformer: Planning as Latent Variable Inference

no code implementations7 Feb 2024 Deqian Kong, Dehong Xu, Minglu Zhao, Bo Pang, Jianwen Xie, Andrew Lizarraga, Yuhao Huang, Sirui Xie, Ying Nian Wu

We introduce the Latent Plan Transformer (LPT), a novel model that leverages a latent space to connect a Transformer-based trajectory generator and the final return.

Emergence of Abstract State Representations in Embodied Sequence Modeling

no code implementations3 Nov 2023 Tian Yun, Zilai Zeng, Kunal Handa, Ashish V. Thapliyal, Bo Pang, Ellie Pavlick, Chen Sun

Decision making via sequence modeling aims to mimic the success of language models, where actions taken by an embodied agent are modeled as tokens to predict.

Decision Making

Learning the Geodesic Embedding with Graph Neural Networks

1 code implementation11 Sep 2023 Bo Pang, Zhongtian Zheng, Guoping Wang, Peng-Shuai Wang

Then, we can compute the geodesic distance between a pair of points using our decoding function, which requires only several matrix multiplications and can be massively parallelized on GPUs.

Graph Neural Network

XGen-7B Technical Report

1 code implementation7 Sep 2023 Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam, Tong Niu, Wojciech Kryściński, Lidiya Murakhovs'ka, Prafulla Kumar Choubey, Alex Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng Wu, Silvio Savarese, Yingbo Zhou, Shafiq Joty, Caiming Xiong

Most open-source LLMs, on the other hand, are limited in their ability to support longer sequence lengths, which is a key requirement for many tasks that require inference over an input context.

2k 8k

Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting

1 code implementation9 Jun 2023 Deqian Kong, Bo Pang, Tian Han, Ying Nian Wu

To search for molecules with desired properties, we propose a sampling with gradual distribution shifting (SGDS) algorithm, so that after learning the model initially on the training data of existing molecules and their properties, the proposed algorithm gradually shifts the model distribution towards the region supported by molecules with desired values of properties.

Drug Discovery

Diverse and Faithful Knowledge-Grounded Dialogue Generation via Sequential Posterior Inference

1 code implementation1 Jun 2023 Yan Xu, Deqian Kong, Dehong Xu, Ziwei Ji, Bo Pang, Pascale Fung, Ying Nian Wu

The capability to generate responses with diversity and faithfulness using factual knowledge is paramount for creating a human-like, trustworthy dialogue system.

Dialogue Generation Response Generation

Few-shot Unified Question Answering: Tuning Models or Prompts?

no code implementations23 May 2023 Srijan Bansal, Semih Yavuz, Bo Pang, Meghana Bhat, Yingbo Zhou

Question-answering (QA) tasks often investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks.

Question Answering Transfer Learning

Unsupervised 3D Point Cloud Representation Learning by Triangle Constrained Contrast for Autonomous Driving

no code implementations CVPR 2023 Bo Pang, Hongchi Xia, Cewu Lu

In this paper, we design the Triangle Constrained Contrast (TriCC) framework tailored for autonomous driving scenes which learns 3D unsupervised representations through both the multimodal information and dynamic of temporal sequences.

Autonomous Driving Representation Learning +2

Learning Probabilistic Models from Generator Latent Spaces with Hat EBM

1 code implementation29 Oct 2022 Mitch Hill, Erik Nijkamp, Jonathan Mitchell, Bo Pang, Song-Chun Zhu

This work proposes a method for using any generator network as the foundation of an Energy-Based Model (EBM).

Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach

no code implementations1 Oct 2022 Leilei Cui, Bo Pang, Zhong-Ping Jiang

This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs).

Autonomous Driving Reinforcement Learning (RL)

BigIssue: A Realistic Bug Localization Benchmark

no code implementations21 Jul 2022 Paul Kassianik, Erik Nijkamp, Bo Pang, Yingbo Zhou, Caiming Xiong

As machine learning tools progress, the inevitable question arises: How can machine learning help us write better code?

BIG-bench Machine Learning Program Repair

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

1 code implementation13 Jul 2022 Bo Pang, Yifan Zhang, Yaoyi Li, Jia Cai, Cewu Lu

In this paper, we propose a genuine group-level contrastive visual representation learning method whose linear evaluation performance on ImageNet surpasses the vanilla supervised learning.

Clustering Contrastive Learning +3

Latent Diffusion Energy-Based Model for Interpretable Text Modeling

2 code implementations13 Jun 2022 Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruiqi Gao, Yixin Zhu, Song-Chun Zhu, Ying Nian Wu

Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling.

End-to-end Dense Video Captioning as Sequence Generation

no code implementations COLING 2022 Wanrong Zhu, Bo Pang, Ashish V. Thapliyal, William Yang Wang, Radu Soricut

Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event.

Ranked #3 on Dense Video Captioning on ViTT (CIDEr metric, using extra training data)

Dense Video Captioning Descriptive

CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

5 code implementations25 Mar 2022 Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, Caiming Xiong

To democratize this, we train and release a family of large language models up to 16. 1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER.

Code Generation Language Modelling +2

Long Document Summarization with Top-down and Bottom-up Inference

no code implementations15 Mar 2022 Bo Pang, Erik Nijkamp, Wojciech Kryściński, Silvio Savarese, Yingbo Zhou, Caiming Xiong

Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.

CGNN: Traffic Classification with Graph Neural Network

no code implementations19 Oct 2021 Bo Pang, Yongquan Fu, Siyuan Ren, Ye Wang, Qing Liao, Yan Jia

Extensive evaluation over real-world traffic data sets, including normal, encrypted and malicious labels, show that, CGNN improves the prediction accuracy by 23\% to 29\% for application classification, by 2\% to 37\% for malicious traffic classification, and reaches the same accuracy level for encrypted traffic classification.

Classification Graph Neural Network +2

Unsupervised Meta-Learning via Latent Space Energy-based Model of Symbol Vector Coupling

no code implementations 5th Workshop on Meta-Learning at NeurIPS 2021 2021 Deqian Kong, Bo Pang, Ying Nian Wu

We propose to learn an energy-based model (EBM) in the latent space of a top-down generative model such that the EBM in the low dimensional latent space is able to be learned efficiently and adapt to each task rapidly.

Meta-Learning Unsupervised Few-Shot Image Classification

Long Document Summarization with Top-Down and Bottom-Up Representation Inference

no code implementations29 Sep 2021 Bo Pang, Erik Nijkamp, Wojciech Maciej Kryscinski, Silvio Savarese, Yingbo Zhou, Caiming Xiong

Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.

Document Summarization

MCMC Should Mix: Learning Energy-Based Model with Flow-Based Backbone

no code implementations ICLR 2022 Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song-Chun Zhu, Ying Nian Wu

However, MCMC sampling of EBMs in high-dimensional data space is generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space.

Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classification

1 code implementation26 Aug 2021 Bo Pang, Ying Nian Wu

The energy term of the prior model couples a continuous latent vector and a symbolic one-hot vector, so that discrete category can be inferred from the observed example based on the continuous latent vector.

Text Generation

Robust Transfer Learning with Pretrained Language Models through Adapters

no code implementations ACL 2021 Wenjuan Han, Bo Pang, YingNian Wu

Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks.

Adversarial Attack Adversarial Robustness +1

Human Pose Regression with Residual Log-likelihood Estimation

3 code implementations ICCV 2021 Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, Cewu Lu

In light of this, we propose a novel regression paradigm with Residual Log-likelihood Estimation (RLE) to capture the underlying output distribution.

3D Human Pose Estimation Multi-Person Pose Estimation +1

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

2 code implementations16 Jul 2021 Bo Pang, Zhong-Ping Jiang

This paper studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques.

reinforcement-learning Reinforcement Learning (RL)

Generative Text Modeling through Short Run Inference

1 code implementation EACL 2021 Bo Pang, Erik Nijkamp, Tian Han, Ying Nian Wu

It is initialized from the prior distribution of the latent variable and then runs a small number (e. g., 20) of Langevin dynamics steps guided by its posterior distribution.

Language Modelling

Trajectory Prediction with Latent Belief Energy-Based Model

1 code implementation CVPR 2021 Bo Pang, Tianyang Zhao, Xu Xie, Ying Nian Wu

Sampling from or optimizing the learned LB-EBM yields a belief vector which is used to make a path plan, which then in turn helps to predict a long-range trajectory.

Self-Driving Cars Trajectory Prediction

PGT: A Progressive Method for Training Models on Long Videos

1 code implementation CVPR 2021 Bo Pang, Gao Peng, Yizhuo Li, Cewu Lu

This progressive training (PGT) method is able to train long videos end-to-end with limited resources and ensures the effective transmission of information.

TDAF: Top-Down Attention Framework for Vision Tasks

no code implementations14 Dec 2020 Bo Pang, Yizhuo Li, Jiefeng Li, Muchen Li, Hanwen Cao, Cewu Lu

Such spatial and attention features are nested deeply, therefore, the proposed framework works in a mixed top-down and bottom-up manner.

Action Recognition object-detection +2

Understanding Guided Image Captioning Performance across Domains

1 code implementation CoNLL (EMNLP) 2021 Edwin G. Ng, Bo Pang, Piyush Sharma, Radu Soricut

Image captioning models generally lack the capability to take into account user interest, and usually default to global descriptions that try to balance readability, informativeness, and information overload.

Descriptive Image Captioning +2

Multimodal Pretraining for Dense Video Captioning

1 code implementation Asian Chapter of the Association for Computational Linguistics 2020 Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, Radu Soricut

First, we construct and release a new dense video captioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos together with time-stamped annotations.

 Ranked #1 on Dense Video Captioning on YouCook2 (ROUGE-L metric, using extra training data)

Dense Video Captioning

Learning Latent Space Energy-Based Prior Model for Molecule Generation

no code implementations19 Oct 2020 Bo Pang, Tian Han, Ying Nian Wu

Deep generative models have recently been applied to molecule design.


ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation

1 code implementation12 Aug 2020 Hanwen Cao, Yongyi Lu, Cewu Lu, Bo Pang, Gongshen Liu, Alan Yuille

In this paper, we further improve spatio-temporal point cloud feature learning with a flexible module called ASAP considering both attention and structure information across frames, which we find as two important factors for successful segmentation in dynamic point clouds.


Learning Latent Space Energy-Based Prior Model

1 code implementation NeurIPS 2020 Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, Ying Nian Wu

Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well.

Anomaly Detection Text Generation

MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC

no code implementations12 Jun 2020 Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song-Chun Zhu, Ying Nian Wu

Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm.

Joint Training of Variational Auto-Encoder and Latent Energy-Based Model

no code implementations CVPR 2020 Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, Ying Nian Wu

This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM).

Anomaly Detection

TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model

1 code implementation CVPR 2020 Bo Pang, Yizhuo Li, Yifan Zhang, Muchen Li, Cewu Lu

As deep learning brings excellent performances to object detection algorithms, Tracking by Detection (TBD) has become the mainstream tracking framework.

Multi-Object Tracking Object +2

Single Image Deraining via Scale-space Invariant Attention Neural Network

no code implementations9 Jun 2020 Bo Pang, Deming Zhai, Junjun Jiang, Xian-Ming Liu

Image enhancement from degradation of rainy artifacts plays a critical role in outdoor visual computing systems.

Image Enhancement Single Image Deraining

Robust Policy Iteration for Continuous-time Linear Quadratic Regulation

no code implementations19 May 2020 Bo Pang, Tao Bian, Zhong-Ping Jiang

This paper studies the robustness of policy iteration in the context of continuous-time infinite-horizon linear quadratic regulation (LQR) problem.

Systems and Control Numerical Analysis Systems and Control Numerical Analysis Optimization and Control

Asynchronous Interaction Aggregation for Action Detection

2 code implementations ECCV 2020 Jiajun Tang, Jin Xia, Xinzhi Mu, Bo Pang, Cewu Lu

We propose the Asynchronous Interaction Aggregation network (AIA) that leverages different interactions to boost action detection.

Action Detection

Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference

no code implementations ECCV 2020 Erik Nijkamp, Bo Pang, Tian Han, Linqi Zhou, Song-Chun Zhu, Ying Nian Wu

Learning such a generative model requires inferring the latent variables for each training example based on the posterior distribution of these latent variables.

A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions

no code implementations CONLL 2019 Jack Hessel, Bo Pang, Zhenhai Zhu, Radu Soricut

Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annotations (e. g., "heat the oil in the pan") improves user experiences.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +1

Neural-based Chinese Idiom Recommendation for Enhancing Elegance in Essay Writing

no code implementations ACL 2019 Yuanchao Liu, Bo Pang, Bingquan Liu

Although the proper use of idioms can enhance the elegance of writing, the active use of various expressions is a challenge because remembering idioms is difficult.

Machine Translation Translation

SParC: Cross-Domain Semantic Parsing in Context

4 code implementations ACL 2019 Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, Dragomir Radev

The best model obtains an exact match accuracy of 20. 2% over all questions and less than10% over all interaction sequences, indicating that the cross-domain setting and the con-textual phenomena of the dataset present significant challenges for future research.

Semantic Parsing Text-To-SQL

Deep RNN Framework for Visual Sequential Applications

1 code implementation CVPR 2019 Bo Pang, Kaiwen Zha, Hanwen Cao, Chen Shi, Cewu Lu

There are mainly two novel designs in our deep RNN framework: one is a new RNN module called Context Bridge Module (CBM) which splits the information flowing along the sequence (temporal direction) and along depth (spatial representation direction), making it easier to train when building deep by balancing these two directions; the other is the Overlap Coherence Training Scheme that reduces the training complexity for long visual sequential tasks on account of the limitation of computing resources.

Future prediction SSIM +1

Human Action Adverb Recognition: ADHA Dataset and A Three-Stream Hybrid Model

no code implementations4 Feb 2018 Bo Pang, Kaiwen Zha, Cewu Lu

We introduce the first benchmark for a new problem --- recognizing human action adverbs (HAA): "Adverbs Describing Human Actions" (ADHA).

Action Recognition Image Captioning +1

Thumbs up? Sentiment Classification using Machine Learning Techniques

no code implementations28 May 2002 Bo Pang, Lillian Lee, Shivakumar Vaithyanathan

We consider the problem of classifying documents not by topic, but by overall sentiment, e. g., determining whether a review is positive or negative.

BIG-bench Machine Learning Classification +3

Cannot find the paper you are looking for? You can Submit a new open access paper.