OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratch

1 code implementation • 19 Sep 2023 • Juntao Li, Zecheng Tang, Yuyang Ding, Pinzheng Wang, Pei Guo, Wangjie You, Dan Qiao, Wenliang Chen, Guohong Fu, Qiaoming Zhu, Guodong Zhou, Min Zhang

This report provides the main details to pre-train an analogous model, including pre-training data processing, Bilingual Flan data collection, the empirical observations that inspire our model architecture design, training objectives of different stages, and other enhancement techniques.

Detoxify Language Model Step-by-Step

1 code implementation • 16 Aug 2023 • Zecheng Tang, Keyan Zhou, Pinzheng Wang, Yuyang Ding, Juntao Li, Minzhang

Detoxification for LLMs is challenging since it requires models to avoid generating harmful content while maintaining the generation capability.

Language Modelling

Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!

1 code implementation • 8 May 2023 • Zecheng Tang, Pinzheng Wang, Keyan Zhou, Juntao Li, Ziqiang Cao, Min Zhang

Diffusion models have been successfully adapted to text generation tasks by mapping the discrete text into the continuous space.

Text Generation

UFNRec: Utilizing False Negative Samples for Sequential Recommendation

1 code implementation • 8 Aug 2022 • Xiaoyang Liu, Chong Liu, Pinzheng Wang, Rongqin Zheng, Lixin Zhang, Leyu Lin, Zhijun Chen, Liangliang Fu

To this end, we propose a novel method that can Utilize False Negative samples for sequential Recommendation (UFNRec) to improve model performance.

Sequential Recommendation