no code implementations • 30 Jan 2025 • Yanxia Deng, Aozhong zhang, Naigang Wang, Selcuk Gurses, Zi Yang, Penghang Yin
Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources.
no code implementations • 28 Dec 2024 • Rongkun Xue, Jing Yang, Yuyang Jiang, Yiming Feng, Zi Yang
Existing local dynamic route planning algorithms, when directly applied to terrain following/terrain avoidance, or dynamic obstacle avoidance for large and medium-sized fixed-wing aircraft, fail to simultaneously meet the requirements of real-time performance, long-distance planning, and the dynamic constraints of large and medium-sized aircraft.
1 code implementation • 13 Dec 2024 • Zi Yang, Haojin Yang, Soumajit Majumder, Jorge Cardoso, Guillermo Gallego
Previous studies have demonstrated that not each sample in a dataset is of equal importance during training.
no code implementations • 30 Sep 2024 • Jiachen Ye, Dingyu Wang, Shaocheng Jia, Xin Pei, Zi Yang, Yi Zhang, S. C. Wong
Real-time estimation of vehicle locations and speeds is crucial for developing many beneficial transportation applications in traffic management and control, e. g., adaptive signal control.
no code implementations • 10 Sep 2024 • Zi Yang
Our proposed methods can identify 0% to 67% of the problems are retrieval focused and 0% to 90% of the problems are holistic understanding focused across 44 existing long context evaluation tasks.
1 code implementation • 2 Jun 2024 • Aozhong zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, Penghang Yin
For example, we achieve a Wikitext2 perplexity of 5. 95 on the LLaMA2-70B model for per-channel INT2 weight quantization without incurring any inference overhead.
1 code implementation • 23 May 2024 • Zi Yang, Ziyue Liu, Samridhi Choudhary, Xinfeng Xie, Cao Gao, Siegfried Kunzmann, Zheng Zhang
Our method also shows $\sim 2\times$ speedup than standard pre-training on a BERT-like code-generation LLM while achieving $4. 23\times$ compression ratio in pre-training.
no code implementations • 21 May 2024 • Chenghao Yang, Zi Yang, Nan Hua
Long-context modeling presents a significant challenge for transformer-based large language models (LLMs) due to the quadratic complexity of the self-attention mechanism and issues with length extrapolation caused by pretraining exclusively on short inputs.
no code implementations • 9 May 2024 • Keyu Chen, Yuan Pang, Zi Yang
In the arena of language model fine-tuning, the traditional approaches, such as Domain-Adaptive Pretraining (DAPT) and Task-Adaptive Pretraining (TAPT), although effective, but computational intensive.
1 code implementation • 11 Mar 2024 • Aozhong zhang, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, Penghang Yin
Within a fixed layer, COMQ treats all the scaling factor(s) and bit-codes as the variables of the reconstruction error.
no code implementations • 10 Jan 2024 • Zi Yang, Nan Hua
As LLMs have become capable of processing more complex types of inputs, researchers have recently studied how to efficiently and affordably process possibly arbitrarily long sequences.
no code implementations • 1 Jun 2023 • Zi Yang, Samridhi Choudhary, Siegfried Kunzmann, Zheng Zhang
To improve the convergence, a layer-by-layer distillation is applied to distill a quantized and tensor-compressed student model from a pre-trained transformer.
no code implementations • 13 May 2022 • Mahdieh Kazemimoghadam, Zi Yang, Lin Ma, Mingli Chen, Weiguo Lu, Xuejun Gu
We proposed to leverage the consistency of organs' anatomical shape and position information in medical images.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Zi Lin, Jeremiah Zhe Liu, Zi Yang, Nan Hua, Dan Roth
Traditional (unstructured) pruning methods for a Transformer model focus on regularizing the individual weights by penalizing them toward zero.
2 code implementations • 27 Jan 2020 • Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le
We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations.
no code implementations • 9 Aug 2019 • Erlei Zhang, Zi Yang, Stephen Seiler, Mingli Chen, Weiguo Lu, Xuejun Gu
These findings indicated that SATPN is promising for effective breast US lesion CAD using small datasets.
no code implementations • 29 Mar 2018 • Aditya Grover, Todor Markov, Peter Attia, Norman Jin, Nicholas Perkins, Bryan Cheong, Michael Chen, Zi Yang, Stephen Harris, William Chueh, Stefano Ermon
We propose a generalization of the best arm identification problem in stochastic multi-armed bandits (MAB) to the setting where every pull of an arm is associated with delayed feedback.
no code implementations • WS 2017 • Khyathi u, Aakanksha Naik, Ch, Aditya rasekar, Zi Yang, Niloy Gupta, Eric Nyberg
In this paper, we describe our participation in phase B of task 5b of the fifth edition of the annual BioASQ challenge, which includes answering factoid, list, yes-no and summary questions from biomedical data.
no code implementations • EMNLP 2017 • Rui Liu, Junjie Hu, Wei Wei, Zi Yang, Eric Nyberg
Deep neural networks for machine comprehension typically utilizes only word or character embeddings without explicitly taking advantage of structured linguistic information such as constituency trees and dependency trees.
Ranked #40 on
Question Answering
on SQuAD1.1 dev