Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference

1 code implementation • 19 Jun 2023 • Junhao Zheng, Qianli Ma, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang, Haibin Chen

Intriguingly, the unified objective can be seen as the sum of the vanilla fine-tuning objective, which learns new knowledge from target data, and the causal objective, which preserves old knowledge from PLMs.

Attribute Causal Inference