Search Results for author: Alessandro Stolfo

Found 8 papers, 5 papers with code

Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study

no code implementations10 Apr 2024 Alessandro Stolfo

We present an empirical study of groundedness in long-form question answering (LFQA) by retrieval-augmented large language models (LLMs).

Long Form Question Answering Retrieval +1

Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?

no code implementations31 Jan 2024 Andreas Opedal, Alessandro Stolfo, Haruki Shirakami, Ying Jiao, Ryan Cotterell, Bernhard Schölkopf, Abulhair Saparov, Mrinmaya Sachan

We find evidence that LLMs, with and without instruction-tuning, exhibit human-like biases in both the text-comprehension and the solution-planning steps of the solving process, but not during the final step which relies on the problem's arithmetic expressions (solution execution).

Reading Comprehension

Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models

1 code implementation23 Oct 2023 Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan

We show that MechanisticProbe is able to detect the information of the reasoning tree from the model's attentions for most examples, suggesting that the LM indeed is going through a process of multi-step reasoning within its architecture in many cases.

A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis

1 code implementation24 May 2023 Alessandro Stolfo, Yonatan Belinkov, Mrinmaya Sachan

Mathematical reasoning in large language models (LMs) has garnered significant attention in recent work, but there is a limited understanding of how these models process and store information related to arithmetic tasks within their architecture.

Arithmetic Reasoning Mathematical Reasoning +2

Distilling Reasoning Capabilities into Smaller Language Models

1 code implementation1 Dec 2022 Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan

In this work, we propose an alternative reasoning scheme, Socratic CoT, that learns a decomposition of the original problem into a sequence of subproblems and uses it to guide the intermediate reasoning steps.

GSM8K Knowledge Distillation +2

A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models

1 code implementation21 Oct 2022 Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, Mrinmaya Sachan

By grounding the behavioral analysis in a causal graph describing an intuitive reasoning process, we study the behavior of language models in terms of robustness and sensitivity to direct interventions in the input space.

Math Mathematical Reasoning

Cannot find the paper you are looking for? You can Submit a new open access paper.