Search Results for author:
Found 14 papers, 2 papers with code
Are Language Model Logits Calibrated?
For example, if the context concerns two equally likely options (e. g., heads or tails for a fair coin), the output probabilities should reflect this.
SEC-QA: A Systematic Evaluation Corpus for Financial QA
The financial domain frequently deals with large numbers of long documents that are essential for daily operations.
Lessons from the Trenches on Reproducible Evaluation of Language Models
Third, we present the Language Model Evaluation Harness (lm-eval): an open source library for independent, reproducible, and extensible evaluation of language models that seeks to address these issues.
DocFinQA: A Long-Context Financial Reasoning Dataset
For large language models (LLMs) to be effective in the financial domain -- where each decision can have a significant impact -- it is necessary to investigate realistic tasks and data.
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
We demonstrate that the current bottleneck in performance is due to LLMs' limited business and financial understanding, highlighting the value of a challenging benchmark for quantitative reasoning within this domain.
Deep Neural Networks Can Learn Generalizable Same-Different Visual Relations
Much of this prior work focuses on training convolutional neural networks to classify images of two same or two different abstract shapes, testing generalization on within-distribution stimuli.
Training Priors Predict Text-To-Image Model Performance
By looking at the subject-verb-object (SVO) triads that underlie these prompts (e. g., "astronaut", "ride", "horse"), we find that the more often an SVO triad appears in the training data, the better the model can generate an image aligned with that triad.
Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex
AlphaZero, an approach to reinforcement learning that couples neural networks and Monte Carlo tree search (MCTS), has produced state-of-the-art strategies for traditional board games like chess, Go, shogi, and Hex.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Unit Testing for Concepts in Neural Networks
Many complex problems are naturally understood in terms of symbolic concepts.
Information-theoretic Probing Explains Reliance on Spurious Features
In this work, we test the hypothesis that the extent to which a feature influences a model's decisions can be predicted using a combination of two factors: The feature's "extractability" after pre-training (measured using information-theoretic probing techniques), and the "evidence" available during fine-tuning (defined as the feature's co-occurrence rate with the label).
Self-play for Data Efficient Language Acquisition
When communicating, people behave consistently across conversational roles: People understand the words they say and are able to produce the words they hear.
Does Data Augmentation Improve Generalization in NLP?
Neural models often exploit superficial features to achieve good performance, rather than deriving more general features.
INTERNAL-CONSISTENCY CONSTRAINTS FOR EMERGENT COMMUNICATION
When communicating, humans rely on internally-consistent language representations.
