Search Results for author:
Found 14 papers, 8 papers with code
PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
We also quantify the effectiveness of reasoning longer with R1 and Gemini Thinking to identify the point beyond which more reasoning is unlikely to improve accuracy on our benchmark.
Substance Beats Style: Why Beginning Students Fail to Code with LLMs
Although LLMs are increasing the productivity of professional programmers, existing work shows that beginners struggle to prompt LLMs to solve text-to-code tasks.
Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark
Several systems have been developed to extract information about characters to aid computational analysis of English literature.
GlyphPattern: An Abstract Pattern Recognition for Vision-Language Models
GlyphPattern evaluates abstract pattern recognition in VLMs, requiring models to understand and judge natural language descriptions of visual patterns.
StarCoder 2 and The Stack v2: The Next Generation
Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size.
Ranked #35 on
Code Generation
on MBPP
Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
These are tasks in which the model is provided a block of code and an instruction to modify the code.
Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
We apply this approach to generate tens of thousands of validated training items for Julia, Lua, OCaml, R, and Racket.
Solving and Generating NPR Sunday Puzzles with Large Language Models
We explore the ability of large language models to solve and generate puzzles from the NPR Sunday Puzzle game show using PUZZLEQA, a dataset comprising 15 years of on-air puzzles.
StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code
We use StudentEval to evaluate 5 Code LLMs and find that StudentEval is a better discriminator of model performance than existing benchmarks.
StarCoder: may the source be with you!
The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15. 5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention.
Ranked #53 on
Code Generation
on MBPP
SantaCoder: don't reach for the stars!
The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code.
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex, CodeGen, and InCoder.
