Search Results for author:
Found 16 papers, 9 papers with code
Mixture of Parrots: Experts improve memorization more than reasoning
On the other hand, we find that on memory-intensive tasks, MoEs can effectively leverage a small number of active parameters with a large number of experts to memorize the data.
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Fulfilling both of these characteristics can pose a great challenge for LLMs. To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1, 140 fine-grained tasks.
Ranked #1 on
Code Generation
on BigCodeBench-Instruct
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry.
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?
In this work, we focus on these counterfeit samples: programs sampled from a language model that 1) have a high enough log-probability to be generated at a moderate temperature and 2) pass weak correctness checks.
StarCoder 2 and The Stack v2: The Next Generation
Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size.
Ranked #32 on
Code Generation
on MBPP
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75% and 81% on input and output prediction, respectively.
Language Agnostic Code Embeddings
Recently, code language models have achieved notable advancements in addressing a diverse array of essential code comprehension and generation tasks.
LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers
Logical reasoning, i. e., deductively inferring the truth value of a conclusion from a set of premises, is an important task for artificial intelligence with wide potential impacts on science, mathematics, and society.
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
Using this data, we develop ReProver (Retrieval-Augmented Prover): an LLM-based prover augmented with retrieval for selecting premises from a vast math library.
StarCoder: may the source be with you!
The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15. 5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention.
Ranked #50 on
Code Generation
on MBPP
Certified Interpretability Robustness for Class Activation Mapping
Interpreting machine learning models is challenging but crucial for ensuring the safety of deep networks in autonomous driving systems.
SantaCoder: don't reach for the stars!
The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code.
ObSynth: An Interactive Synthesis System for Generating Object Models from Natural Language Specifications
We introduce ObSynth, an interactive system leveraging the domain knowledge embedded in large language models (LLMs) to help users design object models from high level natural language prompts.
Min-Max Bilevel Multi-objective Optimization with Applications in Machine Learning
We consider a generic min-max multi-objective bilevel optimization problem with applications in robust machine learning such as representation learning and hyperparameter optimization.
The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements.
Reproducibility Report: La-MAML: Look-ahead Meta Learning for Continual Learning
The Continual Learning (CL) problem involves performing well on a sequence of tasks under limited compute.
