Search Results for author:
Found 8 papers, 5 papers with code
CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?
Recent advancements in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks.
CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
By evaluating 17 popular LLMs using this benchmark, we reveal significant differences in their accuracy and reliability in code generation, offering detailed insights for further improving the code generation capabilities of LLMs.
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts.
Token Alignment via Character Matching for Subword Completion
Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens.
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation.
A Static Evaluation of Code Completion by Large Language Models
Large language models trained on code have shown great potential to increase productivity of software developers.
Greener yet Powerful: Taming Large Code Generation Models with Quantization
Such large models incur significant resource usage (in terms of memory, latency, and dollars) as well as carbon footprint.
Multi-lingual Evaluation of Code Generation Models
Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings.
