A benchmark suite for testing code quality and performance
Find a file
hermes 804c27db29
All checks were successful
CI / Lint (push) Successful in 25s
CI / Test (Python 3.12) (push) Successful in 46s
ci: re-run via runner pickup verification
2026-09-26 20:06:52 -07:00
.forgejo/workflows ci: pin ruff to v0.11.0 to match pre-commit config 2026-09-11 00:05:32 -07:00
Models feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
tests gsd: pre-implementation lint/format cleanup 2026-05-27 04:56:55 -07:00
.gitignore gsd: pre-implementation lint/format cleanup 2026-05-16 00:39:08 -07:00
.pre-commit-config.yaml chore: add basic pre-commit hooks (trailing-whitespace, eof-fixer, ruff) 2026-06-03 17:28:41 -07:00
.quickfix-run.log gsd: pre-implementation lint/format cleanup 2026-05-26 19:59:34 -07:00
=3.8 fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00
__init__.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
async_fetch_all.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
auth.py fix #11: [smoke-test] Test coverage review: 18 source files, 1 test files (5.6% 2026-05-26 20:05:23 -07:00
benchmark_suite.py fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00
binary_search.py gsd: pre-implementation lint/format cleanup 2026-05-16 00:39:08 -07:00
caching_decorator.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
config.json fix #9: Centralize configuration into config.json 2026-05-23 04:21:14 -07:00
config.sh fix #9: Centralize configuration into config.json 2026-05-23 04:21:14 -07:00
data_processing.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
get_status.py gsd: pre-implementation lint/format cleanup 2026-05-16 00:39:08 -07:00
is_even.py gsd: pre-implementation lint/format cleanup 2026-05-16 00:39:08 -07:00
is_greater.py gsd: pre-implementation lint/format cleanup 2026-05-16 00:39:08 -07:00
main.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
math_utils.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
processor.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
pyproject.toml fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00
README.md feat: implement benchmarks #5, #8, #4, #3, #1 2026-05-15 07:23:08 -07:00
requirements.txt fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00
run_benchmark.sh fix #9: Centralize configuration into config.json 2026-05-23 04:21:14 -07:00
run_benchmark_l4.sh fix #9: Centralize configuration into config.json 2026-05-23 04:21:14 -07:00
run_benchmark_l5.sh fix #2: rewrite run_benchmark_l5.sh following L4 pattern 2026-05-23 07:47:59 -07:00
success_by_level.png fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00
sum_numbers.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
task_heatmap.png fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00
test_is_even.py feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
timing_distribution.png fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00
utils.js feat: benchmark implementation with Models, agents, and benchmark suite 2026-05-10 07:08:09 -07:00
utils.py gsd: pre-implementation lint/format cleanup 2026-05-16 00:39:08 -07:00
visualize_results.py fix #7: No result analysis or visualization 2026-05-27 05:00:55 -07:00

Little Coder Bench

A benchmark suite for evaluating small coding language models across 5 difficulty levels.

Setup

pip install -r requirements.txt

Running Benchmarks

Level 1–3 (Python functions)

./run_benchmark.sh

Level 4 (Multi-file scripts)

./run_benchmark_l4.sh

Level 5 (Shell scripts)

./run_benchmark_l5.sh

Difficulty Levels

Level Description Examples
L1 Single function is_even, is_greater, sum_numbers
L2 Utility module math_utils, binary_search
L3 API/service auth.py, data_processing, processor
L4 Multi-file ETL, file ops, multi-file import, async fetch
L5 Shell scripts build, test, system check, status check

Configuration

Edit benchmark_suite.py to configure models, timeouts, and provider settings.