Benchmark method
No results yet. This page defines the method so results can be added without inventing numbers.
What to measure
Section titled “What to measure”- Tokens per second, generated and prompted.
- Time to first token.
- Peak memory use.
- Model, quantisation, and context length.
- Hardware: CPU, GPU, VRAM, RAM, and operating system.
- Runtime and flags.
- Task success on a fixed set of real prompts.
- Failure modes, not just averages.
Minimum record for each run
Section titled “Minimum record for each run”| Field | Example |
|---|---|
| Date | 2026-10-31 |
| Model | |
| Quantisation | |
| Context | |
| Runtime | |
| Flags | |
| Hardware | |
| Tokens per second | |
| Time to first token | |
| Peak memory | |
| Task set | |
| Passed | |
| Failed | |
| Notes |
- Publish the exact model and quantisation.
- Publish the flags and the runtime version.
- Publish the task set, or a representative sample.
- Report failures and outliers.
- Do not compare runs on different hardware and call it a model comparison.
- Date every number. Hardware and software move.
Where results go
Section titled “Where results go”Add one markdown file per run or one table per hardware tier. Link to the raw logs if they can be shared.