mct-bench
I built a CLI that connects model quantization, ONNX export and CPU measurement. On 10,000 Fashion-MNIST test images, W4A8 accuracy fell from 94.76% to 94.22%; the exported ONNX file grew by about 0.18%.
The project
The CLI connects retrieval of a public pretrained CNN with post-training quantization, ONNX export, and evaluation. It measures the baseline and quantized models under the same runtime settings and preserves results even when they show no improvement.
Alongside Markdown and CSV tables, it saves JSON with measurement conditions and raw samples, image indices, asset hashes, and executable ONNX files. Each comparison retains the information needed to repeat it.
How it works
Prepare a model and images
A public trained model is loaded, with separate images for adjusting its conversion and for measuring accuracy.
Vary numerical precision
The original stays as the baseline; variants with coarser numerical values are made independently from it and exported as runnable files.
Compare accuracy, size and time
The exported models run on the same images with the same CPU settings, producing a comparison of accuracy, complete file size and runtime.
Calibration uses the training split, evaluation uses the test split, and every variant sees the same evaluation images. A fixed PyTorch architecture loads the public model's weights; output comparisons and inspection of the actual quantizer bit widths precede measurement.
Both accuracy and time are measured from the exported ONNX files under common CPU settings. Size is measured from the complete file. Timing excludes warmup, varies model execution order, and retains the raw samples.
What the records show
Across 10,000 Fashion-MNIST test images, W4A8 accuracy changed from 94.76% to 94.22%, while the ONNX file grew by approximately 0.18%. A quantization setting alone did not make the distributed file smaller.
The selected export format stores the quantized weight values as floating-point data. By recording numerical precision, storage type, and runtime separately, the comparison also explains why the expected reduction did not occur.
- Evaluation images
- 10,000
- Accuracy · FP32 → W4A8
- 94.76% → 94.22%
- ONNX size change
- Approx. 0.18% larger
What this covers
These results cover one dataset and one model and do not establish a general outcome. They concern the selected ONNX export format and host CPU, rather than packed low-bit storage or acceleration on a target device. Small timing differences on the shared host do not establish a performance advantage.