2018-02-11 11:28 AM
For benchmarking C implementations, the there are a few benchmarks, but they all have their problems. Many benchmarks have memory requirements that are far too high or need functionality not necessarily available.
Some are quite one-sided in what they measure: Whetstone, Dhrystone, Coremark all pose as generic benchmarks, but in reality, at least for 8-bit systems, scores are dominated by one single aspect (floating-point performance for Whetstone, string processing for Dhrystone, integer multiplication for Coremark).So, I deciced to write a new benchmark, stdcbench. I wanted it to be suitable for small systems (4KB of RAM, about 32 KB of Flash). There is a trade-off here, since all the data and code will fit easily into caches on bigger systems, but IMO it is worth it.
The current version consists of 2 modules, which on typical systems
should contribute about equally to the score.c90base:
It benchmarks a commonly-implemented subset of what the standardrequires for freestanding implementations of C90. It consists of threesubmodules:1) Huffman/RLE decompression (adapted from real-world code)2) Integer matrix multiplication (synthetic)3) Insertion sort (adapted from real-world code)c90lib:
Benchmarks the standard library.I consists of two submodules:1) Computation of lnlc-width (adapted from real-world code).2) Peephole optimizer (simplified from real-world code).C99 features (e.g. bool, restrict) are used where available, but not
necessary.Modules can be disabled, as necessary due to implementation limitations.
For the future, module(s) for floating-point are planned. But the current benchmark already is quite usable.So far, stdcbench seems to achieve the goals: benchmark a wide range of important standard C functionality, without giving too much emphasis to any particular aspect.
Here are a few scores from a STM8AF5288 at 16 Mhz:
SDCC 3.7.0 RC1 with optimization for code size (-mstm8 --opt-code-size --max-allocs-per-node 100000), binary size 20953 B:
stdcbench 0.3
stdcbench c90base score: 106
stdcbench c90lib score: 87stdcbench final score: 193SDCC 3.7.0 RC1 with optimization for code speed (-mstm8--opt-code-speed --max-allocs-per-node 100000), binary size 21083 B:
stdcbench 0.3
stdcbench c90base score: 109stdcbench c90lib score: 88stdcbench final score: 197IAR 3.10.1.201 with optimization for code size, binary size 24288 B:
stdcbench 0.3
stdcbench c90base score: 117stdcbench c90lib score: 71stdcbench final score: 188IAR 3.10.1.201 with optimization for code speed, binary size 27268 B:
stdcbench 0.3
stdcbench c90base score: 197stdcbench c90lib score: 100stdcbench final score: 297Cosmic 4.4.4 with optimization for code size, binary size 8564 B:
stdcbench 0.3
stdcbench c90base score: 116stdcbench final score: 116Cosmic 4.4.4 with optimization for code speed, binary size 8598 B:
stdcbench 0.3
stdcbench c90base score: 123stdcbench final score: 123For comparison, here is the output for a high-end 8051-derivative, the C8051F120 at 98 Mhz (compiled with SDCC 3.7.0 RC1 using -mmcs51 --model-large --stack-auto --opt-code-speed --max-allocs-per-node 10000), binary size 13282 B:
stdcbench 0.3
stdcbench c90base score: 96stdcbench final score: 96For Cosmic and the C8051F120, the c90lib module was disabled due to implementation limitations.� That of course results in lower scores and makes the code size uncomparable to IAR/SDCC.
These results are quite interesting when
http://www.colecovision.eu/stm8/compilers.shtml
. In particular, while SDCC is ahead in Dhrystone and Coremark scores, it apparently falls behind in stdcbench scores. On the other hand, SDCC seems to do better in code size for stdcbench.Philipp
� For Cosmic, the problem is the lack of qsort() in the standard library. An issue that could be fixed easily, and I hope it will happen. For the C8051F120, the c90lib module runs out of stack space (the MCS-51 architecture has an 8-bit stack pointer).
#compiler #iar #cosmic #c8051 #sdcc #stm82018-06-14 03:48 AM
A few more scores and details on the benchmark can be found in the SCOPES 2018 paper:
https://dl.acm.org/citation.cfm?id=3207726
Philipp