C compiler benchmarks

[ccowdery9]

Posted on December 18, 2009 at 10:20

C compiler benchmarks

[ccowdery9]

Posted on May 17, 2011 at 13:34

Are there any benchmarks comparing the various C compilers on Cortex?

I'm particularly interested in various remarks saying GCC is poor, so I'd like to see some figures and compiler output examples.... Chris.

[gahelton12]

Posted on May 17, 2011 at 13:34

I've found these benchmarks comparing various compilers including GCC. These benchmarks are several years old and compare the Normal and Thumb mode of an ARM7 core instead of the Thumb2 mode of a Cortex. I would expect the Thumb and Thumb2 results to be similar for a given compiler.

Since these results are years old, these compilers may have changed. For certain, Keil now uses the ARM company RVCT compiler which greatly improved their performance over their original CARM compiler. In any case, GCC appears to perform well in a lot of categories. I would also be interested in seeing a more recent benchmark comparing various compilers. Benchmark reports are attached.

[ccowdery9]

Posted on May 17, 2011 at 13:34

Interesting benchmarks.

I ask because on the one hand you see marketing material claiming that GCC isn't particularly good, yet every time I've examined the assembler output I've found it to be very clever. Further, I often fall foul of the optimiser stripping out redundant code / execution paths (or what it deems to be redundant!). Hence the interest in some actual figures. Certainly the figures so far show GCC to be pretty good in fact! Just need some Cortex benchmarks now. I wonder why ARM wanted their compiler removed from the results? Seems a bit odd. Chris.

[st3]

Posted on May 17, 2011 at 13:34

Quote:

I would expect the Thumb and Thumb2 results to be similar for a given compiler.

I don't think that's a safe assumption at all.

[akaiser9]

Posted on May 17, 2011 at 13:34

From looking at the generated code I've got the impression that GCC does not make much use of r8-r11 in Thumb2 even when register pressure is high. Other Thumb2 extensions like IT (conditional exec) are heavily used though. What about other compilers?

[st3]

Posted on May 17, 2011 at 13:34

The trouble is, ''bechmarks'' can very easily be very misleading!

just because compiler 'A' beats 'B' hands-down in a bubble sort doesn't necessarily mean that the same will be true in any other application; especially not in a small embedded application - as would be typical of a Cortex-M3 application... I think the only really meaningful comparison is to look at how different compilers cope with your particular type(s) of code in your particular type(s) of application.

Quote:

I wonder why ARM wanted their compiler removed from the results? Seems a bit odd.

OK, this is pure speculation, but here's a possibility: I think the ARM compiler is specifically designed as an embedded compiler, but my distinct impression of GCC is that it is not. On that basis, it is quite conceivable that the ARM compiler could appear to score ''poorly'' on bechmarks that bear little relevance to real embedded applications... Again, that's pure speculation - I have no evidence to back that up at all!

[st3]

Posted on May 17, 2011 at 13:34

For specifically embedded benchmarks,have you looked at EEMBC - the Embedded Microprocessor Benchmark Consortium

http://www.eembc.org

? I think they're focussed more on bechmarking chips - but might be worth a look...?

[st3]

Posted on May 17, 2011 at 13:34

Just found this on the EEMBC site:

Quote:

01 June 2009

ARM Announces Support For EEMBC CoreMark Benchmark New open source benchmark provides an alternative to Dhrystone WHAT: ARM today announced their support for the new EEMBC (Embedded Microprocessor Benchmark Consortium) CoreMark processor benchmark. CoreMark is a synthetic benchmark, but carries out meaningful calculations, and is claimed to be a significant improvement on current Dhrystone benchmarks which is overly dependent on the performance of the C library...

http://www.arm.com/news/25152.html

So perhaps you should get hold of that ''open-source benchmark'' and see what the differnt compilers make of it... Interesting to note the bit about ''dependent on the performance of the C library'' - so it's not just about the compiler itself, but also the libraries. Which does somewhat bear-out my suspicion - as a whole lot of the standard 'C' library is of no relevance to the types of applications likely to be found in a Cortex-M3... Also ARM (as Keil) provide (at least) 2 distinct libraries; so you can't just talk about, ''the ARM compiler'' - you would have to say ''the ARM compiler (with standard library)'' or ''the ARM compiler with microlib'', etc. And, of course, the library relies upon user-provided target-specfic code... CoreMark:

http://www.coremark.org/home.php

[ This message was edited by: st7 on 17-12-2009 14:48 ]

[gahelton12]

Posted on May 17, 2011 at 13:34

The Keil compiler used in these benchmarks was the CARM compiler. As can be seen, it was outperformed by the other compilers in most tests (with the exception of the full ''printf'' tests).

However, Keil was bought by ARM company about the same time the last benchmark was run. Since Keil was preparing to use the ARM RVCT compiler, you can't very well do a comparision against two identical compliers in the test. I think a more sensible approach would have been for Keil to deliver the new uVision toolset with the ARM RVCT compiler, and then have the ''ARM'' complier withdrawn before the benchmark was run. Of course, this is all speculation.