Control and management of toolchain and other tools to create reproducible firmware

[Rookie38]

Hello ST Community,

currently I am thinking about how to ensure that different developers working on the same project are able to keep exact results. Building, linking, debugging and flashing process should be reproducible and identical for my purposes. Any recommendations on this?

When using STM32CubeIDE I had the problem that the project files, which contain information about the toolchain etc., are very difficult to manage. (XML files containing absolute paths, .cproject, .project files etc. are changing on every startup of IDE).

Currently I manage in a separate GIT repository tools like:

- GNU ARM Embedded Toolchain

- Make

- STM32CubeProgrammer

- OpenOCD

The firmware is in a separate GIT repository and relies on a Makefile build process. There are only relative paths to the toolchain repository in the Makefiles. This means that as soon as all developers consider the hierarchy of the firmware and the toolchain folder, it should always be possible to reproduce identical results. The Makefile contains further commands for building, flashing firmware, erasing etc.

Now I wanted to ask if there are better approaches to make the full lifecycle of firmware development and testing more traceable than my approach. (According to my researches there are also approaches to outsource and manage the toolchain in Docker Images. I don't see the additional benefit to my approach).

(i could not find a suitable category for questions regarding firmware development)

Best regards

[waclawek.jan]

​Sounds adequate to me.

Fancy to split some hair?​

The question is, how much external dependencies do you have.​ In principle, you always have some: imagine waking up a couple of centuries later, are your backed-up tools going to run on the current 256-bit PC running Windows 1234? Less extremely, make implicitly depends on shell, and there are some other nuanced dependencies of make on system variables, paths, etc.; but that's usually not anything severe to grant the nontrivial effort to mitigate (but can be incredibly annoying if you stumble upon).

That you don't have external sources (e.g. Cuboids) or libraries, probably ​goes without question.

The programmer is a weak point, but again not worth any more hassle.​

JW​

[Pavel A. · Accepted solution]

Well, a cuboid here. Though I use whatever customer wants: Keil, IAR or makefiles.

Agree that the Eclipse CDT .cproject is a mess. But absolute paths? project files changing on every startup? The project paths can be kept relative, with use of the "path variables".

Paths of external dependencies (including Cube libraries) can be managed.

Sometimes Eclipse indeed changes the project files without obvious reason. These changes can be detected with any decent version control tools, reviewed and reverted if needed.

So it can be kept in control.

Toolchains: The CubeIDE allows to choose a toolchain and install more ST or ARM toolchains. These are versioned. You can also make your own archive. The important thing is to make notes which toolchain and libraries are in use.

Docker is great.

[waclawek.jan]

Two schools of thought appear to clash here... We can call one unix-spirited, although I personally am not in friendly terms with the *nixen either; and the other gave us facebook among other things... My experiences with Docker felt exactly like that.

But who am I to judge.

JW

[Rookie38]

Thanks for the input. I will try the way with the separate repositories to manage the toolchain and remaining development tools. In the CubeIDE settings I have seen that it can also be referred to a separate toolchain, but tools such as STM32CubeProgrammer or even the debugger still remain in the plugins installation subfolder of the IDE. A full CubeIDE installation folder versioning via git was my way to ensure that an CubeIDE update does not change of the programmer or debugger versions, which could maybe results in different outcome on controller side. Practically, ST-Link hardware and its firmware versions should also be tracked, these components could have an influence on the reproducibility too.

[Pavel A.]

>  A full CubeIDE installation folder versioning via git

This looks exactly like a case for docker. Only for Linux, unfortunately (but on Windows 10/11 you can run it too, via WSL2). Make a container with ST toolchain or even complete CubeIDE with Java.

> Practically, ST-Link hardware and its firmware versions should also be tracked

Debuggers are not part of a build... but why not. Later you may want to use a J-Link or other adapter.