Convert IDE project to scriptable build

[ashaver]

We have added an STM32CubeIDE project to our monorepo. The monorepo is mostly C++ software built for linux x86_64, using CMake as the build tooling. This software is built inside of docker containers in our CI environment. We would like to be able to build this STM32 project as part of CI.

We need to be able to script the configuration and build of an IDE project without any human interaction. This includes scripting the installation of any tools needed for the build. Ideally we would like to have a "single source of truth" build (meaning not a separate `.project` system from what runs in CI, where they could get out of sync). Furthermore, we'd like to avoid committing generated code to the repo. We can copy down the "Drivers" and other software packages, but we'd like to generate code from the .ioc file rather than commit the generated code. 

I'll sketch out our current workflow, things I've tried, and problems I've encountered. This has to be a thing others are doing, so I don't know if I'm missing documentation somewhere or what.

Development workflow:

1. CMake build project - We have generated code (protobuf) that will need to exist for the STM32 project
2. Copy in relevant Driver files to project area
3. Open CubeIDE to an `stm32` directory in the project as the workspace, then add the project (subdir from there) to the workspace
4. In CubeIDE, run configure from `.ioc` file
5. build project

CMake/Makefile approach:

It seems to me like the IDE supports Makefiles and CMake. Ideally, we could just convert our project to a CMakeLists.txt and with a little bit of tweaking get it to either run as part of our overall build or as a standalone build. Either way would be fine. I can't figure out how to convert an existing project to CMake. There are instructions for starting new projects that way, but it's not clear to me how to convert them, if that's even possible. (edit: I have tried CLion to automatically generate the CMakeLists.txt, but the resulting file takes a bit of guesswork to get into working shape, and even then it wasn't getting dependencies resolved correctly)

Alternatively, I'd be okay with a Makefile based approach, but, again, I can't figure out how to get the IDE to actually generate a Makefile that I could run.

Headless build approach:

I can get the IDE "headless build" mode to build the code, but only if I've already run the generator against the `.ioc` file. It isn't clear to me how to actually run the generator in this headless mode. Additionally, I believe it's not best practices to commit generated code, so I would prefer a solution that can generate as a part of the build process.

Additionally, this is a bit of a frustrating approach because I shouldn't need to have an IDE installed in my CI environment. I should be able to just build the code (the CMake/Makefile approach above). I haven't yet tried to install the IDE into a docker container automatically (i.e. without an install "wizard"), but I had a lot of difficulty with the STM32CubeCLT before I realized that wasn't the right toolkit for me.

[jerry_sandc]

I feel your pain!  I also have to ensure we can build code in CI (headless) environment.  CMake has always been our preferred build system.  I let CMake create the Makefiles. 

I don't know how to run the code generation from the command line.  I suggest you do that once and save the generated code, and manually modify from there.   Consider that the code shouldn't change too much once you've gone through your initial setup.

I created a long post on how I put together a CMake based project.  Its not for the faint of heart.  We setup a cross compile toolchain file for CMake.  We do not call the IDE to build the code, we use the arm compiler/linker directly.

https://community.st.com/t5/stm32cubeide-mcus/my-solution-for-building-project-for-stm32f7-evaluation-board/td-p/594258

If you have any specific questions, let me know.

[ashaver]

Thanks, I think the piece I didn't understand was that the "code generation" step didn't need to be run manually each time. I rather thought it was like `.proto` files where a fair amount of source code was generated each time you build. I'll look through your post and give that a try. 

[Pavel A.]

Why " I shouldn't need to have an IDE installed in my CI environment"?

Does your CI support custom user-provided containers? If yes, you can make a container with all tools installed in it (even CubeIDE, it is not huge) and push to the docker store or whatever.  Then enjoy the headless eclipse build, as you've mentioned above. The ST libraries can be stored in the same container or fetched from the ST's github.

Agree with @jerry_sandc  "I suggest you do that once and save the generated code, and manually modify from there" - the CubeMX part normally isn't needed in the CI - unless you develop some extension of the Cube and want to test the integration. 

[jerry_sandc]

well, ironically, we do install the IDE in our docker image (container), but only use the ARM tools that come with it.

[ashaver]

I have configured such a container. (though the documentation on how to do so is considerably lacking, imo; one needs to set an otherwise undocumented environment variable, LICENSE_ALREADY_ACCEPTED=1, to allow for headless installation of the IDE). 

Still, not a huge fan of the UX here overall. My actual solution was to add "Embedded Tools" extension to VSCode, have that tool generate a CMake build setup. Tweak that enough to be able to build it (their tool didn't handle "linked" source folders for the project). Modify the CMake Generator to produce Eclipse CDT4 project files. Then engineers can open the CMake generated project in the CubeIDE, but CI systems can just build directly from the CMake generated system itself.

It just feels entirely overly-complex. Like if the CubeIDE is going to use makefiles to build the project, maybe there should be a way to expose the makefiles directly and be able to build outside of the IDE? Maybe there is and I just haven't figured out how. Or maybe, like how CLion and VSCode have tools to convert the Eclipse project into a CMake one (and do so poorly), maybe there could be a way to convert an eclipse project into a CMake one? I tried a couple of variations of the `New CMake project`, none of which would convert the existing project, as best I could tell.

[Pavel A.]

>  if the CubeIDE is going to use makefiles to build the project, maybe there should be a way to expose the makefiles directly and be able to build outside of the IDE

Eclipse CDT has two build modes: run a user tool (typically, make with user-provided makefile, or CMake/ninja or whatever) and so called managed builder - which gives all the CDT goodies: automatic scan for files, customizable project properties, configurations etc. The latter optionally can produce makefiles and run them with 'make' - but these generated makefiles are noisy and not quite intended for humans. You don't want to edit them directly and check into the source control. Preferred mode is the "internal builder", when these makefiles are hidden, and the make is replaced by something inside the eclipse. The only two files visible to the user are .project and .cproject. Now, they are horrible abomination of xml, ought to be replaced by CMake scripts (or better) long ago... but the traction is too slow ((

From the CI POV, the headless Eclipse "managed" build is all you need from CubeIDE. If you had a normal makefile/CMake scripts you could just run these.

"External builder" is the default mode after installation, IIRC because the internal builder has some minor bugs, maybe already fixed in the last update. (these bugs don't affect me, so I always switch to internal builder).

For engineering, the main and crucial component of IDE is the debugger. It makes or kills developer's productivity. Editing and building you can do with vi, VS Code... even notepad. Eclipse is fair if you're well used to it.

ST provides a decent free debugger that works well with their MCUs. Generic gdb or OpenOCD are opensource and free... but that's all good about them. 

Thanks ST, they enabled CubeIDE to pick an executable built with anything, and debug it. Actually this produces a new simple CubeIDE project of the 1st kind, which you can reuse, edit and extend later.