The future of STM32CubeCLT

[richmit]

I noticed the following in the recent post announcing STM32CubeIDE for Visual Studio Code 3.8.0:

• CLI tools: A new bundle manager is enabling VS Code to download required CLI tools. This replaces the all-in-one STM32CubeCLT package. ...

Now that STM32CubeCLT is no longer required by the VSCode integration, what is the future of STM32CubeCLT?  In particular, will it continue to be updated and supported?

[mattias norlander · Accepted solution]

Hi @richmit ,

The bundle manager will replace SMT32CubeCLT.
But, we will not deprecate STM32CubeCLT until we have made the bundle manager publicly accessible as a stand-alone tool. This is something we are working on. We also need a bit of introduciton documentation highlighting the main usage scenarios.

In secret we have already shared the bundle manager stand-alone tool with a few customers and the feedback is positive.

Do you see any problems with discontinuing STM32CubeCLT (assuming bundle manager has been launched)? Please share any concerns / questions! That information can help us secure that the user base does not perceive this as a disruptive move.

There are many benefits of the bundle manager:

• Bundle manager only brings the tools you ask for instead of a full package of stuff you may not need...
• Bundle manager can update each tool individually without forcing update of other tools..,
• Bundle manager just adds the "cube" command to the PATH, where as CubeCLT bloats your PATH variable potentially leading to collision with other tools. Bundle manager plays nice!
• Bundle manager can let you work with different GCC versions in different projects without the user having to care about the environmental variables
• Bundle manager is aware of your projects and allows you to call the right CLI tools based on bundle lock-files. Again - you don't have to care!

The cube bundle ... command is today only avialable via VS Code integrated terminal. We have bash script installation method to make it available stand-alone for Win/Lnx/Mac. This allows DevOps to re-create the VS Code project "build and debug" environment on a CI / CD machine with one single command:

cube bundle install --project

That's it. This command will pull download/install/activate only the tools that the proejct requires...

Then to build your project run:

cube cmake ....

The cube prefix before calling cmake ensures that we call the CMake and GCC version that our tools have downloaded instead of whatever you may have on the PATH variable coming...

If you are interested to try an explore, I can provide a bash script allowing you to install the 10MByte bundle manager and put it on PATH... Drop me a PM. :) Always curious to have some feedback...

Note: The bundle manager has been used internally in our "Cube dev organization" to allow developer to share/access many different ST and open-source tools to make collaboration easier!

Kind regards, Mattias

[richmit]

Hi Mattias,

Thank you for your comprehensive reply.  My use cases revolve around command line builds for CI/CD and ad-hoc test/debug.  I think the external bundle manager will not only work for me but be an improvement over the CubeCLT package.

Thanx again,

-mitch.

[Cartu38 OpenDev]

@richmit 
May I ask if you're executing test / debug relying on CI/CD ? I'm a bit curious about.

[richmit]

---

@Cartu38 OpenDev wrote:

@richmit 
May I ask if you're executing test / debug relying on CI/CD ? I'm a bit curious about.

---

I have several stm32 applications that drive DACs as part of a hybrid analog computer. For some applications I use the onboard DAC on STM32G4 MCUs. For others I use a high speed, 16-channel, 16-bit, SPI DAC. This stuff runs on several different targets (STM32G4, STM32L4, & STM32H7), but I unit test everything on a NUCLEO-H753ZI board with an LTC2668 EVM attached. This is permanently connected to a Raspberry Pi equipped with a 16 channel ADC capture board and some relays driven by the Pi's GPIO.

Merging code to "main" trips a git trigger. If the stm32 code was modified, then cmake configure & build flows run for all of my target platforms and for the "unit test" board. The unit test ELFs and some config files are copied to the PI. For each test the ADC relays set appropriately for the test, the board is programmed, DAC outputs are captured, the waveforms are checked, and a test report is written.

I put this together because one of my research interests is developing new numerical methods for differential equations that are resistant to existing methods. Solving the equation with an analog computer gives me a way to test the results. If I have a bug in the DAC drivers, then I'll get incorrect results from the analog computer that may be nearly impossible to detect. Such a situation is super frustrating and inspired me to take the time to build better tests.

-mitch

[JesperEC]

Hi,

Late to the party @mattias norlander but I have a concern.

> Do you see any problems with discontinuing STM32CubeCLT (assuming bundle manager  has been launched)? Please share any concerns / questions! That information can help us secure that the user base does not perceive this as a disruptive move.

The bundle manager sounds like it is very "online" which won't work for us that need to do development on air-gapped networks for various compliance reasons. We are currently not using the STM32 plugins for VS Code for the same reasons because it was too messy to try to pull together all the relevant vsix:es.

We are currently using STM32CubeCLT as a replacement for taking the compiler parts of STM32CubeIDE and separately installing STM32CubeProgrammer. This is also very useful for creating our Docker images for CI.

I guess that the contents of STM32CubeCLT can be downloaded separately (at least GNU Tools for STM32 and STM32CubeProgrammer can) so that we can bring them over the airgap in controlled manner. But it will be an increase of the amount of work we need to do and making sure that all the installers don't trip over each other.

But I fully understand if you do not want to continue to maintain the STM32CubeCLT bundle if it was only intended as a stop gap until the online VS Code experience was on par with STM32CubeMX / STM32CubeIDE.

But an heads up when you release the last version of STM32CubeCLT will help us plan our next step.

     Best regards, Jesper

[mattias norlander]

Tjena Jesper,

Thanks for highlighting this point. You are probably not the only one to have these thoughts. So let's try to cover this use case as well.

The point about VS Code and the hassle of installing the solution in offline environments is known to us. We have several ideas on how to resolve it:

• All-in-one bundle: Create an "offline bundle" collecting all you need (both vsix + bundle files together with an install script)
• Theia / VS Codium: Build an all-in-one IDE based on Theia/VS Codium which we would be allowed to re-package.

Both options have pros & cons. No decision taken, and the back-log is full of higher prio stuff. This is a bit off-topic, your real question was more like: 

"Will the cube bundle manager allow developers working on offline machines to conveniently download/install/activate various CLI tools on CI machines?"

The answer is yes. And this can be done in different ways.

Regardless of how this is done, "re-producibility" is one of the main objectives. Users should not end up with different build/debug behavior CI environment vs dev machines. We must also acknowledge that in order for the bundle manager to become the proper replacement for CubeCLT, it must work with CubeIDE/Eclipse and CubeIDE/VSCode. Let's have a look at those 2x use cases (Eclipse/VSCode:(

We assume that you have at least one internet connected PC that you can use to download/transfer files to you offline network. In the explanaiton above I will use the "Online PC" and "Offline PC" to distinguish between the two types of machines in your environment...

• User Journey1: STM32CubeIDE/Eclipse users wants to setup CI:
  
  • Online PC: Read the CubeIDE Release Note which lists the exact versions of all CLI tools needed in CI environment.
  • Online PC: For each CLI-tool you need, download(not install) the corresponding download which is a zip-file containing some json metadata.

    • cube bundle download gnu-tools-for-stm32@14.3.1+st.2 cmake@4.2.3+st.1 ninja@1.13.2+st.1

  • Online PC to Offline PC: Transfer these X bundles from your online PC to you offline CI machine.
  • Offline CI machine just run this command in the folder where you placed the bundle files:

    • cube bundle install *.bundle

• User Journey2: STM32CubeIDE for VS Code users wants to setup CI:
  • Offline PC to Online PC: Copy the project .settings/bundles*.json file which describes which exact CLI tool versions the project depends on. You don't need to copy the full project code. Only those 2x json-files.
  • Online PC: "cd" into the .settings fodler and execute this command:

    • cube bundle download --project

  • Online PC to Offline PC: That single command read the bundles*.json file and downloaded correct versions of the CLI tools that the proejct depends on into a folder. It did not install it. Now transfer that folder to your offline machine(s)
  • Offline CI machine: Again, just install all bundles

    • cube bundle install *.bundle

As can be seen above the VS Code environment is easier to work with since it produces these .settings/bunles*.json. Therefore you do not explicitly need to check which versions to download. instead just use the --project attribute to let the bundle manager figure that our from the project json-files. The json-files is a manifest listing project CLI tool dependencies. The Eclipse environment does not create these json-files today. Something we may re-visit later. Let's see...

@JesperEC, do you think that would address the needs? Any questions / feedback is appreciated! :)
We have not yet planned to build any prepared docker images yet. Would that add value? At the same time, tool environments are heavily project dependent. Developers may use any combination of versions of GCC, CMake, Ninja, GDB, ST-LINK GDB server, CubeProg, ... At the end I think we need to provide flexibility not all-in-one monolithic deliveries. But the flexible solution must also be easy-to-use the first time and to easy-to-maintain over time.

Sorry for a messy reply. Our official documentation will be easier to read.

I am curious to hear feedback from DevOps who are setting up CI and Docker images to see if there are room for improvements.

Kind regards, Mattias

[JesperEC]

Hej Mattias,

Thanks for the explanation

> , do you think that would address the needs? Any questions / feedback is appreciated! 

I guess this would work as long as the "bundles" are not tainted with user-specific paths and information.

For us it would be important that one person can do the downloading and publish the bundles on internal (offline) storage and other devs can install from there. I guess they have to be pointed to the right location based on what projects they plan to work on.

Even better if you can set an environment variable to the root of storage (or internal web server) and cube bundle download would download from there based on project files - instead of from the internet. Then we can just make sure that internal mirror is kept up to date and no one has to start running around with USB sticks just because some small detail of a project was changed.

> We have not yet planned to build any prepared docker images yet. Would that add value?

I would say no. To a large extent based on the issues you state.

I can't see that someone here would spin up a Docker image with just some specific combination of ST tools. Everything but the most trivial projects will rely on some in-house tooling, credential management for CI, internal CA certificates and what not. It would be easier to take an existing in-house base image that contains these things and then do a Dockerfile like 

FORM company_base_image:1.2
COPY --from=stcontext *.bundle /tmp/bundles
RUN cube bundle install /tmp/bundles/*.bundle

Rather than Docker images I would prefer to have mirrorable repositories of deb and rpm packages for the ST toolchains that I can throw at a apt-get in my Dockerfile or Ansible for non-containerized environments.

This might be hard with VS Code plugins and such but for things like GNU Tools for STM32, STM32CubeProgrammer and the ST-Link udev rules it would be useful. And maybe for STM32CubeIDE and STM32CubeMX as well. The current installers that are deb-files, wrapped in a shell-script, wrapped in a zip-file are messy and require a lot of needlessly duplicated temporary storage.

I would also like to stress that it is a hassle with the current installers that force you to be interactive to accept the click-thru license agreements. In many cases there are variables you can set to inhibit this but I would like to see this working the same across all products and preferably you would only have to accept the license agreement when downloading and then it can be assumed that you did this on behalf of the org. If Legal won't allow you to publish deb-packages without some record of an accepted license agreement, there could be a way in My ST web pages to accept the licenses and obtain an access token that could then be used when mirroring the deb/rpm-files until the license need to change for some reason.

    Vänliga hälsningar / Best regards, Jesper