Cubeide does not start (and 'Spawner Reaper')

[giancarlofi]

I installed cubeide (under linux ver 1.15.1) and started it by creating a workspace.
I created a new project (STM32 project), the 'Target selector' appeared and I selected the nucleo-g474re board.
I gave the project a name, I answered yes to the question 'Initialize all peripher.....".

The "Device Configuration Tool editor is associated ..." window appeared and I answered yes again.

The bar under "Perform Project Creation. Please wait..." moves ...
After several minutes (10) I launch top and see that a 'Spawner Reaper' process is active and uses 100% CPU and 11% memory.
If I give the command 'strace -p PID' the result is "close(510686161) = -1 EBADF (Bad File Descriptor)"

Where can I see the output of 'Spawner Reaper'?
How can I find out what he's doing?

Suggestions?
Thank you

[JohannGlaser]

Same problem here. It seems that this "Spawner Reaper" tries to close all file handles from ~0 to 2^31, and afterwards it continues and executes "make" as usual.

My installation of STM32CubeIDE (Version: 1.15.0 Build: 20695_20240315_1429 (UTC)) worked without a problem a few weeks ago.

[giancarlofi]

Hi JohannGlaser, I have more information. Due to a problem with the ssd disk I had to reinstall the system, with Debian testing [trixie]. Even in a 'clean' system cubeid didn't work. If I tried to install with the 'STM32CubeIDE Debian Linux Installer' python2.7 was missing and the installation did not reach the end, with the 'STM32CubeIDE Generic Linux Installer' I completed the installation but the system blocked when creating a new project for 20 minutes and then for another 20 minutes when I compiled. Now I'm working on a virtual machine with Ubuntu 24.04 and everything works normally. The problem could be in some program of the toochain, perhaps even python.

I attach the list of installed packages, I don't have time to look at them now, but the solution is there.

Ciao

[blinken]

I'm having exactly this problem following an upgrade to Debian Testing (required for KiCad 8.0). STM32CubeIDE v1.15.1 worked well until the upgrade. Now it will start, but building projects involves a ~5 minute wait before any commands are executed. I have reinstalled STM32Cube IDE several times and created a fresh workspace.

During the delay, strace shows that "Spawner Reaper" is attempting to close every file descriptor one by one, consuming 100% CPU in the process.

...

close(160986609) = -1 EBADF (Bad file descriptor)
close(160986610) = -1 EBADF (Bad file descriptor)
close(160986611) = -1 EBADF (Bad file descriptor)
close(160986612) = -1 EBADF (Bad file descriptor)
close(160986613) = -1 EBADF (Bad file descriptor)
close(160986614) = -1 EBADF (Bad file descriptor)

...

The "Spawner Reaper" process is the following command:

/opt/st/stm32cubeide_1.15.1_3//plugins/com.st.stm32cube.ide.jre.linux64_3.1.0.202310310920/jre/bin/java -Dosgi.requiredJavaVersion=1.8 -Dosgi.instance.area.default=@user.home/STM32CubeIDE/workspace_1.15.1 -Declipse.buildId=Version 1.15.1 -DproductMaturityGrade=mm -Djdk.http.auth.tunneling.disabledSchemes= -XX:+UseG1GC -XX:+UseStringDeduplication -Xms256m -Xmx1024m -jar /opt/st/stm32cubeide_1.15.1_3//plugins/org.eclipse.equinox.launcher_1.6.600.v20231106-1826.jar -os linux -ws gtk -arch x86_64 -showsplash -launcher /opt/st/stm32cubeide_1.15.1_3/stm32cubeide -name Stm32cubeide --launcher.library /opt/st/stm32cubeide_1.15.1_3//plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.2.800.v20231003-1442/eclipse_11802.so -startup /opt/st/stm32cubeide_1.15.1_3//plugins/org.eclipse.equinox.launcher_1.6.600.v20231106-1826.jar --launcher.overrideVmargs -exitdata 5801d -vm /opt/st/stm32cubeide_1.15.1_3//plugins/com.st.stm32cube.ide.jre.linux64_3.1.0.202310310920/jre/bin/java -vmargs -Dosgi.requiredJavaVersion=1.8 -Dosgi.instance.area.default=@user.home/STM32CubeIDE/workspace_1.15.1 -Declipse.buildId=Version 1.15.1 -DproductMaturityGrade=mm -Djdk.http.auth.tunneling.disabledSchemes= -XX:+UseG1GC -XX:+UseStringDeduplication -Xms256m -Xmx1024m -jar /opt/st/stm32cubeide_1.15.1_3//plugins/org.eclipse.equinox.launcher_1.6.600.v20231106-1826.jar

I can compile the project outside of eclipse ok with a script similar to below; the build time is <1 second. This may be a workaround.

I'm happy to provide any information or assistance needed to get to the bottom of this.

$ cat write_firmware.sh 
#!/bin/bash
# Requires stlink-tools and the arm toolchain -
# sudo apt install stlink-tools gcc-arm-none-eabi gdb-multiarch

set -e

if [ "$EUID" -eq 0 ]
  then echo "Must not be run as root"
  exit
fi

# Check if the script is run in the correct directory
if [ ! -f "Debug/makefile" ]; then
  echo "Please run the script in the root of the repository"
  exit
fi

echo "## Building firmware"
cd Debug
make -j 16 all

# Check if the firmware is built successfully
if [ ! -f "project-name.bin" ]; then
  echo "Firmware build failed - could not find project-name.bin"
  exit
fi

echo
echo "## Writing firmware"
sudo st-flash write project-name.bin 0x8000000
sleep 1
echo
echo "## Resetting board"
sudo st-flash reset

[JohannGlaser]

Thanks a lot to both of you for the additional information!

Yesterday I didn't provide information about my system. I'm also using Debian as both of you, but I set it to "unstable" (which rarely makes problems). An upgrade from KiCAD 7 to 8 also fell in the same time frame for me, after which STM32CubeIDE stopped working.

The 100% CPU is consumed inside of a Java process, which is part of the STM32CubeIDE. Also, when I googled for this, I found that it is part of Eclipse CDT. For example, see https://www.tabnine.com/code/java/packages/org.eclipse.cdt.utils.spawner .

Its source code https://github.com/eclipse-cdt/cdt/blob/main/core/org.eclipse.cdt.core.native/src/org/eclipse/cdt/utils/spawner/Spawner.java#L509 even has a function "closeUnusedStreams()", but it only closes the 3 standard streams. No sign of this loop.

Now, I'm not 100% sure whether this is actually the culprit, or the same version and/or branch of it, although at line 614 it does have a sub-class Reaper.

And what also makes me curious is the fact, that despite this all being Java code, which did not change at all from before it worked, BUT we updated the Debian packages, how this could be affected.

I'll try to strace a bit more of that _before_ the 100% happens. Perhaps it gives some insight.

[JohannGlaser]

I could investigate further. In https://github.com/eclipse-cdt/cdt/blob/main/core/org.eclipse.cdt.core.native/native_src/unix/exec_unix.c#L87 there is a call to sysconf(_SC_OPEN_MAX) to get the maximum number of open file descriptors. And then below it, there is a while loop closing all these files.

Looking at the strace, I could identify the preceding calls to close() and dup2(), and then the Linux kernel call prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1073741816, rlim_max=1073741816}) = 0. It seems that LibC uses this internally for the sysconf() call above.

This very high number of a bit over a billion (1073741816 = 0x3FFFFFF8) is our problem.

"man prlimit64" gives the informatoin, that a process can increase its rlim_cur up to the value of rlim_max. If it tries to increase it beyond the value in /proc/sys/fs/nr_open it will fail. This file interestingly also states 1073741816.

When executing "ulimit -n" and "getconf OPEN_MAX" on my command line, I get 1024, which is much more reasonable. For the latter command, strace reveals prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=1073741816}) = 0.

The current limit of open files for all processes of STM32CubeIDE are also interesting:

$ grep open /proc/{1108250,1108274,1108527,1108563,1108567,1108625,1108626}/limits
/proc/1108250/limits:Max open files 1024 1073741816 files 
/proc/1108274/limits:Max open files 1073741816 1073741816 files 
/proc/1108527/limits:Max open files 1073741816 1073741816 files 
/proc/1108563/limits:Max open files 1073741816 1073741816 files 
/proc/1108567/limits:Max open files 1073741816 1073741816 files 
/proc/1108625/limits:Max open files 1073741816 1073741816 files 
/proc/1108626/limits:Max open files 1073741816 1073741816 files 

Only the very first has the default soft limit, but all others have the much higher value (same as the hard limit).

I've tried (as root) to

echo 4096 > /proc/sys/fs/nr_open

This keeps the hard limit still at 1073741816 for all running processes. But, when I started STM32CubeIDE, it did not change its soft limits for any of its processes. They all stayed at 1024. Building the project worked as before.

I tried a second workaround by setting the hard limit in my current shell with

ulimit -n -H 4096

and afterwards I executed STM32CubeIDE. Now it seems to again setting the soft limit to the hard limit (4096), but at least it is a much lower value. Building also works fine again.

grep open /proc/{1110516,1110540,1110760,1110796,1110800,1110858,1110859}/limits
/proc/1110516/limits:Max open files 1024 4096 files 
/proc/1110540/limits:Max open files 4096 4096 files 
/proc/1110760/limits:Max open files 4096 4096 files 
/proc/1110796/limits:Max open files 4096 4096 files 
/proc/1110800/limits:Max open files 4096 4096 files 
/proc/1110858/limits:Max open files 4096 4096 files 
/proc/1110859/limits:Max open files 4096 4096 files 

[JohannGlaser · Accepted solution]

TL;DR of the above posting: As a work-around, set the hard limit of allowed open files to a reasonable number before starting STM32CubeIDE.

ulimit -n -H 4096
/opt/stm32cubeide_1.15.0/stm32cubeide &

[blinken]

@JohannGlaser wow, what a simple fix. Thank you so much for spotting this.

I see the same absurd default ulimit -

$ ulimit -n
1024
$ ulimit -n -H
1073741816

Setting it to 4096 before starting Eclipse, as you suggest, fixes the problem.

Digging a little further, I find that several years ago, systemd increased RLIMIT_NOFILE to 1073741816, and this change was overridden in Debian to 524288 because "Not every software is able to deal with such an RLIMIT_NOFILE properly." https://salsa.debian.org/systemd-team/systemd/-/commit/084e84e33a403868b7f84159da745689e2ff0ba9 

On 28th May 2024 that patch was reverted in the Debian systemd package version 256~rc3-3, with the comment

Restore open files limit bump on boot. Broken packages ought to have been fixed by now. (Closes: #1029152)

See https://metadata.ftp-master.debian.org/changelogs//main/s/systemd/systemd_256.1-2_changelog and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1029152 

To restore this limit system-wide, as an alternative to setting ulimit -n -H 4096 before starting Eclipse, edit /etc/security/limits.conf and add the line

*                hard    nofile          524288

Note that on my Debian system with systemd 256 editing /etc/systemd/user.conf or /etc/systemd/system.conf to set the line DefaultLimitNOFile=1024:524288 seems to have no effect (I don't fully understand why).

I will add a comment on the Debian bug report that this change breaks some versions of Eclipse, but I agree with them that this is an Eclipse/STM32CubeIDE bug. Other Linux distros will likely report poor build performance if STM32CubeIDE is not fixed.

[giancarlofi]

wow, work!

thank you all. !!

[JohannGlaser]

@blinken, wow, thanks for digging this out and providing this additional information!

I just wanted to reply to the bug at Debian, but it was not accepted because the bug was already archived. :\

Hi!

From: Luca Boccassi <noreply@salsa.debian.org>
> Broken packages ought to have been fixed by now.

There are still some programs which don't work with such a high limit,
for example Eclipse CDT (used as basis of STM32CubeIDE). See the thread
at
 
https://community.st.com/t5/stm32cubeide-mcus/cubeide-does-not-start-and-spawner-reaper/m-p/690290
for more details.

It is actually a bug in Eclipse CDT, that they use a loop to count all
possible file descriptor numbers. However, increasing the limit
triggered that (hidden) bug.

And it is unlikely that such programs (like STM32CubeIDE or any other
Eclipse CDT based tools for MCUs) will get a fix soon. Some users are
even bound to old versions for whatever other technical reason.

Bye
  Hansi

Now I just wanted to file a bug against Eclipse CDT, but it was already solved in the meantime: https://github.com/eclipse-cdt/cdt/issues/835 . :beaming_face_with_smiling_eyes: Hopefully STM32CubeIDE uses this new CDT for their next version.