STM32MP15 ECO 5.0.0 Kernel - Freezes at deferred probe pending

debugging · ‎2023-10-13

Custom MP157AAA3 board. SD card formatted with ECO 5.0.0 using the SD tools using the files from the starter package. (including rootfs etc..) . Build the TF-A, OP-TEE (in SYSRAM), U-boot and kernel from the ECO 5.0.0 sources.

Kernel generates the messages as below and then feezes. The DT's were generated using CubeMX and customized for the board. It looks like drivers are all waiting on resources to become available, perhaps one single one they all need. Hence the deferred probe.

Any hints/steps how to debug this ?

[12.832984] Run /init as init process
[ 13.390242] platform 50000000.rcc: deferred probe pending
[ 13.395778] platform 5000d000.interrupt-controller: deferred probe pending
[ 13.402749] platform 48002000.dma-router: deferred probe pending
[ 13.408792] platform soc:pinctrl@50002000: deferred probe pending
[ 13.414956] platform soc:pinctrl@54004000: deferred probe pending
[ 13.421084] platform 48000000.dma-controller: deferred probe pending
[ 13.427498] platform 58000000.dma-controller: deferred probe pending
[ 13.433906] platform 5a006000.usbphyc: deferred probe pending
[ 13.439684] platform 50001000.pwr: deferred probe pending
[ 13.445143] platform 40010000.serial: deferred probe pending
[ 13.450835] platform 49000000.usb-otg: deferred probe pending
[ 13.456629] platform 5800d000.usb: deferred probe pending
[ 13.462058] platform 5c004000.rtc: deferred probe pending
[ 13.467502] platform 5800c000.usb: deferred probe pending
[ 13.472960] platform 50028000.thermal: deferred probe pending
[ 13.478739] platform stm32-cpufreq: deferred probe pending
[ 13.484273] amba 58005000.mmc: deferred probe pending
[ 13.489353] amba 58007000.mmc: deferred probe pending
[ 13.494448] platform leds: deferred probe pending

debugging · ‎2023-10-15

Fixed after recreating the linux defconfig (default as per WIKI) and recompilation.

View solution in original post

michiel.tjampens · ‎2023-10-13

If i compare the bootlog with mine...

Yours doesn't mention anything about optee.
The order of things pretty much matches, at the point it stops this is what is started on my board.

[ 0.513823] arm-scmi firmware:scmi: Enabled polling mode TX channel - prot_id:16
[ 0.514131] arm-scmi firmware:scmi: SCMI Notifications - Core Enabled.
[ 0.514257] arm-scmi firmware:scmi: SCMI Protocol v2.0 'ST:' Firmware version 0x0

Might be wrong, but optee and scmi are closely related. The scmi dtsi you're using is also pretty empty. Maybe try matching the dk2 one? I copied that one and removed the nodes that aren't relevant for the 151.

debugging · ‎2023-10-13

Thank. Yes, SCMI is OPTEE secure clock service , and there is RCC and PSCI to deal with as well.

https://trustedfirmware-a.readthedocs.io/en/latest/design/firmware-design.html

https://wiki.st.com/stm32mpu/wiki/SCMI_overview

https://wiki.st.com/stm32mpu/wiki/Regulator_overview#Fixed_regulator

I feel this must be a device tree issue missing something or having something too much. it could be defconfig, but since my uImage boots on your board, that shouldn't be the case.

Perhaps this is related using discrete power instead of a PMIC and the regulator/power nodes in the DT. (I don't use any GPIO, I2C or other type of regulator power switches either). There is no ECO 5.0.0. reference DT for STM32MP non-PMIC based board though. so since power to the peripherals is basically always on only clock and rest would be the culprit,

what i notice it that it waits for a few seconds at [2.822487] Run /init as init process and then dumps the deferred messages. with the first one being [ 13.389771] platform 50000000.rcc: deferred probe pending , so I suspect something with reset and clock.

I updated the tree with the SCMI files.

Which OP-TEE message you don't see ? In the log:

INFO:    Image id=4 loaded: 0x2ffc0000 - 0x2ffc002c
INFO:    OPTEE ep=0x2ffc0000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x0
INFO:          flags=0x0
INFO:          nb_images=0x2
INFO:    BL2: Loading image id 8
INFO:    Loading image id=8 at address 0x2ffc0000
INFO:    Image id=8 loaded: 0x2ffc0000 - 0x2ffd2768
INFO:    BL2: Loading image id 9
INFO:    Loading image id=9 at address 0xde000000
INFO:    Image id=9 loaded: 0xde000000 - 0xde054000

AFAIK this is OPTEE (BL32) loaded in SYSRAM (ID8) and DDR (ID9)

NOTICE:  BL2: Booting BL32

I maintain another tree based on 157a-DK1 files, but it crashes on a a panic early in the boot process which I have not figured out yet. ThIs (the CubeMX one) already reached the kernel. The Include files for SCMI (including the one for the board copied form the DK1) are the same unmodified ones from the ECO 5.0.0. package

<rant>somebody from ST must be reading community posts and at least have some idea or hint to help us ? </rant>

EDIT: a bit of topic and side issue: When I ONLY replace the u-boot device dtb blob in FIP generated with the DK DT files, with the one from the MX tree, the DK generated tree (incl TF-A, OPTEE , unboot.bim and kernel) moves beyond the panic and also boots into the kernel, (with the same deferred frreeze as MX package). I am comparing the DK u-boot DT with the MX uboot. DT

michiel.tjampens · ‎2023-10-14

These are printed during kernel startup of my board. Not sure if the first line is a warning or an error...

The last four are printed close to when yours stops.

[    0.121195] Driver 'scmi-optee' was unable to register with bus_type 'tee' because the bus was not initialized.
...
[    0.427815] optee: probing for conduit method.
[    0.427851] optee: revision 3.19 (afacf356)
[    0.435404] optee: dynamic shared memory is enabled
[    0.437111] optee: initialized driver

debugging · ‎2023-10-14

Thanks, I have updated all files to the latest in the gdrive and cleaned it up by moving all changes into the USER area on order to have a clean set of files to easily identify MX generated data versus changes made.

Fishbone:

1. TF-A DT or OP-TEE DT/conf.mk

Perhaps missing scmi, rcc or pcsi nodes ?

For OP-TEE conf.mk only changed option to build for SYSRAM in conf.mk. Other changes are from the workshop.

Running OPTEE in DRAM makes no difference.

Cube MX generates for TF-A & OP=TEE much fewer RCC settings than DK board DT in stm32mp15xx-dkx.dtsi

The PLL's are also very different between the CubeMX generated and 157aDK1 board's in stm32mp15xx-dkx.dtsi

Though using the DK dtsi clock settings for the Cube MX project causes many problems, perhaps that is because

the clock settings used for the 157A based DK1 are different than CubeMX generated ones

2. U-boot defconfig or DT

defconfig or DT ? change are mostly the same as from workshop. (Except also adding the fix for uart bug)

3. kernel defconfig or DT ?

DT is same as u-boot

My uImage boots on you board ... so unlikely.

4. bootfs or rootfs ext4 issue ?

I use the starter packages ext4 image files. unlikely unless the distribution package contains updates.

5. SD card layout.

u-boot find extlinux.conf and uImage. SD card uses the layout generated with file for157a DK board. Unlikely.

Continue debugging..

in Kernel command line added: root=PARTUUID=e91c4e10-16e6-4c0e-bd0e-77becf4a3582 rootwait rw earlyprintk console=ttySTM0,115200 dyndbg="file drivers/usb/core/hub.c +p" loglevel=8 maxcpus=0 logo.nologo

In DEFCONFIG CONFIG_DYNAMIC_DEBUG=y, but i see only a dew much more additional messages, I do see now:

[ 0.226700] No ATAGs?

3.453689] Warning: unable to open an initial console.
[ 3.463352] Freeing unused kernel image (initmem) memory: 2048K
[ 3.482746] Run /init as init process
[ 3.486437] with arguments:
[ 3.489418] /init
[ 3.491700] with environment:
[ 3.494897] HOME=/
[ 3.497269] TERM=linux

Not sure if that is correct.

debugging · ‎2023-10-15

Fixed after recreating the linux defconfig (default as per WIKI) and recompilation.

Guilam · ‎2024-07-05

Would you have any precisions of what you did to fix the issue? I have the exact same problem with exact same output logs and can't find a way to solve it. And I don't know what you mean by recreating linux defconfig.

This would be a huge help, I have been blocked for days on this issue.

Guilam · ‎2024-07-12

So I finally fixed the problem by switch from Ecosystem 5.0.0 to Ecosystem 5.1.0. There must be a bugfix related to this issue in the most recent releases.

I tried that by switching to 5.0.3 previously but I must had done it wrong considering it did not change anything. But this time I did it right so everything is fine.

debugging · ‎2024-08-08

it a year ago, in my notes I could only find "Rebuild the kernel defconfig file from original multi defconfig". so i went back to copy and use the original multi--v7_ defconfig file that is generated in the instruction when installing the development . Probably the deconfig I had was messed up. I am still at ECO 5.0.