Upgrading Kernel for EV Charging System: Seeking Guidance on Debugging Kernel Boot

[VivekB]

Hello everyone,

We are currently using the v5.15-stm32mp-r2 kernel for our EV charging system and have decided to upgrade to v6.1-stm32mp-r2 for its long-term support. I successfully compiled the v6.1-stm32mp-r2 kernel using Buildroot by replacing the kernel tarball. During the process, I fixed issues in the device tree source (DTS) file, including sram and clock references. After resolving these, I was able to compile the final image.

However, when I flash the image onto the STM32MP157C, the boot process halts at the message "Starting kernel..." with no further logs. This suggests the kernel is crashing early in the boot process.

I suspect the changes I made shouldn’t cause such a critical failure, but debugging is difficult without any output. I am trying to enable early printk for more insight into the kernel’s initialization, but I am struggling to get it working. I am considering adding log traces to the start_kernel function as a next step.

For now, I have not replaced U-Boot or made other major changes. My plan is to modify one component at a time to isolate issues before completing the full upgrade.

Does this approach make sense? If anyone can provide guidance on enabling early printk or suggest other ways to debug early-stage kernel crashes, I would greatly appreciate your input. Let me know if there’s anything I might be missing.

Thank you for your help!

[Olivier GALLIEN]

Hi @VivekB,

If you want to upgrade your kernel from v5.15-stm32mp-r2 (DV4.1) to v6.1-stm32mp-r2 (DV 5.1) you might also consider getting the associated BSP component tf-a, uboot, Optee .. 

ST does not guarantee combability between major versions. 

Refer to STM32 MPU ecosystem release note - stm32mpu-ecosystem-v5 

And as you are upgrading, be aware that a new DV6 is also available. 

STM32 MPU ecosystem release note - stm32mpu

Hope it help 

Olivier 

[VivekB]

Hi Olivier, 

Thanks for your reply. I appreciate you pointing me to the other components in the package for example tf-a, uboot, Optee - but do I need to replace everything in one go? I am planning to take the activity in steps. As a first step I am trying to replace the kernel tarball, and get the system up maybe with minimal support. But at the moment the kernel seems to be crashing. I have got some more debugging information regarding this which is as below: 

The below are the information related to the existing v5.15 kernel and this works:

do_bootm_linux: Function called with flags: 0x100, argc: 1

========== do_bootm_linux Debug Information ==========
Flag: 0x100
Argument Count (argc): 1
argv[0]: C0000000
Images Pointer: ddfe3e14
Kernel Entry Point (images->ep): 0xc00000d0
Initrd Start (images->rd_start): 0xc05f437c
Initrd End (images->rd_end): 0xc240937c
Relocated Initrd Start (images->initrd_start): 0xce1eb000
Relocated Initrd End (images->initrd_end): 0xd0000000
FDT Address (images->ft_addr): c05e4e08
FDT Length (images->ft_len): 62651 bytes
Verification Result (images->verify): -1
======================================================
do_bootm_linux: Preparing kernel (BOOTM_STATE_OS_PREP)
boot_prep_linux: Starting Linux preparation
boot_prep_linux: Using FDT
   Loading Device Tree to ce1d8000, end ce1ea4ba ... OK
boot_prep_linux: Calling board_prep_linux
boot_prep_linux: Completed Linux preparation
do_bootm_linux: Kernel preparation completed (BOOTM_STATE_OS_PREP)
do_bootm_linux: Function called with flags: 0x400, argc: 1

========== do_bootm_linux Debug Information ==========
Flag: 0x400
Argument Count (argc): 1
argv[0]: C0000000
Images Pointer: ddfe3e14
Kernel Entry Point (images->ep): 0xc00000d0
Initrd Start (images->rd_start): 0xc05f437c
Initrd End (images->rd_end): 0xc240937c
Relocated Initrd Start (images->initrd_start): 0xce1eb000
Relocated Initrd End (images->initrd_end): 0xd0000000
FDT Address (images->ft_addr): ce1d8000
FDT Length (images->ft_len): 62651 bytes
Verification Result (images->verify): -1
======================================================
do_bootm_linux: Jumping to kernel (BOOTM_STATE_OS_GO or BOOTM_STATE_OS_FAKE_GO)
Kernel entry point set to: 0xc00000d0
Using default machid: 0x0
Transferring control to Linux at address: 0xc00000d0...
Boot stage marked: RUN_OS.

VBDBG Starting kernel ...

Using FDT at address: 0xce1d8000
Jumping to kernel at address: 0xc00000d0 with machid: 0x0 and boot parameters: 0xce1d8000
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.15.67 (vivekb@vivekb-ThinkBook-16-G6-IRL) (arm-none-linux-gnueabihf-gcc (GNU Toolchain for the A-profile Architecture 10.3-2021.07 (arm-10.29)) 10.3.1 20210621, GNU ld (GNU4
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: STMicroelectronics custom STM32CubeMX board - openstlinux-5.15-yocto-kirkstone-mp1-v22.11.23

 And the below are the logs from the v6.1 which is crashing: 

NOTICE:  Early console setup
NOTICE:  CPU: STM32MP157CAA Rev.Z
NOTICE:  Model: STMicroelectronics custom STM32CubeMX board - openstlinux-5.15-yocto-kirkstone-mp1-v22.11.23
NOTICE:  Bootrom authentication succeeded
NOTICE:  BL2: v2.6-stm32mp1-r2.0(release):2022.02.7-2624-g4f5852e1b6-dirty(4f5852e1)
NOTICE:  BL2: Built : 06:46:31, Dec 11 2024
NOTICE:  TRUSTED_BOARD_BOOT support enabled
NOTICE:  BL2: Booting BL32
optee optee: OP-TEE: revision 3.16 (4f5852e1)


U-Boot 2021.10-stm32mp-r2 (Dec 11 2024 - 06:45:45 +0000)

CPU: STM32MP157CAA Rev.Z
Model: STMicroelectronics custom STM32CubeMX board - openstlinux-5.15-yocto-kirkstone-mp1-v22.11.23
Board: stm32mp1 in trusted mode (st,stm32mp157c-ccb-mx)
DRAM:  512 MiB
optee optee: OP-TEE: revision 3.16 (4f5852e1)
Clocks:
- MPU : 650 MHz
- MCU : 200 MHz
- AXI : 266.500 MHz
- PER : 24 MHz
- DDR : 533 MHz
WDT:   Started with servicing (32s timeout)
MMC:   STM32 SD/MMC: 0, STM32 SD/MMC: 1
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial@40010000
Out:   serial@40010000
Err:   serial@40010000
invalid MAC address 0 in OTP 00:00:00:00:00:00
Net:   
Error: ethernet@5800a000 address not set.
No ethernet found.

Hit any key to stop autoboot:  0 
Boot over mmc1!
Saving Environment to MMC... Writing to redundant MMC(1)... OK
switch to partitions #0, OK
mmc1(part 0) is current device
get_runtime_version c0000000 mmc 1 3 /boot/image.fit
ifname:mmc Reading /boot/bootstatus from 1:3
get_boot_status failed to set block device access method
Loading /boot/image.fit from mmc:1:3
failed to set block device access method
get_runtime_version c4000000 mmc 1 4 /boot/image.fit
ifname:mmc Reading /boot/bootstatus from 1:4
get_boot_status failed to set block device access method
Loading /boot/image.fit from mmc:1:4
failed to set block device access method
get_runtime_version c0000000 mmc 1 2 /boot/image.fit
ifname:mmc Reading /boot/bootstatus from 1:2
** File not found /boot/bootstatus **
failed to read /boot/bootstatus
Loading /boot/image.fit from mmc:1:2
Checking FIT image at 0xc0000000
Using 'indra-ccb-kernel-dtb-ramdisk' configuration
Verifying Hash Integrity ... sha256,rsa2048:ccb+ OK
description = 'version 2052 - v0.8.4
release-I8 (2024-12-11T06:46:48 4f5852e1b6 dev)'
bootm_args[0]:do_bootm bootm_args[1]:C0000000
## Loading kernel from FIT Image at c0000000 ...
   Using 'indra-ccb-kernel-dtb-ramdisk' configuration
   Verifying Hash Integrity ... sha256,rsa2048:ccb+ OK
   Trying 'kernel' kernel subimage
     Description:  kernel zImage
     Created:      2024-12-11   6:46:51 UTC
     Type:         Kernel Image (no loading done)
     Compression:  uncompressed
     Data Start:   0xc00000d0
     Data Size:    6475216 Bytes = 6.2 MiB
     Hash algo:    sha256
     Hash value:   8612853edb6e488df4b6de60a6420b1d0a21f6592d61e7034c962362334aad60
   Verifying Hash Integrity ... sha256+ OK
## Loading ramdisk from FIT Image at c0000000 ...
   Using 'indra-ccb-kernel-dtb-ramdisk' configuration
   Verifying Hash Integrity ... sha256,rsa2048:ccb+ OK
   Trying 'ramdisk' ramdisk subimage
     Description:  rootfs
     Created:      2024-12-11   6:46:51 UTC
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0xc063cea4
     Data Size:    31846400 Bytes = 30.4 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: unavailable
     Entry Point:  unavailable
     Hash algo:    sha256
     Hash value:   489103a7633d620a44381f8789bff9af669aa98c4d13ec76854ceaafcaf183a2
   Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at c0000000 ...
   Using 'indra-ccb-kernel-dtb-ramdisk' configuration
   Verifying Hash Integrity ... sha256,rsa2048:ccb+ OK
   Trying 'fdt' fdt subimage
     Description:  stm32mp157c-ccb-mx.dtb
     Created:      2024-12-11   6:46:51 UTC
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0xc062cfa0
     Data Size:    65097 Bytes = 63.6 KiB
     Architecture: ARM
     Hash algo:    sha256
     Hash value:   58fb1c669170894c5fc136f310a59dc15ae086bdac94a3b7413fd59d55bafe87
   Verifying Hash Integrity ... sha256+ OK
   Booting using the fdt blob at 0xc062cfa0
   XIP Kernel Image (no loading done)
   Loading Ramdisk to ce1a1000, end d0000000 ... OK
do_bootm_linux: Function called with flags: 0x100, argc: 1

========== do_bootm_linux Debug Information ==========
Flag: 0x100
Argument Count (argc): 1
argv[0]: C0000000
Images Pointer: ddfe3e14
Kernel Entry Point (images->ep): 0xc00000d0
Initrd Start (images->rd_start): 0xc063cea4
Initrd End (images->rd_end): 0xc249bea4
Relocated Initrd Start (images->initrd_start): 0xce1a1000
Relocated Initrd End (images->initrd_end): 0xd0000000
FDT Address (images->ft_addr): c062cfa0
FDT Length (images->ft_len): 65097 bytes
Verification Result (images->verify): -1
======================================================
do_bootm_linux: Preparing kernel (BOOTM_STATE_OS_PREP)
boot_prep_linux: Starting Linux preparation
boot_prep_linux: Using FDT
   Loading Device Tree to ce18e000, end ce1a0e48 ... OK
boot_prep_linux: Calling board_prep_linux
boot_prep_linux: Completed Linux preparation
do_bootm_linux: Kernel preparation completed (BOOTM_STATE_OS_PREP)
do_bootm_linux: Function called with flags: 0x400, argc: 1

========== do_bootm_linux Debug Information ==========
Flag: 0x400
Argument Count (argc): 1
argv[0]: C0000000
Images Pointer: ddfe3e14
Kernel Entry Point (images->ep): 0xc00000d0
Initrd Start (images->rd_start): 0xc063cea4
Initrd End (images->rd_end): 0xc249bea4
Relocated Initrd Start (images->initrd_start): 0xce1a1000
Relocated Initrd End (images->initrd_end): 0xd0000000
FDT Address (images->ft_addr): ce18e000
FDT Length (images->ft_len): 65097 bytes
Verification Result (images->verify): -1
======================================================
do_bootm_linux: Jumping to kernel (BOOTM_STATE_OS_GO or BOOTM_STATE_OS_FAKE_GO)
Kernel entry point set to: 0xc00000d0
Using default machid: 0x0
Transferring control to Linux at address: 0xc00000d0...
Boot stage marked: RUN_OS.

VBDBG Starting kernel ...

Using FDT at address: 0xce18e000
Jumping to kernel at address: 0xc00000d0 with machid: 0x0 and boot parameters: 0xce18e000
E/TC:0   Panic 'Watchdog' at core/drivers/stm32_iwdg.c:105 <stm32_iwdg_it_handler>

Is there anything which is obviously wrong here? The only things I changed so far is the sram label and the can clocks in the device tree. But should that cause the kernel to crash? 

Also did the below analysis: 

UBOOT CCB> mmc dev 1

switch to partitions #0, OK
mmc1(part 0) is current device

UBOOT CCB> ext4load mmc 1:2 0xc0000000 /boot/image.fit
38388882 bytes read in 931 ms (39.3 MiB/s)

UBOOT CCB> iminfo 0xc0000000

## Checking Image at c0000000 ...
   FIT image found
   FIT description: Boot Image for Indra CCB Devices
   Created:         2024-12-11   6:46:51 UTC
    Image 0 (kernel)
     Description:  kernel zImage
     Created:      2024-12-11   6:46:51 UTC
     Type:         Kernel Image (no loading done)
     Compression:  uncompressed
     Data Start:   0xc00000d0
     Data Size:    6475216 Bytes = 6.2 MiB
     Hash algo:    sha256
     Hash value:   8612853edb6e488df4b6de60a6420b1d0a21f6592d61e7034c962362334aad60
    Image 1 (fdt)
     Description:  stm32mp157c-ccb-mx.dtb
     Created:      2024-12-11   6:46:51 UTC
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0xc062cfa0
     Data Size:    65097 Bytes = 63.6 KiB
     Architecture: ARM
     Hash algo:    sha256
     Hash value:   58fb1c669170894c5fc136f310a59dc15ae086bdac94a3b7413fd59d55bafe87
    Image 2 (ramdisk)
     Description:  rootfs
     Created:      2024-12-11   6:46:51 UTC
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0xc063cea4
     Data Size:    31846400 Bytes = 30.4 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: unavailable
     Entry Point:  unavailable
     Hash algo:    sha256
     Hash value:   489103a7633d620a44381f8789bff9af669aa98c4d13ec76854ceaafcaf183a2
    Default Configuration: 'indra-ccb-kernel-dtb-ramdisk'
    Configuration 0 (indra-ccb-kernel-dtb-ramdisk)
     Description:  version 2052 - v0.8.4
release-I8 (2024-12-11T06:46:48 4f5852e1b6 dev)
     Kernel:       kernel
     Init Ramdisk: ramdisk
     FDT:          fdt
     Sign algo:    sha256,rsa2048:ccb
     Sign value:   0e283983f7ec81c82b40e569166d5556d5ea2a8c76f8331f9030d352e6ce1dcceb4fa2b4465b4f0843e7b2f32e0b4e786f38bd0127ad96229853bdc6766d16eea71775372256b61f4baa13d8498b3c9af97487dfcebc4d1cb84633f6fc425af4
     Timestamp:    2024-12-11   6:46:51 UTC
## Checking hash(es) for FIT Image at c0000000 ...
   Hash(es) for Image 0 (kernel): sha256+ 
   Hash(es) for Image 1 (fdt): sha256+ 
   Hash(es) for Image 2 (ramdisk): sha256+ 
 
Analysis of FIT Image Details
Kernel:

    Description: kernel zImage
    Start Address: 0xc00000d0
    Size: 6475216 Bytes (6.2 MiB)
    Hash: sha256 verified successfully (sha256+).

Device Tree Blob (DTB):

    Description: stm32mp157c-ccb-mx.dtb
    Start Address: 0xc062cfa0
    Size: 65097 Bytes (63.6 KiB)
    Hash: sha256 verified successfully (sha256+).

Initrd (RAMDisk):

    Description: rootfs
    Start Address: 0xc063cea4
    Size: 31846400 Bytes (30.4 MiB)
    Hash: sha256 verified successfully (sha256+).

Default Configuration:

    The configuration is indra-ccb-kernel-dtb-ramdisk, which includes all the necessary components.

Signatures:

    The sha256,rsa2048:ccb signature was verified successfully.

Memory Layout

From the addresses and sizes:

    Kernel: 0xc00000d0 to 0xc062cfa0 (approximately 6.2 MiB).
    DTB: 0xc062cfa0 to 0xc063cea4 (approximately 63.6 KiB).
    Initrd: 0xc063cea4 to 0xc249bea4 (approximately 30.4 MiB).

No Overlap Detected: The memory layout appears valid, with each component occupying its unique space.
Further Validation Steps

    Verify Kernel in Memory:

        Dump the first few bytes of the kernel to confirm it's loaded correctly:

md 0xc00000d0 10
c00000d0: e1a00000 e1a00000 e1a00000 e1a00000  ................
c00000e0: e1a00000 e1a00000 e1a00000 e1a00000  ................
c00000f0: ea000005 016f2818 00000000 0062cdd0  .....(o.......b.
c0000100: 04030201 45454545 000067f0 e10f9000  ....EEEE.g......

The output of md 0xc00000d0 10 confirms that the kernel is successfully loaded into memory at the expected address (0xc00000d0). Here's an interpretation of the key parts:
Memory Dump Analysis

    e1a00000 Repeated Entries:
        This is ARM NOP (no operation) instructions or padding often used at the start of the kernel for alignment.

    ea000005:
        This is an ARM branch instruction, likely jumping to the kernel's entry point.

    016f2818:
        This is the magic number for ARM zImage kernels. It confirms that the kernel is valid and in the expected format.

    0062cdd0:
        This is likely a reference to an internal address, possibly the kernel's decompression start address.

DTB validation
===========
md 0xc062cfa0 10
c062cfa0: edfe0dd0 49fe0000 38000000 30f20000  .......I...8...0
c062cfb0: 28000000 11000000 10000000 00000000  ...(............
c062cfc0: 190c0000 f8f10000 00000000 00000000  ................
c062cfd0: 00000000 00000000 01000000 00000000  ................

The memory dump of the DTB at 0xc062cfa0 confirms that the device tree blob (DTB) is loaded and valid. Here's the breakdown:
Memory Dump Analysis for DTB

    edfe0dd0:
        This is the magic number for a valid DTB. It indicates the file is in Flattened Device Tree (FDT) format.

    49fe0000:
        This is the total size of the DTB in bytes. The value in hexadecimal translates to 0x49FE, which is approximately 18.5 KB, aligning with the DTB's expected size.

    38000000 and 30f20000:
        These are internal offsets for the DTB structure, such as the offset to the structure block, strings block, and reserved memory.

Validate Initrd (RAMDisk)
md 0xc063cea4 10
c063cea4: 73717368 00000b3d 6759355b 00020000  hsqs=...[5Yg....
c063ceb4: 000000b2 00110004 000300c0 00000004  ................
c063cec4: 6680000f 00000000 01e5e039 00000000  ...f....9.......
c063ced4: 01e5e031 00000000 ffffffff ffffffff  1...............

The memory dump of the initrd (RAMDisk) at 0xc063cea4 confirms that the initrd is loaded correctly. Here's the breakdown:
Memory Dump Analysis for Initrd (RAMDisk)

    73717368:
        This is the magic number for a squashfs filesystem (hsqs in ASCII). It confirms that the initrd is a valid squashfs image.

    Other Data:
        The rest of the data (00000b3d, 6759355b, etc.) represents metadata for the squashfs filesystem, including size and structure information.

    No Empty or Invalid Data:
        The memory content contains valid-looking metadata, indicating the initrd is correctly loaded.

Memory Layout Check

Based on the addresses:

    Kernel: 0xc00000d0 to 0xc062cfa0
    DTB: 0xc062cfa0 to 0xc063cea4
    Initrd: 0xc063cea4 to 0xc249bea4

There are no overlaps, and the memory regions for the kernel, DTB, and initrd appear valid.

Any input is welcome. 

Many thanks in advance.

[UVV]

If I grasped the logs correctly you try to boot optee-3.16 and kernel-6.1. In my experience, you should match optee/kernel version with what comes in the yocto layer branch. Otherwise you might get SCMI compatibility issues between optee and the kernel and get unexpected failures.

The second point in your logs seems to be coming from the watchdog configuration in the optee device tree.

I hope this helps.

[VivekB]

Hiya, thanks for your reply. I was hoping I can pull in the kernel, uboot, optee and tf-A in that order one by one so that I don't have to change all of these components at one go, but seem like that is not the case?