Kernel bootup failure with Kirkstone UBOOT and Mickledore Kernel.

[umarFarooq-emb]

We have changed the Yocto ecosystem of our builds from Kirkstone to mickledore.

Everything works smoothly if all the components from the boot chain are built with the same ecosystem (either Kirkstone or Mickledore). Which means is, if TF-A, U-BOOT, Kernel, and rootfs all are either Kirkstone based or Mickledore based in the the image, the system boots fine.

We have a different test base scenario, the deployed hardwares in the field are currently operational with kirkstone releases, we intend to update the components individually in the hardwares to avoid the software breaks and to enhance the rollback feature in case of update failure.

So, when we update either U-BOOT (kirstone to mickledore) and the kernel is of kirkstone based, the system doesn't boot and watch dog timer keeps resetting the system.
When we keep the U-BOOT at kirkstone version and update the kernel (kirkstone to mickledore), the system doesn't boot and the watchdog timer keeps resetting the system.

This boot fails after the "Starting Kernel ..." logs and nothing appears after these logs.

What we have tested until now is the different Kernel load address that are mentioned as UBOOT_ENTRYPOINT in the yocto variables. In the kirkstone image this address is set to 0xc0800000 while in mickledore it is set to 0xc4000000 . We believed that this conflict in different load addresses in kernel headers might be causing the issue.

We changed the kernel load address to 0xc0800000 in the Mickledore image and tried loading it with the kirkstone U-BOOT that tries loading the kernel from the same address (0xc0800000 ). Unfortunately, this attempt also failed.

Can you please discuss this scenario with your team? and provide us some help on how we can update our ecosystem from kirkstone to mickledore?

Many thanks in advance.

Kind Regards,

Umer

[Christophe Guibout]

Hello @umarFarooq-emb,

U-boot is part of the FIP image, which also includes Op-tee firmwares. The Linux kernel has a strong dependency with secure services provided by Op-tee, so if they are not aligned, it could lead to a crash.

I recommand to keep FIP and Linux images always aligned to avoid this kind of problem.

To update devices on the field, the OpenSTLinux ecosystem supports the A/B mechanism : more information in this wiki page.

BR,

Christophe

[umarFarooq-emb]

Hello Christophe,

Thank you for replying. We already have RAUC based update mechanism implied in our systems, we have no problem updating the system, we just don't want to update all the components in one go. This compromises the roll back feature of the components.

Also, I am now trying that mickledore UBOOT, boots both the kernels (Kirkstone and UBOOT). Can you please test it internally? If this is a feasible solution?

Best Regards,
Umer

[umarFarooq-emb]

Hello Cristophe,

I have attached an image with this comment, that refers to the booting sequence in both the OPENSTLINUX versions.

Our wic image has two dedicated bootfs slots and AB scheme for updating the system is implemented. Rauc is being used to update the bootfs slots. 

The problem is, we want to update one component at a time, like first of all the rootfs shall be updated and the previous kernel (kirkstone) should be able to load it, and it does,

Then the kernel is updated but the uboot from the kirkstone doesn't load the kernel. The kernel shall be rolled back to the other slot and then we cannot update the UBOOT to the mickledore version because it won't be able to boot the kirkstone kernel.

What I have debugged so far is that the mickledore kernel is expecting the clocks from the OPTEE, like the SCMI clocks, but the SP_MIN (in kirkstone) is not able to provide that and the kernel is not able to load.

If the uboot is from the mickledore build, the clocks and the power are handled by the OPTEE as optee takes control of the system resorces as also mentioned in the stm32mp15-scmi.dtsi file. The kirkstone kernel has no routine to call the scmi and get access of the clocks and power. 

If my findings hold true?

This topic carries a high priority on our side, if you could offer any help in this issue?

Thanks in advance.

BR,
Umer