Debug Authentication stuck at "SDM Library version v1.3.0" in CLOSED state

[Dhaval_Shiroya]

Hello,

I am following the blog "Implementing Debug Authentication on STM32H5 using CLI tools" (https://community.st.com/t5/stm32-mcus/implementing-debug-authentication-on-stm32h5-using-cli-tools/ta-p/622308) to provision my STM32H533RE based custom board on which i have configured secure word only(All the peripherals configured as secure) and i have disabled Non Secure Project from "Project Manage" tab of STM32CubeMX, and I am now stuck with the device in CLOSED state with no way to regress.

Environment:

• Board: NUCLEO-H533RE (STM32H533RET6)
• STM32CubeProgrammer: v2.22.0
• STM32CubeH5 Firmware: v1.6.0
• ST-LINK FW: V2J47S7
• SFSP Version: v1.2.0
• OS: Windows

What I did (step by step):

Step 1 — Flashed secure firmware (succeeded with mode=UR):

STM32_Programmer_CLI.exe -c port=SWD mode=UR -e all -d C:\H5\Binary\Sec_FP_Sensor_v1_2_Secure.elf -v

Result: :white_heavy_check_mark: Download verified successfully

Step 2 — Set product state to Provisioning (succeeded):

STM32_Programmer_CLI.exe -c port=SWD mode=HOTPLUG -ob PRODUCT_STATE=0x17

Result: :white_heavy_check_mark: Option Bytes successfully programmed

Step 3 — Provisioned DA OBK with password (succeeded):

STM32_Programmer_CLI.exe -c port=SWD mode=HOTPLUG -sdp DA_ConfigWithPassword.obk

Result: :white_heavy_check_mark: OBKey Provisioned successfully

Note: I initially had a RSSLIB_SETOBKEY_BAD_SIZE error because I was using an 80-byte OBK file (from wrong H5 variant). The correct OBK for STM32H533 from DA_ConfigWithPassword.obk is 108 bytes and worked correctly.

Step 4 — Set product state to CLOSED:

STM32_Programmer_CLI.exe -c port=SWD mode=HOTPLUG -ob PRODUCT_STATE=0x72

Result: Option bytes programmed but then reported Error: failed to reconnect after reset — I understand this is expected as debug port is now locked.

Step 5 — Attempted debug authentication discovery (as per blog):

STM32_Programmer_CLI.exe -c port=SWD mode=HOTPLUG debugauth=2

 Result: :cross_mark: Hangs indefinitely at:

Start Debug Authentication Sequence

Open SDM Lib
SDMOpen                       :   723 : open       : SDM API v1.0
SDMOpen                       :   724 : open       : SDM Library version v1.3.0
-

No further output. Had to Ctrl+C to cancel.

After Ctrl+C, subsequent attempts show:

open_comms                    :   578 : open       : Unable to detect DBGMCU Mailbox

Error:
Debug Authentication: Discovery Failed

What I have tried to recover:

1. Power cycling the board — no change

2. BOOT0 = HIGH + power cycle, then retry debugauth=2 — still stuck at same point

3. Different SWD modes and speeds:

STM32_Programmer_CLI.exe -c port=SWD mode=UR debugauth=2
STM32_Programmer_CLI.exe -c port=SWD speed=100 mode=HOTPLUG debugauth=2
STM32_Programmer_CLI.exe -c port=SWD ap=0 debugauth=2
STM32_Programmer_CLI.exe -c port=SWD ap=2 mode=HOTPLUG debugauth=2

 All stuck at the same SDM Library point.

4. Regression with password, key, and certificate (from the NUCLEO-H533RE ROT_Provisioning scripts):

STM32_Programmer_CLI.exe -c port=SWD per=a key=Keys\key_3_leaf.pem cert=Certificates\cert_leaf_chain.b64 pwd=Binary\password.bin debugauth=1

1. Running regression.bat from STM32Cube_FW_H5_V1.6.0\Projects\NUCLEO-H533RE\ROT_Provisioning\DA\ — script hangs (stuck at the same debugauth step internally).

2. UART connection with BOOT0 HIGH — virtual COM port not responding.

DA Configuration used:

The DA_ConfigWithPassword.xml from the firmware package:

<?xml version="1.0" encoding="UTF-8"?>
<Root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="SCHVerif.xsd">
<Obdata>
<Info>
<Title>DA Config</Title>
<ObDestAddress>0x0ffd0100</ObDestAddress>
<DoEncryption>1</DoEncryption>
<GlobalAlign>16</GlobalAlign>
<MCU>STM32H533/523</MCU>
</Info>
<Hash></Hash>
<DA></DA>
<Text>
<Name>Password</Name>
<Value>0123456789012345</Value>
<MinWidth>4</MinWidth>
<Width>16</Width>
<Default>0000</Default>
</Text>
<Data>
<Name>Reserved</Name>
<Value>0x00000000</Value>
<Width>4</Width>
<Reserved>1</Reserved>
<Default>0x00000000</Default>
</Data>
<Output>
<Name>Output File</Name>
<Value>../Binary/DA_ConfigWithPassword.obk</Value>
</Output>
</Obdata>
</Root>

 Password used: 0123456789012345 (default from ST example)

My Questions:

1. Why is the debug authentication stuck at SDM Library version v1.3.0 and unable to detect the DBGMCU Mailbox?

2. My secure firmware (Sec_FP_Sensor_v1_2_Secure.elf) is a custom application. Does the application firmware need to explicitly call HAL_RSSLIB_Debug_Auth() or similar RSS handler for the DA mailbox to work in CLOSED state? If yes, shouldn't the system bootloader (SFSP v1.2.0) handle DA when booting with BOOT0=HIGH?

3. Is there any other way to regress the device back to OPEN state from CLOSED when the DA mailbox is unresponsive?

4. Is this a known issue with STM32CubeProgrammer v2.22.0 or the SDM Library v1.3.0?

Any help would be greatly appreciated. The device appears to be bricked from a debug perspective.

Thank you.

[Jocelyn RICARD]

Helllo @Dhaval_Shiroya ,

NUCLEO-H533RE is embedding STLink V3, here you say you use V2J47S7.

If you used external connection to connect a STLink V2, you have to connect the reset pin.

Now, when this is not the case the behaviour is usually not this one, just says that it cannot communicate with DA.

One generic advice: when testing DA the first time, just try to perform a discovery in open state, and then a regression when you are in provisioning state (after provisioning the DA).

This allow you to check everything is ok before closing the device.

Best regards

Jocelyn

[Dhaval_Shiroya]

Yes, I am using an external ST-Link V2 debugger to connect to our custom hardware.
Do I need to connect the RST pin to the NRST pin on the board?

[Jocelyn RICARD]

Hi @Dhaval_Shiroya ,

yes this is mandatory to have DA feature working.

Best regards

Jocelyn

[Dhaval_Shiroya]

Thank you for spending your time.

I also tried using the RST pin, but the result was the same.