DMA2 breaks SPI3 communication when performing under Linux context.

[JMein.1]

Hello,

we are working on the ST32MP1. The SPI3 works fine with the CubeMX software combining the DMA2 and FIFO when the board works in engineering mode. But if I perform the same software loaded by Linux (rproc start), the DMA2 SPI stream stops working, if I use the function HAL_SPI_Transmit_DMA() instead of using HAL_SPI_TransmitReceive_DMA(). If for example the amount of bytes to be tranferred is larger than 33, the DMA stops with this number of bytes in NDTR DMA register.

In engineering mode, this works fine with the same setting. I tested a lot (linker, stack, memory, FIFO on/off...) without any idea, why on M4 only mode, all is working fine. Furthermore, if I take 30 bytes, the first amount of data is transmitted by DMA and then the SPI_ISR is inoved around 15 times to send the rest of the data ... With 40 bytes, the DMA IRQ comes ones but after around 15 bytes and the SPI IRQ is not invoked.

Somehow curious but actually, I have no idea what is going wrong using tranmit only in production mode.

Any ideas?

Best regards,

Jan-Otto

[Kevin HUBER]

Hello @JMein.1​ ,

As you said in engineering mode, only the M4 is running so it can use all the resources without encountering problems, but in production mode the linux is also running and also uses some peripherals which might lead to conflict with the M4.

The peripheral assignment is done in the DT files.

This is explained in this wiki page and particularly in the chapter about TF-A and the Linux Kernel: https://wiki.st.com/stm32mpu/wiki/How_to_assign_an_internal_peripheral_to_a_runtime_context#Manual_assignment

If you think you already correctly assigned all your peripherals and still have issues with DMA2 or SPI3,

please tell me on which board are you working and send me:

• the DT of your Linux
• the DT of your TF-A
• and the kernel boot log.

Regards,

Kévin

[JMein.1]

Hello Kevin,

sorry for the late reply. After double checking the IOC and the device tree, I figured out, that there was a wrong assignment in the device tree. Actually, wenn I now start the M4 firmware under Linux running, all works well. Also, if I start from the bootloader (Stop in den bootloader and then load the M4 firmware manually), all works fine.

But when starting from the TFA/Second stage, the ressource actually seems to be reassigned by Linux and the communication of the SPI stops again, although the device tree is now correct.

Is it like this, that the Linux reinitializes something with the M4 firmware, when it is started before the Linux kernel?

Thank you very much.

Best regards,

Jan-Otto

[JMein.1]

Hello Kevin,

after some throwbacks and investigation on the source code and the device tree, everything seems now to be ok (from review point of view).

But the issue with the DMA usage is still present if I boot the M4 software from the second stage bootloader (UBoot). All runs fine (checked by DMA output of the USART on M4 side) but the USART output by DMA stops immediately, when the kernel is starting.

It seems to me, that the kernel takes all of the DMA. Can you have a look into the device tree or any other hint how to prevent the kernel taking the DMA resource (DMA2) of the M4?

I am working on a custom board.

Best regards,

Jan-Otto

[Kevin HUBER]

Hello @JMein.1​ ,

Yes I can have a look into your device tree, but for that you must send them ;).

Please can you send me:

• the Device Tree (DT) of your Linux
• the DT of your TF-A
• and the kernel boot log.

Regards,

Kevin

[Kevin HUBER]

Hi @JMein.1​  ,

We discussed a lot in Private Message, but I think it's better to return to the main post. The goal is to help future users and/or have help from other people of the community.

So I start with my reply to your latest message and I summarize what we found so far, after the Current Status.

Current status

Good to know that the temporary workaround works.

Since I can't understand why your M4 is reset by Linux, I asked an expert of the linux boot and I'm waiting its explanation.

While waiting for his answer, did you tried to add st,auto-boot to your rproc node?

https://wiki.st.com/stm32mpu/wiki/Linux_remoteproc_framework_overview#Automatic_attach_on_Linux_boot

It usually not recommended to use auto-boot, but the text following the DT is interesting:

This mode is recommended to start the resource manager framework during the Linux boot. It prevents a system resource, used by the Cortex-M4 only, from being disabled by Linux at the end of its boot phase.

Otherwise you can simply wait for the response of the expert and I write you if I find something.

Regarding your tentative to get dynamically the resource by using ResMgr_Request, I think this is not relevant because the resource is already allocated to your M4 thanks to the DTS.

Summarize for the community:

I found something weird inside the logs. I do not know why but the M4 firmware seems to be reset by the boot of linux. (One log is attached to this message)

We can see that the M4 firmware is loaded by U-boot with Success:

Load Remote Processor 0 with data@addr=0xc2000000 3348688 bytes: Success!

Then at the beginning of the linux boot, you can see:

[    1.988577] remoteproc remoteproc0: releasing m4
[    1.990527] NET: Registered protocol family 17
[    1.990967] Key type dns_resolver registered
[    1.991098] ThumbEE CPU extension supported.
[    1.991590] registered taskstats version 1
[    1.991626] Loading compiled-in X.509 certificates
[    2.028805] stm32-mdma 58000000.dma-controller: STM32 MDMA driver registered
[    2.034552] stm32-dma 48000000.dma-controller: STM32 DMA driver registered
[    2.044687] remoteproc remoteproc0: releasing m4

The two lines with remoteproc seem to have stopped the m4 firmware.

Then the firmware is loaded again by remoteproc in Linux:

[    3.448134] remoteproc remoteproc0: m4 is available
[    3.458925] remoteproc remoteproc0: attaching to m4
[    3.478827] random: fast init done
[    3.484130] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:spi@4000c000 (ops rproc_srm_dev_ops)
[    3.495034] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:serial@4000e000 (ops rproc_srm_dev_ops)
[    3.511547] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:serial@4000f000 (ops rproc_srm_dev_ops)
[    3.522536] mmc1: new high speed SDXC card at address 0001
[    3.530534] mmcblk1: mmc1:0001 ED4QT 119 GiB
[    3.533681] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:i2c@40012000 (ops rproc_srm_dev_ops)
[    3.545673] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:i2c@40013000 (ops rproc_srm_dev_ops)
[    3.557706] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:i2c@40014000 (ops rproc_srm_dev_ops)
[    3.569781] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:i2c@40015000 (ops rproc_srm_dev_ops)
[    3.581861] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:spi@44005000 (ops rproc_srm_dev_ops)
[    3.593985] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:spi@44009000 (ops rproc_srm_dev_ops)
[    3.606037] rproc-srm-core 10000000.m4:m4_system_resources: bound 10000000.m4:m4_system_resources:dma@48001000 (ops rproc_srm_dev_ops)
[    3.618323]  remoteproc0#vdev0buffer: assigned reserved memory node vdev0buffer@10044000
[    3.628013] virtio_rpmsg_bus virtio0: rpmsg host is online
[    3.632462]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[    3.638284] remoteproc remoteproc0: remote processor m4 is now attached

 You have that in all you traces, which is weird, normally if the M4 is loaded from U-boot, the Linux must not stop and restart it.

This trace is wrote by drivers/remoteproc/remoteproc_core.c

Finally, when you try to start it from linux:

echo start > /sys/class/remoteproc/remoteproc0/state

Nothing happens which means that the M4 firmware what still running.

For the moment I do not know why your M4 firmware is reset by the linux, but once it happens, it seems to break your dma2/spi3/usart2.

 As a temporary fix, I proposed to force the reset of the spi3 and the Dma2 before initializing them again because some reinitialization can fails seems the two interfaces were already initialized.

Please can you edit the main.c of your M4 project in CM4/Core/Src/main.c

and add inside:

  /* USER CODE BEGIN SysInit */
 
  /* USER CODE END SysInit */

The lines

  hspi3.Instance = SPI3;
  HAL_SPI_DeInit(&hspi3);
  HAL_SPI_MspDeInit(&hspi3);
  HAL_NVIC_DisableIRQ(SPI3_IRQn);
  __HAL_RCC_SPI3_FORCE_RESET();
  __HAL_RCC_SPI3_RELEASE_RESET();
  HAL_NVIC_DisableIRQ(DMA2_Stream0_IRQn);
  __HAL_RCC_DMA2_FORCE_RESET();
  __HAL_RCC_DMA2_RELEASE_RESET();

And you said that yes it improves the behavior, even if some issues remains:

the sample code you provided works in the following case:
 
the M4 software runs in the case on consecutive start/stop commands under Linux (Linux booted completely)
- DMA2 for USART2 works
- DMA2 for SPI3 works
 
 
For autostart (boot by UBoot), this fix comes too late because the transmission under DMA context is running during the Linux kernel boot. I can then detect the DMA error and start the recovering. But this will interrupt the actual M4 application and so this is only a workaround to get DMA running again.
 
 
 
One interesting part I observed: If I wait during boot (boot by UBoot) for the ttyRPMSG channel to arrive at the M4 (nearly 5 seconds) or if I delay the start by 5 seconds, the initialization of the M4 DMA2 is done later so it seems, that the last one catching the DMA2 (A7 or M4) will get and hold the DMA2 access.

Regards,

Kevin

[JMein.1]

Hello Kevin,

thank you for your reply and the hint with the autostart set to '1'. This is already in our script to guess, this should be ok.

After some further investigations on the issue, I plotted out the registers at startup (M4 runs correctly) and after Linux boot and first error.

I observed, in case of an error, the DMA mux channel is overwritten by Linux so the communication breaks.

DMAMUX_C8CR was set to 0x0000 002C (and the amount of data to be written stucks @ DMA_S0NDTR = 0x0000 00034) for USART2 and after Linux boot to 0x0000 0000. I guess, this is the main problem also for the SPI3. In Cube MX all is well configured but I am not quite sure, if in the device tree, this is fine as well. Remember: DMA1 is assigned to A7 and DMA2 is assigned to M4.

Is the following part for this configuration correct? And why overwrites Linux on start the before initialized ID 0x0000 002C for the USART2 to and how to prevent this?

&dmamux1{

dma-masters = <&dma1>;

dma-channels = <8>;

status = "okay";

/* USER CODE BEGIN dmamux1 */

/* USER CODE END dmamux1 */

};

Hope, you answer on this well me get rid of the problem 😊 because for me it seems to be the root cause.

Best regards,

Jan-Otto

[JMein.1]

Hello Kevin,

I have a further update on our issue and a potentially fix/workaround on this. But I would like to discuss it here.

After we found out, that the Linux resets on startup of the kernel the DMAMUX1 and lost therefore the already initialized values of the M4, we commented out the following lines in the Linux driver:

File: stm32_dmamux.c

Function: stm32_dmamux_probe

Lines: 273 to 282

rst = devm_reset_control_get(&pdev->dev, NULL);

if (IS_ERR(rst)) {

ret = PTR_ERR(rst);

if (ret == -EPROBE_DEFER)

goto err_clk;

} else {

reset_control_assert(rst);

udelay(2);

reset_control_deassert(rst);

}

This lines lead to the reset of the DMAMUX1. In out setup (M4 booted by UBoot), the break of the SPI3/USART2/DMA2 transfers do not happen anymore.

In your opinion, is this a way to go or do we miss something (constraints)?

Looking forward to your answer.

Best regards,

Jan-Otto

[Kevin HUBER]

Hello @JMein.1​ ,

Good catch! Since it's a shared device, it seems not logical to reset the dmamux in the linux driver.

Thanks for your investigation.

In my opinion you can keep this temporary fix to correct your problem.

I will ask a DMA expert and give him the new information that you provided. I get back to you as soon as I know more.

And if a constraint exist.

Best Regards,

Kévin

[Kevin HUBER · Accepted solution]

Hi @JMein.1​ ,

We have a reply from a DMA expert which explained that it is possible to not reset the device if the two DMAs are not used by the A7 core.

Below a part of his reply:

"We should avoid to reset dmamux when dma2 is used by M4 & resets property is in the device tree.

Note, later in the probe, CxCR register is cleared depending on dma_requests:

/* Reset the dmamux */

for (i = 0; i < stm32_dmamux->dma_requests; i++)

stm32_dmamux_write(stm32_dmamux->iomem, STM32_DMAMUX_CCR(i), 0);

dma requests is the sum of the dma-requests properties found in dma-masters nodes, so dma1 and dma2. If dma2 is disabled for M4 use, then CxCR with x = [8:15] is not cleared."

The proposal patch is attached to this message.

This patch disables the reset if only one or less than one dma are specified in dma-masters:

- } else {
 
+ } else if (count > 1) { /* Don't reset if there is only one dma-master */

 Thanks for your help on this bug. It will be fixed in a next release.

Regards,

Kevin