Kernel panic with poll/select syscalls on virtual UART devices to M4 core when M4 watchdog restarts

[MMaje.1]

I am using 2 virtual UARTs as described in STM32MP1Cube Applications/OpenAMP/OpenAMP_TTY_echo example and watchdog for M4 core.

Problem occurs when I using select or poll syscall to read from the device. Kernel panic occurs with following on dmesg log:

[  930.513060] remoteproc remoteproc0: crash detected in m4: type watchdog
[  930.518272] remoteproc remoteproc0: handling crash #5 in m4
[  930.523785] remoteproc remoteproc0: recovering m4
[  930.531138] rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: rpmsg tty device 0 is removed
[  930.542605] Unable to handle kernel NULL pointer dereference at virtual address 00000018
[  930.552002] rpmsg_tty virtio0.rpmsg-tty-channel.-1.1: rpmsg tty device 1 is removed
[  930.558402] pgd = a2d2820b
[  930.560889] [00000018] *pgd=d1832835, *pte=00000000, *ppte=00000000
[  930.570663] Unable to handle kernel NULL pointer dereference at virtual address 00000018
[  930.578406] rpmsg_tty virtio0.rpmsg-tty-channel.-1.2: rpmsg tty device 2 is removed
[  930.586506] pgd = a2d2820b
[  930.587747] [00000018] *pgd=d1832835, *pte=00000000, *ppte=00000000
[  930.594180] Internal error: Oops: 17 [#1] PREEMPT ARM
[  930.603662] CPU: 0 PID: 938 Comm: /dev/ttyRPMSG0 Tainted: G        W         4.19.94 #1
[  930.611650] Hardware name: STM32 (Device Tree Support)
[  930.616795] PC is at virtio_get_buffer_size+0x8/0x14
[  930.621740] LR is at n_tty_poll+0x1c4/0x1f4
[  930.625908] pc : [<c069090c>]    lr : [<c04cf548>]    psr: a00f0013
[  930.632167] sp : d3585aa8  ip : 00005401  fp : 00000000
[  930.637381] r10: 0000000d  r9 : 00000000  r8 : d3585b58
[  930.642598] r7 : d3585b58  r6 : d2324a80  r5 : 00000000  r4 : d22b9e00
[  930.649117] r3 : 00000000  r2 : e0a35000  r1 : 00000001  r0 : d22bad00
[  930.655639] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  930.662766] Control: 10c53c7d  Table: d1dcc059  DAC: 00000051
[  930.668505] Process /dev/ttyRPMSG0 (pid: 938, stack limit = 0xc868b704)
[  930.675110] Stack: (0xd3585aa8 to 0xd3586000)
[  930.679463] 5aa0:                   c04cf384 d2324a80 d22b9e00 d1b40600 d3585b58 c04cbd3c
[  930.687634] 5ac0: 00001000 00000001 00001000 00000020 0000000c c029abe0 00080006 00000000
[  930.695805] 5ae0: d2324a80 00001000 00000000 00000000 00001000 d3585e20 d3585e18 d3585e1c
[  930.703975] 5b00: d3585e10 d3585e14 d3585e18 00000000 00000000 00000000 00000000 00000000
[  930.712147] 5b20: 00000001 00000001 d3585df4 d3585f58 d3585b50 d3584000 00000000 00000001
[  930.720319] 5b40: 0000c350 c0c05104 c0c04088 3b9aca00 aab93712 000000d8 00000000 000000db
[  930.728490] 5b60: 00000000 d1828b00 00000000 00000000 00000002 d2324a80 000000db 00000000
[  930.736661] 5b80: d3585b58 c029a2e4 d22b9f48 d22b9f48 d22b9f48 d2324a80 000000db 00000000
[  930.744834] 5ba0: d3585b58 c029a2e4 d22b9f40 d22b9f40 d22b9f40 ae64af51 d3585c50 000010c2
[  930.753006] 5bc0: 00000000 d3db0b24 600d0113 d31b7300 d18c1e40 c027099c d26fb400 00000001
[  930.761179] 5be0: 0015000a 00000001 ae64af51 d18c1540 000010c2 c072d0c0 d3585c50 374242ad
[  930.769351] 5c00: 00000004 00000004 0015000a 00000000 ae64af51 c0c167a0 fffffa58 d31b7300
[  930.777524] 5c20: d18c1540 c072ea90 00010f91 d3db0b24 ffffe000 373f10a7 00000000 c0270da8
[  930.785696] 5c40: 00000001 c072ea90 d18c1540 d26fb400 d18c1540 c0c04088 d19dcda4 ce590c86
[  930.793867] 5c60: 000000d7 c072ea90 00000338 0000024a 00000000 00000001 00000004 00000000
[  930.802039] 5c80: d19dd4e4 4918cc2c 000000d7 d26fb400 d18c1540 c0c04088 d19dcda4 ce590c86
[  930.810209] 5ca0: 000000d7 373f10a7 d26fb400 c072f4a8 00000020 d26fb850 d18c1540 4918cc2c
[  930.818382] 5cc0: 30c97bea d18c1540 d26fb400 d32d7780 00000000 d19dcd90 c0c5e198 c0c04088
[  930.826553] 5ce0: d18c1540 c0739268 d18c1540 d3afa490 d31b7300 00480020 d30004c0 c026f78c
[  930.834725] 5d00: c06bbbfc c0a0ce04 0015000a 80150015 c0cbc804 00000002 d3086000 d3087480
[  930.842897] 5d20: c0c96168 d1828b00 c0c10058 600d0193 c0c10058 c0153230 d1828b00 c0c10058
[  930.851068] 5d40: d3585d54 c01534dc d32f5600 c0c04088 d3585d64 c01534f4 d32f5600 c0c04088
[  930.859240] 5d60: d3585d8c c01550c4 00000000 4918cc2c d2585e68 00000001 d2585e74 00000000
[  930.867411] 5d80: c015534c 00000003 00000000 c015e66c 00000000 00000000 c0c775b8 d2585e70
[  930.875584] 5da0: a00d0193 c0c85a10 c0a61594 00000028 c0c85ad0 4918cc2c c0c775b8 00000004
[  930.883755] 5dc0: d3585e0c 0000000d c0c04088 b60fe7b8 00000000 d3585e0c 00000000 c029b850
[  930.891927] 5de0: d357e7c0 d3585f58 00000051 d3585ec8 d3585e8c d3585e0c d3585e10 d3585e14
[  930.900099] 5e00: d3585e18 d3585e1c d3585e20 00001000 00000000 00000000 00000000 00000000
[  930.908271] 5e20: 00000000 00000001 d22b9e58 e0a37258 ffffe000 c0c04088 00000000 d22b9f48
[  930.916443] 5e40: 00000027 e0a37000 d31d7b00 00000001 00000001 d3585ec8 d3585f44 c016a774
[  930.924614] 5e60: d31d7b00 c0c04790 c0c7112c 00000000 00000001 00000100 148414b8 00000000
[  930.932786] 5e80: 14844004 00000000 00000005 00ffffff 00000005 0d96d7ce 000003a2 00000000
[  930.940959] 5ea0: a120b1bb ffffffff 0d96d7ce 00000000 c0c04088 d3585f58 d3585ef8 0000000d
[  930.949131] 5ec0: b60fe7b8 00000000 00000000 c029b604 000003a2 00000000 221b17d2 d3585f1c
[  930.957304] 5ee0: 00000000 00000000 004c4b40 c0184220 00000000 00000000 000003a2 00000000
[  930.965476] 5f00: 22676312 d3585f1c 00000000 4918cc2c 004c4b40 00000008 c0c04088 00000000
[  930.973648] 5f20: 0000014f 0000000d b60fe7b8 00000000 00000000 c029be84 d3585f58 ffffe000
[  930.981819] 5f40: d3585f58 00000000 00000000 00000000 004c4b40 c0c04088 000003a2 00000000
[  930.989992] 5f60: 22676312 d3585f1c b60fe8e0 00000100 d2324a80 c0285b18 00000000 4918cc2c
[  930.998164] 5f80: d3584000 b60fe760 b60fe768 00000000 0000014f c0101204 d3584000 0000014f
[  931.006336] 5fa0: bedb8108 c0101000 b60fe760 b60fe768 0000000d b60fe7b8 00000000 00000000
[  931.014509] 5fc0: b60fe760 b60fe768 00000000 0000014f 0000000d 00000000 00000000 bedb8108
[  931.022681] 5fe0: 0000014f b60fe758 b6b356e5 b6abb6c6 800d0030 0000000d 00000000 00000000
[  931.030867] [<c069090c>] (virtio_get_buffer_size) from [<c04cf548>] (n_tty_poll+0x1c4/0x1f4)
[  931.039295] [<c04cf548>] (n_tty_poll) from [<c04cbd3c>] (tty_poll+0x6c/0x88)
[  931.046338] [<c04cbd3c>] (tty_poll) from [<c029abe0>] (do_select+0x338/0x6b0)
[  931.053464] [<c029abe0>] (do_select) from [<c029b850>] (core_sys_select+0x238/0x464)
[  931.061200] [<c029b850>] (core_sys_select) from [<c029be84>] (sys_pselect6+0x2c0/0x2e8)
[  931.069197] [<c029be84>] (sys_pselect6) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
[  931.077011] Exception stack(0xd3585fa8 to 0xd3585ff0)
[  931.082059] 5fa0:                   b60fe760 b60fe768 0000000d b60fe7b8 00000000 00000000
[  931.090233] 5fc0: b60fe760 b60fe768 00000000 0000014f 0000000d 00000000 00000000 bedb8108
[  931.098403] 5fe0: 0000014f b60fe758 b6b356e5 b6abb6c6
[  931.103452] Code: e8bd4070 eaf91554 e5903000 e59331b8 (e5930018) 
[  931.118477] Internal error: Oops: 17 [#2] PREEMPT ARM
[  931.633915] remoteproc remoteproc0: warning: remote FW shutdown without ack
[  931.652545] remoteproc remoteproc0: stopped remote processor m4
[  931.740546] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:timer@40002000 (ops 0xc0964a34)
[  931.765042] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:dac@40017000 (ops 0xc0964a34)
[  931.785526] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:serial@40018000 (ops 0xc0964a34)
[  931.794608] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:spi@44005000 (ops 0xc0964a34)
[  931.807666] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:adc@48003000 (ops 0xc0964a34)
[  931.816905] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:hash@4c002000 (ops 0xc0964a34)
[  931.834081] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:rng@4c003000 (ops 0xc0964a34)
[  931.844411] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:crc@4c004000 (ops 0xc0964a34)
[  931.853738] rproc-srm-core m4@0:m4_system_resources: bound m4@0:m4_system_resources:cryp@4c005000 (ops 0xc0964a34)
[  931.873706]  m4@0#vdev0buffer: assigned reserved memory node vdev0buffer@10044000
[  931.885776] virtio_rpmsg_bus virtio0: rpmsg host is online
[  931.896159]  m4@0#vdev0buffer: registered virtio0 (type 7)
[  931.900191] remoteproc remoteproc0: remote processor m4 is now up
[  931.916027] ---[ end trace 5aeed22fa411946f ]---
[  931.919330] virtio_rpmsg_bus virtio0: creating channel rpmsg-tty-channel addr 0x0
[  931.934455] Kernel panic - not syncing: Fatal exception

When using read syscall, M4 core restarts without causing kernel panic. Attached files are dmesg logs with read (without panic) and select (with panic) variants and standalone C programs used on Linux side to read from the devices.

[Olivier GALLIEN]

Hi @MMaje.1​ 

We are investigating this issue.

Could you please confirm software version you are using ? ( assume openstlinux-20-02-19 along V1.2 ecosystem )

Thanks

Olivier

[JeanPhilippeR]

Hi @MMaje.1​ 

I compile both read and poller binaries and test them on my target, by using the openAMP_TTY_Echo FW on M4 side.

Run is ok.

I would like to ask you to share your M4 firmware which also enable the M4 watchdog in order to reproduce the issue as the analysis of kernel dmesg is not sufficient?

Thank you

Best regards

Jean-Philippe

[MMaje.1]

@Community member​  @JeanPhilippeR​ ​ I reproduced the behavior on stock STM32MP157C-DK2 kit with Starter Package image v1.2.0 today.

I'm sending s STM32CubeIDE project with watchdog enabled that I can use to reproduce this.

I can also see some crash happening when using cat program on Linux to read from tty device

$ stty -i icrnl -F /dev/ttyRPMSG0
$ cat /dev/ttyRPMSG0 &
$ echo "test" > /dev/ttyRPMSG0
test
$ echo "watchdog" > /dev/ttyRPMSG0
[  105.516938] remoteproc remoteproc0: crash detected in m4: type watchdog
...
[  105.785312] [<c013eaa0>] (get_work_pool) from [<c013f3a8>] (__flush_work+0x44/0x224)
[  105.793043] [<c013f3a8>] (__flush_work) from [<c0584768>] (n_tty_read+0x284/0x8d0)
[  105.800612] [<c0584768>] (n_tty_read) from [<c057f208>] (tty_read+0x90/0xe4)
[  105.807662] [<c057f208>] (tty_read) from [<c02a0fec>] (__vfs_read+0x2c/0x15c)
[  105.814794] [<c02a0fec>] (__vfs_read) from [<c02a11ac>] (vfs_read+0x90/0x15c)
[  105.821928] [<c02a11ac>] (vfs_read) from [<c02a1798>] (ksys_read+0x5c/0xdc)
[  105.828888] [<c02a1798>] (ksys_read) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
[  105.836448] Exception stack(0xccbc7fa8 to 0xccbc7ff0)
[  105.841500] 7fa0:                   0000006c 7ff00000 00000003 b6e9c000 00020000 00000000
[  105.849681] 7fc0: 0000006c 7ff00000 00020000 00000003 00000003 00000000 00020000 00000000
[  105.857856] 7fe0: 00000003 becfd9a0 b6f477b7 b6ed46c6
[  105.862909] Code: 012fff1e e59f000c ea238613 e3c110ff (e5910000) 
[  105.868997] ---[ end trace 5c5cd1536719a678 ]---

[JeanPhilippeR]

Hi @MMaje.1​ 

Thank you for detailed information.

I'm able to reproduce the same with your Virtual watchdog project which force the watchdog to occur.

Based on this setup we will investigate the A7 behavior to avoid for crashing.

I will let you know.

Best regards

Jean-Philippe

[JeanPhilippeR · Accepted solution]

Hi @MMaje.1​ 

The issue has been analysed and fix.

When watchdog occurs on M4 side, the tty device removal and the file close on Linux side are asynchronous, so need to manage this cross case.

Please fins patch to be applied on Linux kernel in attachment.

Best regards

Jean-Philippe

[mishuk]

Hi @JeanPhilippeR​ !

Are you going to send your patch upstream to main Linux kernel developers ?

Regards,

Mikhail

[JeanPhilippeR]

Hi @mishuk​ 

Currently we are working to upstream rpmsg_tty.c to Linux kernel, and this is not finalized, as many discussion with maintainers.

We expect to be completed by end of the year, and for sure it will include this patch.

Best regards

Jean-Philippe