Is it possible to create multiple virtual COM ports using one USB?

Yes, possible

Thanks.

Please guide me or share some material

I think you are trying to make a "USB composite device application". It's a place to start, even if it's not exactly what you want.

https://programmerall.com/article/97601906547/

There are many hits on Google for "stm32 composite usb device". Here is one:

https://github.com/GaryLee/stm32f103_dual_vcp

I am using threadX. Is it the same way to do in that