The compiler doesn't decide. It is part of the CPU and it is designed to hold proximate and recently used memory content. Perhaps take a course on micro-controller architectures, or a book on the topic. Think of it as taking a handful of candy out of a bag so you can pick the pieces from your hand rather than picking individual pieces from the bag each time.
The 4KB RAM/FLASH also doesn't make much sense, so I don't know what documentation you're reading.
Different memory runs faster and is closer to the core (less bus transactions, less delay interacting with other bus users). The TCM is close to the core, code and data placed here can be fetched efficiently, so isn't cached, and the cache can focus in serving the slow memories.
The FLASH has a different caching/fetching mechanism by using very wide access to the array, it is slow, but can fetch 128-bits at a time, first word is expensive, subsequent word(s) are free. It holds a small array of recently fetched word lines.
The LINKER can be used to place things in different memory area, you would use a scatter file or linker script to direct this. On the compiler side you could pass information to the linker via #pragma or attribute directives. Check you compiler, linker and tool chain documentation.
