I was playing with SDRAM and found a very strange cache behavior. Can anyone help me understand what is happening?
I am using STM32F7508-DK board. The SDRAM is connected with FMC.
I am basically going over a small 2-d tile inside a 2-d array. The tile is 4*8*8 Byte, or 8 cache line.
I am accessing every word (4Byte) inside the tile, accessing 64 elements.
The tile is much smaller than the L1 cache size (4KB), so no eviction should occur, and the code should experience 8 cache misses (cold miss) + 56 cache hits.
This is exactly what I see if I visit the memory consecutively.
If I access with a stride, it works as expected when the total array size is small, but suddenly stops to work as expected with larger array (but I still only access the same-sized tile!).
Below is my simplified code:
#define ROW_SIZE 8
#define COL_SIZE 128 // 256
#define TILE 8
slow_mem uint32_t A1[ROW_SIZE][COL_SIZE];
int main(void)
{
init();
HAL_EnableFMCMemorySwapping();
for (int i = 0; i < TILE; ++i) {
for (int j = 0; j < TILE; ++j) {
gpioSet();
A1[i][j]++;
gpioReset();
}
}
SCB_CleanInvalidateDCache();
for (int i = 0; i < TILE; ++i) {
for (int j = 0; j < TILE; ++j) {
gpioSet();
A1[j][i]++;
gpioReset();
}
}
return 0;
}slow_mem places the A1 array inside the SDRAM. SDRAM is mapped to 0x60000000 to enable cache. Cache is enabled and SDRAM RBURST is turned off.
As you can see, I only ever access A[0][0] -- A[7][7].
The first loop accesses in the order of A[0][0]->A[0][1]->... and the second loop in A[0][0]->A[1][0]->...
Because the cache line size is 32B, each row of the tile (e.g., A[i][0] -- A[i][7]) fits exactly one cache line.
Again, according to what I learned from the architecture class, both should have 8 cache misses and 56 cache hits, and this is indeed what I see when COL_SIZE=128.
Figure 1. Memory access for the first loop, with COL_SIZE=128, 256.
Figure 2. Memory access for the second loop, with COL_SIZE=128.
However, when I increase the COL_SIZE, the result does not make sense anymore.
Figure 3. Memory access for the second loop, with COL_SIZE=256
Just to reiterate, I only ever access A[0][0]--A[7][7]! Why would the COL_SIZE even matter if it is 128 or 256? Also, what is the device doing on Figure 3? Why is each pulse even wider than the cache miss case? The behavior is very interesting and very counter-intuitive to me.
I am digging this behavior because essentially, this behavior leads to a slowdown if I do column-major access of a large array stored in SDRAM. It has a real performance implication! Also, I somehow became obsessed with this problem so I really want to know the answer :).
Changing COL_SIZE from 128 to 256 makes A1 larger than the L1 cache size. I wonder if that fact is somehow relevant (Still, I only access a tiny tile of it, so I don't see how it can matter).
Another theory is that the hardware prefetcher is somehow messing things up. Although, I still do not understand how a prefetcher would degrade a performance in such a simple and universal case of accessing a matrix column-major inside a tile. Also, I thought prefetchers do not directly evict the cache (and instead write data inside a dedicated HW structure such as stream buffer).
Can anyone help me explain this phenomenon? Can anyone confirm or deny if it is the prefetcher's problem?
Any discussion is highly appreciated.
Thank you!
