UART how the overflow works ?

[Xenon02]

Hello ! 

I've been wondering, how UART works in receiving side ? I have like maybe 4/5 questions ? 

-> What if I have a RX buffer that is full ? The flag is on and I didn't read the data. Then the new data comes after sometime, will it overwrite my data and still the flag will be on ? 

-> Is the data in the RX buffer loaded bit after bit or is it loaded whole word (8 bits) in one go and not bit by bit ? 

-> How do I know when I type HAL_UART_Transmit and after that the peripheral will be sending data and after transmit command I have the command HAL_UART_Receive, so how do I know it was fast enough to read the whole data without loosing any new or old data ? Like it was to late although the commands were one after another.

-> What if I used IT or DMA, they are non blocking, and questions like above like what if I type Transmit and Receive will I make it in time ? or what if I didn't read data and the buffer sets the flag and new data comes. 

These questions came out of curiosity because I read something about how UARTs looks like conceptually but I wondered how it looks like in practice. Because there are many types of commands in UART and wondered "what if" stuff. Also I know that RX works all the time and not only when I type HAL_UART_Receive so I wondered how it works when I have full data or when I type HAL_UART_Receive after HAL_UART_Transmit, will it be fast enough to read full buffor because this Receive commands reads full buffors and not bit by bit data. And what if I type HAL_UART_Receive after some time .. Yea I didn't know how to check it so maybe asking here won't be a problem ;> Sorry if it is. 

[TDK]

-> What if I have a RX buffer that is full ? The flag is on and I didn't read the data. Then the new data comes after sometime, will it overwrite my data and still the flag will be on ? 

The data is not overwritten. The new byte is lost and the ORE flag is set. The ORE flag must be cleared before the peripheral starts working correctly again.

-> Is the data in the RX buffer loaded bit after bit or is it loaded whole word (8 bits) in one go and not bit by bit ? 

It is loaded one word at a time. Typically one word is one byte, but it can be 9 bits, for instance.

-> How do I know when I type HAL_UART_Transmit and after that the peripheral will be sending data and after transmit command I have the command HAL_UART_Receive, so how do I know it was fast enough to read the whole data without loosing any new or old data ? Like it was to late although the commands were one after another.

You need to know your system. Does data start coming in the middle of a HAL_UART_Transmit call? If so, you'll need to use HAL_UART_Receive_IT or HAL_UART_Receive_DMA and call it prior to the transmit call so that data is received.

-> What if I used IT or DMA, they are non blocking, and questions like above like what if I type Transmit and Receive will I make it in time ? or what if I didn't read data and the buffer sets the flag and new data comes. 

If you use IT, you need to read the byte in the interrupt handler. If you don't, OVR will be set. With DMA, it is automatically transferred into memory and you can read it at a later time.

You're free to ask questions here, but these points are covered in detail in the reference manual as well.

The OREDIS bit can also disable this error flag being set, but data will still be lost.

[Xenon02]

Holy guacamolly.

So there are 2 types of error of the same thing for IT it is OVR and without IT is ORE ? 

I've got 4 more questions (of course I will flag your reply as a solution) :

-> So there are 2 registers in UART for RX ? One that takes bit by bit and the other register that takes the whole data at once ? This makes sense I think because instead of having time before 1 bit comes I have time before a whole word comes if it makes any sense (before 1 bit vs before 1 word comes to read the actual data). 

-> So when ORE flag or OVR flag is set then Receiver perihperal doesn't take any new data (the register that takes bit after bit won't take any data so RX is disconnected) ? 

-> What if I let's say want to read 50 characters but the transmiter sends only 3 characters ? Like I send a command, and I want to receive 50 characters (just by mistake), and the transmiter only sends 3 characters like OK\0, I read the data OK\0 but the transmiter doesn't send next data but somehow I received 47 characters of "\0". Does it mean I read all the time the old data ? Weird it must be because it restarts the RX flag so it is all the time waiting for new data hmmmm. 

-> I tried to read from manual but like in the picture you've sent, it says that it has shift register and RDR register I didn't much understand why there were two registers and it says that during the ORE the shift register is overwritten and that data is lost hmmm so the RX is not disconnected and the data is still received by the shifter register but not by the RX register ? There are also so many flags that I at some point get lost in what is what and what should I think of and what not. 

[Tesla DeLorean]

Peripheral registers are not "memory" but a window into logic and state-machines.

There is a shift register in the background, once it is full it gets moved to the receive data register, if that is empty.

Most STM32 will generate an interrupt for each byte received, and you have around one byte time to recover it before the next potentially arrives. The HAL clouds this a bit because you tell it how large the buffer expectation is. In a more natural implementation you'd receive a byte at a time, and process via a buffer or stateful analysis, so you can understand where the data is complete. For example an AT command spanning a few bytes and terminated by a carriage return / line feed pair ( <CR><LF> )

I'd imagine there are some college level texts explaining how UARTs work and how peripherals are designed / built.

[TDK]

-> So there are 2 registers in UART for RX ? One that takes bit by bit and the other register that takes the whole data at once ? This makes sense I think because instead of having time before 1 bit comes I have time before a whole word comes if it makes any sense (before 1 bit vs before 1 word comes to read the actual data). 

It's ORE in both cases. Same flag, same bit, just a typo in my response. (The overrun flag is called OVR on other peripherals, not sure why it's called ORE here instead.)

-> So when ORE flag or OVR flag is set then Receiver perihperal doesn't take any new data (the register that takes bit after bit won't take any data so RX is disconnected) ? 

Effectively, yes. The actual behavior is a little more complicated. There is a shift register, and that keeps getting updated, but the data never makes it into the DR register. The RX pin is never disconnected.

-> What if I let's say want to read 50 characters but the transmiter sends only 3 characters ? Like I send a command, and I want to receive 50 characters (just by mistake), and the transmiter only sends 3 characters like OK\0, I read the data OK\0 but the transmiter doesn't send next data but somehow I received 47 characters of "\0". Does it mean I read all the time the old data ? Weird it must be because it restarts the RX flag so it is all the time waiting for new data hmmmm. 

If the transmitter only sends 3 characters, that is all you will read. You won't "somehow" read additional 0 characters. You can use the IDLE event to determine when transmission has stopped in the event of variable-length transmission. For example, this is what HAL_UARTEx_ReceiveToIdle_DMA does.

-> I tried to read from manual but like in the picture you've sent, it says that it has shift register and RDR register I didn't much understand why there were two registers and it says that during the ORE the shift register is overwritten and that data is lost hmmm so the RX is not disconnected and the data is still received by the shifter register but not by the RX register ? There are also so many flags that I at some point get lost in what is what and what should I think of and what not. 

You don't have to understand the low-level hardware architecture to start. It sounds like you plan to use HAL. There are UART examples in the CubeMX repository, that's a good place to start. Consider connecting RX and TX (i.e. loopback mode) and verifying you can receive what you're sending.

You do of course need some level of understanding. One common mistake people make here is not understanding that receive needs to be active (i.e. HAL_UART_Receive* needs to be called) before the characters are sent.

If you create the correct code, ORE should never become set.

[Xenon02]

>"It's ORE in both cases. Same flag, same bit, just a typo in my response. (The overrun flag is called OVR on other peripherals, not sure why it's called ORE here instead.)"

I think the answers were mixed but ok :D 
Good to know that both are the same I thought that there are 2 different ;D 

>"Effectively, yes. The actual behavior is a little more complicated. There is a shift register, and that keeps getting updated, but the data never makes it into the DR register. The RX pin is never disconnected."

Aha so as I thought there are 2 registers !!! 
One that is updated and the one that has the actual word. So when ORE/OVR is active then this shift register gets all the time updated but the DR register will not be updated until the ORE is cleared. 

But here comes another problem what if I clear ORE, but didn't clear the RX flag which says it is full but the shift register still gets new word but is half way in it (like it has 4 bit and they other 4 are old or something), then the DR register sees that RX is active because only ORE is cleared, so it takes new data and welp,it is mixed 4 bits new with some 4 bits old data. So there is not full data transaction. 

I think maybe I am wrong dunno. All of it is just a speculation. 

>"If the transmitter only sends 3 characters, that is all you will read. You won't "somehow" read additional 0 characters. You can use the IDLE event to determine when transmission has stopped in the event of variable-length transmission. For example, this is what HAL_UARTEx_ReceiveToIdle_DMA does."

Well in my case it happened that it sent "OK\0" but the receive array was the size of 50 like : receive[50], and I wrote in HAL_UART_Receive to receive sizeof(receive), so yea, 50 words to receive but you see transmiter only sends "OK\0" which is only 3 words. In my case the rest 47 places were filled with "\0" dunno why. Is it that the UART_receive got error because the timeout took him out of the loop of receiving and it somehow itself filled with "\0" ? 

>"You don't have to understand the low-level hardware architecture to start. It sounds like you plan to use HAL. There are UART examples in the CubeMX repository, that's a good place to start. Consider connecting RX and TX (i.e. loopback mode) and verifying you can receive what you're sending.

You do of course need some level of understanding. One common mistake people make here is not understanding that receive needs to be active (i.e. HAL_UART_Receive* needs to be called) before the characters are sent.

If you create the correct code, ORE should never become set."

To be honest I sometime get mixed, I know or maybe I heard that somethimes you can transmit and receive at the same time ? So I was wondering how it works or what if I type :
HAL_UART_Transmit()
HAL_UART_Receive()

And what if this receive takes to much time, like I know I don't calculate it but I usually don't know if I can receive the data later or right now after Transmit command or what if I have some interrupt that is after transmit command, and it takes some time to get out of interrupt, and when the interrupt is done it reads then next command after hal transmit which is hal receive. 

So many "what ifs". And I heard to read a lot from manual or to understand why there are some weird options, so I tried to understand it how it is made in logic gates level. But it is very overwhelming. I also know to read manuals but there are so many flags and devices that I don't know what is what. So I don't know how to read these stuff ;) 
But I want to try though. 

[Xenon02]

>I'd imagine there are some college level texts explaining how UARTs work and how peripherals are designed / built.

I've read how basic uart works from some websites but they were simple that didn't consider whether we read it in time or not. I don't know if I can send links to these websites so I just mention them for now. 

>Peripheral registers are not "memory" but a window into logic and state-machines. 

Window ? I imagined it that there is shift register which just updates all the time and DR register that takes the full data from shift register all at once. So like two step to update RX register so that it updates every 8 bits and not every 1 bit so there is time to take data. 

>For example an AT command spanning a few bytes and terminated by a carriage return / line feed pair ( <CR><LF> )

I just didn't understand why I got so many "\0" when I only requested it to send me back "OK\0". But accidently requested 50 bytes from RX, but of course it sends 3 bytes which is "OK\0". So I wondered if RX just spitted ERROR and stopped doing UART_Receive and just filled the actual array with "\0" or what. 

[Pavel A.]

Let's put it into proportion. USB, CAN, Ethernet are much more complex than the UART. If UART confuses you... too bad. Cheer up.

[Xenon02]

I mean I know it looks lame ... I also know it is just simple two commands, I was just wondering how it always gets information in time, usually I think nobody thinks about it but I just wondered that's all so these questions came up. 
Like didn't you think like when I use a command to transmit how much time do I have to get data ? or will the command after transmit which is receive will be read and interpreted on time ? 

I know there are more complex systems and sorry if my confusions seemed disappointing to the point where simple UART made me confused. I mean those speeds made me wonder how my code just makes it in time everytime :D Because I've learned that timing is very important in logic gate systems. So I tried to understand how I just got it in time like 1ms or smaller for bigger baund rate it is just unimagible for me that using two commands one after another I was able to send transmit and read the receive hal command, interpret it and read the RX buffer while the transmit was sending the data during that time where code was reading receive hal command, interpreting it etc.

So fast (i know there are faster just saying). 
Your comment didn't cheered me up ;> But I understand how it looks like. 
I apologize and sorry for my questions and how they look like ;> 

[Pavel A.]

But @TDK already has answered your question. Please read his reply again. Think what if the device on the other end of the UART does not wait at all and echoes back each sent character?  Unfortunately the HAL library for UART is not quite usable for many real life cases. You'll have to roll your own bi-directional communication, look for suitable examples.