Any experts on LWIP and the netconn API? (not looking for free; can pay)

[PHolt.1]

Hi All,

I am implementing a simple web server. The code is all over the place. It came with ST Cube IDE, and an identical version with same comments is here

https://github.com/particle-iot/lwip/blob/master/contrib/apps/httpserver/httpserver-netconn.c

However, they assume incoming data (or the part of it of interest) is contained in just the one buffer. The code is like this

> err_t res = netconn_recv(conn, &inbuf);

> if ((res == ERR_OK) && (inbuf != NULL))

> {

>  if (netconn_err(conn) == ERR_OK) 

>  {

>   netbuf_data(inbuf, (void**)&buf, &buflen);

>   /* Is this an HTTP GET command? (only check the first 5 chars, since

>   there are other formats for GET, and we're keeping it very simple )*/

>   if ((buflen >=5) && (strncmp(buf, "GET /", 5) == 0))

>   {

netconn_recv appears to be a blocking function which tells you some data has arrived.

netconn_err is a macro.

netbuf_data reads in a buffer. There is no way to know how big this buffer is, but you are told how much data is in there.

The problem I have is that I am extending this "server" (on which I have spent at least a week full-time and which works solidly) with a file upload feature, and that obviously involves getting a bit more incoming data than just a few strings. There are almost no examples on the web of using netconn for receiving a data stream - in this case up to 2MB.

I am wondering if the buffer pointed to by netbuf_data is actually one of these, defined in lwipopts.h

>/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */

>#define PBUF_POOL_SIZE      8

>

>/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */

>// If small, set it so that say 3 of them hold a complete largest packet (1500+stuff)

>#define PBUF_POOL_BUFSIZE    512

which would mean no more than 512 bytes. It kind of makes sense since all the netconn code refers to pbufs.

There is also a function netbuf_next and I don't know how it fits into this scheme.

I have a suspicion this scheme involves a linked list of buffers (up to 8 if my suspicion above is right; also why the double indirection in the buffer address??) and you somehow have to step through them if you want to receive a lot of data.

Basically I need some code which implements data receive, with a timeout. The timeout I can do in various ways, with a FreeRTOS timer being the most "modern" way. The buffer from which I will be writing the eventual file (FLASH filesystem) is 512 bytes long. I can write less but can't write more.

On a related topic, am I right in saying that if

LWIP_TCPIP_CORE_LOCKING=1

then the LWIP API is thread safe for raw, netconn and sockets, whereas if

LWIP_TCPIP_CORE_LOCKING=0

then the LWIP API is thread safe for netconn and sockets only? There is a vast range of disagreement online about this topic.

Thank you in advance for any comments.

If you feel like writing the data receive code for some money, please PM me.

[PHolt.1]

As expected, I never got a reply here, but it has been worked out, and I posted it here

https://www.eevblog.com/forum/microcontrollers/any-experts-on-lwip-and-the-netconn-api-(not-looking-for-free-can-pay)/msg4341529/#msg4341529

[Piranha]

> the LWIP API is thread safe for raw

"raw API functions are not protected from concurrent access"

Does it say that anywhere? No, you invented that! It is never thread safe.

> There is a vast range of disagreement online about this topic.

The internet indeed is full of junk, but I haven't seen a single note that says that lwIP raw API is ever thread safe. Can you show some examples?

> Makes one wonder what exactly LWIP_TCPIP_CORE_LOCKING=1 does

It enables that feature so that the application and some internal code can lock the core thread and call RAW API directly instead of scheduling callbacks to the core thread, which are implemented by message passing. Again you have not read anything:

"Use LWIP_TCPIP_CORE_LOCKING. All "raw" APIs functions can be called when lwIP core lock is aquired, see LOCK_TCPIP_CORE() and UNLOCK_TCPIP_CORE()."

> using LWIP_TCPIP_CORE_LOCKING=1 is "more efficient"

With the core locking application just locks a single mutex and can execute a group of functions. With the message queue based callbacks it has to create a new message and wait for a callback to be executed for every function call. If you would have read at least the official minimum, you would have known that!

So you have fought through and learned the Netconn API. Wasn't it easier to read the documentation? It even states clearly:

"TX/RX handling based on Network buffers (containing Packet buffers (PBUF)) to avoid copying data around."

> It should have been obvious that LWIP cannot be zero copy on the way out because it has to re-pack packets like that into MTU units, and short of a very clever DMA controller one can't do that on the fly.

So you've mixed together an application API, TCP protocol, lwIP internals and a ETH driver... Obviously, if you call netconn_write() with NETCONN_COPY, then it isn't zero-copy. In other cases the stack mostly just chains another PBUF in front of your data, because it needs to add Ethernet+IP+TCP/UDP headers in front if your data. Then that PBUF chain is passed to a hardware driver, which obviously depends on your implementation. For CubeF4 the old driver before the v1.27.0 did a memory copying to dedicated buffers, but the new rewritten driver is implemented as a zero-copy type.

> this is supplied by ST but also found in various places online e.g.

Why are you getting things from various places, when they can be taken from the official sources?

> With the ST port of LWIP you get a thing called HTTPD which I believe doesn't actually work; I terminated a subcontracted project using that when it went well past 100hrs with not much working.

It's not from ST, but from the official lwIP. And, while not a top-noch implementation, it definitely works. It took me few days to get all of those features working. Obviously the lower layers have to work perfectly, otherwise there is no point in wondering why higher layers don't work.

[Piranha]

> also why the double indirection in the buffer address??

Now this is completely ridiculous. So it turns out you don't know even the C programming language and/or don't understand the pointers.

> I do have one example of RAW, in some code produced by someone else

> fd = socket(AF_INET, SOCK_RAW, IP_PROTO_ICMP);

40 years of experience, at least a year of dealing with Ethernet and lwIP and still not recognizing the socket API, which comes from BSD from a year 1983. Even more - not distinguishing the difference of an API and socket type.

It's impossible to deal with complex solutions without understanding the basics...