AZURE RTOS: FileX: send web page from MCU flash without FAT - a missing feature?

[tjaekel]

Meanwhile STM goes with Microsoft AZURE RTOS - ok, fine, feature rich, obvious!

But "I miss a feature": open and read a file from the MCU internal flash memory (as a const file content, as array of characters, without a need for a sector based, FAT based, a file system).

In the past, on STM MCUs and FW (CubeMX drivers and demos) - it was possible to have web pages for a HTTPD daemon (web server) located in MCU Flash memory and read the file data bytes and send it as browser response.

Now, with AZURE RTOS - all seems to be stored on a FAT (sector based) file system. Why so complicated?!

Details:

I have ported the STM32U575-EV USB_ECM project to a NUCLEO-U575ZI-Q board. It needs the external SD Card! So, I had to put the web page files on external SD Card (with external SD Card adapter for a NUCELO board). I want to get rid of this SD Card and have my web pages sitting in the MCU Flash memory.

OK, I understand, AZURE FileX is very flexible, it can also support SDRAM as file system MEDIA, even flash memories (NAND/NOR), e.g via levelX. I know all this, e.g.:
Introduction to FILEX - stm32mcu

I know: AZURE FileX can use SRAM, Flash Memories NAND/NOR (assuming also MCU internal flash - but very delicate to use MCU Flash with code in parallel) as File System Storage. But it remains a fact: it is handled as a sector based, FAT (or exFAT) file system (on every media).

But it means also: you have to "format", "partition" such a file system: and you had to copy the web pages from MCU flash into the SRAM/SDRAM/Flash file system storage (wasting memory: already available in Flash - why to "copy"?).

And it consumes a lot of time, it wastes a lot of space (e.g. needing a Boot record, a FAT directory table, files allocating "clusters" (numbers of sectors) etc. And my web page would be "doubled": it sits already in MCU Flash, but I had to copy it into a (formatted) file system on boot time, e.g. to SRAM media.

I miss this:

The web browser (in NetXDuo - Addons Web Server) uses (only) functions like:

fx_file_open(), fx_file_read(), fx_file_close() and fx_directory_information_get()

Technically, the web server just wants to open a file (in order to send as a request response), it needs to know the size of the file and it might read the file in chunks (splitting into smaller packets via network).

But all these functions require a FAT file system: it goes at the end to a media for the file system (SRAM, NAND/NOR or SD Card). And it is slow (going through the FAT system).

I do not need this "complexity": instead: the "file" sits already in my Flash Memory (MCU), without any overhead, e.g. without having a FAT table, Boot sector, organized in "clusters" and "sectors". The function should read just the bytes of a string (which acts like the content of a "file", a byte stream reading from).

Solution/intention

Due to fact, that AZURE FileX does not support to read a "file" - without a FAT system - from a read-only memory (as a stream of bytes) - I am tending towards this approach:

• I will implement all the functions needed by the web server, e.g. fx_file_open(), fx_file_read(), fx_file_close() and fx_directory_information_get() - again
• but instead of using a FAT file system (with sectors) - I read just a stream of bytes (from a "file", stored in MCU flash, from a byte array as a const string)
• no need to have a FAT File System, no need to have sectors - save the space by just allocating the Flash memory for the "file" content (as done in the past)
• all these functions are "high-level" and "abstracted" from a real FAT system: a fx_file_read() could read also my byte string from flash memory, without to go via FAT, sector numbers etc.

I want to have a web browser, dealing with "files" but not with the overhead to have a real FAT file system, not wasting memory space, not delaying boot time by copying files from MCU flash into File System Media, not delaying real-time response by going through a file system...

It should not be a big deal, to implement these "few" functions to read a "file" as a stream of characters from my MCU flash: the function calls can work in the same way, with same parameters and behavior (but bypassing any FAT file system).

It was possible before in STM FW to do this approach - now, with changing to AZURE RTOS, this feature is gone.

Why is such real-time (and memory space optimized) "feature" missing (in AZURE RTOS)?

[tjaekel · Accepted solution]

So, I have now my web pages read from internal MCU flash memory working:
I have implemented the minimal features to read a file sent via web server. I use it on a NUCLE-U5757ZI-Q board - for USB ECM - now without a need to have an SD Card socket (Nucleo board without HW extension).

It works this way:

The web server, implemented in file "nx_web_http_server.c" wants to open, get file size, read and close a file (as web page to be sent).

I have implemented similar functions, e.g.

UINT  fxs_file_open(FX_MEDIA *media_ptr, FX_FILE *file_ptr, CHAR *file_name, UINT open_type);
UINT  fxs_directory_information_get(FX_MEDIA *media_ptr, CHAR *directory_name,
                                    UINT *attributes, ULONG *size,
                                    UINT *year, UINT *month, UINT *day,
                                    UINT *hour, UINT *minute, UINT *second);
UINT  fxs_file_read(FX_FILE *file_ptr, VOID *buffer_ptr, ULONG request_size, ULONG *actual_size);
UINT  fxs_file_close(FX_FILE *file_ptr);

See the name as fxs_...: they substitute the original FileX functions calls like this:

#ifdef FLASH_PAGES
    status =  fxs_file_open(server_ptr -> nx_web_http_server_media_ptr, &(server_ptr -> nx_web_http_server_file), server_ptr -> nx_web_http_server_request_resource, FX_OPEN_FOR_READ);
#else
    status =  fx_file_open(server_ptr -> nx_web_http_server_media_ptr, &(server_ptr -> nx_web_http_server_file), server_ptr -> nx_web_http_server_request_resource, FX_OPEN_FOR_READ);
#endif

Define the macro FLASH_PAGES on project settings: it will read now the web page content from a character array, stored in MCU Flash memory. Not anymore via FAT file system, not anymore from SD card. Way faster and saving a lot of space.

If you want to disable completely the SD card initialization and use just the MCU Flash:
see in file "app_netxduo.c" this code:

  /* Open the SD disk driver */
#ifndef FLASH_PAGES
  if (fx_media_open(&SDMedia, "STM32_SDIO_DISK", fx_stm32_sd_driver,
                    0, DataBuffer, sizeof(DataBuffer)) != FX_SUCCESS)
  {
    Error_Handler();
  }
#endif

Also the LED toggle works ("page2").

Here the two files you need: "fileXS.h":

#ifndef __FILEXS_H__
#define __FILEXS_H__

//#include "tx_port.h"
//#include "fx_api.h"

#include "fx_api.h"
#include "fx_system.h"
#include "fx_file.h"

typedef struct {
	CHAR *file_name;
	ULONG file_size;
	const unsigned char *file_content;
} FileXSContentDescriptor;

UINT  fxs_file_open(FX_MEDIA *media_ptr, FX_FILE *file_ptr, CHAR *file_name, UINT open_type);
UINT  fxs_directory_information_get(FX_MEDIA *media_ptr, CHAR *directory_name,
                                    UINT *attributes, ULONG *size,
                                    UINT *year, UINT *month, UINT *day,
                                    UINT *hour, UINT *minute, UINT *second);
UINT  fxs_file_read(FX_FILE *file_ptr, VOID *buffer_ptr, ULONG request_size, ULONG *actual_size);
UINT  fxs_file_close(FX_FILE *file_ptr);

#endif

and "fileXS.c":

/*
 * fileXS.c
 *
 *  Created on: Jan 29, 2024
 *      Author: tj
 */

#include "fileXS.h"

/* create the file content, on flash memory */

static const unsigned char _FILE_0_Content[] =
"enter a URL as IPADDR/index.html"
;

static const unsigned char _FILE_1_Content[] =
"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\r\n\
<html xmlns:v=\"urn:schemas-microsoft-com:vml\" xmlns:o=\"urn:schemas-microsoft-com:office:office\" xmlns:w=\"urn:schemas-microsoft-com:office:word\" xmlns=\"http://www.w3.org/TR/REC-html40\">\r\n\
<head>\r\n\
<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\">\r\n\
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\r\n\
<title> \r\n\
USB ECM Web Page\r\n\
</title>\r\n\
<style>\r\n\
#more {display: none;}\r\n\
ol{\r\n\
margin-left: -1.3em;\r\n\
}\r\n\
ol:before {\r\n\
content: '- ';\r\n\
margin: 26px;\r\n\
}\r\n\
</style>\r\n\
</head>\r\n\
<body style=\"\" lang=\"EN-US\" link=\"blue\" vlink=\"blue\">\r\n\
<h2>Welcome to the Web Page</h2>\r\n\
<a href=\"page2.html\">Page 2</a>\r\n\
</body>\r\n\
</html>\r\n"
;

static const unsigned char _FILE_2_Content[] =
"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\r\n\
<html xmlns:v=\"urn:schemas-microsoft-com:vml\" xmlns:o=\"urn:schemas-microsoft-com:office:office\" xmlns:w=\"urn:schemas-microsoft-com:office:word\" xmlns=\"http://www.w3.org/TR/REC-html40\">\r\n\
<head>\r\n\
<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\">\r\n\
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\r\n\
<title> \r\n\
USB ECM Web Page\r\n\
</title>\r\n\
<style>\r\n\
#more {display: none;}\r\n\
ol{\r\n\
margin-left: -1.3em;\r\n\
}\r\n\
ol:before {\r\n\
content: '- ';\r\n\
margin: 26px;\r\n\
}\r\n\
</style>\r\n\
</head>\r\n\
<body style=\"\" lang=\"EN-US\" link=\"blue\" vlink=\"blue\">\r\n\
<h2>Welcome to the Page 2</h2>\r\n\
<form method=\"post\" action=\"/page2.html\">\r\n\
<input value=\"LED_ON\" name=\"led\" type=\"radio\">LED ON<br>\r\n\
<input value=\"LED_OFF\" name=\"led\" type=\"radio\">LED OFF<br>\r\n\
<br>\r\n\
<input value=\"Send\" type=\"submit\"></form>\r\n\
<p><a href=\"index.html\">back to home</a></p>\r\n\
</body>\r\n\
</html>\r\n"
;

static FileXSContentDescriptor contentDescriptor[] = {
		{
				"/",							/* just "/", just IP address entered - sent as "text/plain" */
				sizeof(_FILE_0_Content) -1,		/* without NUL */
				_FILE_0_Content
		},
		{
				"/index.html",
				sizeof(_FILE_1_Content) -1,		/* without NUL */
				_FILE_1_Content
		},
		{
				"/page2.html",
				sizeof(_FILE_2_Content) -1,
				_FILE_2_Content
		}
};

static FX_FILE fxs_file = {
		.fx_file_id = -1
};

static int findDescriptor(CHAR *file_name)
{
	int i;
	for (i = 0; i < (sizeof(contentDescriptor) / sizeof(FileXSContentDescriptor)); i++)
	{
		if (strcmp(file_name, contentDescriptor[i].file_name) == 0)
		{
			return i;
		}
	}

	return -1;
}

/* implement the FileX functions needed */

UINT  fxs_file_open(FX_MEDIA *media_ptr, FX_FILE *file_ptr, CHAR *file_name, UINT open_type)
{
	int i;

	if (fxs_file.fx_file_id != -1)
		return(FX_ALREADY_CREATED);					//already open

	i = findDescriptor(file_name);
	if ( i != -1)
	{
		fxs_file.fx_file_id = i;
		fxs_file.fx_file_current_logical_offset = 0;
		fxs_file.fx_file_current_file_size = contentDescriptor[i].file_size;

		file_ptr->fx_file_id = i;
		return(FX_SUCCESS);
	}

	return(FX_NOT_FOUND);
}

UINT  fxs_directory_information_get(FX_MEDIA *media_ptr, CHAR *directory_name,
                                    UINT *attributes, ULONG *size,
                                    UINT *year, UINT *month, UINT *day,
                                    UINT *hour, UINT *minute, UINT *second)
{
	int i;

	i = findDescriptor(directory_name);
	if ( i != -1)
	{
		if (size)
			*size = contentDescriptor[i].file_size;

		return(FX_SUCCESS);
	}

	return(FX_NOT_FOUND);
}

UINT  fxs_file_read(FX_FILE *file_ptr, VOID *buffer_ptr, ULONG request_size, ULONG *actual_size)
{
	ULONG rdSize;

	if (fxs_file.fx_file_id == -1)
	{
		*actual_size = 0;
		return(FX_NOT_OPEN);
	}

	if (request_size > (fxs_file.fx_file_current_file_size - fxs_file.fx_file_current_logical_offset))
		rdSize = fxs_file.fx_file_current_file_size - fxs_file.fx_file_current_logical_offset;
	else
		rdSize = request_size;

	if (rdSize)
	{
		memcpy(buffer_ptr, contentDescriptor[fxs_file.fx_file_id].file_content + fxs_file.fx_file_current_logical_offset, rdSize);

		*actual_size = rdSize;
		fxs_file.fx_file_current_logical_offset += rdSize;

		return(FX_SUCCESS);
	}

	return(FX_END_OF_FILE);
}

UINT  fxs_file_close(FX_FILE *file_ptr)
{
	fxs_file.fx_file_id = -1;

	return(FX_SUCCESS);
}

See the definition of the web page files there as well.

Take both files to your project and use the fxs_... functions instead of the original fx_... functions (via macro).

[tjaekel]

My reply was removed: I see "This reply was marked as spam and has been removed." Why?

It works now without SD Card:

Here what to do:

"nx_web_http_server.c":

    /* Open the specified file for reading.  */
#ifdef FLASH_PAGES
    status =  fxs_file_open(server_ptr -> nx_web_http_server_media_ptr, &(server_ptr -> nx_web_http_server_file), server_ptr -> nx_web_http_server_request_resource, FX_OPEN_FOR_READ);
#else
    status =  fx_file_open(server_ptr -> nx_web_http_server_media_ptr, &(server_ptr -> nx_web_http_server_file), server_ptr -> nx_web_http_server_request_resource, FX_OPEN_FOR_READ);
#endif

Disable completely the use of SD card, in "app_netxduo.c":

  /* Open the SD disk driver */
#ifndef FLASH_PAGES
  if (fx_media_open(&SDMedia, "STM32_SDIO_DISK", fx_stm32_sd_driver,
                    0, DataBuffer, sizeof(DataBuffer)) != FX_SUCCESS)
  {
    Error_Handler();
  }
#endif

Here the two files you need. Define the macro FLASH_PAGES on project!

"fileXS.h":

#ifndef __FILEXS_H__
#define __FILEXS_H__

//#include "tx_port.h"
//#include "fx_api.h"

#include "fx_api.h"
#include "fx_system.h"
#include "fx_file.h"

typedef struct {
	CHAR *file_name;
	ULONG file_size;
	const unsigned char *file_content;
} FileXSContentDescriptor;

UINT  fxs_file_open(FX_MEDIA *media_ptr, FX_FILE *file_ptr, CHAR *file_name, UINT open_type);
UINT  fxs_directory_information_get(FX_MEDIA *media_ptr, CHAR *directory_name,
                                    UINT *attributes, ULONG *size,
                                    UINT *year, UINT *month, UINT *day,
                                    UINT *hour, UINT *minute, UINT *second);
UINT  fxs_file_read(FX_FILE *file_ptr, VOID *buffer_ptr, ULONG request_size, ULONG *actual_size);
UINT  fxs_file_close(FX_FILE *file_ptr);

#endif

 "fileXS.c":

/*
 * fileXS.c
 *
 *  Created on: Jan 29, 2024
 *      Author: tj
 */

#include "fileXS.h"

/* create the file content, on flash memory */

static const unsigned char _FILE_0_Content[] =
"enter a URL as IPADDR/index.html"
;

static const unsigned char _FILE_1_Content[] =
"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\r\n\
<html xmlns:v=\"urn:schemas-microsoft-com:vml\" xmlns:o=\"urn:schemas-microsoft-com:office:office\" xmlns:w=\"urn:schemas-microsoft-com:office:word\" xmlns=\"http://www.w3.org/TR/REC-html40\">\r\n\
<head>\r\n\
<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\">\r\n\
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\r\n\
<title> \r\n\
USB ECM Web Page\r\n\
</title>\r\n\
<style>\r\n\
#more {display: none;}\r\n\
ol{\r\n\
margin-left: -1.3em;\r\n\
}\r\n\
ol:before {\r\n\
content: '- ';\r\n\
margin: 26px;\r\n\
}\r\n\
</style>\r\n\
</head>\r\n\
<body style=\"\" lang=\"EN-US\" link=\"blue\" vlink=\"blue\">\r\n\
<h2>Welcome to the Web Page</h2>\r\n\
<a href=\"page2.html\">Page 2</a>\r\n\
</body>\r\n\
</html>\r\n"
;

static const unsigned char _FILE_2_Content[] =
"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\r\n\
<html xmlns:v=\"urn:schemas-microsoft-com:vml\" xmlns:o=\"urn:schemas-microsoft-com:office:office\" xmlns:w=\"urn:schemas-microsoft-com:office:word\" xmlns=\"http://www.w3.org/TR/REC-html40\">\r\n\
<head>\r\n\
<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\">\r\n\
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\r\n\
<title> \r\n\
USB ECM Web Page\r\n\
</title>\r\n\
<style>\r\n\
#more {display: none;}\r\n\
ol{\r\n\
margin-left: -1.3em;\r\n\
}\r\n\
ol:before {\r\n\
content: '- ';\r\n\
margin: 26px;\r\n\
}\r\n\
</style>\r\n\
</head>\r\n\
<body style=\"\" lang=\"EN-US\" link=\"blue\" vlink=\"blue\">\r\n\
<h2>Welcome to the Page 2</h2>\r\n\
<form method=\"post\" action=\"/page2.html\">\r\n\
<input value=\"LED_ON\" name=\"led\" type=\"radio\">LED ON<br>\r\n\
<input value=\"LED_OFF\" name=\"led\" type=\"radio\">LED OFF<br>\r\n\
<br>\r\n\
<input value=\"Send\" type=\"submit\"></form>\r\n\
<p><a href=\"index.html\">back to home</a></p>\r\n\
</body>\r\n\
</html>\r\n"
;

static FileXSContentDescriptor contentDescriptor[] = {
		{
				"/",							/* just "/", just IP address entered - sent as "text/plain" */
				sizeof(_FILE_0_Content) -1,		/* without NUL */
				_FILE_0_Content
		},
		{
				"/index.html",
				sizeof(_FILE_1_Content) -1,		/* without NUL */
				_FILE_1_Content
		},
		{
				"/page2.html",
				sizeof(_FILE_2_Content) -1,
				_FILE_2_Content
		}
};

static FX_FILE fxs_file = {
		.fx_file_id = -1
};

static int findDescriptor(CHAR *file_name)
{
	int i;
	for (i = 0; i < (sizeof(contentDescriptor) / sizeof(FileXSContentDescriptor)); i++)
	{
		if (strcmp(file_name, contentDescriptor[i].file_name) == 0)
		{
			return i;
		}
	}

	return -1;
}

/* implement the FileX functions needed */

UINT  fxs_file_open(FX_MEDIA *media_ptr, FX_FILE *file_ptr, CHAR *file_name, UINT open_type)
{
	int i;

	if (fxs_file.fx_file_id != -1)
		return(FX_ALREADY_CREATED);					//already open

	i = findDescriptor(file_name);
	if ( i != -1)
	{
		fxs_file.fx_file_id = i;
		fxs_file.fx_file_current_logical_offset = 0;
		fxs_file.fx_file_current_file_size = contentDescriptor[i].file_size;

		file_ptr->fx_file_id = i;
		return(FX_SUCCESS);
	}

	return(FX_NOT_FOUND);
}

UINT  fxs_directory_information_get(FX_MEDIA *media_ptr, CHAR *directory_name,
                                    UINT *attributes, ULONG *size,
                                    UINT *year, UINT *month, UINT *day,
                                    UINT *hour, UINT *minute, UINT *second)
{
	int i;

	i = findDescriptor(directory_name);
	if ( i != -1)
	{
		if (size)
			*size = contentDescriptor[i].file_size;

		return(FX_SUCCESS);
	}

	return(FX_NOT_FOUND);
}

UINT  fxs_file_read(FX_FILE *file_ptr, VOID *buffer_ptr, ULONG request_size, ULONG *actual_size)
{
	ULONG rdSize;

	if (fxs_file.fx_file_id == -1)
	{
		*actual_size = 0;
		return(FX_NOT_OPEN);
	}

	if (request_size > (fxs_file.fx_file_current_file_size - fxs_file.fx_file_current_logical_offset))
		rdSize = fxs_file.fx_file_current_file_size - fxs_file.fx_file_current_logical_offset;
	else
		rdSize = request_size;

	if (rdSize)
	{
		memcpy(buffer_ptr, contentDescriptor[fxs_file.fx_file_id].file_content + fxs_file.fx_file_current_logical_offset, rdSize);

		*actual_size = rdSize;
		fxs_file.fx_file_current_logical_offset += rdSize;

		return(FX_SUCCESS);
	}

	return(FX_END_OF_FILE);
}

UINT  fxs_file_close(FX_FILE *file_ptr)
{
	fxs_file.fx_file_id = -1;

	return(FX_SUCCESS);
}

 It works fine, even the LED toggle works (via "page2.html"). See the definition of you web pages.

No need to have an SD Card, no need to have a FAT/FAT32 file system... Much faster as FileX would do...

[tjaekel]

The code for entire project is here:

GitHub - tjaekel/NUCLEO-U575ZI-Q_USB_ECM_noSDCard: NUCLEO-U5757ZI-Q board for USB_ECM without SD Card

[tjaekel]

A better project for USB ECM with NUCLEO-U575ZI-Q is here:

GitHub - tjaekel/STM32U5xx_USB_ECM: NUCLEO-U575ZI-Q with USB ECM, without SD Card

I have setup this now as a complete stand alone project, not anymore so "messy" as taken from CubeMX, removing (or excluding from compile) files which are not needed.