USBX Audio Class 2.0 microphone won't send data to host

[nico23]

I've developed on an STM32U5A9J-DK using ThreadX, which, via USBX, creates a loopback audio interface between the PC and the device in the way: PC->speaker on device->microphone on device->PC.

The devices are correctly seen and are working, as I'm able to enumerate them on the host and the device correctly receives audio frame,s and I'm able to process them into the device.

Now, the issue is that I'm unable to send the audio packets back to the host.

My implementation is pretty simple: I'm using the USBD_AUDIO_PlaybackStreamFrameDone function to read the audio frames and send them back to the microphone interface with

VOID USBD_AUDIO_PlaybackStreamFrameDone(UX_DEVICE_CLASS_AUDIO_STREAM *audio_play_stream,
                                        ULONG length)
{
  UCHAR *read_buffer;
  ULONG read_length;
  
  UCHAR *write_buffer;
  ULONG write_length;
  
  AUDIO_FRAME_MSG frame_msg;
  AUDIO_FRAME_BUFFER *frame_buf;

  /* -------------------------------------------------------------------------- */
  /* STEP 1: Get Access to the Received Frame (Speaker Data)                    */
  /* -------------------------------------------------------------------------- */
  if (ux_device_class_audio_read_frame_get(audio_play_stream, &read_buffer, &read_length) != UX_SUCCESS)
  {
      return; /* Should not happen if this callback is called */
  }

  /* Only process if we actually received bytes */
  if (length > 0 && read_length > 0)
  {
    /* ==================================================================== */
    /* LOGIC A: Send to TouchGFX Queue (User Application)                   */
    /* ==================================================================== */
    frame_buf = get_free_audio_frame();
    if (frame_buf != NULL && read_length <= MAX_AUDIO_FRAME_SIZE)
    {
      ux_utility_memory_copy(frame_buf->data, read_buffer, read_length);
      frame_buf->length = read_length;

      frame_msg.frame_ptr = frame_buf;
      
      if (tx_queue_send(&audio_processing_queue, &frame_msg, TX_NO_WAIT) != TX_SUCCESS)
      {
        release_audio_frame(frame_buf);
      }
    }

    /* ==================================================================== */
    /* LOGIC B: Loopback to Microphone (Direct Forwarding)                  */
    /* ==================================================================== */
    /* Check if Mic is active and stream is available */
    if (g_mic_stream != UX_NULL)
    {
        /* 2. Get Write Buffer: Request an empty buffer from Mic Stream FIFO */
        if (ux_device_class_audio_write_frame_get(g_mic_stream, &write_buffer, &write_length) == UX_SUCCESS)
        {
            /* 3. Process/Copy: Copy from Speaker Read Buffer to Mic Write Buffer */
            /* Ensure we don't overflow the Mic buffer (min of read/write lengths) */
            ULONG copy_len = (read_length < write_length) ? read_length : write_length;
            
            ux_utility_memory_copy(write_buffer, read_buffer, copy_len);

            /* 4. Commit Write: Tell Mic Stream the buffer is ready to send */
            ux_device_class_audio_write_frame_commit(g_mic_stream, copy_len);
        }
    }
  }

  /* -------------------------------------------------------------------------- */
  /* STEP 5: Free the Read Buffer (Crucial Step!)                               */
  /* -------------------------------------------------------------------------- */
  /* We must release the Speaker buffer so USBX can receive the next packet.    */
  ux_device_class_audio_read_frame_free(audio_play_stream);
}

(the first section is because I'm running a TouchGFX application as well, and the audio is sent to the GUI thread for visualization of the waveform, which works just fine)

For some reason, the ux_device_class_audio_write_frame_get(g_mic_stream, &write_buffer, &write_length) function always returns UX_BUFFER_OVERFLOW, even at the very first run when the micro is turned on (so the buffer should be empty)

I've tried to wipe the audio buffer of the microphone with

VOID USBD_AUDIO_RecordStreamChange(UX_DEVICE_CLASS_AUDIO_STREAM *audio_stream,
                                   ULONG alternate_setting)
{
  UX_SLAVE_ENDPOINT *endpoint;
  UCHAR *buffer;
  ULONG length;
  UINT status = UX_SUCCESS;

  if (alternate_setting == 0)
  {
    /* Stop Transmission */
    g_mic_stream = UX_NULL; // Invalidate the global handle

    endpoint = audio_stream->ux_device_class_audio_stream_endpoint;
    if (endpoint != UX_NULL)
    {
      _ux_device_stack_transfer_all_request_abort(endpoint, UX_ABORTED);
    }
  }
  else
  {
    /* Start Transmission */
    g_mic_stream = audio_stream; // Publish the stream handle

    /* Enable the class to accept write commands */
    status = ux_device_class_audio_transmission_start(audio_stream);

    if (ux_device_class_audio_write_frame_get(audio_stream, &buffer, &length) == UX_SUCCESS)
    {
        /* Fill with zeros (Silence) */
        ux_utility_memory_set(buffer, 0, length);
        
        /* Commit the silent frame so the thread has something to send */
        ux_device_class_audio_write_frame_commit(audio_stream, length);
    }

    if(status != UX_SUCCESS)
    	return;
  }
}

In this case, ux_device_class_audio_write_frame_get works for about two cycles (with the commit function being called), but on the host, no audio is received, and after the two cycles, it returns the UX_BUFFER_OVERFLOW issue.

Because I'm implementing a loopback interface, I have the same configuration (bitrate, channels, etc) on both IN and OUT interfaces, and the two FIFO queues have the same sizes (actually, I've tried to increase the microphone queue, as below, but got the same issue)

#define USBD_MAX_EP0_SIZE                             64U
#define USBD_AUDIO_REC_EPIN_HS_MPS                    28U  
HAL_PCDEx_SetTxFiFo(&hpcd_USB_OTG_HS, 0, USBD_MAX_EP0_SIZE/4);
HAL_PCDEx_SetTxFiFo(&hpcd_USB_OTG_HS, 1, USBD_AUDIO_REC_EPIN_HS_MPS);

 in the same way, the parameters for the frame buffer are the same for both interfaces

// speaker
ux_device_class_audio_stream_parameter_max_frame_buffer_nb = 3U
ux_device_class_audio_stream_parameter_max_frame_buffer_size = 28U
// microphone
ux_device_class_audio_stream_parameter_max_frame_buffer_nb = 3U
ux_device_class_audio_stream_parameter_max_frame_buffer_size = 28U

The thing I noticed is that, in the speaker interface, I've defined  ux_device_class_audio_stream_parameter_thread_entry = ux_device_class_audio_read_thread_entry;  and if I pause the firmware, I'm seeing the audio thread for the speaker with a high number of executions (as it should be).

Instead, even if I defined ux_device_class_audio_stream_parameter_thread_entry = ux_device_class_audio_write_thread_entry; for some reason it seems the thread is never executed (count is always 0).

Any idea on why the audio packets are not sent to the host (probably why the audio thread for mthe icrophone is not executed)?

[nico23]

Investigating more, it seems that the host (my Windows PC) doesn't send requests to access new microphone data (Windows never sends a single Isochronous IN token) and this is why the buffer is filled instantly.

Because Windows does not seem to request an audio packet from the microphone, the thread 

audio_stream_parameter[audio_stream_index].ux_device_class_audio_stream_parameter_thread_entry
	= ux_device_class_audio_write_thread_entry;

 is never called, and it is always SUSPENDED

I'm thinking the issue is somehow linked to the clock source not configured in the correct way as, I think I'm correctly filling and asking to send the microphone audio data

About the clock, I'm initializing both the speaker and the microphone with the same clock source

VOID USBD_AUDIO_SetControlValues(VOID)
{
  /* USER CODE BEGIN USBD_AUDIO_SetControlValues */
  /* Initialize audio 2.0 control values.  */
  audio_control[0].ux_device_class_audio20_control_cs_id                = USBD_AUDIO_PLAY_CLOCK_SOURCE_ID;
  audio_control[0].ux_device_class_audio20_control_sampling_frequency   = 0;
  audio_control[0].ux_device_class_audio20_control_sampling_frequency_cur = USBD_AUDIO_FREQ_48_K;
  audio_control[0].ux_device_class_audio20_control_sampling_frequency_range = sampling_freq_range;
  audio_control[0].ux_device_class_audio20_control_fu_id                = USBD_AUDIO_PLAY_FEATURE_UNIT_ID;
  audio_control[0].ux_device_class_audio20_control_mute[0]              = 0;
  audio_control[0].ux_device_class_audio20_control_volume_min[0]        = VOLUME_SPEAKER_MIN;
  audio_control[0].ux_device_class_audio20_control_volume_max[0]        = VOLUME_SPEAKER_MAX;
  audio_control[0].ux_device_class_audio20_control_volume_res[0]        = VOLUME_SPEAKER_RES;
  audio_control[0].ux_device_class_audio20_control_volume[0]            = VOLUME_SPEAKER_DEFAULT;

  audio_control[1].ux_device_class_audio20_control_cs_id                = USBD_AUDIO_REC_CLOCK_SOURCE_ID;
  audio_control[1].ux_device_class_audio20_control_sampling_frequency   = 0;
  audio_control[1].ux_device_class_audio20_control_sampling_frequency_cur = USBD_AUDIO_FREQ_48_K;
  audio_control[1].ux_device_class_audio20_control_sampling_frequency_range = sampling_freq_range;
  audio_control[1].ux_device_class_audio20_control_fu_id                = USBD_AUDIO_REC_FEATURE_UNIT_ID; // ID 38 (0x26)
  audio_control[1].ux_device_class_audio20_control_mute[0]              = 0;
  audio_control[1].ux_device_class_audio20_control_volume_min[0]        = 0;
  audio_control[1].ux_device_class_audio20_control_volume_max[0]        = 100;
  audio_control[1].ux_device_class_audio20_control_volume_res[0]        = 1;
  audio_control[1].ux_device_class_audio20_control_volume[0]            = 50;
  /* USER CODE END USBD_AUDIO_SetControlValues */

  return;
}

with

static UCHAR sampling_freq_range[] = {
    0x01, 0x00,              // wNumSubRanges = 1
    0x80, 0xBB, 0x00, 0x00,  // dMIN = 48000
    0x80, 0xBB, 0x00, 0x00,  // dMAX = 48000
    0x01, 0x00, 0x00, 0x00   // dRES = 1
};

On the descriptor, I have the clock configuration for the speaker

  /* Speaker Clock Source */
  pSpeakerCSDesc = ((USBD_AUDIOClockSourceDescTypeDef *)(pConf + *Sze));
  pSpeakerCSDesc->bLength = (uint8_t)sizeof(USBD_AUDIOClockSourceDescTypeDef);
  pSpeakerCSDesc->bDescriptorType = UX_DEVICE_CLASS_AUDIO_CS_INTERFACE;
  pSpeakerCSDesc->bDescriptorSubtype = UX_DEVICE_CLASS_AUDIO20_AC_CLOCK_SOURCE;
  pSpeakerCSDesc->bClockID = USBD_AUDIO_PLAY_CLOCK_SOURCE_ID;
  pSpeakerCSDesc->bmAttributes = 0x01U;
  pSpeakerCSDesc->bmControls = 0x01U;
  pSpeakerCSDesc->bAssocTerminal = 0x00U;
  pSpeakerCSDesc->iClockSource = 0x00U;
  *Sze += (uint32_t)sizeof(USBD_AUDIOClockSourceDescTypeDef);

but it's not set for the microphone. If I try to add it for the microphone in the same way using the same configuration but with  static USBD_AUDIOClockSourceDescTypeDef *pMicCSDesc; the descriptor seems to fail as Windows no more recognize the microphone and the speaker is shown as impossible to initialize with Error 10.

I've also tried to change the USBD_FrameWork_AssignEp for the microphone from USBD_EP_TYPE_ISOC|USBD_EP_ATTR_ISOC_NOSYNC, which is the one used on the speaker to 

USBD_FrameWork_AssignEp(pdev,
    USBD_AUDIO_REC_EPIN_ADDR,
    USBD_EP_TYPE_ISOC | USBD_EP_ATTR_ISOC_ASYNC,
    USBD_AUDIO_REC_EPIN_HS_MPS);

But again, nothing changes, and audio data on the microphone buffer is not requested by the host.

I've also tried to change the clock ID for the microphone, instead of using the same as the speaker. So, I've used

#define USBD_AUDIO_PLAY_CLOCK_SOURCE_ID               0x18U
#define USBD_AUDIO_REC_CLOCK_SOURCE_ID                0x19U

instead

#define USBD_AUDIO_PLAY_CLOCK_SOURCE_ID               0x18U
#define USBD_AUDIO_REC_CLOCK_SOURCE_ID                USBD_AUDIO_PLAY_CLOCK_SOURCE_ID

and the same clockID is changed in the descriptor because it is linked with

pMicITDesc->bCSourceID = USBD_AUDIO_REC_CLOCK_SOURCE_ID;

pMicOTDesc->bCSourceID = USBD_AUDIO_REC_CLOCK_SOURCE_ID;

But still Error 10 code when trying to connect to the host.

From the USB trace using Wireshark, I'm seeing that there's no

SET_INTERFACE
  Interface = Microphone AS interface
  AlternateSetting = 1

but, when in debug, I'm seeing that the callback

audio_stream_parameter[audio_stream_index].ux_device_class_audio_stream_parameter_callbacks.ux_device_class_audio_stream_change
	= USBD_AUDIO_RecordStreamChange;

// in USBD_AUDIO_RecordStreamChange
status = ux_device_class_audio_transmission_start(audio_stream);

is being correctly called

Is there a particular thing I have to watch/set for the clock source on a microphone USB Audio device?

[nico23]

And... as my previous post https://community.st.com/t5/stm32-mcus-embedded-software/usbx-audio-class-frame-done-callback-not-called/td-p/855262 even this time the issue was a bug on the USBX Audio API

I'm pretty impressed that nobody is facing these issues... Maybe nobody is using them?

Anyway, in the function UINT _ux_device_class_audio_transmission_start(UX_DEVICE_CLASS_AUDIO_STREAM *stream)

    /* Check if there is frame to send (underflow).  */
    if (stream -> ux_device_class_audio_stream_transfer_pos -> ux_device_class_audio_frame_length == 0)
        return(UX_BUFFER_OVERFLOW);

The code is wrong because it always returns UX_BUFFER_OVERFLOW

The fix is to set

    /* Check if there is frame to send (underflow).  */
    if (stream -> ux_device_class_audio_stream_transfer_pos -> ux_device_class_audio_frame_length > 0)
        return(UX_BUFFER_OVERFLOW);

And everything, magically, starts working

Maybe it's the magic of Christmas

[FBL]

Hi @nico23 

An internal ticket 224405 is submitted to dedicated team to allow check only when there is data pending (not even getting started) 

Would you attach minimum firmware to reproduce the issue ?

[nico23]

At the moment, I'm unable to produce a minimum firmware as the whole setup includes the loopback configuration.

I'm planning to open-source the project when it is working properly.

Anyway, another thing I've done to fix the issue I haven't mentioned in this post (besides the USBX API) was to get rid of the clock configuration in the microphone SetControlValues

audio_control[1].ux_device_class_audio20_control_cs_id                = USBD_AUDIO_REC_CLOCK_SOURCE_ID;
  audio_control[1].ux_device_class_audio20_control_sampling_frequency   = 0;
  audio_control[1].ux_device_class_audio20_control_sampling_frequency_cur = USBD_AUDIO_FREQ_48_K;
  audio_control[1].ux_device_class_audio20_control_sampling_frequency_range = sampling_freq_range;

and switch in the descriptor the USBD_EP_ATTR_ISOC_NOSYNC to USBD_EP_ATTR_ISOC_ASYNC for the microphone

Correct code:

      if (pdev->Speed == USBD_HIGH_SPEED)
      {
        /* Assign OUT Endpoint */
        USBD_FrameWork_AssignEp(pdev, USBD_AUDIO_PLAY_EPOUT_ADDR,
                                USBD_EP_TYPE_ISOC|USBD_EP_ATTR_ISOC_NOSYNC,
                                USBD_AUDIO_PLAY_EPOUT_HS_MPS);

        /* Assign IN Endpoint */
        USBD_FrameWork_AssignEp(pdev, USBD_AUDIO_REC_EPIN_ADDR,
                                USBD_EP_TYPE_ISOC|USBD_EP_ATTR_ISOC_ASYNC,
                                USBD_AUDIO_REC_EPIN_HS_MPS);

      }
      else
      {
        /* Assign OUT Endpoint */
        USBD_FrameWork_AssignEp(pdev, USBD_AUDIO_PLAY_EPOUT_ADDR,
                                USBD_EP_TYPE_ISOC|USBD_EP_ATTR_ISOC_NOSYNC,
                                USBD_AUDIO_PLAY_EPOUT_FS_MPS);
        
        /* Assign IN Endpoint */
        USBD_FrameWork_AssignEp(pdev, USBD_AUDIO_REC_EPIN_ADDR,
                                USBD_EP_TYPE_ISOC|USBD_EP_ATTR_ISOC_ASYNC,
                                USBD_AUDIO_REC_EPIN_FS_MPS);
      }