I don't understand why you are using 24 bits in 32 bit frame. The PDM format is not aware of the frame length. That way you are dropping 8 bits from each 32 bits received. That must affect the sound quality. You should use all bits in the frame.