Describe the bug
The rate error > 200 for capture device hw:0,99 measured by alsa_conformance_test for firmware w/ IGO

To Reproduce
Run the following command and check the rate error:

alsa_conformance_test -Chw:0,99 -c4 -fS32_LE

Reproduction Rate
How often does the issue happen ? always

Expected behavior
The rate error should be low (e.g. < 1 measured on other TGL board w/o IGO)

Impact
The performance of audio would drop

Environment
tgl-13 branch with IGO patches & IGO binary
Issue still reproduced with PR #4575

Screenshots or console output
---------PRINT PARAMS---------
PCM name: hw:0,99
card: sofrt5682 [sof-rt5682]
device: DMIC (*) []
stream: CAPTURE
merge_threshold_t: 0.000100
merge_threshold_sz: 240
access type: MMAP_INTERLEAVED
format: S32_LE
channels: 4
rate: 48000 fps
period time: 5333 us
period size: 256 frames
buffer time: 85333 us
buffer size: 4096 frames
---------TIMER RESULT---------
Total_time(s) Counts Averages(s)
snd_pcm_open 0.199339453 2 0.099670
snd_pcm_hw_params 0.004043824 2 0.002022
snd_pcm_hw_params_any 0.000015971 2 0.000008
snd_pcm_sw_params 0.000001639 2 0.000001
snd_pcm_prepare 0.000029138 2 0.000015
snd_pcm_start 0.000916965 2 0.000458
snd_pcm_avail 2.023577778 77075 0.000026
precision: 0.000000001
----------RUN RESULT----------
number of recorders: 1
number of points: 153
step average: 316.394737
step min: 8
step max: 768
step median: 284
step standard deviation: 195.473385
rate: 47906.655167
rate error: 285.501755
zero channels: 0 0 0 0
number of underrun: 0
number of overrun: 0