Skip to content

CUFFT_INTERNAL_ERROR #2956

@programindz

Description

@programindz

Describe the bug

I'm trying to fine-tune the 'lang-id-voxlingua107-ecapa' embedding model, for 21 indian languages but as soon as the training hits i get the cufft error. I initially thought this could be problem because of small duration audios, so i filtered out them. This error still exists:

Is there any lower limit for audio duration we need the audios to be in??

Expected behaviour

This is the error:

Traceback (most recent call last):
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 310, in
lid_brain.fit(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1575, in fit
self._fit_train(train_set=train_set, epoch=epoch, enable=enable)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1400, in _fit_train
loss = self.fit_batch(batch)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1199, in fit_batch
outputs = self.compute_forward(batch, sb.Stage.TRAIN)
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 78, in compute_forward
feats, lens = self.prepare_features(batch.sig, stage)
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 49, in prepare_features
wavs, lens = self.hparams.wav_augment(wavs, lens)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/augmenter.py", line 392, in forward
output, output_lengths = self.augment(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/augmenter.py", line 252, in augment
out = augment_fun(next_input[idx, ...])
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/time_domain.py", line 409, in forward
rev_waveform = reverberate(waveforms, rir_waveform, rescale_amp="avg")
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/processing/signal_processing.py", line 372, in reverberate
waveforms = convolve1d(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/processing/signal_processing.py", line 287, in convolve1d
result = fft.rfft(waveform) * fft.rfft(kernel)
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

To Reproduce

No response

Environment Details

No response

Relevant Log Output

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions