Describe the bug
Context
We are performing a custom ASR training based on CommonVoice/CTC recipe.
For this test, the dataset is composed of 25k records for Train, 3k records for Val, 3,2k records for Test (dynamic_batch_length 80).
Our complete dataset is composed of 220k records. We are training on 3 x A100 40Gb.
Problem
With Speechbrain 1.0.2
- using 1 GPU, each epoch last around 2:30, and val 1:20:
100%|██████████| 523/523 [02:35<00:00, 3.37it/s, train_loss=2.72]
100%|██████████| 417/417 [01:20<00:00, 5.17it/s]
- using 3 GPUs, each epoch last also 1:10, and val 1:15s (usually 2.5 time faster as expected):
100%|██████████| 175/175 [01:10<00:00, 2.49it/s, train_loss=3.36]
100%|██████████| 417/417 [01:14<00:00, 5.62it/s]
With Speechbrain 1.1.0
- using 1 GPU, each epoch last around 3:45, and val 0:40:
100%|██████████| 523/523 [03:44<00:00, 2.33it/s, train_loss=2.72]
100%|██████████| 417/417 [00:38<00:00, 10.80it/s]
- using 3 GPUs, each epoch last also 3:45, and val 0:40 (no change!):
100%|██████████| 175/175 [03:47<00:00, 1.30s/it, train_loss=3.37]
100%|██████████| 417/417 [00:36<00:00, 11.45it/s]
Conclusion
Although the validation step appears more efficient, single-GPU training performance has decreased by 50%, and multi-GPU training has become inefficient. This issue prevents us from upgrading to SpeechBrain 1.1.0, as our training times are three times longer compared to SpeechBrain 1.0.2 with the same dataset, recipe, hyperparameters, and torch/transformers versions.
Expected behaviour
Equal or better training performance
To Reproduce
Training command line (for multi-GPU)
CUDA_VISIBLE_DEVICES=0,1,2 torchrun --nproc_per_node=3 --master_port=25000 /data/sb/wav2vec2/train_with_wav2vec.py /data/sb/wav2vec2/hyperparams_train_w2v2.yaml --wav2vec2_folder=/data/wav2vec2_checkpoints/ --seed=1234 --skip_prep=True --precision=bf16 --wav2vec2_hub=microsoft/wavlm-large --language=en --max_batch_length_train=80.0 --max_batch_length_val=20.0 --number_of_epochs=2 --test_beam_search={kenlm_model_path: "/data/lm/lm_train.arpa"} --token_type=bpe --avoid_if_longer_than=10.0 --data_folder=/data/dataset_test --train_csv=/data/dataset_test/dataset_train.csv --valid_csv=/data/dataset_test/dataset_val.csv --test_csv=/data/dataset_test/dataset_test.csv --output_neurons=60 --output_folder=/data/output --find_unused_parameters
hyperparams_train file:
# ################################
# Model: wav2vec2 + DNN + CTC
# Augmentation: SpecAugment
# Authors: Titouan Parcollet 2021
# ################################
# Seed needs to be set at top of yaml, before objects with parameters are made
seed: 1234
__set_seed: !apply:speechbrain.utils.seed_everything [!ref <seed>]
output_folder: !PLACEHOLDER
test_wer_file: !ref <output_folder>/wer_test.txt
save_folder: !ref <output_folder>/save
train_log: !ref <output_folder>/train_log.txt
# URL for the biggest Fairseq english wav2vec2 model.
wav2vec2_hub: !PLACEHOLDER
wav2vec2_folder: !ref <save_folder>/wav2vec2_checkpoint
# Data files
data_folder: !PLACEHOLDER # e.g, /localscratch/cv-corpus-5.1-2020-06-22/fr
train_tsv_file: !ref <data_folder>/train.tsv # Standard CommonVoice .tsv files
dev_tsv_file: !ref <data_folder>/dev.tsv # Standard CommonVoice .tsv files
test_tsv_file: !ref <data_folder>/test.tsv # Standard CommonVoice .tsv files
accented_letters: True
language: !PLACEHOLDER # use 'it' for Italian, 'rw' for Kinyarwanda, 'en' for english
train_csv: !ref <save_folder>/train.csv
valid_csv: !ref <save_folder>/dev.csv
test_csv: !ref <save_folder>/test.csv
skip_prep: True # Skip data preparation
# We remove utterance slonger than 10s in the train/dev/test sets as
# longer sentences certainly correspond to "open microphones".
avoid_if_longer_than: !PLACEHOLDER
####################### Training Parameters ####################################
number_of_epochs: !PLACEHOLDER
lr: 1.0
lr_wav2vec: 0.0001
precision: bf16 # bf16, fp16 or fp32
sample_rate: 16000
ckpt_interval_minutes: 30 # save checkpoint every N min
# We use Dynamic Batching here as CV english is quite big.
# Or turn it off (but training speed will decrease)
dynamic_batching: True
max_batch_length_train: 200
max_batch_length_val: 50 # we reduce it as the beam is much wider (VRAM)
num_bucket: 200
shuffle: True # if true re-creates batches at each epoch shuffling examples.
batch_ordering: random
max_batch_ex: 256
num_workers: 4
dynamic_batch_sampler_train:
max_batch_length: !ref <max_batch_length_train>
num_buckets: !ref <num_bucket>
shuffle: !ref <shuffle>
batch_ordering: !ref <batch_ordering>
max_batch_ex: !ref <max_batch_ex>
dynamic_batch_sampler_valid:
max_batch_length: !ref <max_batch_length_val>
num_buckets: !ref <num_bucket>
shuffle: !ref <shuffle>
batch_ordering: !ref <batch_ordering>
max_batch_ex: !ref <max_batch_ex>
# If dynamic batching is False, we use these instead
batch_size: 12
test_batch_size: 4
sorting: ascending
dataloader_options:
batch_size: !ref <batch_size>
num_workers: !ref <num_workers>
test_dataloader_options:
batch_size: !ref <test_batch_size>
num_workers: 6
# BPE parameters
token_type: bpe # ["unigram", "bpe", "char"]
character_coverage: 1.0
####################### Model Parameters #######################################
# activation: !name:torch.nn.LeakyReLU
wav2vec_output_dim: 1024
dnn_neurons: 1024
freeze_wav2vec: False
freeze_feature_extractor: False
dropout: 0.15
warmup_steps: 500
# Outputs
output_neurons: 28 # BPE size, index(blank/eos/bos) = 0
# Decoding parameters
# Be sure that the bos and eos index match with the BPEs ones
blank_index: 0
bos_index: 1
eos_index: 2
# Decoding parameters
test_beam_search:
blank_index: !ref <blank_index>
beam_size: 100
beam_prune_logp: -12.0
token_prune_min_logp: -1.2
prune_history: False
topk: 1
alpha: 1.0
beta: 0.5
# To use n-gram LM for decoding, follow steps in README.md.
# kenlm_model_path: none
#
# Functions and classes
#
epoch_counter: !new:speechbrain.utils.epoch_loop.EpochCounter
limit: !ref <number_of_epochs>
############################## Augmentations ###################################
# Speed perturbation
speed_perturb: !new:speechbrain.augment.time_domain.SpeedPerturb
orig_freq: !ref <sample_rate>
speeds: [95, 100, 105]
# Frequency drop: randomly drops a number of frequency bands to zero.
drop_freq: !new:speechbrain.augment.time_domain.DropFreq
drop_freq_low: 0
drop_freq_high: 1
drop_freq_count_low: 1
drop_freq_count_high: 3
drop_freq_width: 0.05
# Time drop: randomly drops a number of temporal chunks.
drop_chunk: !new:speechbrain.augment.time_domain.DropChunk
drop_length_low: 1000
drop_length_high: 2000
drop_count_low: 1
drop_count_high: 5
# Augmenter: Combines previously defined augmentations to perform data augmentation
wav_augment: !new:speechbrain.augment.augmenter.Augmenter
min_augmentations: 3
max_augmentations: 3
augment_prob: 1.0
augmentations: [
!ref <speed_perturb>,
!ref <drop_freq>,
!ref <drop_chunk>]
############################## Models ##########################################
enc: !new:speechbrain.nnet.containers.Sequential
input_shape: [null, null, !ref <wav2vec_output_dim>]
linear1: !name:speechbrain.nnet.linear.Linear
n_neurons: !ref <dnn_neurons>
bias: True
bn1: !name:speechbrain.nnet.normalization.BatchNorm1d
activation: !new:torch.nn.LeakyReLU
drop: !new:torch.nn.Dropout
p: !ref <dropout>
linear2: !name:speechbrain.nnet.linear.Linear
n_neurons: !ref <dnn_neurons>
bias: True
bn2: !name:speechbrain.nnet.normalization.BatchNorm1d
activation2: !new:torch.nn.LeakyReLU
drop2: !new:torch.nn.Dropout
p: !ref <dropout>
linear3: !name:speechbrain.nnet.linear.Linear
n_neurons: !ref <dnn_neurons>
bias: True
bn3: !name:speechbrain.nnet.normalization.BatchNorm1d
activation3: !new:torch.nn.LeakyReLU
# for sb 1.0.2 : wav2vec2: !new:speechbrain.lobes.models.huggingface_transformers.wav2vec2.Wav2Vec2
wav2vec2: !new:speechbrain.integrations.huggingface.wav2vec2.Wav2Vec2
source: !ref <wav2vec2_hub>
output_norm: True
freeze: !ref <freeze_wav2vec>
freeze_feature_extractor: !ref <freeze_feature_extractor>
save_path: !ref <wav2vec2_folder>
#####
# Uncomment this block if you prefer to use a Fairseq pretrained model instead
# of a HuggingFace one. Here, we provide an URL that is obtained from the
# Fairseq github for the multilingual XLSR.
#
#wav2vec2_url: https://dl.fbaipublicfiles.com/fairseq/wav2vec/xlsr_53_56k.pt
#wav2vec2: !new:speechbrain.lobes.models.fairseq_wav2vec.FairseqWav2Vec2
# pretrained_path: !ref <wav2vec2_url>
# output_norm: True
# freeze: False
# save_path: !ref <save_folder>/wav2vec2_checkpoint/model.pt
#####
ctc_lin: !new:speechbrain.nnet.linear.Linear
input_size: !ref <dnn_neurons>
n_neurons: !ref <output_neurons>
log_softmax: !new:speechbrain.nnet.activations.Softmax
apply_log: True
ctc_cost: !name:speechbrain.nnet.losses.ctc_loss
blank_index: !ref <blank_index>
modules:
wav2vec2: !ref <wav2vec2>
enc: !ref <enc>
ctc_lin: !ref <ctc_lin>
model: !new:torch.nn.ModuleList
- [!ref <enc>, !ref <ctc_lin>]
model_opt_class: !name:torch.optim.Adadelta
lr: !ref <lr>
rho: 0.95
eps: 1.e-8
# for sb1.0.2 : wav2vec_opt_class: !name:torch.optim.AdamW
wav2vec_opt_class: !name:torch.optim.Adam
lr: !ref <lr_wav2vec>
lr_annealing_model: !new:speechbrain.nnet.schedulers.NewBobScheduler
initial_value: !ref <lr>
improvement_threshold: 0.0025
annealing_factor: 0.8
patient: 0
lr_annealing_wav2vec: !new:speechbrain.nnet.schedulers.NewBobScheduler
initial_value: !ref <lr_wav2vec>
improvement_threshold: 0.0025
annealing_factor: 0.9
patient: 0
checkpointer: !new:speechbrain.utils.checkpoints.Checkpointer
checkpoints_dir: !ref <save_folder>
recoverables:
wav2vec2: !ref <wav2vec2>
model: !ref <model>
scheduler_model: !ref <lr_annealing_model>
scheduler_wav2vec: !ref <lr_annealing_wav2vec>
counter: !ref <epoch_counter>
train_logger: !new:speechbrain.utils.train_logger.FileTrainLogger
save_file: !ref <train_log>
error_rate_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
cer_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
split_tokens: True
Environment Details
Ubuntu 22.04.3
3 x Nvidia A100 GPU
Driver Version: 545.23.08
CUDA Version: 12.3
Python 3.11.13
torch==2.2.2
torchaudio==2.2.2
torchvision==0.17.2
torchcodec==0.0.3
transformers==4.51.3
Relevant Log Output
Additional Context
At each launch, we have the following warning (that seems without impact using sb1.0.2):
[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Without the --find_unused_parameters option, training stops immediately with this error:
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument 'find_unused_parameters=True' to 'torch.nn.parallel.DistributedDataParallel', and by making sure all 'forward' function outputs participate in calculating loss.
Describe the bug
Context
We are performing a custom ASR training based on CommonVoice/CTC recipe.
For this test, the dataset is composed of 25k records for Train, 3k records for Val, 3,2k records for Test (dynamic_batch_length 80).
Our complete dataset is composed of 220k records. We are training on 3 x A100 40Gb.
Problem
With Speechbrain 1.0.2
With Speechbrain 1.1.0
Conclusion
Although the validation step appears more efficient, single-GPU training performance has decreased by 50%, and multi-GPU training has become inefficient. This issue prevents us from upgrading to SpeechBrain 1.1.0, as our training times are three times longer compared to SpeechBrain 1.0.2 with the same dataset, recipe, hyperparameters, and torch/transformers versions.
Expected behaviour
Equal or better training performance
To Reproduce
Training command line (for multi-GPU)
CUDA_VISIBLE_DEVICES=0,1,2 torchrun --nproc_per_node=3 --master_port=25000 /data/sb/wav2vec2/train_with_wav2vec.py /data/sb/wav2vec2/hyperparams_train_w2v2.yaml --wav2vec2_folder=/data/wav2vec2_checkpoints/ --seed=1234 --skip_prep=True --precision=bf16 --wav2vec2_hub=microsoft/wavlm-large --language=en --max_batch_length_train=80.0 --max_batch_length_val=20.0 --number_of_epochs=2 --test_beam_search={kenlm_model_path: "/data/lm/lm_train.arpa"} --token_type=bpe --avoid_if_longer_than=10.0 --data_folder=/data/dataset_test --train_csv=/data/dataset_test/dataset_train.csv --valid_csv=/data/dataset_test/dataset_val.csv --test_csv=/data/dataset_test/dataset_test.csv --output_neurons=60 --output_folder=/data/output --find_unused_parametershyperparams_train file:
Environment Details
Ubuntu 22.04.3
3 x Nvidia A100 GPU
Driver Version: 545.23.08
CUDA Version: 12.3
Python 3.11.13
torch==2.2.2
torchaudio==2.2.2
torchvision==0.17.2
torchcodec==0.0.3
transformers==4.51.3
Relevant Log Output
Additional Context
At each launch, we have the following warning (that seems without impact using sb1.0.2):
[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())Without the
--find_unused_parametersoption, training stops immediately with this error:RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument 'find_unused_parameters=True' to 'torch.nn.parallel.DistributedDataParallel', and by making sure all 'forward' function outputs participate in calculating loss.