Skip to content

Unable to Pretrain BERT using run_pretraining_lamb.sh #510

Description

@Esaada

Since my issue was closed and wasn't reopen when I comment:
I followed the steps described in the quickstart in Bert's Readme:
[(https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT#quick-start-guide.)]
After a long preprocessing fight , I finally got to pertaining part I used this line:
bash scripts/run_pretraining_lamb.sh 64 8 8 7.5e-4 5e-4 fp32 true 8 2000 200 7820 100 128 512 large

I'm getting two errors;

Traceback (most recent call last):
File "/workspace/bert/run_pretraining.py", line 30, in
import utils.dllogger_class
ModuleNotFoundError: No module named 'utils.dllogger_class'

I tried to google it but I got only java stuff..
And :

Error! /results/tf_bert_pretraining_lamb_large_fp32_gbs165536_gbs232768_200514070735/phase_1/model.ckpt-7038.meta directory missing. Please mount correctly

of course that directory :/results/tf_bert_pretraining_lamb_large_fp32_gbs165536_gbs232768_200514070735/phase_1/
does exists.
The fix:
#502
Did not solve my problem.
To be sure, I even re-cloned the repo.
Still getting the same errors.

I'm using 8 V100 (not DGX) with 16gb.
The container is as requested.
**tf version:**1.14
I'll be happy for your help.
Thanks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions