Skip to content

Adding DLLogger #469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nvpstr merged 3 commits into from
May 7, 2020
Merged

Adding DLLogger #469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eb8e823
Merge pull request #7 from NVIDIA/master
swethmandava Apr 24, 2020
c94b73f
Adding DLLogger and specificying v1.1 in the readme results for clarity
Apr 24, 2020
b7903f0
Inference throughput computation removing outliers, eval when eval sc…
Apr 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion TensorFlow/LanguageModeling/BERT/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ ARG FROM_IMAGE_NAME=nvcr.io/nvidia/tensorflow:19.10-py3
FROM ${FROM_IMAGE_NAME}

RUN apt-get update && apt-get install -y pbzip2 pv bzip2 libcurl4 curl

RUN pip install --upgrade pip
RUN pip install toposort networkx pytest nltk tqdm html2text progressbar
RUN pip --no-cache-dir --no-cache install git+https://github.com/NVIDIA/dllogger

WORKDIR /workspace
RUN git clone https://github.com/openai/gradient-checkpointing.git
Expand Down
40 changes: 20 additions & 20 deletions TensorFlow/LanguageModeling/BERT/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,34 +38,34 @@ This repository provides a script and recipe to train the BERT model for TensorF
* [Training accuracy results](#training-accuracy-results)
* [Pre-training accuracy: single-node](#pre-training-accuracy-single-node)
* [Pre-training accuracy: multi-node](#pre-training-accuracy-multi-node)
* [Fine-tuning accuracy for SQuAD: NVIDIA DGX-2 (16x V100 32G)](#fine-tuning-accuracy-for-squad-nvidia-dgx-2-16x-v100-32g)
* [Fine-tuning accuracy for SQuAD v1.1: NVIDIA DGX-2 (16x V100 32G)](#fine-tuning-accuracy-for-squad-v1.1-nvidia-dgx-2-16x-v100-32g)
* [Training stability test](#training-stability-test)
* [Pre-training SQuAD stability test: NVIDIA DGX-2 (512x V100 32G)](#fine-tuning-squad-stability-test-nvidia-dgx-2-512x-v100-32g)
* [Fine-tuning SQuAD stability test: NVIDIA DGX-2 (16x V100 32G)](#fine-tuning-squad-stability-test-nvidia-dgx-2-16x-v100-32g)
* [Pre-training SQuAD v1.1 stability test: NVIDIA DGX-2 (512x V100 32G)](#pre-training-squad-v1.1-stability-test-nvidia-dgx-2-512x-v100-32g)
* [Fine-tuning SQuAD v1.1 stability test: NVIDIA DGX-2 (16x V100 32G)](#fine-tuning-squad-v1.1-stability-test-nvidia-dgx-2-16x-v100-32g)
* [Training performance results](#training-performance-results)
* [Training performance: NVIDIA DGX-1 (8x V100 16G)](#training-performance-nvidia-dgx-1-8x-v100-16g)
* [Pre-training training performance: single-node on 16G](#pre-training-training-performance-single-node-on-16g)
* [Pre-training training performance: multi-node on 16G](#pre-training-training-performance-multi-node-on-16g)
* [Fine-tuning training performance for SQuAD on 16G](#fine-tuning-training-performance-for-squad-on-16g)
* [Fine-tuning training performance for SQuAD v1.1 on 16G](#fine-tuning-training-performance-for-squad-v1.1-on-16g)
* [Training performance: NVIDIA DGX-1 (8x V100 32G)](#training-performance-nvidia-dgx-1-8x-v100-32g)
* [Pre-training training performance: single-node on 32G](#pre-training-training-performance-single-node-on-32g)
* [Fine-tuning training performance for SQuAD on 32G](#fine-tuning-training-performance-for-squad-on-32g)
* [Fine-tuning training performance for SQuAD v1.1 on 32G](#fine-tuning-training-performance-for-squad-v1.1-on-32g)
* [Training performance: NVIDIA DGX-2 (16x V100 32G)](#training-performance-nvidia-dgx-2-16x-v100-32g)
* [Pre-training training performance: single-node on DGX-2 32G](#pre-training-training-performance-single-node-on-dgx-2-32g)
* [Pre-training training performance: multi-node on DGX-2 32G](#pre-training-training-performance-multi-node-on-dgx-2-32g)
* [Fine-tuning training performance for SQuAD on DGX-2 32G](#fine-tuning-training-performance-for-squad-on-dgx-2-32g)
* [Fine-tuning training performance for SQuAD v1.1 on DGX-2 32G](#fine-tuning-training-performance-for-squad-v1.1-on-dgx-2-32g)
* [Inference performance results](#inference-performance-results)
* [Inference performance: NVIDIA DGX-1 (1x V100 16G)](#inference-performance-nvidia-dgx-1-1x-v100-16g)
* [Pre-training inference performance on 16G](#pre-training-inference-performance-on-16g)
* [Fine-tuning inference performance for SQuAD on 16G](#fine-tuning-inference-performance-for-squad-on-16g)
* [Fine-tuning inference performance for SQuAD v1.1 on 16G](#fine-tuning-inference-performance-for-squad-v1.1-on-16g)
* [Inference performance: NVIDIA DGX-1 (1x V100 32G)](#inference-performance-nvidia-dgx-1-1x-v100-32g)
* [Pre-training inference performance on 32G](#pre-training-inference-performance-on-32g)
* [Fine-tuning inference performance for SQuAD on 32G](#fine-tuning-inference-performance-for-squad-on-32g)
* [Fine-tuning inference performance for SQuAD v1.1 on 32G](#fine-tuning-inference-performance-for-squad-v1.1-on-32g)
* [Inference performance: NVIDIA DGX-2 (1x V100 32G)](#inference-performance-nvidia-dgx-2-1x-v100-32g)
* [Pre-training inference performance on DGX-2 32G](#pre-training-inference-performance-on-dgx-2-32g)
* [Fine-tuning inference performance for SQuAD on DGX-2 32G](#fine-tuning-inference-performance-for-squad-on-dgx-2-32g)
* [Fine-tuning inference performance for SQuAD v1.1 on DGX-2 32G](#fine-tuning-inference-performance-for-squad-v1.1-on-dgx-2-32g)
* [Inference performance: NVIDIA Tesla T4 (1x T4 16G)](#inference-performance-nvidia-tesla-t4-1x-t4-16g)
* [Fine-tuning inference performance for SQuAD on Tesla T4 16G](#fine-tuning-inference-performance-for-squad-on-tesla-t4-16g)
* [Fine-tuning inference performance for SQuAD v1.1 on Tesla T4 16G](#fine-tuning-inference-performance-for-squad-v1.1-on-tesla-t4-16g)
- [Release notes](#release-notes)
* [Changelog](#changelog)
* [Known issues](#known-issues)
Expand Down Expand Up @@ -692,7 +692,7 @@ Our results were obtained by running the `scripts/run_pretraining_lamb.sh` train

Note: Time to train includes upto 16 minutes of start up time for every restart. Experiments were run on clusters with a maximum wall clock time of 8 hours.

###### Fine-tuning accuracy for SQuAD: NVIDIA DGX-2 (16x V100 32G)
###### Fine-tuning accuracy for SQuAD v1.1: NVIDIA DGX-2 (16x V100 32G)

Our results were obtained by running the `scripts/run_squad.sh` training script in the TensorFlow 19.08-py3 NGC container on NVIDIA DGX-2 with 16x V100 32G GPUs.

Expand All @@ -704,15 +704,15 @@ Our results were obtained by running the `scripts/run_squad.sh` training script

##### Training stability test

###### Pre-training stability test: NVIDIA DGX-2 (512x V100 32G)
###### Pre-training SQuAD v1.1 stability test: NVIDIA DGX-2 (512x V100 32G)

The following tables compare `Final Loss` scores across 5 different training runs with different seeds, for both FP16. The runs showcase consistent convergence on all 5 seeds with very little deviation.

| **FP16, 512x GPUs** | **seed 1** | **seed 2** | **seed 3** | **seed 4** | **seed 5** | **mean** | **std** |
|:-----------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
|Final Loss |1.57 |1.598 |1.614 |1.583 |1.584 |1.5898|0.017 |

###### Fine-tuning SQuAD stability test: NVIDIA DGX-2 (16x V100 32G)
###### Fine-tuning SQuAD v1.1 stability test: NVIDIA DGX-2 (16x V100 32G)

The following tables compare `F1` scores across 5 different training runs with different seeds, for both FP16 and FP32 respectively. The runs showcase consistent convergence on all 5 seeds with very little deviation.

Expand Down Expand Up @@ -764,7 +764,7 @@ Our results were obtained by running the `run.sub` training script in the Tensor

Note: The respective values for FP32 runs that use a batch size of 16, 2 in sequence lengths 128 and 512 respectively are not available due to out of memory errors that arise.

###### Fine-tuning training performance for SQuAD on 16G
###### Fine-tuning training performance for SQuAD v1.1 on 16G

Our results were obtained by running the `scripts/run_squad.sh` training script in the TensorFlow 19.08-py3 NGC container on NVIDIA DGX-1 with 8x V100 16G GPUs. Performance (in sentences per second) is the mean throughput from 2 epochs.

Expand Down Expand Up @@ -795,7 +795,7 @@ Our results were obtained by running the `scripts/run_pretraining_lamb.sh` train

Note: The respective values for FP32 runs that use a batch size of 48, 8 in sequence lengths 128 and 512 respectively are not available due to out of memory errors that arise.

###### Fine-tuning training performance for SQuAD on 32G
###### Fine-tuning training performance for SQuAD v1.1 on 32G

Our results were obtained by running the `scripts/run_squad.sh` training script in the TensorFlow 19.08-py3 NGC container on NVIDIA DGX-1 with 8x V100 32G GPUs. Performance (in sentences per second) is the mean throughput from 2 epochs.

Expand Down Expand Up @@ -848,7 +848,7 @@ Our results were obtained by running the `run.sub` training script in the Tensor
| 64 | 512 | 8,4 | 19766.57 | 6029.48 | 3.28 | 51.56 | 54.83 |


###### Fine-tuning training performance for SQuAD on DGX-2 32G
###### Fine-tuning training performance for SQuAD v1.1 on DGX-2 32G

Our results were obtained by running the `scripts/run_squad.sh` training script in the TensorFlow 19.08-py3 NGC container on NVIDIA DGX-2 with 16x V100 32G GPUs. Performance (in sentences per second) is the mean throughput from 2 epochs.

Expand Down Expand Up @@ -876,7 +876,7 @@ Our results were obtained by running the `scripts/run_pretraining_lamb.sh` scrip
|:-----:|:-------:|:-------:|:-------:|:-------------:|
|128 |8, 8 |349.51 | 104.31 | 3.35 |

###### Fine-tuning inference performance for SQuAD on 16G
###### Fine-tuning inference performance for SQuAD v1.1 on 16G

Our results were obtained by running the `scripts/finetune_inference_benchmark.sh` script in the TensorFlow 19.08-py3 NGC container on NVIDIA DGX-1 with 1x V100 16G GPUs. Performance numbers (throughput in sentences per second and latency in milliseconds) were averaged from 1024 iterations. Latency is computed as the time taken for a batch to process as they are fed in one after another in the model ie no pipelining.

Expand Down Expand Up @@ -945,7 +945,7 @@ Our results were obtained by running the `scripts/run_pretraining_lamb.sh` scrip
|:-----:|:-------:|:-------:|:-------:|:-------------:|
|128 |8, 8 |345.50 | 101.84 | 3.39 |

###### Fine-tuning inference performance for SQuAD on 32G
###### Fine-tuning inference performance for SQuAD v1.1 on 32G

Our results were obtained by running the `scripts/finetune_inference_benchmark.sh` training script in the TensorFlow 19.08-py3 NGC container on NVIDIA DGX-1 with 1x V100 32G GPUs. Performance numbers (throughput in sentences per second and latency in milliseconds) were averaged from 1024 iterations. Latency is computed as the time taken for a batch to process as they are fed in one after another in the model ie no pipelining.

Expand Down Expand Up @@ -1014,7 +1014,7 @@ Our results were obtained by running the `scripts/run_pretraining_lamb.sh` scrip
|:-----:|:-------:|:-------:|:-------:|:-------------:|
|128 |8, 8 |366.24 | 107.88 | 3.39 |

###### Fine-tuning inference performance for SQuAD on DGX-2 32G
###### Fine-tuning inference performance for SQuAD v1.1 on DGX-2 32G

Our results were obtained by running the `scripts/finetune_inference_benchmark.sh` training script in the TensorFlow 19.08-py3 NGC container on NVIDIA DGX-2 with 1x V100 32G GPUs. Performance numbers (throughput in sentences per second and latency in milliseconds) were averaged from 1024 iterations. Latency is computed as the time taken for a batch to process as they are fed in one after another in the model ie no pipelining.

Expand Down Expand Up @@ -1073,7 +1073,7 @@ BERT BASE FP32

##### Inference performance: NVIDIA Tesla T4 (1x T4 16G)

###### Fine-tuning inference performance for SQuAD on Tesla T4 16G
###### Fine-tuning inference performance for SQuAD v1.1 on Tesla T4 16G

Our results were obtained by running the `scripts/finetune_inference_benchmark.sh` training script in the TensorFlow 19.08-py3 NGC container on NVIDIA Tesla T4 with 1x T4 16G GPUs. Performance numbers (throughput in sentences per second and latency in milliseconds) were averaged from 1024 iterations. Latency is computed as the time taken for a batch to process as they are fed in one after another in the model ie no pipelining.

Expand Down
10 changes: 9 additions & 1 deletion TensorFlow/LanguageModeling/BERT/run_classifier.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@
import horovod.tensorflow as hvd
import time
from utils.utils import LogEvalRunHook, LogTrainRunHook
import utils.dllogger_class
from dllogger import Verbosity
from utils.create_glue_data import *
import numpy as np

Expand Down Expand Up @@ -58,6 +60,9 @@
"The output directory where the model checkpoints will be written.")

## Other parameters
flags.DEFINE_string(
"dllog_path", "bert_dllog.json",
"filename where dllogger writes to")

flags.DEFINE_string(
"init_checkpoint", None,
Expand Down Expand Up @@ -425,6 +430,7 @@ def input_fn():

def main(_):
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
dllogging = utils.dllogger_class.dllogger_class(FLAGS.dllog_path)

if FLAGS.horovod:
hvd.init()
Expand Down Expand Up @@ -622,13 +628,15 @@ def main(_):
tf.compat.v1.logging.info("Latency Confidence Level 100 (ms) = %0.2f", cf_100 * 1000)
tf.compat.v1.logging.info("Latency Average (ms) = %0.2f", avg * 1000)
tf.compat.v1.logging.info("Throughput Average (sentences/sec) = %0.2f", ss_sentences_per_second)
dllogging.logger.log(step=(), data={"throughput_train": ss_sentences_per_second}, verbosity=Verbosity.DEFAULT)
tf.compat.v1.logging.info("-----------------------------")


output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt")
with tf.io.gfile.GFile(output_eval_file, "w") as writer:
tf.compat.v1.logging.info("***** Eval results *****")
for key in sorted(result.keys()):
dllogging.logger.log(step=(), data={key: float(result[key])}, verbosity=Verbosity.DEFAULT)
tf.compat.v1.logging.info(" %s = %s", key, str(result[key]))
writer.write("%s = %s\n" % (key, str(result[key])))

Expand Down Expand Up @@ -684,7 +692,6 @@ def main(_):
predict_hooks[-1].count * FLAGS.predict_batch_size)
tf.compat.v1.logging.info("Total Inference Time W/O Overhead = %0.2f for Sentences = %d", predict_time_wo_overhead,
(predict_hooks[-1].count - predict_hooks[-1].skipped) * FLAGS.predict_batch_size)

tf.compat.v1.logging.info("Summary Inference Statistics on TEST SET")
tf.compat.v1.logging.info("Batch size = %d", FLAGS.predict_batch_size)
tf.compat.v1.logging.info("Sequence Length = %d", FLAGS.max_seq_length)
Expand All @@ -696,6 +703,7 @@ def main(_):
tf.compat.v1.logging.info("Latency Confidence Level 100 (ms) = %0.2f", cf_100 * 1000)
tf.compat.v1.logging.info("Latency Average (ms) = %0.2f", avg * 1000)
tf.compat.v1.logging.info("Throughput Average (sentences/sec) = %0.2f", ss_sentences_per_second)
dllogging.logger.log(step=(), data={"throughput_val": ss_sentences_per_second}, verbosity=Verbosity.DEFAULT)
tf.compat.v1.logging.info("-----------------------------")


Expand Down
17 changes: 13 additions & 4 deletions TensorFlow/LanguageModeling/BERT/run_ner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@
import time
import horovod.tensorflow as hvd
from utils.utils import LogEvalRunHook, LogTrainRunHook
import utils.dllogger_class
from dllogger import Verbosity

flags = tf.flags

Expand Down Expand Up @@ -54,6 +56,10 @@
"vocab_file", None,
"The vocabulary file that the BERT model was trained on.")

flags.DEFINE_string(
"dllog_path", "bert_dllog.json",
"filename where dllogger writes to")

flags.DEFINE_string(
"init_checkpoint", None,
"Initial checkpoint (usually from a pre-trained BERT model)."
Expand Down Expand Up @@ -545,10 +551,9 @@ def metric_fn(per_example_loss, label_ids, logits):
f = tf_metrics.f1(label_ids, predictions, num_labels, [1, 2], average="macro")
#
return {
"eval_precision": precision,
"eval_recall": recall,
"eval_f": f,
# "eval_loss": loss,
"precision": precision,
"recall": recall,
"f1": f,
}

eval_metric_ops = metric_fn(per_example_loss, label_ids, logits)
Expand Down Expand Up @@ -609,6 +614,7 @@ def result_to_pair(predict_line, pred_ids, id2label, writer, err_writer):

def main(_):
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
dllogging = utils.dllogger_class.dllogger_class(FLAGS.dllog_path)

if FLAGS.horovod:
hvd.init()
Expand Down Expand Up @@ -749,6 +755,7 @@ def main(_):
(num_train_steps - training_hooks[-1].skipped) * global_batch_size)
tf.compat.v1.logging.info("Throughput Average (sentences/sec) with overhead = %0.2f", avg_sentences_per_second)
tf.compat.v1.logging.info("Throughput Average (sentences/sec) = %0.2f", ss_sentences_per_second)
dllogging.logger.log(step=(), data={"throughput_train": ss_sentences_per_second}, verbosity=Verbosity.DEFAULT)
tf.compat.v1.logging.info("-----------------------------")

if FLAGS.do_eval and master_process:
Expand All @@ -774,6 +781,7 @@ def main(_):
tf.compat.v1.logging.info("***** Eval results *****")
for key in sorted(result.keys()):
tf.compat.v1.logging.info(" %s = %s", key, str(result[key]))
dllogging.logger.log(step=(), data={key: float(strresult[key])}, verbosity=Verbosity.DEFAULT)
writer.write("%s = %s\n" % (key, str(result[key])))
if FLAGS.do_predict and master_process:
predict_examples = processor.get_test_examples(FLAGS.data_dir)
Expand Down Expand Up @@ -850,6 +858,7 @@ def main(_):
tf.compat.v1.logging.info("Latency Confidence Level 100 (ms) = %0.2f", cf_100 * 1000)
tf.compat.v1.logging.info("Latency Average (ms) = %0.2f", avg * 1000)
tf.compat.v1.logging.info("Throughput Average (sentences/sec) = %0.2f", ss_sentences_per_second)
dllogging.logger.log(step=(), data={"throughput_val": ss_sentences_per_second}, verbosity=Verbosity.DEFAULT)
tf.compat.v1.logging.info("-----------------------------")

tf.compat.v1.logging.info('Reading: %s', test_labels_file)
Expand Down
Loading