Skip to content

BERT module update #502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
swethmandava merged 7 commits into from
May 18, 2020
Merged

BERT module update #502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
edef331
Merge pull request #8 from NVIDIA/master
swethmandava May 13, 2020
58a3ed6
trtis to triton update
May 13, 2020
398dc78
amp env variable to amp api
May 13, 2020
cfc2395
merge conflicts
May 13, 2020
942b096
change default dllog path, disable horovod for 1 gpu
May 14, 2020
bd5ce85
Updating notebooks to have relative paths instead of absolute
May 18, 2020
5876314
fixing xla fragmentation in latest container
May 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions TensorFlow/LanguageModeling/BERT/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ARG FROM_IMAGE_NAME=nvcr.io/nvidia/tensorflow:19.10-py3
ARG FROM_IMAGE_NAME=nvcr.io/nvidia/tensorflow:20.03-tf1-py3

FROM ${FROM_IMAGE_NAME}

Expand All @@ -13,10 +13,11 @@ RUN git clone https://github.com/attardi/wikiextractor.git
RUN git clone https://github.com/soskek/bookcorpus.git
RUN git clone https://github.com/titipata/pubmed_parser


RUN pip3 install /workspace/pubmed_parser

#Copy the perf_client over
ARG TRTIS_CLIENTS_URL=https://github.com/NVIDIA/tensorrt-inference-server/releases/download/v1.5.0/v1.5.0_ubuntu1804.clients.tar.gz
ARG TRTIS_CLIENTS_URL=https://github.com/NVIDIA/triton-inference-server/releases/download/v1.12.0/v1.12.0_ubuntu1804.clients.tar.gz
RUN mkdir -p /workspace/install \
&& curl -L ${TRTIS_CLIENTS_URL} | tar xvz -C /workspace/install

Expand Down
8 changes: 4 additions & 4 deletions TensorFlow/LanguageModeling/BERT/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ This repository provides a script and recipe to train the BERT model for TensorF
* [Multi-node](#multi-node)
* [Inference process](#inference-process)
* [Inference Process With TensorRT](#inference-process-with-tensorrt)
* [Deploying the BERT model using TensorRT Inference Server](#deploying-the-bert-model-using-tensorrt-inference-server)
* [Deploying the BERT model using Triton Inference Server](#deploying-the-bert-model-using-triton-inference-server)
* [BioBERT](#biobert)
- [Performance](#performance)
* [Benchmarking](#benchmarking)
Expand Down Expand Up @@ -619,9 +619,9 @@ I0312 23:14:00.550973 140287431493376 run_squad.py:1397] 0 Inference Performance
### Inference Process With TensorRT
NVIDIA TensorRT is a platform for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime that delivers low latency and high-throughput for deep learning inference applications. More information on how to perform inference using TensorRT can be found in the subfolder [./trt/README.md](trt/README.md)

### Deploying the BERT model using TensorRT Inference Server
### Deploying the BERT model using Triton Inference Server

The [NVIDIA TensorRT Inference Server](https://github.com/NVIDIA/tensorrt-inference-server) provides a datacenter and cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or gRPC endpoint, allowing remote clients to request inferencing for any number of GPU or CPU models being managed by the server. More information on how to perform inference using `TensorRT Inference Server` can be found in the subfolder `./trtis/README.md`.
The [NVIDIA Triton Inference Server](https://github.com/NVIDIA/triton-inference-server) provides a datacenter and cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or gRPC endpoint, allowing remote clients to request inferencing for any number of GPU or CPU models being managed by the server. More information on how to perform inference using `Triton Inference Server` can be found in the subfolder `./triton/README.md`.

### BioBERT

Expand Down Expand Up @@ -1153,7 +1153,7 @@ September 2019

July 2019
- Results obtained using 19.06
- Inference Studies using TensorRT Inference Server
- Inference Studies using Triton Inference Server

March 2019
- Initial release
Expand Down
38 changes: 20 additions & 18 deletions TensorFlow/LanguageModeling/BERT/notebooks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,24 +41,10 @@ Here is a short description of each relevant file:
### 2.a Build the BERT TensorFlow NGC container:
To run the notebook you first need to build the Bert TensorFlow container using the following command from the main directory of this repository:

``` bash
```bash
docker build . --rm -t bert
```
### 2.b Dataset

We need to download the vocabulary and the bert_config files:

``` python3
python3 /workspace/bert/data/bertPrep.py --action download --dataset google_pretrained_weights # Includes vocab
```

This is only needed during fine-tuning in order to download the Squad dataset:

``` python3
python3 /workspace/bert/data/bertPrep.py --action download --dataset squad
```

### 2.c Start of the NGC container to run inference:
### 2.b Start of the NGC container to run inference:
Once the image is built, you need to run the container with the `--publish
0.0.0.0:8888:8888` option to publish Jupyter's port `8888` to the host machine
at port `8888` over all network interfaces (`0.0.0.0`):
Expand All @@ -74,7 +60,23 @@ nvidia-docker run \
-it bert:latest bash
```

Then you can use the following command within the BERT Tensorflow container under
### 2.c Dataset

We need to download the vocabulary and the bert_config files:

```python3
python3 /workspace/bert/data/bertPrep.py --action download --dataset google_pretrained_weights # Includes vocab
```

This is only needed during fine-tuning in order to download the Squad dataset:

```python3
python3 /workspace/bert/data/bertPrep.py --action download --dataset squad
```

### 2.d Starting Jupyter Notebook

Now you can use the following command within the BERT Tensorflow container under
`/workspace/bert`:

```bash
Expand Down Expand Up @@ -123,7 +125,7 @@ Here is a short description of the relevant file:
### 2.a Build the BERT TensorFlow NGC container:
To run the notebook you first need to build the Bert TensorFlow container using the following command from the main directory of this repository:

``` bash
```bash
docker build . --rm -t bert
```
### 2.b Start of the NGC container to run inference:
Expand Down
29 changes: 18 additions & 11 deletions TensorFlow/LanguageModeling/BERT/notebooks/bert_squad_tf_finetuning.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@
"import os\n",
"import sys\n",
"\n",
"data_dir = '/workspace/bert/data/download'\n",
"data_dir = '../data/download'\n",
"\n",
"# SQuAD json for training\n",
"train_file = os.path.join(data_dir, 'squad/v1.1/train-v1.1.json')\n",
Expand Down Expand Up @@ -141,7 +141,7 @@
"|BERTBASE |12 encoder| 768| 12|4 x 768|512|110M|\n",
"|BERTLARGE|24 encoder|1024| 16|4 x 1024|512|330M|\n",
"\n",
"We will large use pre-trained models avaialble on NGC (NVIDIA GPU Cluster, https://ngc.nvidia.com).\n",
"We will use large pre-trained models avaialble on NGC (NVIDIA GPU Cluster, https://ngc.nvidia.com).\n",
"There are many configuration available, in particular we will download and use the following:\n",
"\n",
"**bert_tf_large_fp16_384**\n",
Expand All @@ -164,7 +164,7 @@
"outputs": [],
"source": [
"# bert_tf_large_fp16_384\n",
"DATA_DIR_FP16 = '/workspace/bert/data/download/pretrained_model_fp16'\n",
"DATA_DIR_FP16 = data_dir + '/pretrained_model_fp16'\n",
"!mkdir -p $DATA_DIR_FP16\n",
"!wget -nc -q --show-progress -O $DATA_DIR_FP16/bert_for_tensorflow.zip \\\n",
"https://api.ngc.nvidia.com/v2/models/nvidia/bert_for_tensorflow/versions/1/zip\n",
Expand All @@ -184,9 +184,9 @@
"metadata": {},
"outputs": [],
"source": [
"notebooks_dir = '/workspace/bert/notebooks'\n",
"notebooks_dir = '../notebooks'\n",
"\n",
"working_dir = '/workspace/bert'\n",
"working_dir = '..'\n",
"if working_dir not in sys.path:\n",
" sys.path.append(working_dir)\n",
"\n",
Expand Down Expand Up @@ -257,7 +257,10 @@
"if 'f' not in tf.flags.FLAGS: \n",
" tf.app.flags.DEFINE_string('f', '', 'kernel')\n",
"FLAGS = flags.FLAGS\n",
"# FLAGS.verbose_logging = True\n",
"\n",
"verbose_logging = True\n",
"# Set to True if the dataset has samples with no answers. For SQuAD 1.1, this is set to False\n",
"version_2_with_negative = False\n",
"\n",
"# The total number of n-best predictions to generate in the nbest_predictions.json output file.\n",
"n_best_size = 20\n",
Expand Down Expand Up @@ -536,7 +539,10 @@
" end_logits=end_logits))\n",
"\n",
"eval_time_elapsed = time.time() - eval_start_time\n",
"eval_time_wo_startup = eval_hooks[-1].total_time\n",
"\n",
"time_list = eval_hooks[-1].time_list\n",
"time_list.sort()\n",
"eval_time_wo_startup = sum(time_list[:int(len(time_list) * 0.99)])\n",
"num_sentences = eval_hooks[-1].count * predict_batch_size\n",
"avg_sentences_per_second = num_sentences * 1.0 / eval_time_wo_startup\n",
"\n",
Expand All @@ -554,7 +560,8 @@
"run_squad.write_predictions(eval_examples, eval_features, all_results,\n",
" n_best_size, max_answer_length,\n",
" do_lower_case, output_prediction_file,\n",
" output_nbest_file, output_null_log_odds_file)\n",
" output_nbest_file, output_null_log_odds_file,\n",
" version_2_with_negative, verbose_logging)\n",
"\n",
"tf.logging.info(\"Inference Results:\")\n",
"\n",
Expand Down Expand Up @@ -585,7 +592,7 @@
"metadata": {},
"outputs": [],
"source": [
"!python /workspace/bert/data/download/squad/v1.1/evaluate-v1.1.py \\\n",
"!python ../data/download/squad/v1.1/evaluate-v1.1.py \\\n",
" $predict_file \\\n",
" $output_dir/predictions.json"
]
Expand All @@ -596,7 +603,7 @@
"source": [
"## 6. What's next\n",
"\n",
"Now that you have fine-tuned a BERT model you may want to take a look ad the run_squad script which containd more options for fine-tuning."
"Now that you have fine-tuned a BERT model you may want to take a look at the run_squad script which containd more options for fine-tuning."
]
}
],
Expand All @@ -616,7 +623,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
"version": "3.6.9"
}
},
"nbformat": 4,
Expand Down
25 changes: 16 additions & 9 deletions TensorFlow/LanguageModeling/BERT/notebooks/bert_squad_tf_inference.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,10 +162,10 @@
"import os\n",
"import sys\n",
"\n",
"notebooks_dir = '/workspace/bert/notebooks'\n",
"data_dir = '/workspace/bert/data/download'\n",
"notebooks_dir = '../notebooks'\n",
"data_dir = '../data/download'\n",
"\n",
"working_dir = '/workspace/bert'\n",
"working_dir = '../'\n",
"if working_dir not in sys.path:\n",
" sys.path.append(working_dir)"
]
Expand Down Expand Up @@ -239,7 +239,7 @@
"metadata": {},
"outputs": [],
"source": [
"#input_file = '/workspace/bert/data/download/squad/v2.0/dev-v2.0.json'"
"#input_file = '../data/download/squad/v2.0/dev-v2.0.json'"
]
},
{
Expand Down Expand Up @@ -272,14 +272,14 @@
"outputs": [],
"source": [
"# bert_tf_v2_large_fp32_384\n",
"DATA_DIR_FP32='/workspace/bert/data/download/finetuned_model_fp32'\n",
"DATA_DIR_FP32 = data_dir + '/finetuned_model_fp32'\n",
"!mkdir -p $DATA_DIR_FP32\n",
"!wget -nc -q --show-progress -O $DATA_DIR_FP32/bert_tf_v2_large_fp32_384.zip \\\n",
"https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_v2_large_fp32_384/versions/1/zip\n",
"!unzip -n -d $DATA_DIR_FP32/ $DATA_DIR_FP32/bert_tf_v2_large_fp32_384.zip \n",
" \n",
"# bert_tf_v2_large_fp16_384\n",
"DATA_DIR_FP16='/workspace/bert/data/download/finetuned_model_fp16'\n",
"DATA_DIR_FP16 = data_dir + '/finetuned_model_fp16'\n",
"!mkdir -p $DATA_DIR_FP16\n",
"!wget -nc -q --show-progress -O $DATA_DIR_FP16/bert_tf_v2_large_fp16_384.zip \\\n",
"https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_v2_large_fp16_384/versions/1/zip\n",
Expand Down Expand Up @@ -363,6 +363,10 @@
" tf.app.flags.DEFINE_string('f', '', 'kernel')\n",
"FLAGS = flags.FLAGS\n",
"\n",
"verbose_logging = True\n",
"# Set to True if the dataset has samples with no answers. For SQuAD 1.1, this is set to False\n",
"version_2_with_negative = False\n",
"\n",
"# The total number of n-best predictions to generate in the nbest_predictions.json output file.\n",
"n_best_size = 20\n",
"\n",
Expand Down Expand Up @@ -501,7 +505,9 @@
"\n",
"eval_time_elapsed = time.time() - eval_start_time\n",
"\n",
"eval_time_wo_startup = eval_hooks[-1].total_time\n",
"time_list = eval_hooks[-1].time_list\n",
"time_list.sort()\n",
"eval_time_wo_startup = sum(time_list[:int(len(time_list) * 0.99)])\n",
"num_sentences = eval_hooks[-1].count * predict_batch_size\n",
"avg_sentences_per_second = num_sentences * 1.0 / eval_time_wo_startup\n",
"\n",
Expand All @@ -519,7 +525,8 @@
"run_squad.write_predictions(eval_examples, eval_features, all_results,\n",
" n_best_size, max_answer_length,\n",
" do_lower_case, output_prediction_file,\n",
" output_nbest_file, output_null_log_odds_file)\n",
" output_nbest_file, output_null_log_odds_file,\n",
" version_2_with_negative, verbose_logging)\n",
"\n",
"tf.logging.info(\"Inference Results:\")\n",
"\n",
Expand Down Expand Up @@ -569,7 +576,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
"version": "3.6.9"
}
},
"nbformat": 4,
Expand Down
10 changes: 8 additions & 2 deletions TensorFlow/LanguageModeling/BERT/notebooks/bert_squad_tf_inference_colab.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,10 @@
" tf.app.flags.DEFINE_string('f', '', 'kernel')\n",
"FLAGS = flags.FLAGS\n",
"\n",
"verbose_logging = True\n",
"# Set to True if the dataset has samples with no answers. For SQuAD 1.1, this is set to False\n",
"version_2_with_negative = False\n",
"\n",
"# The total number of n-best predictions to generate in the nbest_predictions.json output file.\n",
"n_best_size = 20\n",
"\n",
Expand Down Expand Up @@ -678,7 +682,9 @@
"\n",
"eval_time_elapsed = time.time() - eval_start_time\n",
"\n",
"eval_time_wo_startup = eval_hooks[-1].total_time\n",
"time_list = eval_hooks[-1].time_list\n",
"time_list.sort()\n",
"eval_time_wo_startup = sum(time_list[:int(len(time_list) * 0.99)])\n",
"num_sentences = eval_hooks[-1].count * predict_batch_size\n",
"avg_sentences_per_second = num_sentences * 1.0 / eval_time_wo_startup\n",
"\n",
Expand Down Expand Up @@ -757,7 +763,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
"version": "3.6.9"
}
},
"nbformat": 4,
Expand Down
8 changes: 4 additions & 4 deletions TensorFlow/LanguageModeling/BERT/notebooks/biobert_ner_tf_inference.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,8 @@
"import os\n",
"import sys\n",
"\n",
"notebooks_dir = '/workspace/bert/notebooks'\n",
"working_dir = '/workspace/bert'\n",
"notebooks_dir = '../notebooks'\n",
"working_dir = '../'\n",
"if working_dir not in sys.path:\n",
" sys.path.append(working_dir)"
]
Expand Down Expand Up @@ -267,7 +267,7 @@
"outputs": [],
"source": [
"# biobert_uncased_base_ner_disease\n",
"DATA_DIR_FP16='/workspace/bert/data/download/finetuned_model_fp16'\n",
"DATA_DIR_FP16 = '../data/download/finetuned_model_fp16'\n",
"!mkdir -p $DATA_DIR_FP16\n",
"!wget -nc -q --show-progress -O $DATA_DIR_FP16/biobert_uncased_base_ner_disease.zip \\\n",
"https://api.ngc.nvidia.com/v2/models/nvidia/biobert_uncased_base_ner_disease/versions/1/zip\n",
Expand Down Expand Up @@ -602,7 +602,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
"version": "3.6.9"
}
},
"nbformat": 4,
Expand Down
16 changes: 13 additions & 3 deletions TensorFlow/LanguageModeling/BERT/optimization.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,15 @@ def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, hvd=None,

if hvd is not None and (num_accumulation_steps == 1 or (not allreduce_post_accumulation)):
optimizer = hvd.DistributedOptimizer(optimizer, sparse_as_dense=True, compression=Compression.fp16 if use_fp16 or manual_fp16 else Compression.none)
if manual_fp16 or use_fp16:
loss_scale_manager = tf.contrib.mixed_precision.ExponentialUpdateLossScaleManager(init_loss_scale=2**32, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, decr_ratio=0.5)
if use_fp16:
loss_scaler = tf.train.experimental.DynamicLossScale(initial_loss_scale=2**32, increment_period=1000, multiplier=2.0)
optimizer = tf.train.experimental.enable_mixed_precision_graph_rewrite(optimizer, loss_scaler)
loss_scale_value = tf.identity(loss_scaler(), name="loss_scale")
if manual_fp16:
loss_scale_manager = tf.contrib.mixed_precision.ExponentialUpdateLossScaleManager(init_loss_scale=2 ** 32,
incr_every_n_steps=1000,
decr_every_n_nan_or_inf=2,
decr_ratio=0.5)
optimizer = tf.contrib.mixed_precision.LossScaleOptimizer(optimizer, loss_scale_manager)

tvars = tf.trainable_variables()
Expand Down Expand Up @@ -149,7 +156,10 @@ def update(accum_vars):
update_op = tf.cond(update_step,
lambda: update(accum_vars), lambda: tf.no_op())

new_global_step = tf.cond(tf.math.logical_and(update_step, tf.cast(hvd.allreduce(tf.cast(batch_finite, tf.int32)), tf.bool)), lambda: global_step+1, lambda: global_step)
new_global_step = tf.cond(tf.math.logical_and(update_step,
tf.cast(hvd.allreduce(tf.cast(batch_finite, tf.int32)), tf.bool)) if hvd is not None else batch_finite,
lambda: global_step+1,
lambda: global_step)
new_global_step = tf.identity(new_global_step, name='step_update')
train_op = tf.group(update_op, [global_step.assign(new_global_step)])
else:
Expand Down
Loading