Skip to content

Nvpstr/fed7ba #561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nv-kkudrynski merged 4 commits into from
Jun 12, 2020
Merged

Nvpstr/fed7ba #561

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
66ed01d
[FastPitch/PyT] README updates
nvpstr Jun 8, 2020
23cc1cd
[FastPitch/PyT] Small README fixes
nvpstr Jun 12, 2020
fed7ba9
[ConvNets/TF] Updating RN50, Adding ResNext and SE-ResNext
nvpstr Jun 12, 2020
f11884b
minor readme updates
nv-kkudrynski Jun 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
258 changes: 119 additions & 139 deletions PyTorch/SpeechSynthesis/FastPitch/README.md
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion PyTorch/SpeechSynthesis/FastPitch/fastpitch/transformer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def __init__(self, demb):

def forward(self, pos_seq, bsz=None):
sinusoid_inp = torch.ger(pos_seq, self.inv_freq)
pos_emb = torch.cat([sinusoid_inp.sin(), sinusoid_inp.cos()], dim=-1)
pos_emb = torch.cat([sinusoid_inp.sin(), sinusoid_inp.cos()], dim=1)
if bsz is not None:
return pos_emb[None, :, :].expand(bsz, -1, -1)
else:
Expand Down
2 changes: 1 addition & 1 deletion PyTorch/SpeechSynthesis/FastPitch/fastpitch/transformer_jit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def __init__(self, demb):

def forward(self, pos_seq, bsz: Optional[int] = None):
sinusoid_inp = torch.ger(pos_seq, self.inv_freq)
pos_emb = torch.cat([sinusoid_inp.sin(), sinusoid_inp.cos()], dim=-1)
pos_emb = torch.cat([sinusoid_inp.sin(), sinusoid_inp.cos()], dim=1)
if bsz is not None:
return pos_emb[None, :, :].expand(bsz, -1, -1)
else:
Expand Down
3 changes: 3 additions & 0 deletions PyTorch/SpeechSynthesis/FastPitch/inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,9 @@ def main():
'pitch_tgt': None,
'pitch_transform': build_pitch_transformation(args)}

if args.torchscript:
gen_kw.pop('pitch_transform')

all_utterances = 0
all_samples = 0
all_letters = 0
Expand Down
3 changes: 1 addition & 2 deletions PyTorch/SpeechSynthesis/FastPitch/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,7 @@ def forward(self, spect, sigma=1.0):
class FastPitch__forward_is_infer(_FastPitchJIT):
def forward(self, inputs, input_lengths, pace: float = 1.0,
dur_tgt: Optional[torch.Tensor] = None,
pitch_tgt: Optional[torch.Tensor] = None,
pitch_transform: Optional[bool] = None):
pitch_tgt: Optional[torch.Tensor] = None):
return self.infer(inputs, input_lengths, pace=pace,
dur_tgt=dur_tgt, pitch_tgt=pitch_tgt)
else:
Expand Down
1 change: 1 addition & 0 deletions PyTorch/SpeechSynthesis/FastPitch/scripts/inference_benchmark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ for PRECISION in fp16 fp32; do
--include-warmup \
--batch-size ${BSZ} \
--repeats 1000 \
--torchscript \
-i phrases/benchmark_8_128.tsv
done
done
14 changes: 8 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ These examples, along with our NVIDIA deep learning software stack, are provided
The examples are organized first by framework, such as TensorFlow, PyTorch, etc. and second by use case, such as computer vision, natural language processing, etc. We hope this structure enables you to quickly locate the example networks that best suit your needs. Here are the currently supported models:

### Computer Vision
- __ResNet-50__ [[MXNet](https://github.com/NVIDIA/DeepLearningExamples/tree/master/MxNet/Classification/RN50v1.5)] [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets)] [[TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/RN50v1.5)]
- __ResNext__ [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets)]
- __SE-ResNext__ [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets)]
- __ResNet-50__ [[MXNet](https://github.com/NVIDIA/DeepLearningExamples/tree/master/MxNet/Classification/RN50v1.5)] [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets)] [[TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/ConvNets)]
- __ResNext__ [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets)] [[TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/ConvNets)]
- __SE-ResNext__ [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets)] [[TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/ConvNets)]
- __SSD__ [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/SSD)] [[TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Detection/SSD)]
- __Mask R-CNN__ [[PyTorch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/MaskRCNN)] [[TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/Segmentation/MaskRCNN)] [[TensorFlow 2](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/Segmentation/MaskRCNN)]
- __U-Net(industrial)__ [[TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Segmentation/UNet_Industrial)]
Expand Down Expand Up @@ -65,7 +65,7 @@ The examples are organized first by framework, such as TensorFlow, PyTorch, etc.
## Feature Matrix
| Models | Framework | DALI | AMP | Multi-GPU | Multi-Node | TensorRT | ONNX | Triton | TF-TRT |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |------------- |------------- |------------- |------------- |
| [ResNet50 v1.5](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5) |PyTorch | Yes | Yes | Yes | - | - | - | - | - |
| [ResNet-50 v1.5](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5) |PyTorch | Yes | Yes | Yes | - | - | - | - | - |
| [ResNeXt101-32x4d](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnext101-32x4d) |PyTorch | Yes | Yes | Yes | - | - | - | - | - |
| [SE-ResNeXt101-32x4d](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/se-resnext101-32x4d) |PyTorch | Yes | Yes | Yes | - | - | - | - | - |
| [SSD300 v1.1](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/SSD) |PyTorch | Yes | Yes | Yes | - | - | - | - | - |
Expand All @@ -79,9 +79,11 @@ The examples are organized first by framework, such as TensorFlow, PyTorch, etc.
| [FastPitch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch) | PyTorch | N/A | Yes | Yes | - | - | - | - | - |
| [GNMT v2](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/GNMT) |PyTorch | N/A | Yes | Yes | - | - | - | - | - |
| [Transformer](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/Transformer) |PyTorch | N/A | Yes | Yes | - | - | - | - | - |
| [ResNet-50 v1.5](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/RN50v1.5) |TensorFlow | Yes | Yes | Yes | - | - | - | - | - |
| [ResNet-50 v1.5](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/ConvNets/resnet50v1.5) |TensorFlow | Yes | Yes | Yes | - | - | - | - | - |
| [ResNeXt101-32x4d](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/ConvNets/resnext101-32x4d) |TensorFlow | Yes | Yes | Yes | - | - | - | - | - |
| [SE-ResNeXt101-32x4d](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TEnsorFlow/Classification/ConvNets/se-resnext101-32x4d) |TensorFlow | Yes | Yes | Yes | - | - | - | - | - |
| [SSD320 v1.2](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Detection/SSD) | TensorFlow | N/A | Yes | Yes | - | - | - | - | - |
| [BERT](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT) |TensorFlow | N/A | Yes | Yes | Yes | Yes | - | [Yes](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT/trtis) | Yes |
| [BERT](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT) |TensorFlow | N/A | Yes | Yes | Yes | Yes | - | [Yes](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT/triton) | Yes |
| [BioBert](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT/biobert) | TensorFlow | N/A | Yes | Yes | - | - | - | - | - |
| [Transformer-XL](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/Transformer-XL) |TensorFlow | N/A | Yes | Yes | - | - | - | - | - |
| [Neural Collaborative Filtering](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Recommendation/NCF) |TensorFlow | N/A | Yes | Yes | - | - | - | - | - |
Expand Down
0 ...orFlow/Classification/RN50v1.5/.gitignore → ...orFlow/Classification/ConvNets/.gitignore
View file
Open in desktop
File renamed without changes.
0 ...ssification/RN50v1.5/dllogger/__init__.py → ...rFlow/Classification/ConvNets/.gitmodules
View file
Open in desktop
File renamed without changes.
0 ...rFlow/Classification/RN50v1.5/.style.yapf → ...rFlow/Classification/ConvNets/.style.yapf
View file
Open in desktop
File renamed without changes.
8 changes: 8 additions & 0 deletions TensorFlow/Classification/ConvNets/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
ARG FROM_IMAGE_NAME=nvcr.io/nvidia/tensorflow:20.03-tf1-py3
FROM ${FROM_IMAGE_NAME}

ADD requirements.txt .
RUN pip install -r requirements.txt

ADD . /workspace/rn50v15_tf
WORKDIR /workspace/rn50v15_tf
0 TensorFlow/Classification/RN50v1.5/LICENSE → TensorFlow/Classification/ConvNets/LICENSE
View file
Open in desktop
File renamed without changes.
68 changes: 68 additions & 0 deletions TensorFlow/Classification/ConvNets/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Resnet-family Convolutional Neural Networks for Image Classification in Tensorflow

In this repository you will find implementation of Resnet and its variations for image
classification

## Table Of Contents

* [Models](#models)
* [Validation accuracy results](#validation-accuracy-results)
* [Training performance results](#training-performance-results)
* [Training performance: NVIDIA DGX-1 (8x V100 16G)](#training-performance-nvidia-dgx-1-(8x-v100-16G))
* [Release notes](#release-notes)
* [Changelog](#changelog)


## Models

The following table provides links to where you can find additional information on each model:

| **Model** | **Link**|
|-----------|---------|
| resnet50 | [README](./resnet50v1.5/README.md) |
| resnext101-32x4d | [README](./resnext101-32x4d/README.md) |
| se-resnext101-32x4d | [README](./se-resnext101-32x4d/README.md) |

## Validation accuracy results

Our results were obtained by running the applicable training scripts in the tensorflow-20.03-tf1-py3 NGC container
on NVIDIA DGX-1 with (8x V100 16G) GPUs. The specific training script that was run is documented in the corresponding model's README.

The following table shows the validation accuracy results of the
three classification models side-by-side.


| **arch** | **AMP Top1** | **AMP Top5** | **FP32 Top1** | **FP32 Top5** |
|:-:|:-:|:-:|:-:|:-:|
| resnet50 | 78.35 | 94.21 | 78.34 | 94.21 |
| resnext101-32x4d | 80.21 | 95.00 | 80.21 | 94.99 |
| se-resnext101-32x4d | 80.87 | 95.35 | 80.84 | 95.37 |

## Training performance results

### Training performance: NVIDIA DGX-1 (8x V100 16G)

Our results were obtained by running the applicable
training scripts in the tensorflow-20.03-tf1-py3 NGC container
on NVIDIA DGX-1 with (8x V100 16G) GPUs.
Performance numbers (in images per second)
were averaged over an entire training epoch.
The specific training script that was run is documented
in the corresponding model's README.

The following table shows the training accuracy results of the
three classification models side-by-side.


| **arch** | **Mixed Precision** | **Mixed Prcesision XLA** | **FP32** | **Mixed Precision speedup** | **XLA Mixed Precision speedup**|
|:-:|:-:|:-:|:-:|:-:|:-:|
| resnet50 | 8277.91 img/s | 9485.21 img/s | 2785.81 img/s | 2.97x | 1.14x |
| resnext101-32x4d | 3151.81 img/s | 4231.42 img/s | 1055.82 img/s | 2.98x | 1.34x |
| se-resnext101-32x4d | 2168.40 img/s | 3297.39 img/s | 921.38 img/s | 2.35x | 1.52x |

## Release notes

### Changelog
June 2020
- ConvNets repo restructurization
- Initial release of ResNext and SE-Resnext
127 changes: 127 additions & 0 deletions TensorFlow/Classification/ConvNets/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-

# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import os

import warnings
warnings.simplefilter("ignore")

import tensorflow as tf

import horovod.tensorflow as hvd
import dllogger

from utils import hvd_utils
from runtime import Runner
from model.resnet import model_architectures

from utils.cmdline_helper import parse_cmdline

if __name__ == "__main__":

tf.logging.set_verbosity(tf.logging.ERROR)

FLAGS = parse_cmdline(model_architectures.keys())
hvd.init()

if hvd.rank() == 0:
log_path = os.path.join(FLAGS.results_dir, FLAGS.log_filename)
os.makedirs(FLAGS.results_dir, exist_ok=True)

dllogger.init(
backends=[
dllogger.JSONStreamBackend(verbosity=dllogger.Verbosity.VERBOSE, filename=log_path),
dllogger.StdOutBackend(verbosity=dllogger.Verbosity.VERBOSE)
]
)
else:
dllogger.init(backends=[])
dllogger.log(data=vars(FLAGS), step='PARAMETER')

runner = Runner(
# ========= Model HParams ========= #
n_classes=1001,
architecture=FLAGS.arch,
input_format='NHWC',
compute_format=FLAGS.data_format,
dtype=tf.float32 if FLAGS.precision == 'fp32' else tf.float16,
n_channels=3,
height=224,
width=224,
distort_colors=False,
log_dir=FLAGS.results_dir,
model_dir=FLAGS.model_dir if FLAGS.model_dir is not None else FLAGS.results_dir,
data_dir=FLAGS.data_dir,
data_idx_dir=FLAGS.data_idx_dir,
weight_init=FLAGS.weight_init,
use_xla=FLAGS.use_xla,
use_tf_amp=FLAGS.use_tf_amp,
use_dali=FLAGS.use_dali,
gpu_memory_fraction=FLAGS.gpu_memory_fraction,
gpu_id=FLAGS.gpu_id,
seed=FLAGS.seed
)

if FLAGS.mode in ["train", "train_and_evaluate", "training_benchmark"]:
runner.train(
iter_unit=FLAGS.iter_unit,
num_iter=FLAGS.num_iter,
run_iter=FLAGS.run_iter,
batch_size=FLAGS.batch_size,
warmup_steps=FLAGS.warmup_steps,
log_every_n_steps=FLAGS.display_every,
weight_decay=FLAGS.weight_decay,
lr_init=FLAGS.lr_init,
lr_warmup_epochs=FLAGS.lr_warmup_epochs,
momentum=FLAGS.momentum,
loss_scale=FLAGS.loss_scale,
label_smoothing=FLAGS.label_smoothing,
mixup=FLAGS.mixup,
use_static_loss_scaling=FLAGS.use_static_loss_scaling,
use_cosine_lr=FLAGS.use_cosine_lr,
is_benchmark=FLAGS.mode == 'training_benchmark',
)

if FLAGS.mode in ["train_and_evaluate", 'evaluate', 'inference_benchmark']:

if FLAGS.mode == 'inference_benchmark' and hvd_utils.is_using_hvd():
raise NotImplementedError("Only single GPU inference is implemented.")

elif not hvd_utils.is_using_hvd() or hvd.rank() == 0:

runner.evaluate(
iter_unit=FLAGS.iter_unit if FLAGS.mode != "train_and_evaluate" else "epoch",
num_iter=FLAGS.num_iter if FLAGS.mode != "train_and_evaluate" else 1,
warmup_steps=FLAGS.warmup_steps,
batch_size=FLAGS.batch_size,
log_every_n_steps=FLAGS.display_every,
is_benchmark=FLAGS.mode == 'inference_benchmark',
export_dir=FLAGS.export_dir
)

if FLAGS.mode == 'predict':
if FLAGS.to_predict is None:
raise ValueError("No data to predict on.")

if not os.path.isfile(FLAGS.to_predict):
raise ValueError("Only prediction on single images is supported!")

if hvd_utils.is_using_hvd():
raise NotImplementedError("Only single GPU inference is implemented.")

elif not hvd_utils.is_using_hvd() or hvd.rank() == 0:
runner.predict(FLAGS.to_predict)
2 changes: 1 addition & 1 deletion ...Classification/RN50v1.5/model/__init__.py → ...Classification/ConvNets/model/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@

from model import layers
from model import blocks
from model import resnet_v1_5
from model import resnet
0 ...ication/RN50v1.5/model/blocks/__init__.py → ...ication/ConvNets/model/blocks/__init__.py
View file
Open in desktop
File renamed without changes.
70 changes: 15 additions & 55 deletions ...ion/RN50v1.5/model/blocks/conv2d_block.py → ...ion/ConvNets/model/blocks/conv2d_block.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ def conv2d_block(
data_format='NHWC',
conv2d_hparams=None,
batch_norm_hparams=None,
name='conv2d'
name='conv2d',
cardinality=1,
):

if not isinstance(conv2d_hparams, tf.contrib.training.HParams):
Expand All @@ -44,8 +45,7 @@ def conv2d_block(
raise ValueError("The paramater `conv2d_hparams` is not of type `HParams`")

with tf.variable_scope(name):

if mode != 'SAME_RESNET':
if cardinality == 1:
net = layers.conv2d(
inputs,
n_channels=n_channels,
Expand All @@ -56,58 +56,18 @@ def conv2d_block(
use_bias=not use_batch_norm,
trainable=is_training,
kernel_initializer=conv2d_hparams.kernel_initializer,
bias_initializer=conv2d_hparams.bias_initializer,
)

else: # Special padding mode for ResNet models
if strides == (1, 1):

net = layers.conv2d(
inputs,
n_channels=n_channels,
kernel_size=kernel_size,
strides=strides,
padding='SAME',
data_format=data_format,
use_bias=not use_batch_norm,
trainable=is_training,
kernel_initializer=conv2d_hparams.kernel_initializer,
bias_initializer=conv2d_hparams.bias_initializer,
)

else:
rate = 1 # Unused (for 'a trous' convolutions)

kernel_height_effective = kernel_size[0] + (kernel_size[0] - 1) * (rate - 1)

pad_h_beg = (kernel_height_effective - 1) // 2
pad_h_end = kernel_height_effective - 1 - pad_h_beg

kernel_width_effective = kernel_size[1] + (kernel_size[1] - 1) * (rate - 1)

pad_w_beg = (kernel_width_effective - 1) // 2
pad_w_end = kernel_width_effective - 1 - pad_w_beg

padding = [[0, 0], [pad_h_beg, pad_h_end], [pad_w_beg, pad_w_end], [0, 0]]

if data_format == 'NCHW':
padding = [padding[0], padding[3], padding[1], padding[2]]

padded_inputs = tf.pad(inputs, padding)

net = layers.conv2d(
padded_inputs, # inputs,
n_channels=n_channels,
kernel_size=kernel_size,
strides=strides,
padding='VALID',
data_format=data_format,
use_bias=not use_batch_norm,
trainable=is_training,
kernel_initializer=conv2d_hparams.kernel_initializer,
bias_initializer=conv2d_hparams.bias_initializer,
)

bias_initializer=conv2d_hparams.bias_initializer)
else:
group_filter = tf.get_variable(
name=name + 'group_filter',
shape=[3, 3, n_channels // cardinality, n_channels],
trainable=is_training,
dtype=tf.float32)
net = tf.nn.conv2d(inputs,
group_filter,
strides=strides,
padding='SAME',
data_format=data_format)
if use_batch_norm:
net = layers.batch_norm(
net,
Expand Down
Loading