Skip to content

Commit b09ce98

Browse files
nvpstrnvpstr
committed
Updating GNMT/PyT
1 parent e470c21 commit b09ce98

41 files changed

Lines changed: 781 additions & 556 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎PyTorch/Translation/GNMT/Dockerfile‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,25 @@
1-
FROM nvcr.io/nvidia/pytorch:19.05-py3
1+
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Permission is hereby granted, free of charge, to any person obtaining a copy
4+
# of this software and associated documentation files (the "Software"), to deal
5+
# in the Software without restriction, including without limitation the rights
6+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
# copies of the Software, and to permit persons to whom the Software is
8+
# furnished to do so, subject to the following conditions:
9+
#
10+
# The above copyright notice and this permission notice shall be included in all
11+
# copies or substantial portions of the Software.
12+
#
13+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19+
# SOFTWARE.
20+
21+
ARG FROM_IMAGE_NAME=nvcr.io/nvidia/pytorch:19.05-py3
22+
FROM ${FROM_IMAGE_NAME}
223

324
ENV LANG C.UTF-8
425
ENV LC_ALL C.UTF-8

‎PyTorch/Translation/GNMT/LICENSE‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
MIT License
2-
31
Copyright (c) 2017 Elad Hoffer
42
Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
53

‎PyTorch/Translation/GNMT/README.md‎

Lines changed: 67 additions & 104 deletions
Large diffs are not rendered by default.

‎PyTorch/Translation/GNMT/img/diagram.png‎

-3.46 KB
Loading

‎PyTorch/Translation/GNMT/launch.py‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,75 @@
1+
# From PyTorch:
2+
#
3+
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
4+
# Copyright (c) 2016- Facebook, Inc (Adam Paszke)
5+
# Copyright (c) 2014- Facebook, Inc (Soumith Chintala)
6+
# Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
7+
# Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
8+
# Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
9+
# Copyright (c) 2011-2013 NYU (Clement Farabet)
10+
# Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
11+
# Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
12+
# Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
13+
#
14+
# From Caffe2:
15+
#
16+
# Copyright (c) 2016-present, Facebook Inc. All rights reserved.
17+
#
18+
# All contributions by Facebook:
19+
# Copyright (c) 2016 Facebook Inc.
20+
#
21+
# All contributions by Google:
22+
# Copyright (c) 2015 Google Inc.
23+
# All rights reserved.
24+
#
25+
# All contributions by Yangqing Jia:
26+
# Copyright (c) 2015 Yangqing Jia
27+
# All rights reserved.
28+
#
29+
# All contributions from Caffe:
30+
# Copyright(c) 2013, 2014, 2015, the respective contributors
31+
# All rights reserved.
32+
#
33+
# All other contributions:
34+
# Copyright(c) 2015, 2016 the respective contributors
35+
# All rights reserved.
36+
#
37+
# Caffe2 uses a copyright model similar to Caffe: each contributor holds
38+
# copyright over their contributions to Caffe2. The project versioning records
39+
# all such contribution and copyright details. If a contributor wants to further
40+
# mark their specific copyright on a particular contribution, they should
41+
# indicate their copyright solely in the commit message of the change when it is
42+
# committed.
43+
#
44+
# All rights reserved.
45+
#
46+
# Redistribution and use in source and binary forms, with or without
47+
# modification, are permitted provided that the following conditions are met:
48+
#
49+
# 1. Redistributions of source code must retain the above copyright
50+
# notice, this list of conditions and the following disclaimer.
51+
#
52+
# 2. Redistributions in binary form must reproduce the above copyright
53+
# notice, this list of conditions and the following disclaimer in the
54+
# documentation and/or other materials provided with the distribution.
55+
#
56+
# 3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories America
57+
# and IDIAP Research Institute nor the names of its contributors may be
58+
# used to endorse or promote products derived from this software without
59+
# specific prior written permission.
60+
#
61+
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
62+
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
63+
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
64+
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
65+
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
66+
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
67+
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
68+
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
69+
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
70+
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
71+
# POSSIBILITY OF SUCH DAMAGE.
72+
173
r"""
274
`torch.distributed.launch` is a module that spawns up multiple distributed
375
training processes on each of the training nodes.

‎PyTorch/Translation/GNMT/scripts/docker/build.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,23 @@
11
#!/bin/bash
22

3+
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
4+
#
5+
# Permission is hereby granted, free of charge, to any person obtaining a copy
6+
# of this software and associated documentation files (the "Software"), to deal
7+
# in the Software without restriction, including without limitation the rights
8+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
# copies of the Software, and to permit persons to whom the Software is
10+
# furnished to do so, subject to the following conditions:
11+
#
12+
# The above copyright notice and this permission notice shall be included in all
13+
# copies or substantial portions of the Software.
14+
#
15+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
# SOFTWARE.
22+
323
docker build . --rm -t gnmt:latest

‎PyTorch/Translation/GNMT/scripts/docker/interactive.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,23 @@
11
#!/bin/bash
22

3+
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
4+
#
5+
# Permission is hereby granted, free of charge, to any person obtaining a copy
6+
# of this software and associated documentation files (the "Software"), to deal
7+
# in the Software without restriction, including without limitation the rights
8+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
# copies of the Software, and to permit persons to whom the Software is
10+
# furnished to do so, subject to the following conditions:
11+
#
12+
# The above copyright notice and this permission notice shall be included in all
13+
# copies or substantial portions of the Software.
14+
#
15+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
# SOFTWARE.
22+
323
nvidia-docker run --init -it --rm --ipc=host -v $PWD:/workspace/gnmt/ gnmt bash

‎PyTorch/Translation/GNMT/scripts/filter_dataset.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,23 @@
1+
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Permission is hereby granted, free of charge, to any person obtaining a copy
4+
# of this software and associated documentation files (the "Software"), to deal
5+
# in the Software without restriction, including without limitation the rights
6+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
# copies of the Software, and to permit persons to whom the Software is
8+
# furnished to do so, subject to the following conditions:
9+
#
10+
# The above copyright notice and this permission notice shall be included in all
11+
# copies or substantial portions of the Software.
12+
#
13+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19+
# SOFTWARE.
20+
121
import argparse
222
from collections import Counter
323

‎PyTorch/Translation/GNMT/scripts/tests/inference.sh‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,26 @@
11
#!/bin/bash
22

3+
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
4+
#
5+
# Permission is hereby granted, free of charge, to any person obtaining a copy
6+
# of this software and associated documentation files (the "Software"), to deal
7+
# in the Software without restriction, including without limitation the rights
8+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
# copies of the Software, and to permit persons to whom the Software is
10+
# furnished to do so, subject to the following conditions:
11+
#
12+
# The above copyright notice and this permission notice shall be included in all
13+
# copies or substantial portions of the Software.
14+
#
15+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
# SOFTWARE.
22+
23+
324
set -e
425

526
DATASET_DIR='data/wmt16_de_en'

‎PyTorch/Translation/GNMT/scripts/tests/reference_performance‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

0 commit comments

Comments
 (0)