[JIT] Optionally validate nvfuser outputs after execution #74361
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
@torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
[ghstack-poisoned]
CI Flow Status⚛️ CI FlowRuleset - Version:
|
Contributor
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit 478c54b (more details on the Dr. CI page):
🕵️♀️ 2 failures not recognized by patterns:The following CI failures may be due to changes from the PR
This comment was automatically generated by Dr. CI (expand for details).Please report bugs/suggestions to the (internal) Dr. CI Users group. |
facebook-github-bot
added
the
oncall: jit
davidberard98
added a commit
that referenced
this pull request
Mar 17, 2022
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
ghstack-source-id: d41601c
Pull Request resolved: #74361
…tion"
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
[ghstack-poisoned]
Contributor
Author
|
@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
…tion"
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
Differential Revision: [D34975310](https://our.internmc.facebook.com/intern/diff/D34975310)
[ghstack-poisoned]
davidberard98
added a commit
that referenced
this pull request
Mar 17, 2022
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
ghstack-source-id: 86899be
Pull Request resolved: #74361
davidberard98
changed the title
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback.
```python
import torch
def callback(x, y, graph):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
print(graph)
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(True, callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
Differential Revision: [D34975310](https://our.internmc.facebook.com/intern/diff/D34975310)
[ghstack-poisoned]
Contributor
Author
|
@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback.
```python
import torch
def callback(x, y, graph):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
print(graph)
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(True, callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
Differential Revision: [D34975310](https://our.internmc.facebook.com/intern/diff/D34975310)
[ghstack-poisoned]
davidberard98
added a commit
that referenced
this pull request
Mar 21, 2022
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
ghstack-source-id: 6224fcd
Pull Request resolved: #74361
Contributor
Author
|
@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
eellison
approved these changes
Mar 21, 2022
Contributor
eellison
left a comment
eellison
left a comment
There was a problem hiding this comment.
Sweet!! LGTM - just a couple small comments. CC @jjsjann123 @kevinstephano may want to review as well
| auto copied_graph = fusion_node->g(attr::Subgraph)->copy(); | ||
| EraseShapeInformation(copied_graph); | ||
| enableAliasCopyNodes(copied_graph, copied_graph->block()); | ||
| InterpreterState{Code(copied_graph, "fallback_cuda_fuser")}.run(stack); |
Contributor
There was a problem hiding this comment.
We should really cache and save the InterpreterState code here. This fallback actually does get run in real networks, and this whole processes happens on each invocation
Contributor
There was a problem hiding this comment.
That probably shouldn't be in this pr - but something to do soon
| struct CudaFuserComparisonCallback { | ||
| using callback_type = | ||
| std::function<void(const Stack&, const Stack&, const std::string&)>; | ||
| bool run_fallback; |
Contributor
There was a problem hiding this comment.
When would this ever be false ?
Contributor
Author
There was a problem hiding this comment.
@eellison if you just want to collect the graph (e.g. for logging fusion groups), without running the fallback
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback.
```python
import torch
def callback(x, y, graph):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
print(graph)
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(True, callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
Differential Revision: [D34975310](https://our.internmc.facebook.com/intern/diff/D34975310)
[ghstack-poisoned]
davidberard98
added a commit
that referenced
this pull request
Mar 25, 2022
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
ghstack-source-id: dbd66e3
Pull Request resolved: #74361
Contributor
Author
|
@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
jjsjann123
reviewed
Mar 28, 2022
| TORCH_INTERNAL_ASSERT(stack.size() >= inputs_size); | ||
| stack_copy = Stack(); | ||
| stack_copy->insert( | ||
| stack_copy->end(), stack.begin(), stack.end() - inputs_size); |
Collaborator
There was a problem hiding this comment.
Should we assert on !stack_copy.empty()? Since we later (L297) only use to determine whether we re-run fallback.
Contributor
Author
There was a problem hiding this comment.
stack_copy is a c10::optional<> so conversion to bool will return true as long as it has a value - I can change the later if statement to if (stack_copy.has_value()) for clarity
Collaborator
There was a problem hiding this comment.
sorry I missed the optional part... code looks fine as-is.
jjsjann123
approved these changes
Mar 28, 2022
| TORCH_INTERNAL_ASSERT(stack.size() >= inputs_size); | ||
| stack_copy = Stack(); | ||
| stack_copy->insert( | ||
| stack_copy->end(), stack.begin(), stack.end() - inputs_size); |
Collaborator
There was a problem hiding this comment.
sorry I missed the optional part... code looks fine as-is.
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback.
```python
import torch
def callback(x, y, graph):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
print(graph)
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(True, callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
Differential Revision: [D34975310](https://our.internmc.facebook.com/intern/diff/D34975310)
[ghstack-poisoned]
davidberard98
added a commit
that referenced
this pull request
Mar 31, 2022
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation.o
```python
import torch
def callback(amt, x, y):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
ghstack-source-id: 51f3a0e
Pull Request resolved: #74361
Contributor
Author
|
@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
This was referenced Mar 31, 2022
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback.
```python
import torch
def callback(x, y, graph):
for i in range(len(x)-amt, len(x)):
print(x[i])
print(y[i])
print(graph)
with torch.jit.fuser("fuser2"):
torch._C._jit_nvfuser_set_comparison_callback(True, callback)
torch.jit.script
def g(x, y):
z = torch.add(x, y)
return torch.sin(z)
def f(x, y, a):
z = torch.add(x, y)
return g(torch.relu(z), a)
f_s = torch.jit.script(f)
x = torch.rand((10, 10), dtype=torch.half).cuda()
y = torch.rand((10, 10), dtype=torch.half).cuda()
a = torch.rand((10, 10), dtype=torch.half).cuda()
f_s(x, y, a)
f_s(x, y, a)
f_s(x, y, a)
```
Differential Revision: [D34975310](https://our.internmc.facebook.com/intern/diff/D34975310)
[ghstack-poisoned]
Contributor
Author
|
@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 1, 2022
Summary: Pull Request resolved: #74361 This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback. ```python import torch def callback(x, y, graph): for i in range(len(x)-amt, len(x)): print(x[i]) print(y[i]) print(graph) with torch.jit.fuser("fuser2"): torch._C._jit_nvfuser_set_comparison_callback(True, callback) torch.jit.script def g(x, y): z = torch.add(x, y) return torch.sin(z) def f(x, y, a): z = torch.add(x, y) return g(torch.relu(z), a) f_s = torch.jit.script(f) x = torch.rand((10, 10), dtype=torch.half).cuda() y = torch.rand((10, 10), dtype=torch.half).cuda() a = torch.rand((10, 10), dtype=torch.half).cuda() f_s(x, y, a) f_s(x, y, a) f_s(x, y, a) ``` Test Plan: Imported from OSS Reviewed By: eellison Differential Revision: D34975310 Pulled By: davidberard98 fbshipit-source-id: 2379c9a6f371cd58da6a187c1f16882f3923ab24
jjsjann123
pushed a commit
to jjsjann123/nvfuser
that referenced
this pull request
Oct 29, 2022
Summary: Pull Request resolved: pytorch/pytorch#74361 This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback. ```python import torch def callback(x, y, graph): for i in range(len(x)-amt, len(x)): print(x[i]) print(y[i]) print(graph) with torch.jit.fuser("fuser2"): torch._C._jit_nvfuser_set_comparison_callback(True, callback) torch.jit.script def g(x, y): z = torch.add(x, y) return torch.sin(z) def f(x, y, a): z = torch.add(x, y) return g(torch.relu(z), a) f_s = torch.jit.script(f) x = torch.rand((10, 10), dtype=torch.half).cuda() y = torch.rand((10, 10), dtype=torch.half).cuda() a = torch.rand((10, 10), dtype=torch.half).cuda() f_s(x, y, a) f_s(x, y, a) f_s(x, y, a) ``` Test Plan: Imported from OSS Reviewed By: eellison Differential Revision: D34975310 Pulled By: davidberard98 fbshipit-source-id: 2379c9a6f371cd58da6a187c1f16882f3923ab24 (cherry picked from commit 96c87992c65f5e6bb1bdd51791682dd837af99b4)
jjsjann123
pushed a commit
to jjsjann123/nvfuser
that referenced
this pull request
Nov 10, 2022
Summary: Pull Request resolved: pytorch/pytorch#74361 This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback. ```python import torch def callback(x, y, graph): for i in range(len(x)-amt, len(x)): print(x[i]) print(y[i]) print(graph) with torch.jit.fuser("fuser2"): torch._C._jit_nvfuser_set_comparison_callback(True, callback) torch.jit.script def g(x, y): z = torch.add(x, y) return torch.sin(z) def f(x, y, a): z = torch.add(x, y) return g(torch.relu(z), a) f_s = torch.jit.script(f) x = torch.rand((10, 10), dtype=torch.half).cuda() y = torch.rand((10, 10), dtype=torch.half).cuda() a = torch.rand((10, 10), dtype=torch.half).cuda() f_s(x, y, a) f_s(x, y, a) f_s(x, y, a) ``` Test Plan: Imported from OSS Reviewed By: eellison Differential Revision: D34975310 Pulled By: davidberard98 fbshipit-source-id: 2379c9a6f371cd58da6a187c1f16882f3923ab24 (cherry picked from commit 96c87992c65f5e6bb1bdd51791682dd837af99b4)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
This adds an optional validation after executing an NVFuser node, which checks that the output is the same as the unfused implementation. Then the outputs and the graph are reported via a callback.
Differential Revision: D34975310