Skip to content

Add binary to benchmark model load speed #74700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
SS-JIA wants to merge 12 commits into from
Closed

Add binary to benchmark model load speed #74700

SS-JIA wants to merge 12 commits into from

Conversation

@SS-JIA
Copy link
Contributor

@SS-JIA SS-JIA commented Mar 24, 2022 •
edited
Loading

Stack from ghstack:

Differential Revision: D35124881

This was referenced Mar 24, 2022
Closed
Closed
SS-JIA added a commit that referenced this pull request Mar 24, 2022
@SS-JIA
Add binary to benchmark model load speed
90e0e63 
ghstack-source-id: 927e467
Pull Request resolved: #74700
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Mar 24, 2022 •
edited
Loading

🔗 Helpful links

💊 CI failures summary and remediations

As of commit 1e13a90 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@SS-JIA
Copy link
Contributor Author

SS-JIA commented Mar 24, 2022

@SS-JIA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

SS-JIA pushed a commit that referenced this pull request Mar 24, 2022
Add binary to benchmark model load speed
616aa96 
ghstack-source-id: 0c072e2
Pull Request resolved: #74700
@SS-JIA SS-JIA requested review from beback4u and kimishpatel March 24, 2022 23:28
beback4u
beback4u approved these changes Mar 24, 2022
View reviewed changes
binaries/load_benchmark_torch.cc
#include <vector>

#include <ATen/ATen.h>
#include "caffe2/core/timer.h"
Copy link
Contributor

@beback4u beback4u Mar 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we re-organize the order of includes?

@SS-JIA SS-JIA mentioned this pull request Mar 25, 2022
Closed
@SS-JIA
Copy link
Contributor Author

SS-JIA commented Mar 25, 2022

@SS-JIA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@SS-JIA
Copy link
Contributor Author

SS-JIA commented Mar 25, 2022

@SS-JIA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

SS-JIA pushed a commit that referenced this pull request Mar 25, 2022
Add binary to benchmark model load speed
6cc7aad 
ghstack-source-id: 29acc17
Pull Request resolved: #74700
@SS-JIA
Copy link
Contributor Author

SS-JIA commented Mar 25, 2022

@SS-JIA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@SS-JIA
Copy link
Contributor Author

SS-JIA commented Mar 28, 2022

@SS-JIA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

SS-JIA pushed a commit that referenced this pull request Mar 28, 2022
Add binary to benchmark model load speed
9f0d3a4 
ghstack-source-id: e184a83
Pull Request resolved: #74700
@SS-JIA
Copy link
Contributor Author

SS-JIA commented Mar 28, 2022

@SS-JIA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@SS-JIA SS-JIA mentioned this pull request Mar 28, 2022
Closed
SS-JIA pushed a commit that referenced this pull request Mar 28, 2022
Add binary to benchmark model load speed
1eeb42a 
ghstack-source-id: 91691b1
Pull Request resolved: #74700
@SS-JIA
Copy link
Contributor Author

SS-JIA commented Mar 28, 2022

@SS-JIA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot pushed a commit that referenced this pull request Mar 30, 2022
@SS-JIA @facebook-github-bot
Add binary to benchmark model load speed (#74700)
96f9914 
Summary: Pull Request resolved: #74700

Test Plan:
Imported from OSS

Some results running this benchmark for a quantized CPU xirp14b model on a Pixel 5:

```
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "46749"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19261"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19235"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19396"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19486"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19562"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19566"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19559"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19632"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19938"}
```

Some results running this benchmark for the Vulkan xirp20a model on Pixel 5, after pre-loading the Context:

```
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "38664"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19921"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20316"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20255"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20219"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20329"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20463"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "21072"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20668"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20889"}
```

Without pre-loading Context:

```
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "70850"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19867"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20211"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20039"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20082"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20268"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20363"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "21103"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20511"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20528"}

```

Reviewed By: mrshenli

Differential Revision: D35124881

Pulled By: SS-JIA

fbshipit-source-id: 0f093e4aa45d69c538a4fe2003e0d5617d72b97a
pytorchmergebot pushed a commit that referenced this pull request Mar 30, 2022
@SS-JIA @pytorchmergebot
Add binary to benchmark model load speed (#74700)
be71777 
Summary: Pull Request resolved: #74700

Test Plan:
Imported from OSS

Some results running this benchmark for a quantized CPU xirp14b model on a Pixel 5:

```
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "46749"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19261"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19235"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19396"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19486"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19562"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19566"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19559"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19632"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19938"}
```

Some results running this benchmark for the Vulkan xirp20a model on Pixel 5, after pre-loading the Context:

```
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "38664"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19921"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20316"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20255"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20219"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20329"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20463"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "21072"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20668"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20889"}
```

Without pre-loading Context:

```
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "70850"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "19867"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20211"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20039"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20082"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20268"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20363"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "21103"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20511"}
PyTorchObserver {"type": "NET", "unit": "us", "metric": "latency", "value": "20528"}

```

Reviewed By: mrshenli

Differential Revision: D35124881

Pulled By: SS-JIA

fbshipit-source-id: 0f093e4aa45d69c538a4fe2003e0d5617d72b97a
(cherry picked from commit 96f9914)
@github-actions
Copy link
Contributor

github-actions bot commented Mar 30, 2022

Hey @SS-JIA.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

NesrineMHB pushed a commit to NesrineMHB/pytorch that referenced this pull request Apr 7, 2022
Add binary to benchmark model load speed
2ac038a 
ghstack-source-id: c34962a
Pull Request resolved: pytorch/pytorch#74700
@facebook-github-bot facebook-github-bot deleted the gh/SS-JIA/46/head branch April 30, 2022 14:17
@WBobby WBobby mentioned this pull request Aug 17, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kimishpatel kimishpatel Awaiting requested review from kimishpatel

1 more reviewer

@beback4u beback4u beback4u approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@SS-JIA @facebook-github-bot @beback4u