[RPC profiling] Add tests to ensure RPC profiling works on single threaded server #44923

rohan-varma · 2020-09-18T00:36:41Z

Stack from ghstack:

Fallback to CPU when remote end does not have CUDA for profiling #44967 Fallback to CPU when remote end does not have CUDA for profiling
[RPC profiling] Add tests to ensure RPC profiling works on single threaded server #44923 [RPC profiling] Add tests to ensure RPC profiling works on single threaded
server
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.
server**
[RPC profiling] Extend RPC profiling to support async function execution over RPC. #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC.
[RPC profiling] Don't wrap toHere() calls with profiling #44655 [RPC profiling] Don't wrap toHere() calls with profiling
[RPC profiling] Allow disableProfiler() to be called from another thread. #44653 [RPC profiling] Allow disableProfiler() to be called from another thread.
Remove thread_local RecordFunctionGuard from profiler. #44646 Remove thread_local RecordFunctionGuard from profiler.

server

This ensures that RPC profiling works in single-threaded server
scenarios and that we won't make the assumption that we'll have multiple
threads when working on this code. For example, this assumption resulted in a
bug in the previous diff (which was fixed).

Differential Revision: D23691304

…eaded server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed) Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

…eaded server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed) Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) ghstack-source-id: 112341818 Pull Request resolved: #44923

codecov · 2020-09-18T07:12:12Z

Codecov Report

Merging #44923 into gh/rohan-varma/175/base will decrease coverage by 0.00%.
The diff coverage is 72.50%.

@@                     Coverage Diff                     @@
##           gh/rohan-varma/175/base   #44923      +/-   ##
===========================================================
- Coverage                    68.04%   68.03%   -0.01%     
===========================================================
  Files                          393      393              
  Lines                        51019    51085      +66     
===========================================================
+ Hits                         34714    34757      +43     
- Misses                       16305    16328      +23

Impacted Files	Coverage Δ
...orch/testing/_internal/distributed/rpc/rpc_test.py	`26.62% <72.50%> (+1.07%)`	⬆️
torch/testing/_internal/expecttest.py	`77.55% <0.00%> (-1.03%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9250e79...9afb200. Read the comment docs.

dr-ci · 2020-09-18T07:12:24Z

💊 CI failures summary and remediations

As of commit 9afb200 (more details on the Dr. CI page):

1/1 failures possibly* introduced in this PR
- 1/1 non-CircleCI failure(s)

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm3.7-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 34 times.

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

…eaded server Pull Request resolved: #44923 This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed) ghstack-source-id: 112548171 Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/)

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

lw · 2020-09-22T18:31:19Z

torch/testing/_internal/distributed/rpc/rpc_test.py


+    @single_threaded_process_group_agent
+    @dist_init
+    def test_profiler_with_sync_rpc_udf_single_threaded(self):


In the tests above you renamed the test method to a helper function and then created two new test methods, with different decorators, which invoked that helper function. However in this test case, and in the one that follows, you duplicated the test code. To avoid duplication could we also use the helper method approach in these cases?

@lw I see, so do you mean that the following lines (which are repeated across the 2 tests as you mentioned):

self._profiler_test_with_rpc(RPCExecMode.SYNC, my_sleep_func, args=(1,)) self._profiler_test_with_rpc(RPCExecMode.SYNC, my_sleep_func, args=(1,), use_record_function=True)

should be deduped and turned into a single test function?

Yes, exactly. This test case here is rather short so de-duplicating doesn't bring a huge advantage, but as we go further down in this file the test cases become longer and more complex.

What I am suggesting is, for example, to do this:

def _profiler_with_sync_rpc_udf_helper(self): self._profiler_test_with_rpc(RPCExecMode.SYNC, my_sleep_func, args=(1,)) self._profiler_test_with_rpc(RPCExecMode.SYNC, my_sleep_func, args=(1,), use_record_function=True) @dist_init def test_profiler_with_sync_rpc_udf(self): self._profiler_with_sync_rpc_udf_helper() @single_threaded_process_group_agent @dist_init def test_profiler_with_sync_rpc_udf_single_threaded(self): self._profiler_with_sync_rpc_udf_helper()

And a similar thing for the tests that follow.

Sounds good, just updated.

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

lw

Awesome, thanks!

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

… single threaded server" server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server** * #44664 [RPC profiling] Extend RPC profiling to support async function execution over RPC. * #44655 [RPC profiling] Don't wrap toHere() calls with profiling * #44653 [RPC profiling] Allow disableProfiler() to be called from another thread. * #44646 Remove thread_local RecordFunctionGuard from profiler. server This ensures that RPC profiling works in single-threaded server scenarios and that we won't make the assumption that we'll have multiple threads when working on this code. For example, this assumption resulted in a bug in the previous diff (which was fixed). Differential Revision: [D23691304](https://our.internmc.facebook.com/intern/diff/D23691304/) [ghstack-poisoned]

facebook-github-bot · 2020-09-25T22:15:26Z

This pull request has been merged in 7c5436d.

rohan-varma requested review from mrshenli, pritamdamania87 and zhaojuanmao as code owners September 18, 2020 00:36

pritamdamania87 approved these changes Sep 18, 2020

View reviewed changes

rohan-varma mentioned this pull request Sep 18, 2020

Fallback to CPU when remote end does not have CUDA for profiling #44967

Closed

rohan-varma added 2 commits September 21, 2020 12:37

rohan-varma added 2 commits September 22, 2020 00:28

lw reviewed Sep 22, 2020

View reviewed changes

rohan-varma added 2 commits September 23, 2020 14:23

lw approved these changes Sep 24, 2020

View reviewed changes

rohan-varma added 2 commits September 24, 2020 09:45

facebook-github-bot closed this in 7c5436d Sep 25, 2020

facebook-github-bot added the merged label Sep 25, 2020

facebook-github-bot deleted the gh/rohan-varma/175/head branch September 29, 2020 14:23

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[RPC profiling] Add tests to ensure RPC profiling works on single threaded server #44923

[RPC profiling] Add tests to ensure RPC profiling works on single threaded server #44923

Uh oh!

rohan-varma commented Sep 18, 2020 •

edited

Loading

Uh oh!

codecov bot commented Sep 18, 2020 •

edited

Loading

Uh oh!

dr-ci bot commented Sep 18, 2020 •

edited

Loading

Uh oh!

lw Sep 22, 2020

Uh oh!

rohan-varma Sep 22, 2020 •

edited

Loading

Uh oh!

lw Sep 23, 2020

Uh oh!

rohan-varma Sep 23, 2020

Uh oh!

lw left a comment

Uh oh!

facebook-github-bot commented Sep 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[RPC profiling] Add tests to ensure RPC profiling works on single threaded server #44923

[RPC profiling] Add tests to ensure RPC profiling works on single threaded server #44923

Uh oh!

Conversation

rohan-varma commented Sep 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dr-ci bot commented Sep 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

ci.pytorch.org: 1 failed

Uh oh!

lw Sep 22, 2020

Choose a reason for hiding this comment

Uh oh!

rohan-varma Sep 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lw Sep 23, 2020

Choose a reason for hiding this comment

Uh oh!

rohan-varma Sep 23, 2020

Choose a reason for hiding this comment

Uh oh!

lw left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rohan-varma commented Sep 18, 2020 •

edited

Loading

codecov bot commented Sep 18, 2020 •

edited

Loading

dr-ci bot commented Sep 18, 2020 •

edited

Loading

rohan-varma Sep 22, 2020 •

edited

Loading