Skip to content

Commit f94ae3b

Browse files
bddppqbddppq
authored
Update from facebook (#7696)
* Fix handling of empty batches in SumReduceDimsOp As titled * Deferrable async_scheduling finishRun fix Proper order of finishing run operations in deferrable_async_scheduling net * Simplify exception handling in async_scheduling Simplify exception handling, no need to busy wait, thread that processes the last task can finish the run * [C2]worker_coordinator_memorize_worker_ids As titled. This is related to T28689868, where the number of blobs we want to create is equal to the number of worker ids * Add unit test for nets with no type set * Ignore total length argument in sympolic_pad_packed_sequence 1- There was a mistake in the code that total_length was added to the wrong symbolic function (pack_padded_sequence) instead of (pad_packed_sequence) 2- No need to throw an exception if total_length is given since it is only used to enable data_parallel training on multi-gpus and doesn't have anything to do with onnx export, so just ignore it. https://fburl.com/tk4gciqp * Add support for MKLDNN to async_scheduling Just add MKLDNN as a possible CPU option to async_scheduling's pool function * [AuFL][ensemble] support branch output for prediction This diff supports using predictions from different branches and thus enables model ensembling (not fully independent). * Fix a bug in add_loss in layer_model_helper As titled. * Support lradaption for adam 1.lr adaption operator 2.apply to dense adam * Perf tweaks for async_scheduling Restore single pool option + remove unnecessary (no-ops) calls * add quantization to SparseSimdAdagradOp add a bunch of quantization signatures to SparseSimdAdagradOp, implementations to come next * [sr] [codemod] Change all SR callsites to use new API @allow-large-files This diff refactors all callsites of SR to use the slightly changed API introduced in the diff below. Really what this means is that you need to include the correct header. Also if you were using `ClientFactory::newFactory` you need to not prefix it with `ClientFactory::`. ``` cd ~/fbsource/fbcode find ./ -type f -exec sed -i -e 's:#include "servicerouter/client/cpp2/ClientFactory.h":#include "servicerouter/client/cpp2/ServiceRouter.h":' -e 's:#include <servicerouter/client/cpp2/ClientFactory.h>:#include <servicerouter/client/cpp2/ServiceRouter.h>:' -e 's/ClientFactory::newFactory(/newFactory(/g' {} \; ``` Also manually fixed spots that couldn't be done automatically (or broke because they depended on transitive includes). * Back out "Fix handling of empty batches in SumReduceDimsOp" Original commit changeset: 282da1730cc2 This commit is blocking the Github->fbcode sync, which really needs to get merged ASAP. D7881937 which this diff depends on will be reverted in the sync D7990948 which causes this to break. The sync diff cannot be patched with this reversion because it must be landed against base revision 5c8c099 , and D7881937 must not be included in the sync diff because it is breaking GPU tests that are not available in sandcastle : https://ci.pytorch.org/jenkins/job/caffe2-builds/job/py2-cuda8.0-cudnn6-ubuntu16.04-test/3638/console for one example. * Add the flow to support operator benchmark 1) generate model with the operator 2) upload to everstore 3) generate model spec into json file 4) start running the benchmark * [tum][gpu] Connect DPM trainer with flow and unit tests This diff: - Fix some small bugs for Yiming's recent changes to parallelizer, so it suits real use cases. - Add correct tags to the TUM code, so we can do data parallel transform - pass extra info when instantiation. - add unit test for using DPM in TUM model After this diff, we can do simple box, multi-gpu fully-sync trainer for TUM in Fblearner workflow, but may still need to do speed benchmarking. * w/o normalized lradaption for adam dense only The previous lr adaption includes a normalization step when performing the dot product operation. This is not exactly same as what is proposed in the paper. I add normalization as an option. Without it, the operator performs exactly what the paper proposed. With the option, we add the normalization step * [fb] Use SharedPromise in DeferrableAsyncSchedulingNet This code is to simplify DeferrableAsyncSchedulingNet by removing condition variable + small fixes * [tum] implement cuda sparseLengthsMean and LengthsMean as title * Adding an optional parameter to allow use of protobufs in InferShapesAndTypes function. Adding an optional parameter to allow use of protobufs in InferShapesAndTypes function. * Move feature_to_index to FeatureSpec.feature_to_index move feature_to_index to FeatureSpec.feature_to_index to avoid override other fields * [Caffe2] Rename bytes_moved to bytes_written Just a rename in preparation for supporting bytes_read. * [c2] fix ReduceFrontSumOp for empty case by setting 0 otherwise, it may use the results from last iteration when it's empty batch. * [Caffe2] [Int8] Improve Intel CPU performance * [Easy] Improve PrependDim op logging as titled * DBFileReader expand db_path using os.path.expanduser(..) Since there are a lot of possible use cases of `DBFileReader` to read from user home path, like `~/local/sample.db`, I want to save people's trouble of calling `os.path.expanduser(db_path)` themselves. * [Caffe2] Add bytes_read to cost structure We're adding analytical read bytes to cost functions. This extends the structure accordingly for all CostInference defined operators. Additionally, some small bug fixes were performed: 1) Cost functions now extract type information of operands instead of assuming float * Fix sleef on aarch64 for hhvm @bypass-lint Rename flag * Remove duplicated part in caffe2/ideep/operators/conv_op.cc should be sync error * Rename test helper function test_adagrad_sparse_helper to adagrad_sparse_test_helper to avoid confusing pytest
1 parent 2cb096a commit f94ae3b

40 files changed

+1293
-329
lines changed

‎aten/src/ATen/cpu/vec256/vec256_float.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66
#include <sleef.h>
77
#endif
88

9+
#include <iostream>
10+
911
namespace at {
1012
namespace vec256 {
1113
namespace {

‎binaries/bench_gen/bench_gen.py‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
#!/usr/bin/env python
2+
3+
from __future__ import absolute_import
4+
from __future__ import division
5+
from __future__ import print_function
6+
from __future__ import unicode_literals
7+
8+
import argparse
9+
10+
from caffe2.python.model_helper import ModelHelper
11+
from caffe2.python.predictor import mobile_exporter
12+
from caffe2.python import workspace, brew
13+
14+
15+
def parse_kwarg(kwarg_str):
16+
key, value = kwarg_str.split('=')
17+
try:
18+
value = int(value)
19+
except ValueError:
20+
try:
21+
value = float(value)
22+
except ValueError:
23+
pass
24+
return key, value
25+
26+
27+
def main(args):
28+
# User defined keyword arguments
29+
kwargs = {"order": "NCHW"}
30+
kwargs.update(dict(args.kwargs))
31+
32+
model = ModelHelper(name=args.benchmark_name)
33+
34+
op_type = args.operator # assumes a brew type op name
35+
input_name = args.input_name
36+
output_name = args.output_name
37+
38+
iters = int(args.iters)
39+
for i in range(iters):
40+
input_blob_name = input_name + (str(i) if i > 0 and args.chain else '')
41+
output_blob_name = output_name + str(i + 1)
42+
add_op = getattr(brew, op_type)
43+
add_op(model, input_blob_name, output_blob_name, **kwargs)
44+
if args.chain:
45+
input_name, output_name = output_name, input_name
46+
47+
workspace.RunNetOnce(model.param_init_net)
48+
49+
init_net, predict_net = mobile_exporter.Export(
50+
workspace, model.net, model.params
51+
)
52+
53+
if args.debug:
54+
print("init_net:")
55+
for op in init_net.op:
56+
print(" ", op.type, op.input, "-->", op.output)
57+
print("predict_net:")
58+
for op in predict_net.op:
59+
print(" ", op.type, op.input, "-->", op.output)
60+
61+
with open(args.predict_net, 'wb') as f:
62+
f.write(predict_net.SerializeToString())
63+
with open(args.init_net, 'wb') as f:
64+
f.write(init_net.SerializeToString())
65+
66+
67+
if __name__ == "__main__":
68+
parser = argparse.ArgumentParser(
69+
description="Utilitity to generate Caffe2 benchmark models.")
70+
parser.add_argument("operator", help="Caffe2 operator to benchmark.")
71+
parser.add_argument("-b", "--blob",
72+
help="Instantiate a blob --blob name=dim1,dim2,dim3",
73+
action='append')
74+
parser.add_argument("--context", help="Context to run on.", default="CPU")
75+
parser.add_argument("--kwargs", help="kwargs to pass to operator.",
76+
nargs="*", type=parse_kwarg, default=[])
77+
parser.add_argument("--init_net", help="Output initialization net.",
78+
default="init_net.pb")
79+
parser.add_argument("--predict_net", help="Output prediction net.",
80+
default="predict_net.pb")
81+
parser.add_argument("--benchmark_name",
82+
help="Name of the benchmark network",
83+
default="benchmark")
84+
parser.add_argument("--input_name", help="Name of the input blob.",
85+
default="data")
86+
parser.add_argument("--output_name", help="Name of the output blob.",
87+
default="output")
88+
parser.add_argument("--iters",
89+
help="Number of iterations to run the operator.",
90+
default="1")
91+
parser.add_argument("-d", "--debug", help="Print debug information.",
92+
action='store_true')
93+
parser.add_argument("-c", "--chain",
94+
help="Chain ops together (create data dependencies)",
95+
action='store_true')
96+
args = parser.parse_args()
97+
main(args)

‎binaries/benchmark_helper.cc‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -69,12 +69,16 @@ void setDeviceType(caffe2::NetDef* net_def, caffe2::DeviceType& run_dev) {
6969

7070
void setOperatorEngine(caffe2::NetDef* net_def, const string& backend) {
7171
if (backend != "builtin") {
72-
string engine = backend == "nnpack" ? "NNPACK"
73-
: backend == "eigen" ? "EIGEN"
74-
: backend == "mkl"
75-
? "MKLDNN"
76-
: backend == "cuda" ? "CUDA"
77-
: backend == "default" ? "" : "NONE";
72+
string engine = backend == "nnpack"
73+
? "NNPACK"
74+
: backend == "eigen" ? "EIGEN"
75+
: backend == "mkl" ? "MKLDNN"
76+
: backend == "cuda"
77+
? "CUDA"
78+
: backend == "dnnlowp" ? "DNNLOWP"
79+
: backend == "dnnlowp_16"
80+
? "DNNLOWP_16"
81+
: backend == "default" ? "" : "NONE";
7882
CAFFE_ENFORCE(engine != "NONE", "Backend is not supported");
7983
for (int i = 0; i < net_def->op_size(); i++) {
8084
caffe2::OperatorDef* op_def = net_def->mutable_op(i);

‎caffe2/core/net_async_base.cc‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,11 @@ CAFFE2_DEFINE_bool(
3636
true,
3737
"Select next non-busy stream");
3838

39+
CAFFE2_DEFINE_bool(
40+
caffe2_net_async_use_single_pool,
41+
false,
42+
"Use single pool for all devices");
43+
3944
namespace caffe2 {
4045

4146
thread_local std::vector<int> AsyncNetBase::stream_counters_;
@@ -109,7 +114,12 @@ std::shared_ptr<TaskThreadPool> AsyncNetBase::pool_getter(
109114

110115
std::shared_ptr<TaskThreadPool> AsyncNetBase::pool(
111116
const DeviceOption& device_option) {
112-
if (device_option.device_type() == CPU) {
117+
if (FLAGS_caffe2_net_async_use_single_pool) {
118+
return pool_getter(cpu_pools_, CPU, -1, num_workers_);
119+
}
120+
if (device_option.device_type() == CPU ||
121+
device_option.device_type() == MKLDNN ||
122+
device_option.device_type() == IDEEP) {
113123
auto numa_node_id = device_option.numa_node_id();
114124
CAFFE_ENFORCE(
115125
numa_node_id >= -1 &&
@@ -141,8 +151,8 @@ int AsyncNetBase::stream(int task_id) {
141151
do {
142152
stream_id = stream_counters_[gpu_id]++;
143153
stream_counters_[gpu_id] %= FLAGS_caffe2_streams_per_gpu;
144-
} while (!isStreamFree(task_id, stream_id) &&
145-
FLAGS_caffe2_net_async_check_stream_status);
154+
} while (FLAGS_caffe2_net_async_check_stream_status &&
155+
!isStreamFree(task_id, stream_id));
146156
}
147157
return stream_id;
148158
}
@@ -226,6 +236,16 @@ void AsyncNetBase::asyncWait(
226236
first_op->WaitEvents(events, stream_id);
227237
}
228238

239+
void AsyncNetBase::reset() {
240+
for (auto& op : GetOperators()) {
241+
op->ResetEvent();
242+
}
243+
#ifdef CAFFE2_USE_EXCEPTION_PTR
244+
std::unique_lock<std::mutex> exception_lock(exception_mutex_);
245+
caught_exception_ = nullptr;
246+
#endif // CAFFE2_USE_EXCEPTION_PTR
247+
}
248+
229249
void AsyncNetBase::storeExceptionPtr() {
230250
#ifdef CAFFE2_USE_EXCEPTION_PTR
231251
std::unique_lock<std::mutex> exception_lock(exception_mutex_);
@@ -236,6 +256,12 @@ void AsyncNetBase::storeExceptionPtr() {
236256
}
237257

238258
void AsyncNetBase::run(int task_id, int stream_id) {
259+
// Optionally insert async wait ops,
260+
// skip when using --caffe2_net_async_finish_chain -
261+
// all parents are guaranteed to be finished
262+
if (!FLAGS_caffe2_net_async_finish_chain) {
263+
asyncWait(task_id, stream_id, parents(task_id));
264+
}
239265
std::string err_msg;
240266
for (auto& op_id : chains_[task_id]) {
241267
auto& op = operators_[op_id];

‎caffe2/core/net_async_base.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,14 @@
1313
#include "caffe2/utils/proto_utils.h"
1414
#include "caffe2/utils/thread_pool.h"
1515

16+
CAFFE2_DECLARE_int(caffe2_streams_per_gpu);
17+
CAFFE2_DECLARE_bool(caffe2_net_async_finish_chain);
18+
CAFFE2_DECLARE_int(caffe2_net_async_max_gpus);
19+
CAFFE2_DECLARE_int(caffe2_net_async_max_numa_nodes);
20+
CAFFE2_DECLARE_int(caffe2_net_async_cpu_pool_size);
21+
CAFFE2_DECLARE_bool(caffe2_net_async_check_stream_status);
22+
CAFFE2_DECLARE_bool(caffe2_net_async_use_single_pool);
23+
1624
namespace caffe2 {
1725

1826
class AsyncNetExecutorHelper;
@@ -63,6 +71,8 @@ class AsyncNetBase : public NetBase {
6371

6472
bool isStreamFree(int task_id, int stream_id) const;
6573

74+
virtual void reset();
75+
6676
// Operator/task graph
6777
std::vector<OperatorBase*> operators_;
6878
std::vector<dag_utils::OperatorNode> operator_nodes_;

‎caffe2/core/net_async_polling.cc‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,6 @@ void AsyncPollingNet::schedule(int task_id) {
6464
task_timers_[task_id]->MicroSeconds());
6565
}
6666

67-
// Non-blocking wait, setups scheduling of dependent async computations;
68-
// canSchedule ensures that there's no busy wait,
69-
// for CUDA events we need to insert CUDA event synchronization to ensure
70-
// that async CUDA computations are executed in correct order
71-
asyncWait(task_id, stream_id, parents(task_id));
7267
try {
7368
if (FLAGS_caffe2_dag_net_collect_stats) {
7469
Timer run_time;

‎caffe2/core/net_async_scheduling.cc‎

Lines changed: 40 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,9 @@ AsyncSchedulingNet::AsyncSchedulingNet(
1515
}
1616

1717
void AsyncSchedulingNet::reset() {
18+
AsyncNetBase::reset();
19+
1820
processed_tasks_num_ = 0;
19-
cleanup_ = false;
2021
success_ = true;
2122

2223
for (auto task_id = 0; task_id < tasksNum(); ++task_id) {
@@ -37,8 +38,10 @@ void AsyncSchedulingNet::schedule(int task_id) {
3738
const auto& device_option = event(task_id).GetDeviceOption();
3839
pool(device_option)->run([this, task_id]() {
3940
if (success_) {
40-
int stream_id = stream(task_id);
41-
asyncWait(task_id, stream_id, parents(task_id));
41+
int stream_id = 0;
42+
if (FLAGS_caffe2_streams_per_gpu > 1) {
43+
stream_id = stream(task_id);
44+
}
4245
try {
4346
run(task_id, stream_id);
4447
} catch (const std::exception& e) {
@@ -51,9 +54,14 @@ void AsyncSchedulingNet::schedule(int task_id) {
5154
for (auto child_id : children(task_id)) {
5255
int parent_count = updateParentCount(child_id);
5356
if (parent_count == 0) {
54-
if (!success_ || cleanup_ ||
55-
FLAGS_caffe2_net_async_always_schedule_child ||
56-
canSchedule(child_id)) {
57+
// Schedule a child if:
58+
// - there is failure, we skip an op execution and finish the job
59+
// - forced scheduling though --caffe2_net_async_always_schedule_child
60+
// - --caffe2_net_async_finish_chain is set, in this case parents are
61+
// guaranteed to be finished
62+
// - in all other cases, check parents with canSchedule
63+
if (!success_ || FLAGS_caffe2_net_async_always_schedule_child ||
64+
FLAGS_caffe2_net_async_finish_chain || canSchedule(child_id)) {
5765
schedule(child_id);
5866
} else {
5967
const auto& device_option = event(child_id).GetDeviceOption();
@@ -64,37 +72,8 @@ void AsyncSchedulingNet::schedule(int task_id) {
6472
}
6573
}
6674

67-
if (success_) {
68-
if (task_count == tasksNum()) {
69-
// All tasks are finished, polling thread is sleeping;
70-
// only one thread enters here
71-
finalizeEvents();
72-
finishRun();
73-
return;
74-
}
75-
} else {
76-
// Before setting running_ to false and notifying waiters we need to
77-
// 1. Ensure that only one thread does the cleanup
78-
// 2. Ensure that all other pending tasks in workers and polling threads
79-
// are finished and
80-
// 3. Ensure that all tasks that were not scheduled have their events set
81-
{
82-
std::unique_lock<std::mutex> cleanup_lock(cleanup_mutex_);
83-
if (cleanup_) {
84-
return;
85-
}
86-
cleanup_ = true;
87-
}
88-
89-
// Errors are not recoverable and happen in exceptional cases,
90-
// ok to busy wait
91-
while (processed_tasks_num_ != tasksNum()) {
92-
}
93-
94-
// Make sure all events are set, wait for scheduled events
75+
if (task_count == tasksNum()) {
9576
finalizeEvents();
96-
97-
// Notify observers and waiters
9877
finishRun();
9978
}
10079
});
@@ -110,7 +89,7 @@ void AsyncSchedulingNet::pollAndSchedule(int task_id) {
11089
// - parents are ready
11190
// - we failed / cleanup started (no ops will run)
11291

113-
if (can_schedule || cleanup_ || !success_ || parent_failed) {
92+
if (can_schedule || !success_ || parent_failed) {
11493
schedule(task_id);
11594
} else {
11695
const auto& device_option = event(task_id).GetDeviceOption();
@@ -128,24 +107,38 @@ int AsyncSchedulingNet::updateParentCount(int child_id) {
128107
}
129108

130109
void AsyncSchedulingNet::finishRun() {
110+
{
111+
std::unique_lock<std::mutex> lock(running_mutex_);
112+
running_ = false;
113+
}
114+
131115
// notify observers and waiters
132116
StopAllObservers();
133-
running_ = false;
134117
running_cv_.notify_all();
135118
}
136119

137-
bool AsyncSchedulingNet::DoRunAsync() {
138-
std::unique_lock<std::mutex> lock(running_mutex_);
139-
CAFFE_ENFORCE(!running_, "Concurrent RunAsync calls");
140-
running_ = true;
141-
reset();
120+
bool AsyncSchedulingNet::RunAsync() {
121+
try {
122+
std::unique_lock<std::mutex> lock(running_mutex_);
123+
if (running_) {
124+
LOG(ERROR) << "Detected concurrent runs";
125+
return false;
126+
}
127+
running_ = true;
128+
reset();
142129

143-
StartAllObservers();
130+
StartAllObservers();
144131

145-
for (auto task_id = 0; task_id < tasksNum(); ++task_id) {
146-
if (parents(task_id).empty()) {
147-
schedule(task_id);
132+
for (auto task_id = 0; task_id < tasksNum(); ++task_id) {
133+
if (parents(task_id).empty()) {
134+
schedule(task_id);
135+
}
148136
}
137+
} catch (const std::exception& e) {
138+
LOG(ERROR) << "Exception while starting an async run: " << e.what();
139+
finalizeEvents();
140+
finishRun();
141+
return false;
149142
}
150143

151144
if (tasksNum() == 0) {

‎caffe2/core/net_async_scheduling.h‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,11 @@ class AsyncSchedulingNet : public AsyncNetBase {
1515
void Wait() override;
1616

1717
protected:
18-
bool DoRunAsync() override;
18+
bool RunAsync() override;
1919

2020
void pollAndSchedule(int task_id);
2121
void schedule(int task_id);
22-
void reset();
22+
void reset() override;
2323
virtual void finishRun();
2424
int updateParentCount(int child_id);
2525

@@ -28,9 +28,6 @@ class AsyncSchedulingNet : public AsyncNetBase {
2828
std::atomic<bool> running_;
2929
std::atomic<bool> success_;
3030

31-
std::mutex cleanup_mutex_;
32-
std::atomic<bool> cleanup_;
33-
3431
std::atomic<int> processed_tasks_num_;
3532

3633
DISABLE_COPY_AND_ASSIGN(AsyncSchedulingNet);

0 commit comments

Comments
 (0)