gradient

for which one?

The comment is on line 69, and referring to that op: tf.image.resize with method=ResizeMethod.NEAREST.

Note that the other two ops are loss functions, so (in training) their back-prop paths aways end up getting used. I believe, though I'm not totally certain, that the nondeterminism is introduced in the gradient of those functions too, but I didn't mention it because of this fact that they'll always be used in real models with the gradient (at least in training, even if the model is later used only in inference with no gradient).

When the exception-throwing code is added, it may end up being only added for the gradient kernel because that's the only path that is actually injecting nondeterminism. Whatever the functionality it is, the documentation will be updated to match.

One question here is whether Keras is in scope here as well? I believe in some conversations with @fchollet that there might be some non-determinism introduced in the Keras framework as well

So the other source of randomness in Keras (besides variable initializers) is going to be layer calls (like dropout).

We intend these to rely on stateless ops, seeded by a seed argument in call. That seed would be either provided by a per-model stateful RNG when calling the model, or if left unspecified would be autogenerated by a global RNG.

Both variable initializers and layers currently use legacy random ops, which are made deterministic when enabling determinism (see section "Random ops"). Stateless ops are always deterministic. So Keras will be OK.

In support of what Reed wrote above, in my experience, all nondeterminism seen (and resolved) when using Keras have been due to underlying issues with non-keras TensorFlow.

Would we also do these inconsistency checks for the 2 environ variables mentioned above?

No, because unlike the tf.data flag, these environmental variables are treated as False by default. If users enable deterministic using this API, we don't want to require users to also set environmental variables to avoid an error.

Also, the environment variables will be deprecated, and their functionality will not be further enhanced.

Custom ops? What op set is the determinism guarantee for?

Good point. Since we do not control custom ops, users must ensure custom ops (if any) are deterministic.

I think this should be "Allow users to enable deterministic behavior in TensorFlow."

This could be misleading. You don't have to use exactly the same hardware instance, just the same hardware configuration. So, in the single accelerator case, that might be the same accelerator silicon architecture. I assume that we're not addressing multi-accelerator determinism here. That's another topic.

I suggest: Use the same hardware configuration in every run.

In the case of GPUs, this includes CUDA version, cuDNN version, and also (even, possibly) driver version. A change in any of these parameters between runs could lead to non-bit-exactness between runs.

Added CUDA version. Didn't add others for brevity (they are covered by the "etc")

Python's random module is not nondeterministic. As far as I know, it supports full deterministic functionality, but to enable determinism the user just has to initialize it (as she would for any other compute process from which she desired deterministic functionality).

Suggested change: Not use constructs outside TensorFlow that are nondeterministic, such as Python’s random module (without appropriate use of its ability to be initialized/seeded) or using multiple threads/processes in ways that influence TensorFlow’s behavior.

Used a slightly shorter version.

There is at least one case (that I'm aware of) where third-party multi-GPU training systems (in this case Horovod) can introduce nondeterminism unless they are configured specifically to operate deterministically.

Assuming such systems have a dependency on TensorFlow, they should query deterministic_execution_enabled and if it's true, either enable determinism or raise a warning/error. I don't think it's worth mentioning here.

I would change life sciences to medicine.

An op either operates nondeterministically, or throws and exception. If we miss an exception, then the nondeterminism will prevent a user from running their model deterministically with determinism enabled. So adding a valid exception throw will not realistically break anything.

When deterministic functionality is added, it's usually added with a test to confirm deterministic functionality of the op (at the op level). Testing of deterministic gradients is done using an approach that I call gradient injection.

I wonder if it's worth clarifying here:

When deterministic execution is enabled, tf.errors.UnimplementedError will be thrown if a nondeterministic code path through an op would otherwise be traversed (either in eager mode or during graph execution). Note that it may still be possible to simply construct a graph containing ops that have nondeterministic code paths through them without the error being thrown.

I added the sentences

Certain ops will only raise an error for certain input shapes or attributes. Depending on the op, in graph mode, the error will either be raised when the op is constructed or when the op is run.

The first sentence addresses your first point, as it implies that only certain nondeterministic codepaths raise errors. For you second point: it depends on whether the op raises an error when it is constructed or run. I think both are acceptable, so in the RFC I left this up to each op's author.

increases -> increase

Alternatively, suggest: "deterministic functionality, enabled by deterministic ops, increases"

"For ops which we have not yet implemented a deterministic version, a NotImplementedError will be raised" (grammar) -> "A tf.errors.UnimplementedError will be raised by ops for which we have not yet implemented a deterministic version."

Also, throughout, NotImplementedError -> tf.errors.UnimplementedError

"Certain ops will only raise an error for certain input shapes or attributes." -> "Some ops will only raise an error on a subset of input shapes, attributes, data types, or data-paths through the op."

"Use same hardware configuration ..." -> "Use the same hardware configuration ..."

"Use the same software environment every run ..." -> "Use the same software environment in every run ..." in or on - consistent with last point.

Most forward tests are too small and/or not designed in some other way to test for determinism. For example, many tests use integers even when exercising floating-point data-paths (which will miss floating-point-non-associative-rounding-error-based nondeterminism). To make a test that is likely to exercise nondeterminism in a forward-path, it needs to use the op in a natural way, which most unit tests don't, or don't do much of. Most test cases are too small to exercise the kind of nondeterminism we want to protect against, which shows up with test cases that span asynchronous compute engines.

For the backwards paths (which is where nondeterminism usually shows up), the way that unit tests are written, comparing analytical and numerical Jacobian matrices, will not catch nondeterminism. This is because the analytical Jacobian matrix does not capture the backprop effect on real upstream gradients. The existing tests literally cannot see the nondeterminism. This is one of the factors that motivated me to develop the gradient injection approach to testing backprop determinism. Apart from that, even if the analytical Jacobians could capture the nondeterminism, the test cases would have to be made prohibitively large. The traditional method of gradient function testing, using the Jacobians, requires that the test cases be necessarily small (because the size of the Jacobians explodes in O(Nin*Nout), where N is the number of elements).

Automatic generation of effective determinism tests is a hard problem that I've been thinking about and discussing with others for some time. It will be an interesting problem to solve.

How about a test that automatically and randomly (based on a seed that makes its test case reproducible) creates a test case for one op path every time it runs. It would be a very complicated test because it would need to know, ultimately, how to use every op in TensorFlow. This test could be run at regular intervals, outside of the main CI process (I don't know what you call that; nightly?) and failures could be used to catch op-determinism regressions. It wouldn't automatically catch and block the introduction of the regression, but it would, over time, randomly re-audit every op. That test could be put in place initially for a small number of ops, to test on all devices, data types, etc and more ops could be added to it over time. It would also need to know details of which op paths/configs would throw d9m-unimplemented exceptions. It might read that from a YAML file. It could keep a record of its progress in a single file somewhere: seed, op, params, result.

I've been discussing this with @nluehr. I completely misunderstood the proposal. You want to add this feature under op calls. Then the test suite becomes a ton of pre-generated op calls. This makes sense. Sorry I misunderstood and went off on a tangent.

Okay, so then here are the things I'm worried about:

1. Floats that that have no fractional component. Solution: add/sub a fractional component
2. Calls from the gradient checker where the upstream gradient input is just 1.0. Solution: replace the input with a fractional random number between 0.0 and 1.0.
3. Tensors are too small to cause compute to span asynchronous processors. There doesn't seem to be a simple/automated solution for this problem, and all backward op runs (based on gradient_checker.compute_gradient) are going to, necessarily, have tiny tensors.

Another class of important stimulus that would likely be missing from existing op tests is where the relationship between input elements specifically triggers the most-common cause of nondeterminism (atomic reduction between asynchronous processes). For example, in segment reduction it's necessary for indices to be repeated more than once (with a large input tensor).

If it was possible to solve the problem of making tensors larger (while not breaking the rules of the op API), then the other half of the problem could be solved by randomizing control inputs (such as segment IDs), but then that would also need to be done in such as way as to not break the rules of the op API.

I'm not sure what you originally thought I was proposing, but your issues in the first post make sense (although I didn't understand quite the Jacobian complaint) and your suggestions in the fourth post would fix most of them.

I wonder if we can somehow enlarge the tensors by tiling them on an arbitrary dimension (or trying every dimension in sequence, to make each dimension 10x larger). This is a lot trickier when there are multiple inputs, like Add and MatMul, but we can always catch errors and simply not test that op for determinism with this strategy.

I think it's probably infeasible to for this technique to catch cases where the relationship between input elements cause probable nondeterminism. I doubt we would have caught segment reductions with this technique, but we can try. We can rely on other methods if necessary, like looking for uses of atomics and testing real-world models.

I think your idea in the third post of having a test suite that reads from a YAML file is reasonable as well, although it requires a small amount of per-op work. Conceptually this is very similar to the approach I suggested, except the configs are hand-written instead of automatically generated based on input shapes and attributes in unit tests. Also in my approach, the determinism tests are run during the unit tests, but perhaps instead the input shapes/attributes/dtypes should be written to a file and afterwards the determinism tests can be run. Then we could augment the file with hand-written tests as well, so that we can give larger inputs that are more likely to demonstrate nondeterminism.

I don't think directed testing is redundant at all. The automated testing approach, described in the previous point, might (maybe) capture some small subset of issues, at potentially great additional compute cost, but it would miss most of the nondeterminism that we're trying to defend against.

A potentially more efficient and cost-effective way of defending against regressions might be to automatically flag PRs that contain suspect code, such as the use of CUDA atomic operations or sharding processes across CPU threads; then review those and/or require directed determinism tests (or tf.errors.UnimplementedError tests) for them.

It's true, though, that once an op is proven to be thoroughly deterministic using a human-developed directed test, it's very unlikely that the op will somehow start functioning nondeterministically again. So the non-automated, directed tests do primarily provide evidence that a previously nondeterministic op has been made to operate deterministically.

As you mentioned, it's unlikely the special mode would catch errors for an op which has an explicit unit tests, so there is slight redundancy. But I rephrased the sentence to avoid the word "redundancy".

I don't think it is worth it to flag PRs which have potentially unsafe constructs like atomics, since I think the effort to set that up will be high and there will be false positives. Others may disagree. /CC @sanjoy @pkanwar23