Add Activations by JimClarke5 · Pull Request #123 · tensorflow/java

JimClarke5 · 2020-10-01T19:01:28Z

This is a redone PR to make sure the Activations branch did not rely on the Initializers branch.

I have removed all the no-arg CTORs. The original motivation was that I found previously in looking at Keras layers is that objects like “activations”, “loss” and “metric" most likely would need to be constructed
before the TensorFlow Ops is available within the model, hence I make constructors without Ops and used setTF() so the Model could set it later. The model itself hides Ops tf from the user. We have decided to address this requirement later when we start looking at Model.

I fixed HardSigmoid, as I forgot to apply a clipByValue operation on the final result. Also fixed the Unit test.

I have verified all the Unit tests against their TF Python corresponding methods.

For Softmax, we eliminated the IllegalArgmentException on 1D input that was in the Keras version.

In Swish, there is a comment on not handling the grad argument from the corresponding Python class. This is a Python optimization and will not be implemented in Java, at least at this time.

Fixed dependencies in pom.xml

…hod. This allows the NAME to be used elsewhere instead of hardcoding the string.

…coding the string. added methods isFloating(), isInteger(), isNUmeric(), isBoolean() and isString()

…ng OP_NAME" to each generated operation.

…EntropyWitLogits() Added tf.nn.sparesSoftmaxCrossEntropyWithLogits() and tf.nn.raw.sparesSoftmaxCrossEntropyWithLogits() Added tf.nn.sigmoidCrossEntropyWithLogits()

…Logits to org.tensorflow.op.nn.raw

…XXXX";

fix javadoc, fix casts

…yWithLogits.java to nn.raw, added new versions of these to NnOps

…maxCrossEntropyWithLogits()

…x JavaDoc. Change from snake case to camel case.

…Java files for some Ops. This also resulted in new generated source that are also committed.

…nerics.

…gmoidCrossEntropyWithLogits.java, and SparseSoftmaxCrossEntropyWithLogits.java under package org.tensorflow.op.nn in

…asy inclusion of a default optimizer. Cleaned up JavaDoc

Craigacp

Tiny things to do with casing.

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/ReLUTest.java

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/SoftmaxTest.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Activation.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ELU.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Exponential.java

deansher · 2020-10-03T17:02:50Z

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ReLU.java

+        return LeakyRelu.create(tf.scope(), input, LeakyRelu.alpha(alpha));
+      }
+      if (threshold != 0) {
+        negativePart =


What would you think about moving both the declaration and the initialization of negativePart down to immediately before it is used? I realize that would require a change to keep input as the original input instead of reusing it for intermediate calculations, but personally I'd see that as a feature not a bug.

Perhaps rename negativePart -> amountBelowThreshold? For one thing, negativePart is vague to me. For another, it's always positive.

I am not sure if this is used here as a math term, but I found Wolfram MathWorld - Negative Part. Perhaps @Craigacp and @karllessard might want to comment.

Good point, that Wolfram definition does show why negativePart was used in the first place. That said, I'd still advocate that in the presence of threshold, it's a confusing name.

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ReLU.java

deansher · 2020-10-03T17:38:52Z

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ReLU.java

+    if (clipMax) {
+      Operand<T> lmaxValue = tf.dtypes.cast(tf.constant(maxValue), dataType);
+      Operand<T> zero = tf.dtypes.cast(tf.constant(0), dataType);
+      input = tf.clipByValue(input, zero, lmaxValue);


Is zero still the right lower clip if there's a threshold?

The numbers less than the threshold get converted to zero before the clipMax is applied.
So, there is no need to move the lower bound on the clipByValue operation.

if (threshold != 0) { // computes input for input > threshold else 0 Greater greater = tf.math.greater(input, tf.dtypes.cast(tf.constant(threshold), dataType)); input = tf.math.mul(input, tf.dtypes.cast(greater, dataType));

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/SELU.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Sigmoid.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ELU.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/HardSigmoid.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/SELU.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Softmax.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Swish.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Tanh.java

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/ELUTest.java

deansher · 2020-10-05T12:00:25Z

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/ELUTest.java

+
+  /** Test of ELU call method */
+  @Test
+  public void testCallFloat() {


Perhaps also consolidate this sort of test logic into a helper class? E.g.

assertFloat32Output(tf -> new ELU<Float32>(tf), {1f, -0.86466473f, 3f, -0.9816844f, -0.63212055f, 2f, -0.95021296f, 4f}, {1, -2, 3, -4, -1, 2, -3, 4})

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Activation.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ELU.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Exponential.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ReLU.java

karllessard · 2020-10-15T02:29:04Z

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/ReLU.java

+    if (threshold != 0) {
+      // computes input for input > threshold else 0
+      Greater greater = tf.math.greater(input, tf.dtypes.cast(tf.constant(threshold), dataType));
+      input = tf.math.mul(input, tf.dtypes.cast(greater, dataType));


I would try to avoid changing the value of a method parameter, better use a new variable.

karllessard

@JimClarke5 , it looks like this PR can almost be merged, can you please go through all remaining comments and resolve them, either by a fix or by an answer? Thanks

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Softmax.java

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Tanh.java

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/SwishTest.java

Misc fixes to JavaDoc. In ReLU, change to assign to new variable 'lInput' rather than change the 'input' parameter.

…ng type, as they would no longer compile.

JimClarke5 · 2020-10-21T20:12:47Z

I have just pushed the changes requested, the biggest was adding TFloating in the core-api.

tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/types/TFloat64.java

karllessard

@Craigacp @deansher , there are a few discussions you've started in this PR that is still open, for me it looks like minor changes but can you can please have a quick check and give us your blessing to merge it in its current state or not?

deansher

Ship it! This is a huge step forward.

karllessard · 2020-10-24T16:19:22Z

Thanks @JimClarke5 !

JimClarke5 added 30 commits July 28, 2020 16:26

Initial checkin of Keras Optimzers and helper classes.

ef0ce67

Fixed dependencies in pom.xml

Added static final NAME to replace hardcoded String in the create met…

9c113a7

…hod. This allows the NAME to be used elsewhere instead of hardcoding the string.

Changed of method to use the DataType NAME attribute rather than hard…

824d487

…coding the string. added methods isFloating(), isInteger(), isNUmeric(), isBoolean() and isString()

Added method WriteFieldWithInitializer to output a "final static Stri…

07a83a5

…ng OP_NAME" to each generated operation.

Added tf.nn.softmaxCrossEntropyWitLogits() and tf.nn.raw.softmaxCross…

3d26831

…EntropyWitLogits() Added tf.nn.sparesSoftmaxCrossEntropyWithLogits() and tf.nn.raw.sparesSoftmaxCrossEntropyWithLogits() Added tf.nn.sigmoidCrossEntropyWithLogits()

Moved SoftmaxCrossEntropyWithLogits and SparseSoftmaxCrossEntropyWith…

11cda5f

…Logits to org.tensorflow.op.nn.raw

Generated classes now have public static final String OP_NAME = "XXXX…

9c7dfaa

…XXXX";

Generated classes now have public static final String OP_NAME = "XXXX…

84f49db

…XXXX";

fix dependencies for other Tensorflow Java modules

208b84a

formatting fix

3913161

Fix ctors with name to properly pass the name to the the super ctor.

b5a7c0f

change asserts to IllegalArgumentException

fcba0a5

fix javadoc, fix casts

change asserts to IllegalArgumentException

960cfc3

Moved back to tests

d37298a

Moved SoftmaxCrossEntropyWithLogits.java and SparseSoftmaxCrossEntrop…

c68812c

…yWithLogits.java to nn.raw, added new versions of these to NnOps

Deleted files that are not necessary yet

6b8eb26

Added nn.raw group for softmaxCrossEntropyWithLogits() and sparseSoft…

6515c24

…maxCrossEntropyWithLogits()

Added nn.raw group for softmaxCrossEntropyWithLogits() and sparseSoft…

76d0fe5

…maxCrossEntropyWithLogits()

Merge branch 'master' into master

d2201df

Refactor NN into individual operations under org.tensorflow.op.nn. Fi…

ab379d1

…x JavaDoc. Change from snake case to camel case.

Refactor NN into individual operations under org.tensorflow.op.nn. Fi…

889d67e

…x JavaDoc. Change from snake case to camel case.

Reformatted code

515b799

Added sub scope

5a9fe37

Miscellaneous fixes based on review comments.

8d21dd7

Fixed op_generator.cc to remove a spurious new line in the generated …

4c3cc78

…Java files for some Ops. This also resulted in new generated source that are also committed.

Changed back to non-generic Operand until we resolve how to handle ge…

44f530f

…nerics.

Regenerated due to creation of SoftmaxCrossEntropyWithLogits.java, Si…

b8d3ac2

…gmoidCrossEntropyWithLogits.java, and SparseSoftmaxCrossEntropyWithLogits.java under package org.tensorflow.op.nn in

change snake case to camel case. format code

c32fc5b

clean upd warning, format code

171cd2f

Added Adamax, Ftrl, and Nadam Optimizers. Added Optimizers enum for e…

e9c3134

…asy inclusion of a default optimizer. Cleaned up JavaDoc

added tests for 1D and 3D input

c7d0477

Craigacp requested changes Oct 2, 2020

View reviewed changes

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/ReLUTest.java Outdated Show resolved Hide resolved

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/SoftmaxTest.java Outdated Show resolved Hide resolved

Change snake case to camel case

de0e610

deansher suggested changes Oct 3, 2020

View reviewed changes

deansher suggested changes Oct 4, 2020

View reviewed changes

JavaDoc fixes

63c1f00

deansher reviewed Oct 5, 2020

View reviewed changes

karllessard mentioned this pull request Oct 14, 2020

Add Losses #129

Merged

karllessard reviewed Oct 15, 2020

View reviewed changes

karllessard reviewed Oct 19, 2020

View reviewed changes

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Softmax.java Outdated Show resolved Hide resolved

tensorflow-framework/src/main/java/org/tensorflow/framework/activations/Tanh.java Show resolved Hide resolved

KartikChugh previously approved these changes Oct 20, 2020

View reviewed changes

tensorflow-framework/src/test/java/org/tensorflow/framework/activations/SwishTest.java Show resolved Hide resolved

JimClarke5 added 5 commits October 21, 2020 15:04

Add TFloating family

2302cc5

Add JavaDoc

4c44c62

Changed to TFloating where appropriate.

ef29af9

Misc fixes to JavaDoc. In ReLU, change to assign to new variable 'lInput' rather than change the 'input' parameter.

Remove the test of int arguments for those classes changed to TFloati…

7519436

…ng type, as they would no longer compile.

Remove the test of int arguments for those classes changed to TFloati…

27c1126

…ng type, as they would no longer compile.

JimClarke5 dismissed KartikChugh’s stale review via 27c1126 October 21, 2020 20:10

JimClarke5 mentioned this pull request Oct 21, 2020

Implements both Tensor and NdArray interfaces from same instance #92

Closed

karllessard reviewed Oct 22, 2020

View reviewed changes

tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/types/TFloat64.java Show resolved Hide resolved

JimClarke5 added 3 commits October 22, 2020 09:56

Make LeakyRelu visible so that it is included in tf.nn.

b83f94f

Remove TNumber import

c59e905

Add tf.nn.leakyRelu operation

ebbcc4f

karllessard approved these changes Oct 23, 2020

View reviewed changes

deansher approved these changes Oct 24, 2020

View reviewed changes

Craigacp approved these changes Oct 24, 2020

View reviewed changes

karllessard approved these changes Oct 24, 2020

View reviewed changes

karllessard merged commit 6f3ec0f into tensorflow:master Oct 24, 2020

JimClarke5 deleted the Activation branch October 27, 2020 16:34

Conversation

JimClarke5 commented Oct 1, 2020

Uh oh!

Craigacp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karllessard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JimClarke5 commented Oct 21, 2020

Uh oh!

Uh oh!

karllessard left a comment

Choose a reason for hiding this comment

Uh oh!

deansher left a comment

Choose a reason for hiding this comment

Uh oh!

karllessard commented Oct 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants