[Debugger Plugin] Fix bugs related to binary string tensors #1200

caisq · 2018-05-20T03:45:54Z

Binary strings (strings that cannot be encoded in JSON, as is the case
with output tensors of Summary ops) are now converted to a printable
format using Python's binascii.b2a_qp().
To prevent overloading the frontend and frontend-backend
communication, long string elements are truncated at 40 bytes.
Display dtype 'object' as 'string' for clarity.

* Binary strings (strings that cannot be encoded in JSON, as is the case with output tensors of Summary ops) is now converted to a printable format using Python's `binascii.b2a_qp()`. * To prevent overloading the frontend and frontend-backend communication, long string elements are truncated at 40 bytes. * Display dtype 'object' as 'string' for clarity.

caisq · 2018-05-20T03:52:31Z

@chihuahua See screenshot for the effect of this CL.

caisq · 2018-05-21T14:18:19Z

@chihuahua @nfelt It seems that the Travis test jobs are failing for unrelated reasons:

Traceback (most recent call last):
File "/home/travis/.bazel-output-base/bazel-sandbox/7413091653513360136/execroot/org_tensorflow_tensorboard/bazel-out/k8-fastbuild/bin/tensorboard/plugins/pr_curve/pr_curves_plugin_test.runfiles/org_tensorflow_tensorboard/tensorboard/plugins/pr_curve/pr_curves_plugin_test.py", line 27, in
import tensorflow as tf
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tensorflow/init.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tensorflow/python/init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

jart · 2018-05-21T16:36:09Z

If you run bazel test //tensorboard/... and it works, I'll use admin privileges to merge anyway.

This is a known issue. Kokoro switched to GCC 4.9 because NVidia and Docker, which isn't compatible with RHEL, CentOS, Ubuntu14, Travis, etc. I got them to reconsider that decision last week. We're still waiting on the new nightlies to go live. The same thing happened back in January. I'm reaching that point where I'm about to roll my own binaries. Would you support me?

chihuahua

Thank you Shanqing for fixing this bug!

chihuahua · 2018-05-23T06:11:02Z

tensorboard/plugins/debugger/tensor_helper.py

+def translate_dtype(dtype):
+  """Translate numpy dtype into a string.
+
+  The 'object' type is understood as a tenosrflow string and translated into


nit: TensorFlow

chihuahua · 2018-05-23T06:11:38Z

tensorboard/plugins/debugger/tensor_helper.py

+  """Process a buffer for human-readable display.
+
+  This function performs the following operation on each of the buffers in `s`.
+    1. Truncate input buffer if the length of the buffer is greater than


Maybe briefly offer some rationale on why we truncate in the first place.

chihuahua · 2018-05-23T06:15:17Z

tensorboard/plugins/debugger/tensor_helper.py

+    length = len(s)
+    if length > limit:
+      return (binascii.b2a_qp(s[:limit]) +
+              b' (length-%s truncated at %s bytes)' % (length, limit))


nit: I think both should be %ds.

chihuahua · 2018-05-23T06:15:42Z

tensorboard/plugins/debugger/tensor_helper.py

+    length = len(s)
+    if length > limit:
+      return (binascii.b2a_qp(s[:limit]) +
+              b' (length-%s truncated at %s bytes)' % (length, limit))


also nit: Any reason for this to be a binary string specifically (prefixed with b')?

This is for Python2-3 compatibility. Without the b, the string will end up being a unicode string in Python3 and cannot be concatenated with the output of binascii.b2a_qp.

caisq

Thanks a lot for the review! @chihuahua

caisq · 2018-05-23T16:58:29Z

tensorboard/plugins/debugger/tensor_helper.py

+def translate_dtype(dtype):
+  """Translate numpy dtype into a string.
+
+  The 'object' type is understood as a tenosrflow string and translated into


caisq · 2018-05-23T16:58:37Z

tensorboard/plugins/debugger/tensor_helper.py

+  """Process a buffer for human-readable display.
+
+  This function performs the following operation on each of the buffers in `s`.
+    1. Truncate input buffer if the length of the buffer is greater than


caisq · 2018-05-23T16:58:42Z

tensorboard/plugins/debugger/tensor_helper.py

+    length = len(s)
+    if length > limit:
+      return (binascii.b2a_qp(s[:limit]) +
+              b' (length-%s truncated at %s bytes)' % (length, limit))


This is for Python2-3 compatibility. Without the b, the string will end up being a unicode string in Python3 and cannot be concatenated with the output of binascii.b2a_qp.

caisq · 2018-05-23T17:15:16Z

@jart thanks for the info on the test failures. I've run the tests and made sure the tests all pass under both py2 and py3. So I'll merge the PR now.

@chihuahua Thanks again for your review!!

caisq · 2018-05-24T03:27:44Z

@jart This PR is ready for merging.

jart · 2018-05-24T03:30:10Z

I feel guilty merging when Travis is red, but it appears to be a pylint error, and we just removed that in #1207.

caisq · 2018-05-24T14:16:52Z

Thanks, @jart!

caisq added 2 commits May 19, 2018 23:40

remove debug print statements

3bd2608

caisq requested a review from chihuahua May 20, 2018 03:51

chihuahua approved these changes May 23, 2018

View reviewed changes

caisq commented May 23, 2018

View reviewed changes

caisq added 2 commits May 23, 2018 13:02

respond to review comments

b3c53a1

fix a test in py3

a787adc

jart merged commit 5f7331f into tensorflow:master May 24, 2018

[Debugger Plugin] Fix bugs related to binary string tensors #1200

[Debugger Plugin] Fix bugs related to binary string tensors #1200

Uh oh!

Conversation

caisq commented May 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

caisq commented May 20, 2018

Uh oh!

caisq commented May 21, 2018

Uh oh!

jart commented May 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chihuahua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caisq left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caisq commented May 23, 2018

Uh oh!

caisq commented May 24, 2018

Uh oh!

jart commented May 24, 2018

Uh oh!

caisq commented May 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

caisq commented May 20, 2018 •

edited

Loading

jart commented May 21, 2018 •

edited

Loading