[Tensorflow 2.0 nightly]: invalid info file ->packages/tensorboard/manager.py -> ValueError: incompatible version: {'cache_key': ..)

## Environment information (required)

Please run `diagnose_tensorboard.py` (link below) in the same
environment from which you normally run TensorFlow/TensorBoard, and
paste the output here:

## Diagnostics

<details>
<summary>Diagnostics output</summary>

``````
--- check: autoidentify
INFO: diagnose_tensorboard.py version 393931f9685bd7e0f3898d7dcdf28819fef54c43

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=6, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='Fabien-Tarrades-MacBook-Pro.local', release='18.6.0', version='Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tb-nightly==1.15.0a20190725
INFO: installed: tf-nightly-2.0-preview==2.0.0.dev20190725
INFO: installed: tensorflow-estimator-2.0-preview==1.14.0.dev2019072500

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.15.0a20190725'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '2.0.0-dev20190725'
INFO: tensorflow.__git_version__: 'v1.12.1-7023-g416942991e'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/bin/tensorboard\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): 'Fabien-Tarrades-MacBook-Pro.local'

--- check: stat_tensorboardinfo
INFO: directory: /var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=12916413176, st_dev=16777220, st_nlink=5, st_uid=501, st_gid=20, st_size=160, st_atime=1564128545, st_mtime=1564128545, st_ctime=1564128545)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.7.1
appnope==0.1.0
asn1crypto==0.24.0
astor==0.8.0
astroid==2.2.5
atomicwrites==1.3.0
attrs==19.1.0
autopep8==1.4.4
backcall==0.1.0
bleach==3.1.0
cachetools==2.1.0
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
cloudpickle==1.2.1
colorama==0.4.1
cryptography==2.3.1
cycler==0.10.0
dask==2.1.0
decorator==4.4.0
defusedxml==0.6.0
dill==0.3.0
entrypoints==0.3
future==0.17.1
gast==0.2.2
gitdb2==2.0.5
GitPython==2.1.11
google-api-core==1.13.0
google-auth==1.6.3
google-cloud-bigquery==1.17.0
google-cloud-core==1.0.2
google-cloud-storage==1.16.1
google-pasta==0.1.7
google-resumable-media==0.3.2
googleapis-common-protos==1.5.10
grpcio==1.22.0
h5py==2.9.0
idna==2.8
imageio==2.5.0
ipykernel==4.10.0
ipython==7.6.1
ipython-genutils==0.2.0
ipywidgets==7.4.1
isort==4.3.21
jedi==0.13.3
Jinja2==2.10.1
jsonschema==3.0.1
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.0
jupyter-core==4.5.0
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.4
jupyter-nbextensions-configurator==0.4.0
jupyterlab==0.34.9
jupyterlab-launcher==0.13.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.4.1
lime==0.1.1.36
lxml==4.3.4
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.0
mccabe==0.6.1
mistune==0.8.4
more-itertools==7.0.0
nbconvert==5.4.0
nbdime==1.0.7
nbformat==4.4.0
networkx==2.3
notebook==5.6.0
numpy==1.14.5
olefile==0.46
opt-einsum==2.3.2
pandas==0.24.2
pandocfilters==1.4.2
parso==0.5.0
patsy==0.5.1
pep8==1.7.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
pip==19.1.1
pluggy==0.6.0
prometheus-client==0.7.1
promise==2.2.1
prompt-toolkit==2.0.9
protobuf==3.8.0
psutil==5.6.3
ptyprocess==0.6.0
py==1.8.0
pyasn1==0.4.5
pyasn1-modules==0.2.4
pycodestyle==2.5.0
pycparser==2.19
pydocstyle==3.0.0
pyflakes==2.1.1
Pygments==2.4.2
pylama==7.4.3
pylint==2.3.1
pyOpenSSL==19.0.0
pyparsing==2.4.0
pyrsistent==0.14.11
PySocks==1.7.0
pytest==3.6.2
python-dateutil==2.8.0
pytz==2019.1
PyWavelets==1.0.3
PyYAML==5.1.1
pyzmq==18.0.0
qtconsole==4.5.2
requests==2.22.0
rsa==3.4.2
scikit-image==0.14.0
scikit-learn==0.20.1
scipy==1.1.0
seaborn==0.9.0
Send2Trash==1.5.0
setuptools==41.0.1
six==1.12.0
smmap2==2.0.5
snowballstemmer==1.9.0
statsmodels==0.10.1
tb-nightly==1.15.0a20190725
tensorflow-estimator-2.0-preview==1.14.0.dev2019072500
tensorflow-metadata==0.14.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
tf-nightly-2.0-preview==2.0.0.dev20190725
tfds-nightly==1.1.0.dev201907250105
toolz==0.10.0
tornado==4.5.3
tqdm==4.32.2
traitlets==4.3.2
typed-ast==1.3.4
urllib3==1.24.2
watermark==1.8.1
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.5
wheel==0.33.4
widgetsnbextension==3.4.2
wrapt==1.11.2

``````

</details>

### Next steps

For browser-related issues, please additionally specify:

  - Browser type and version (e.g., Chrome 64.0.3282.140): Firefox 68.0 (64 bits) on MacOS
  - Screenshot, if it’s a visual issue: None

## Issue description

Please describe the bug as clearly as possible. How can we reproduce the
problem without additional resources (including external data files and
proprietary Python modules)?

I am using TF 2.0 nightly on MacOS:

```
python 3.6.6
tb-nightly                1.15.0a20190725          pypi_0    pypi
tensorflow-estimator-2-0-preview 1.14.0.dev2019072500          pypi_0    pypi
tensorflow-metadata       0.14.0                   pypi_0    pypi
tf-nightly-2-0-preview    2.0.0.dev20190725          pypi_0    pypi
tfds-nightly              1.1.0.dev201907250105          pypi_0    pypi
```

I am creating a model in keras, convert it to an tf.estimator and train it using:

```
estimator = tf.keras.estimator.model_to_estimator(
    keras_model = model,
    config=training_config
)

tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
```

I can start TensorBoard in a cell:

```
# Load the TensorBoard notebook extension
%load_ext tensorboard
%tensorboard  --logdir   {FLAGS.model_dir} 
```

![image](https://user-images.githubusercontent.com/12021701/61939210-0cb8c600-af93-11e9-9683-9ee8742c3117.png)

It is working but some stuff is crashing because of "invalid info file".

```
W0726 10:09:01.353978 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-96299.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 96299, 'port': 6006, 'start_time': 1564000650, 'version': '1.15.0a20190724'}
W0726 10:09:01.357082 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-82744.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 82744, 'port': 6006, 'start_time': 1563876282, 'version': '1.15.0a20190722'}
W0726 10:09:01.868635 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-96299.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 96299, 'port': 6006, 'start_time': 1564000650, 'version': '1.15.0a20190724'}
W0726 10:09:01.872291 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-82744.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 82744, 'port': 6006, 'start_time': 1563876282, 'version': '1.15.0a20190722'}
W0726 10:09:02.373405 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-96299.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 96299, 'port': 6006, 'start_time': 1564000650, 'version': '1.15.0a20190724'}
W0726 10:09:02.374543 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-82744.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 82744, 'port': 6006, 'start_time': 1563876282, 'version': '1.15.0a20190722'}
W0726 10:09:02.879035 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-96299.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 96299, 'port': 6006, 'start_time': 1564000650, 'version': '1.15.0a20190724'}
W0726 10:09:02.880595 4583781824 manager.py:321] invalid info file: '/var/folders/l7/00kxfwvs0vbbqxtrp3rpf3yh0000gn/T/.tensorboard-info/pid-82744.info'
Traceback (most recent call last):
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 316, in get_all
    info = _info_from_string(contents)
  File "/Users/tarrade/anaconda-release/conda-env/env_gcp_dl_2_0_nightly/lib/python3.6/site-packages/tensorboard/manager.py", line 155, in _info_from_string
    raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyZXN1bHRzL01vZGVscy9NbmlzdC90Zl8xXzEyL2VzdGltYXRvci92My9ja3B0LyJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiIvVXNlcnMvdGFycmFkZS9EZXNrdG9wL1dvcmsvRGF0YV9TY2llbmNlL1R1dG9yaWFsc19Db2Rlcy9QeXRob24vcHJval9ETF9tb2RlbHNfYW5kX3BpcGVsaW5lc193aXRoX0dDUCJ9', 'db': '', 'logdir': 'results/Models/Mnist/tf_1_12/estimator/v3/ckpt/', 'path_prefix': '', 'pid': 82744, 'port': 6006, 'start_time': 1563876282, 'version': '1.15.0a20190722'}

```

Is this a TensorBoard/Tensorflow mismatch ? Issue on Keras/TensorFlow side or from tf.estimator ?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Tensorflow 2.0 nightly]: invalid info file ->packages/tensorboard/manager.py -> ValueError: incompatible version: {'cache_key': ..) #2467

Environment information (required)

Diagnostics

Next steps

Issue description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Tensorflow 2.0 nightly]: invalid info file ->packages/tensorboard/manager.py -> ValueError: incompatible version: {'cache_key': ..) #2467

Description

Environment information (required)

Diagnostics

Next steps

Issue description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions