@pytorchbot retest this please

It looks like the CI hook-up for libtorch is still not working. I strongly advise downloading the Docker image and testing off of that.

Yes, I’m doing that now. Sorry for the trial and error

I don't mind :) The advice is mostly for your sanity ;)

Ok, it looks like fails are genuine fails now (Can only index tensors with integral scalars (got CPUDoubleType). Linking issues seem to be gone.

I get a failure in a python multiprocessing test in pr/pytorch-linux-xenial-cuda9-cudnn7-py2, independent from the PR. I'll rerun the tests.

On the other hand, I need a suggestion for the failure in pr/pytorch-linux-xenial-py3-clang5-asan.

Building CXX object contrib/meter/CMakeFiles/test-meter.dir/test/basic.cc.o
09:01:03 src/ATen/test/CMakeFiles/scalar_test.dir/build.make:62: recipe for target 'src/ATen/test/CMakeFiles/scalar_test.dir/scalar_test.cpp.o' failed
09:01:03 g++-5: internal compiler error: Killed (program cc1plus)

It looks like when asan is activated the compiler crashes. Does this ring any bell @ezyang ?

Yes, I know exactly what the ASAN problem is. Look at the bottom of build.sh here:

if [[ "$BUILD_ENVIRONMENT" == *asan* ]]; then
  export ASAN_OPTIONS=detect_leaks=0:symbolize=1
  # Disable Valgrind tests in run_aten_tests.sh; otherwise
  # we'll be valgrind'ing an ASAN'ed binary!  ASANity.
  export VALGRIND=0

  sudo apt-get update
  sudo apt-get install clang-5.0

  export PATH="/usr/lib/llvm-5.0/bin:$PATH"

  # TODO: Figure out how to avoid hard-coding these paths
  LD_LIBRARY_PATH=/usr/lib/llvm-5.0/lib/clang/5.0.0/lib/linux \
    CC="sccache clang" \
    CXX="sccache clang++" \
    LDSHARED="clang --shared" \
    LDFLAGS="-stdlib=libstdc++" \
    CFLAGS="-fsanitize=address -shared-libasan" \
    NO_CUDA=1 \
    python setup.py install

  export LD_PRELOAD=/usr/lib/llvm-5.0/lib/clang/5.0.0/lib/linux/libclang_rt.asan-x86_64.so

else
  python setup.py install

fi

To ensure that the build smoketests properly load the ASAN runtime, we are fucking around with LD_PRELOAD. But your code is building with gcc, not clang. That's not right, and linking in clang's ASAN runtime is definitely bad juju.

I think the right thing to do is to make sure libtorch gets built with clang, and with ASAN. That means figuring out how to apply the flags in the invocation above to the cmake invocation, and also doing the build BEFORE we run the LD_PRELOAD. Another way we can make things safer is if we make the LD_PRELOAD opt-in for commands and then manually reapply it in the necessary cases (I think it was necessary in all of the cases, but not sure.)

Re the other test, it does seem like

09:30:37 ======================================================================
09:30:37 FAIL: test_backwards_fork (__main__.TestMultiprocessing)
09:30:37 backwards() should succeed when called before and after a fork
09:30:37 ----------------------------------------------------------------------
09:30:37 Traceback (most recent call last):
09:30:37   File "test_multiprocessing.py", line 436, in test_backwards_fork
09:30:37     self.assertFalse(p.is_alive())
09:30:37 AssertionError: True is not false
09:30:37 

has been flaky.

@pytorchbot retest this please

Thank you @ezyang! What I don't get is why the error is showing up on a build that does not involve libtorch, as libtorch is only built on pr/pytorch-linux-xenial-cuda9-cudnn7-py3 and pr/pytorch-linux-trusty-py3.6-gcc7.2. Could it be leftover CMakeCache.txt or the like?

Thanks for the hint @ezyang, I think I finally figured it out.

The crash was coming from gcc during the tools/run_aten_tests.sh step, on the asan build that does not involve libtorch. Note that ATen tests (that are built for all builds except *cuda*) are always built with gcc, even in asan builds.

Since the asan build has an export LD_PRELOAD after the main build, LD_PRELOAD is also seen from within run_aten_tests.sh.
It's always been that way even before this PR, but apparently the fact that we're using catch now in aten tests makes gcc crash. Why wasn't it crashing before? Not sure.

I now unset LD_PRELOAD in a subshell (to avoid having effects downstream) right before executing run_aten_tests.sh:

( unset LD_PRELOAD; time tools/run_aten_tests.sh )

This way the build succeeds and tests pass.

Current failures are unrelated, I tried re-testing but it didn't help. I'll try later.

It's not been a good weekend for CI.

@pytorchbot retest this please

@pytorchbot retest this please

Thanks @ezyang. Just a note: ATen should also get Catch2 as a submodule.