# NOTE [ Data Loader Multiprocessing Shutdown Logic ]
    #
    # Preliminary:
    #
    # Our data model looks like this (queues are indicated with curly brackets):
    #
    #                main process                              ||
    #                     |                                    ||
    #               {index_queue}                              ||
    #                     |                                    ||
    #              worker processes                            ||     DATA
    #                     |                                    ||
    #            {worker_result_queue}                         ||     FLOW
    #                     |                                    ||
    #      pin_memory_thread of main process                   ||   DIRECTION
    #                     |                                    ||
    #               {data_queue}                               ||
    #                     |                                    ||
    #                data output                               \/
    #
    # P.S. `worker_result_queue` and `pin_memory_thread` part may be omitted if
    #      `pin_memory=False`.
    #
    #
    # Terminating multiprocessing logic requires very careful design. In
    # particular, we need to make sure that
    #
    #   1. The iterator gracefully exits the workers when its last reference is
    #      gone.
    #
    #      In this case, the workers should be gracefully exited because the
    #      main process may still need to continue to run, and we want cleaning
    #      up code in the workers to be executed (e.g., releasing GPU memory).
    #      Naturally, we implement the shutdown logic in `__del__` of
    #      DataLoaderIterator.
    #
    #      We delay the discussion on the logic in this case until later.
    #
    #   2. The iterator exits the workers when the problem ends
    #
    #      We set all workers and `pin_memory_thread` to have `daemon=True`.
    #      Doing so means that when the program ends, it shuts the workers down
    #      with a SIGTERM. `pin_memory_thread` will exit too, but by a different
    #      mechanism.
    #
    #      You may ask, why don't we just not set the workers as daemonic, and
    #      gracefully exit using the same logic as we have in `__del__` when the
    #      iterator gets deleted (see 1 above)? The answer requires a bit
    #      understanding of Python multiprocessing design. As of Python 3.7, for
    #      reasons I have yet to understand, in a subprocess, Python runs the
    #      given function (e.g., the `target` argument passed to a `mp.Process`)
    #      using this pattern (unrelated code removed for clarity):
    #
    #          # These are run the sub-process
    #          try:
    #              user_provided_function()
    #          finally:
    #              multiprocessing.util._exit_function()
    #
    #      In `_exit_function`, Python joins all non-daemonic subprocesses of
    #      this process (which is a subprocess of a Python process itself), and
    #      sends SIGTERM to the daemonic ones. Therefore, if a DataLoader is
    #      used in a subprocess (i.e., used in `user_provided_function` above),
    #      and an error is raised containing frames that references the
    #      DataLoaderIter (Python exception traces keeps local objects in
    #      relevant frames alive), workers will be joined in `_exit_function`
    #      before the `__del__` is called (which starts the shutdown logic). And
    #      unfortunately the DataLoaderIter process will hang. E.g., such errors
    #      can be timeout, or arbitrary error if users hold a reference to an
    #      iterator.
    #
    #      For context, `_exit_function` is also registered as an `atexit` call.
    #      So I really don't understand the need to do this in a finally block
    #      The code dates back to 2008 and there is no comment on the original
    #      PEP 371 or patch https://bugs.python.org/issue3050 (containing both
    #      the finally block and the `atexit` registration) that explains this.
    #
    #      Another choice is to just shutdown workers with logic in 1 above
    #      whenever we see an error in `next`. This isn't ideal because
    #        a. It prevents users from using try-catch to resume data loading.
    #        b. It doesn't prevent hanging if users have references to the
    #           iterator.
    #
    #   3. All processes exit if any of them die unexpectedly (e.g., error,
    #      SIGKILL).
    #
    #      As shown above, the workers are set as daemonic children of the main
    #      process. However, automatic cleaning-up of such child processes only
    #      happen if the parent process exits gracefully (e.g., SIGTERM). So we
    #      must ensure that each process will exit even the process that should
    #      send/receive data to/from it were killed, i.e.,
    #
    #        a. A process won't hang when getting from a queue.
    #
    #           Even with carefully designed data dependencies (i.e., a `put()`
    #           always corresponding to a `get()`), hanging on `get()` can still
    #           happen when data in queue is corrupted (e.g., due to
    #           `cancel_join_thread` or unexpected exit).
    #
    #           For child exit, we register SIGCHLD handler on main process,
    #           which checks if any of the workers fail in the (Python) handler.
    #           See DataLoader.cpp.
    #
    #           For `.get()` calls where the sender(s) is not the workers, we
    #           guard them with timeouts, and check the status of the sender
    #           when timeout happens:
    #             + in the workers, the `ManagerWatchdog` class checks the main
    #               process status.
    #             + if `pin_memory=True`, when getting from `pin_memory_thread`,
    #               check `pin_memory_thread` status periodically until `.get()`
    #               returns or see that `pin_memory_thread` died.
    #
    #        b. A process won't hang when putting into a queue;
    #
    #           We use `mp.Queue` which has a separate background thread to put
    #           objects. The background thread is usually automatically joined
    #           when the process exits.
    #
    #           However, in case that the receiver has ended abruptly while
    #           reading from the pipe, the join will hang forever. Therefore,
    #           for both `worker_result_queue` (worker -> main process/pin_memory_thread)
    #           and each `index_queue` (main process -> worker), we use
    #           `q.cancel_join_thread()` in sender process before any `q.put` to
    #           prevent this automatic join.
    #
    #           Moreover, having all queues called `cancel_join_thread` makes
    #           implementing graceful shutdown logic in `__del__` much easier.
    #           It won't need to get from any queue, which would also need to be
    #           guarded by periodic status checks.
    #
    #           Note that this may leave corrupted data in the queue, but we
    #           don't care about the data anyways once we are shutting down.
    #
    #
    # Now let's get back to 1:
    #   how we gracefully exit the workers when the last reference to the
    #   iteartor is gone.
    #
    # To achieve this, we implement the following logic along with the design
    # choices mentioned above:
    #
    # [pin_memory_thread] and [worker processes]
    #   When getting from queues,
    #     if get a `None`, exit.
    #     if get anything else or time out, check `done_event`,
    #        if set, keep getting until see the `None`, then exit.
    #        otherwise, process the data.
    #
    # [main process]
    #   In the DataLoader Iter's `__del__`
    #     a. Set `done_event` (shared with `pin_memory_thread` and workers).
    #
    #        Note: from here on, the workers & `pin_memory_thread` may exit at
    #              any time after they receive `None`.
    #
    #     b. Exit `pin_memory_thread`
    #          i.   Put `None` in `worker_result_queue`.
    #          ii.  Join the `pin_memory_thread`.
    #
    #     c. Exit the workers.
    #          i.   Put `None` in each worker's `index_queue`.
    #          ii.  Join the workers.
    #
    #        Note: This has to be after (b) because it may leave corrupted data
    #              in `worker_result_queue`, which `pin_memory_thread` reads
    #              from.
    #
    #   Note: If `pin_memory=False`, there is no `pin_memory_thread` and (b)
    #         can be omitted
    #
    # NB: `done_event`s isn't strictly needed. E.g., we can just check for
    #     `None` from `index_queue`, but it allows us to skip wasting resources
    #     processing indices already in `index_queue` if we are already shutting
    #     down.

 Exception ignored in: <function _DataLoaderIter.__del__ at 0x7facf66760d0>
 Traceback (most recent call last):
   File "/private/home/ssnl/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 412, in __del__
     self._shutdown_workers()
   File "/private/home/ssnl/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 408, in _shutdown_worke

     self.pin_memory_thread.join()
   File "/private/home/ssnl/miniconda3/lib/python3.7/threading.py", line 1032, in join
     self._wait_for_tstate_lock()
   File "/private/home/ssnl/miniconda3/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
     elif lock.acquire(block, timeout):
 KeyboardInterrupt

36: Exception ignored in: <function _DataLoaderIter.__del__ at 0x7f214fa412f0>
36: Traceback (most recent call last):
36:   File "/private/home/ssnl/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 416, in __
del__
36:     self._shutdown_workers()
36:   File "/private/home/ssnl/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 401, in _s
hutdown_workers
36:     self.worker_result_queue.join_thread()
36:   File "/private/home/ssnl/miniconda3/lib/python3.7/multiprocessing/queues.py", line 145, in join_thread
36:     self._jointhread()
36:   File "/private/home/ssnl/miniconda3/lib/python3.7/multiprocessing/util.py", line 189, in __call__
36:     res = self._callback(*self._args, **self._kwargs)
36:   File "/private/home/ssnl/miniconda3/lib/python3.7/multiprocessing/queues.py", line 192, in _finalize_join
36:     thread.join()
36:   File "/private/home/ssnl/miniconda3/lib/python3.7/threading.py", line 1032, in join
36:     self._wait_for_tstate_lock()
36:   File "/private/home/ssnl/miniconda3/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
36:     elif lock.acquire(block, timeout):
36: KeyboardInterrupt