Skip to content

Commit adc8731

Browse files
ssnliotamudelta
ssnl
authored and
iotamudelta
committed
Add IterableDataset (pytorch#19228)
Summary: This is a modified version of pytorch#14705 since commit structure for that PR is quite messy. 1. Add `IterableDataset`. 3. So we have 2 data loader mods: `Iterable` and `Map`. 1. `Iterable` if the `dataset` is an instance of `IterableDataset` 2. `Map` o.w. 3. Add better support for non-batch loading (i.e., `batch_size=None` and `batch_sampler=None`). This is useful in doing things like bulk loading. 3. Refactor `DataLoaderIter` into two classes, `_SingleProcessDataLoaderIter` and `_MultiProcessingDataLoaderIter`. Rename some methods to be more generic, e.g., `get_batch` -> `get_data`. 4. Add `torch.utils.data.get_worker_info` which returns worker information in a worker proc (e.g., worker id, dataset obj copy, etc.) and can be used in `IterableDataset.__iter__` and `worker_init_fn` to do per-worker configuration. 5. Add `ChainDataset`, which is the analog of `ConcatDataset` for `IterableDataset`. 7. Import torch.utils.data in `torch/__init__.py` 9. data loader examples and documentations 10. Use `get_worker_info` to detect whether we are in a worker process in `default_collate` Closes pytorch#17909, pytorch#18096, pytorch#19946, and some of pytorch#13023 Pull Request resolved: pytorch#19228 Reviewed By: bddppq Differential Revision: D15058152 fbshipit-source-id: 9e081a901a071d7e4502b88054a34b450ab5ddde
1 parent c9fe379 commit adc8731

File tree

15 files changed

+1752
-443
lines changed

15 files changed

+1752
-443
lines changed

‎docs/source/data.rst‎

Lines changed: 398 additions & 1 deletion
Large diffs are not rendered by default.

‎docs/source/notes/cuda.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,7 @@ also preserve :class:`torch.device` and :class:`torch.dtype` of a Tensor).
262262
y_cpu = torch.ones_like(x_cpu)
263263
y_gpu = torch.zeros_like(x_gpu)
264264

265+
.. _cuda-memory-pinning:
265266

266267
Use pinned memory buffers
267268
^^^^^^^^^^^^^^^^^^^^^^^^^

‎docs/source/notes/multiprocessing.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
.. _multiprocessing-best-practices:
2+
13
Multiprocessing best practices
24
==============================
35

@@ -20,6 +22,8 @@ memory and will only send a handle to another process.
2022
This allows to implement various training methods, like Hogwild, A3C, or any
2123
others that require asynchronous operation.
2224

25+
.. _multiprocessing-cuda-note:
26+
2327
CUDA in multiprocessing
2428
-----------------------
2529

0 commit comments

Comments
 (0)