Thanks! Could you add a test in test_dataloader.py as well?

Personally I don't think that relying on iterator consumption order is a good idea. One accident next call can ruin everything.

@ssnl Done

@pytorchbot add to whitelist

Sorry I forgot to enable CI on this PR. Should be testing now.

@AlexanderRadionov do you have any benchmarks for the new data loader? Me and @colesbury are worried that it will lower the throughput, and it's important for training on e.g. 8x Volta GPUs.

I made some syntetic tests, just iterate by dataloader. And i have not see any issue.
Here are code:

import time
from datetime import datetime
import numpy as np
import random
import torch
from torch.utils.data import DataLoader
from torch.utils.data import Dataset

GLOBAL_SEED = 1024

def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

GLOBAL_WORKER_ID = None
def worker_init_fn(worker_id):
    global GLOBAL_WORKER_ID
    GLOBAL_WORKER_ID = worker_id
    set_seed(GLOBAL_SEED + worker_id)


class RandomAugmentationDataset(Dataset):
    def __init__(self, data):
        self.data = data

    def __getitem__(self, item):
        global GLOBAL_WORKER_ID
        return item, GLOBAL_WORKER_ID, self.data[item] + np.random.random_sample(self.data[item].shape)

    def __len__(self):
        return self.data.shape[0]

set_seed(GLOBAL_SEED)
data = np.random.random_sample((12800, 12800))
dataset = RandomAugmentationDataset(data)

s = time.time()
for i in range(0, 11):
    dataloader = DataLoader(dataset, batch_size=16, shuffle=True, num_workers=3, worker_init_fn=worker_init_fn, ind_worker_queue=True)
    for i, (items, worker_ids, batch) in enumerate(dataloader):
        pass
e = time.time()
print("time", e-s)

I run it with ind_worker_queue=True and ind_worker_queue=False several times.
False (default):
time 17.274786710739136
time 16.919730186462402
True (overrided):
time 16.521615266799927
time 16.571028470993042

I think my changes should not cause slowdown because individual worker behavior is not default.

I'm thinking more of a real ImageNet-like workload that actually has to read from disk, and some requests may get stalled for longer than others, slowing down the whole loading process (unlike previously).

I tried to emulate read time difference in my original code posted at head of merge request (random sleep). Here are results:

False (default):
time 123.02024149894714
time 123.15355587005615

True (overrided):
time 126.91952013969421
time 126.23874187469482

Do you think is it problem for non-default behavior? @colesbury @apaszke

@apaszke Yes the process will be slowed. But have reproducible results is very important. Do you have suggestions on how to do multiprocess deterministic dataloading in a faster way?

@AlexanderRadionov what was the duration of the sleep you've used? I'd still want to try this on a real machine before merging, but I think it's ok. The only change I'd request would be to remove the flag, and make this mode always enabled.

@apaszke I've used sleep random [0.001; 0.003]s duration.

Also I tested on real dataset data. Dataset have 96265 images, total 8,7G.

False (default)
3 workers: 237s
6 workers: 178s

True (overrided)
3 workers: 242s
6 workers: 177s

LGTM. I just want to run a real-world ImageNet benchmark and should be good to merge

@apaszke I have no enough resources to run ImageNet benchmark.

@AlexanderRadionov no worries, I wasn't requesting that from you. I was going to do it myself

Just FYI, another strategy to ensuring determinacy is storing PRNG states on shared memory. It doesn't requires fixing worker assignment, though its implementation is complex and serialization / deserialization overheads are incurred.

A PR to Chainer chainer/chainer#4230 uses this strategy, because fixing process assignment is impossible in Chainer as it uses multiprocessing.Pool.

@grafi-tt I'm not sure how this would work. You need to ensure that the sequence of locks for the PRNG will be the same at every run, but this is very hard or prohibitively slow if you sample multiple times while loading a data item (e.g. sample a mask, sample a rotation, ...).

@apaszke It's simple: cache the state. When a worker process is resumed, a PRNG state is deserialized from shared memory (by e.g. numpy.RandomState.set_state()). When the worker process yields a result, the PRNG state is serialized to shared memory (by e.g. numpy.RandomState.get_state()). During the worker is running, there is no overhead.

@grafi-tt oh I see. This makes sense but is still not ideal:

• you still need a bunch of synchronization, and everyone needs to lock the shared memory and put their random state in it, while everyone else is just waiting (this is a fairly expensive memcpy)
• you really want to make the random sequences in different workers to seem independent of each other. If I understand correctly you will can end up with this behavior with two workers:
  • worker 1 starts, reads state 0
  • worker 2 starts, reads state 0
  • both workers do some work and sample e.g. twice. since both started from the same random state, they both end up in state 1
  • worker 1 finishes, saves state 1
  • worker 2 finishes, saves state 1
  • then the process repeats, and the sampling results are identical in both workers.

@pytorchbot test this please

@pytorchbot retest this please

@pytorchbot retest this please

is this on pytorch 3.1?

@ssnl then were is this?

@ssnl when will it be released. I'd be really nice to have determinism in the data loader.

@AlexanderRadionov @ssnl @apaszke I agree that having reproducible results is very important, but that doesn't mean that you have to synchronize workers in the dataloader. Multiprocessing are working non-deterministically in nature and if you try to have some unnatural synchronization there must be some performance trade-off whether it's trivial or not. Also the synchronization makes the code more complicated and harder to maintain.

I think what you really need is a deterministic dataset, rather than a deterministic data loader.

set_seed(GLOBAL_SEED)

class SomeDataset(Datset):
    def __init__(self, data):
        self.data = data
        self.random_numbers = make_random_numbers_in_advance(len(self.data))

    def __getitem__(self, idx):
        random_number = self.random_numbers[idx]
        # do some works with random_number and self.data[idx]
        return processed_item