For large jobs with long epochs, it is necessary to checkpoint during the middle of an epoch, especially in a shared cluster setting with job time limit. It would be nice to be able to start at exactly where the training stopped and continue sampling for data afterwards.

I can see that directly trying to checkpoint dataloader involves lots of problems that's tied to saving the underlying dataset, etc. It's not a maintainable solution.

A compromise could be to save and retrieve the information as to which indices have already been sampled, and use those as part of the dataloder constructor, so that when the new iterator starts it avoids sampling the consumed data points.

I have searched through the forums for a few Q&As and it seems there is no clean solution.