From reading here I understood that I have to use all=True if I want to get all results, and then do as_list=False to get a generator to iterate over. But in fact this does not work well together. By specifying all=True it first starts fetching all results and generator does not call anything with results as pages are coming in.

I think all=True should just work with as_list=False. If I understand correctly, all=True is a noop in that case because as_list=False already implies it?