@davidcavazos it seems like these are the ones that are causing timeouts. Do you have any suspicions for which one might be the culprit or should I just split these up even further to see what happens? One thing I do notice is that some of the TF versions have a bigger bump than others - not sure if going to a new minor version could be causing problems. I do believe this TF version supports python 3.7 and 3.8 though and even if it didn't that wouldn't timeout

This is a great question! TensorFlow releases tend to be pretty large, and due to the way that pip resolves dependencies, it can take a long time to resolve dependencies without a constraints file. I know Dataflow comes with a pre-installed version of TensorFlow. I suspect we're maybe updating to a more recent version which causes delays in the workers setup. Splitting each PPAI sample into their own should help. However! We're updating the land cover sample (#8365) with an already up-to-date version of TF so you can safely skip that :)