@ezyang @dzhulgakov please review

yes, I've tested e2e, tests are here onnxbot/onnx-fb-universe#600. I may have to make some small tweaks to make sure that I can land all the pieces without breaking tests in between

Looks good! (I wonder if we can make the code shorter somehow though :)

@ezyang yeah I think there probably is an opportunity to share some more code, but I didn't go for it right away because

1 - there's a small constant number (3) of RNN types, so the scope of the messiness is limited in that dimension

2 - there's a small constant number of features, and this is pretty much the last of them, so the messiness shouldn't get much worse in this dimension either