Transformers are Multi-State RNNs

transformers can be thought of as infi- nite multi-state RNNs (MSRNNs)