RNNs with unitary matrices can be probably already built with Tensorflow or MxNet.