Rotational Unit of Memory: A Phase-coding Recurrent Neural Network with Associative Memory

Rumen Dangovski*, Li Jing*, Marin Soljacic
Massachusetts Institute of Technology

Demo

Different model's behavior on Copying memory task. The vector represents the hidden state of each model through time. They show different behaviors on reading, waiting and writing period.

Abstract

The concepts of unitary evolution matrices and associative memory have boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art performance in a variety of sequential tasks. However, RNN still have a limited capacity to manipulate long-term memory. To bypass this weakness the most successful applications of RNN use external techniques such as attention mechanisms. In this paper we propose a novel RNN model that unifies the state-of-the-art approaches: Rotational Unit of Memory (RUM). The core of RUM is its rotational operation, which is, naturally, a unitary matrix, providing architectures with the power to learn long-term dependencies by overcoming the vanishing and exploding gradients problem. Moreover, the rotational unit also serves as associative memory. We evaluate our model on synthetic memorization, question answering and language modeling tasks. RUM learns the Copying Memory task completely and improves the state-of-the-art result in the Recall task. RUM's performance in the bAbI Question Answering task is comparable to that of models with attention mechanism. We also improve the state-of-the-art result to 1.189 bits-per-character (BPC) loss in the Character Level Penn Treebank (PTB) task, which is to signify the applications of RUM to real-world sequential data. The universality of our construction, at the core of RNN, establishes RUM as a promising approach to language modeling, speech recognition and machine translation.

Citation

Bibilographic information

Rumen Dangovski*, Li Jing*, Marin Soljacic. "Rotational Unit of Memory: A Phase-coding Recurrent Neural Network with Associative Memory." arXiv preprint arXiv: 1706.09537 (2017)

@inproceedings{jing2017a,
    title={Rotational Unit of Memory: A Phase-coding Recurrent Neural Network with Associative Memory},
    author={Dangovski, Rumen and Jing, Li and Soljacic, Marin},
    journal={arXiv preprint arXiv: 1710.09537},
    year={2017}
}

Related Projects

Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs

Gated Orthogonal Recurrent Unit: On Learning to Forget