Generative Temporal Models with Memory
Abstract
We consider the general problem of modeling temporal data with long-range dependencies,
wherein new observations are fully or partially predictable based on
temporally-distant, past observations. A sufficiently powerful temporal model
should separate predictable elements of the sequence from unpredictable elements,
express uncertainty about those unpredictable elements, and rapidly identify
novel elements that may help to predict the future. To create such models,
we introduce Generative Temporal Models augmented with external memory systems.
They are developed within the variational inference framework, which provides
both a practical training methodology and methods to gain insight into the
models’ operation. We show, on a range of problems with sparse, long-term temporal
dependencies, that these models store information from early in a sequence,
and reuse this stored information efficiently. This allows them to perform substantially
better than existing models based on well-known recurrent neural networks,
like LSTMs.