CNEL Seminar
“Dual-stream External Memory for Long-term Dependency”
Wednesday, Sept. 27 at 3:00pm
NEB 589

Abstract

Long-term dependency is one of the most important problems in the study of time series. Numerous strategies have been advanced to address this issue. Among them, the gating mechanism stores information and compress them into a single hidden state. In contrast, external memory mechanisms tackle long-term dependencies by retaining and incorporating all historical information. Additionally, alternative approaches center their efforts on modulating the hidden state, either through new restrictions or adjusting the training methodology. These innovative methodologies have garnered substantial acclaim for their remarkable success across diverse application domains.

We study the long-term dependency issue in a linear recurrent neural network and make an analogy to the gamma memory, which is a “myopic” system. We find that the recurrent neural network has the same trade-off between the memory depth and resolution, which limits its power of retrieving information from the past. To transcend this limitation, we introduce a novel architecture – the dual-stream external memory architecture – aimed at disrupting this trade-off. Within this framework, we leverage the attention mechanism to interconnect these two parallel streams of memory. It explains the failure of attention mechanismS in transformerS when applied to time series, particularly in the context of chaotic systems. Importantly, it also demonstrates remarkable efficacy on neural machine translation tasks, underscoring its versatility and potential impact.

Biography

Ran Dou is a Ph.D. student in the Department of Electrical & Computer Engineering at the University of Florida. He has been a research assistant at the Computational NeuroEngineering Laboratory (CNEL) since 2020. His research interests include signal processing and machine learning.