Concrete Recurrent Neural Network Architectures

Abstract

After describing the RNN abstraction, we are now in place to discuss specific instantiations of it. Recall that we are interested in a recursive function s_i = R(x_i; 1_i1/ such that s_i encodes the sequence x_1:n. We will present several concrete instantiations of the abstract RNN architecture, providing concrete definitions of the functions R and O. These include the Simple RNN (SRNN), the Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU).