Online segment to segment neural transduction

Conference item

We introduce an online neural sequence to sequence model that learns to alternate between encoding and decoding segments of the input as it is read. By independently tracking the encoding and decoding representations our algorithm permits exact polynomial marginalization of the latent segmentation during training, and during decoding beam search is employed to find the best alignment path together with the predicted output sequence. Our model tackles the bottleneck of vanilla encoder-decoders that have to read and memorize the entire input sequence in their fixed-length hidden states before producing any output. It is different from previous attentive models in that, instead of treating the attention weights as output of a deterministic function, our model assigns attention weights to a sequential latent variable which can be marginalized out and permits online generation. Experiments on abstractive sentence summarization and morphological inflection show significant performance gains over the baseline encoder-decoders.

Authors: Yu, L; Buys, J; Blunsom, P

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Association for Computational Linguistics
• Host title: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
• Journal: Empirical Methods on Natural Language Processing Conference 2016
• Pages: 1307–1316
• Publication date: 2016-11-01
• Acceptance date: 2016-07-29
• Keywords: cs.AI; cs.CL; cs.NE