Tackling sequence to sequence mapping problems with neural networks

Thesis

In Natural Language Processing (NLP), it is important to detect the relationship between two sequences or to generate a sequence of tokens given another observed sequence. We call the type of problems on modelling sequence pairs as sequence to sequence (seq2seq) mapping problems. A lot of research has been devoted to finding ways of tackling these problems, with traditional approaches relying on a combination of hand-crafted features, alignment models, segmentation heuristics, and external linguistic resources. Although great progress has been made, these traditional approaches suffer from various drawbacks, such as complicated pipeline, laborious feature engineering, and the difficulty for domain adaptation. Recently, neural networks emerged as a promising solution to many problems in NLP, speech recognition, and computer vision. Neural models are powerful because they can be trained end to end, generalise well to unseen examples, and the same framework can be easily adapted to a new domain.

The aim of this thesis is to advance the state-of-the-art in seq2seq mapping problems with neural networks. We explore solutions from three major aspects: investigating neural models for representing sequences, modelling interactions between sequences, and using unpaired data to boost the performance of neural models. For each aspect, we propose novel models and evaluate their efficacy on various tasks of seq2seq mapping.

Chapter 2 covers the relevant literature on neural networks. Following this, in Chapter 3 we explore the usefulness of distributed sentence models in seq2seq mapping problems by testing them in the task of answer sentence selection. We also empirically compare the performance of distributed sentence models based on different types of neural networks. Chapter 4 presents a neural sequence transduction model that learns to alternate between encoding and decoding segments of the input as it is read. The model not only outperforms the encoder-decoder model significantly on various tasks such as machine translation and sentence summarisation, but also is capable of predicting outputs online during decoding. In Chapter 5, we propose to incorporate abundant unpaired data using the noisy channel model---with the component models parameterised by recurrent neural networks---and present a tractable and effective beam search decoder.

Authors: Yu, L

• DOI: 10.5287/ora-yrvmy281r
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford