Phonological Feature Based Mispronunciation Detection and Diagnosis using Multi-Task DNNs and Active Learning

Conference item

This paper presents a phonological feature based computer aided pronunciation training system for the learners of a new language (L2). Phonological features allow analysing the learners’ mispronunciations systematically and rendering the feedback more effectively. The proposed acoustic model consists of a multi-task deep neural network, which uses a shared representation for estimating the phonological features and HMM state probabilities. Moreover, an active learning based scheme is proposed to efficiently deal with the cost of annotation, which is done by expert teachers, by selecting the most informative samples for annotation. Experimental evaluations are carried out for German and Italian native-speakers speaking English. For mispronunciation detection, the proposed feature-based system outperforms conventional GOP measure and classifier based methods, while providing more detailed diagnosis. Evaluations also demonstrate the advantage of active learning based sampling over random sampling.

Authors: Arora, V; Lahiri, A; Reetz, H

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: International Speech Communication Association
• Host title: Interspeech 2017: Situated Interaction
• Journal: Interspeech
• Series: Proceedings of the Annual Conference of the International Speech Communication Association
• Publication date: 2017-08-20
• Acceptance date: 2017-05-22
• Event location: Stockholm
• Event start date: 2017-08-20
• Event end date: 2017-08-24
• DOI: 10.21437/Interspeech.2017-1350
• ISSN: 1990-9772
• Keywords: active learning; phonological features; computer-aided pronunciation training; multi-task DNNs