Committee:

Dr. Chin-Hui Lee, ECE, Chair , Advisor

Dr. David Anderson, ECE

Dr. Elliot Moore, ECE

Dr. Marco Siniscalchi, Univ of Enna

Dr. Jin Liu, Modern Languages

Abstract:

The objective of the proposed research is to improve mispronunciation detection of Mandarin and enrich diagnostic feedback for second language learners. The problem is tackled from the perspective of acoustic modeling and verification of phones and tones. For the acoustic modeling part, speech attributes and soft targets are respectively proposed to help resolve phone and tone's hard-assignments labels, which are not optimal for describing irregular non-native pronunciations. Subsequently, multi-source information or better trained acoustic model can provide more accurate features for mispronunciation detectors. For the verification part, pronunciation representation, usually calculated by frame-level averaging in a DNN, is now learned by BLSTM, which directly uses sequential context information to embed a sequence of pronunciation scores into a pronunciation vector to improve the performance of mispronunciation detectors. Finally, with the help of posterior scores generated by different classifiers and interpretable decision trees, we can visualize non-native mispronunciations and provide comprehensive feedback, including articulation manner, place, and pitch contour-related diagnostic information, to help non-native learners improve their pronunciation quality.