PhD Dissertation Proposal by DU Cunxiao | On the Learning of Fully Non-Autoregressive Machine Translation

In this thesis proposal, I study the non-autoregressive neural machine translation (NAT) problem.

NAT models generate the tokens in the target sequence concurrently by removing the dependencies among the target tokens.

While NAT models have superior computational efficiency, the main challenge to NAT is the multi-modality problem, i.e., the problem that when a source sequence has multiple correct translations, NAT models with their standard log-likelihood loss function cannot easily handle these multiple modes. In this proposal we present two pieces of work with new loss functions addressing the multi-modality problem.

We first propose a new training objective named order-agnostic cross entropy OaXE for NAT models. OaXE improves the standard cross-entropy loss to ameliorate the effect of word reordering, which is a common source to the multi-modality problem. We also extend OaXE by only allowing reordering between $n$-gram phrases and still requiring a strict match of word order within the phrases. Extensive experiments on NAT benchmarks across language pairs and data scales demonstrate the effectiveness and universality of our approaches.

Cunxiao DU is a second year Ph.D in Department of Computer Science at SMU, advised by Professor Jing Jiang and Dr. Zhaopeng Tu from Tencent AI Lab.