Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation

Abstract

We propose an efficient inference procedure for non-autoregressive machine translation that iteratively refines translation purely in the continuous space. Given a continuous latent variable model for machine translation This allows us to use gradient-based optimization to find the target sentence at inference time that approximately maximizes its marginal probability. As each refinement step only involves computation in the latent space of low dimensionality (we use 8 in our experiments), we avoid computational overhead incurred by existing non-autoregressive inference procedures that often refine in token space. We compare our approach to a recently proposed EM-like inference procedure We evaluate our approach on WMT'14 EnDe, WMT'16 RoEn and IWSLT'16 DeEn, and observe two advantages over the EM-like inference:

Topics & Concepts