Parallel Sentence Mining by Constrained Decoding
Pinzhen Chen, Nikolay Bogoychev, Kenneth Heafield, Faheem Kirefu
Abstract
We present a novel method to extract parallel sentences from two monolingual corpora, using neural machine translation. Our method relies on translating sentences in one corpus, but constraining the decoding by a prefix tree built on the other corpus. We argue that a neural machine translation system by itself can be a sentence similarity scorer and it efficiently approximates pairwise comparison with a modified beam search. When benchmarked on the BUCC shared task, our method achieves results comparable to other submissions.
Topics & Concepts
Computer scienceMachine translationDecoding methodsNatural language processingSentenceArtificial intelligenceTreebankPairwise comparisonTask (project management)Similarity (geometry)Encoding (memory)Tree (set theory)PrefixTranslation (biology)Speech recognitionAlgorithmParsingImage (mathematics)Mathematical analysisBiochemistryLinguisticsChemistryMessenger RNAEconomicsManagementMathematicsGenePhilosophyNatural Language Processing TechniquesTopic ModelingSemantic Web and Ontologies