Improvement of End-to-End Offline Handwritten Mathematical Expression Recognition by Weakly Supervised Learning
Thanh-Nghia Truong, Cuong Tuan Nguyen, Khanh Minh Phan, Masaki Nakagawa
Abstract
This paper presents an improvement in recognizing offline handwritten mathematical expressions (HMEs) by deep neural networks. We train it end-to-end using weakly supervised learning. The network has three parts: an encoder using a Convolutional Neural Network to encode high-level features from an input HME image; a decoder using gated recurrent units with attention to parse the high-level features and generate an output expression in the LaTeX format; and a symbol classifier to improve the localization and classification of the high-level features. Besides, we use the model ensemble method for the beam search process to average the probabilities from multiple models. For the dataset of the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014 and 2016, we have achieved an expression recognition rate of 53.65% and 51.96% correspondingly, which is 6 points better than without weakly supervised learning. Furthermore, when ensembling several models, the recognition rate of our method is increased to 55.68% for the CROHME 2014 testing set.