A Tree-Structured Decoder for Image-to-Markup Generation
Jianshu Zhang, Jun Du, Yongxin Yang, Yi-Zhe Song, Si Wei, Li-Rong Dai
Abstract
Recent encoder-decoder approaches typically employ string decoders to convert images into serialized strings for image-to-markup. However, \nfor tree-structured representational markup, string \nrepresentations can hardly cope with the structural complexity. In this work, we first show \nvia a set of toy problems that string decoders \nstruggle to decode tree structures, especially as \nstructural complexity increases, we then propose \na tree-structured decoder that specifically aims \nat generating a tree-structured markup. Our decoders works sequentially, where at each step a \nchild node and its parent node are simultaneously \ngenerated to form a sub-tree. This sub-tree is consequently used to construct the final tree structure \nin a recurrent manner. Key to the success of our \ntree decoder is twofold, (i) it strictly respects the \nparent-child relationship of trees, and (ii) it explicitly outputs trees as oppose to a linear string. \nEvaluated on both math formula recognition and \nchemical formula recognition, the proposed tree \ndecoder is shown to greatly outperform strong \nstring decoder baselines.