Sketch2Vis: Generating Data Visualizations from Hand-drawn Sketches with Deep Learning
Zhongwei Teng, Quchen Fu, Jules White, Douglas C. Schmidt
Abstract
Data visualization has become a vital tool to help people understand the driving forces behind real-world phenomena. Although the learning curve of visualization tools have been reduced, domain experts still often require significant amounts of training to use them effectively. To reduce this learning curve even further, this paper proposes Sketch2Vis, a novel solution using deep learning techniques and tools to generate the source code for multi-platform data visualizations automatically from hand-drawn sketches provided by domain experts, which is similar to how an expert might sketch on a cocktail napkin and ask a software engineer to implement the sketched visualization.This paper explores key challenges (such as model training) in generating visualization code from hand-drawn sketches since acquiring a large dataset of sketches paired with visualization source code is often prohibitively complicated. We present solutions for these problems and conduct experiments on three baseline models that demonstrate the feasibility of generating visualizations from hand-drawn sketches. The best models tested reach a structural accuracy of 95% in generating correct data visualization code from hand-drawn sketches of visualizations.