A consensus privacy metrics framework for synthetic data
Lisa Pilgram, Fida K. Dankar, Jörg Drechsler, Mark Elliot, Josep Domingo‐Ferrer, Paul Francis, Murat Kantarcıoğlu, Linglong Kong, Bradley Malin, Krishnamurty Muralidhar, Puja Myles, Fabian Praßer, Jean Louis Raisaro, Chao Yan, Khaled El Emam
Abstract
Synthetic data generation is a promising approach for sharing data for secondary purposes in sensitive sectors. However, to meet ethical standards and legislative requirements, it is necessary to demonstrate that the privacy of the individuals upon which the synthetic records are based is adequately protected. Through an expert consensus process, we developed a framework for privacy evaluation in synthetic data. The most commonly used metrics measure similarity between real and synthetic data and are assumed to capture identity disclosure. Our findings indicate that they lack precise interpretation and should be avoided. There was consensus on the importance of membership and attribute disclosure, both of which involve inferring personal information. The framework provides recommendations to effectively measure these types of disclosures, which also apply to differentially private synthetic data if the privacy budget is not close to zero. We further present future research opportunities to support widespread adoption of synthetic data.