An Umbrella Review of Reporting Quality in CHI Systematic Reviews: Guiding Questions and Best Practices for HCI
Katja Rogers, Teresa Hirzle, Sukran Karaosmanoglu, Paula T. Palomino, Ekaterina Durmanova, Seiji Isotani, Lennart E. Nacke
Abstract
Systematic reviews (SRs) are vital to gathering and structuring knowledge, yet descriptions of their procedures are often inadequate. In human–computer interaction (HCI), SRs are still uncommon but gaining momentum, which prompted us to explore how SRs are reported at CHI—the flagship HCI conference venue. To assess the reporting quality of CHI reviews that aim for a systematic approach, we conducted an umbrella review and applied reporting guidelines for SRs (PRISMA and ENTREQ) to our corpus. We contribute the first exploration of how well SRs at CHI meet guidelines for reporting quality, showcasing strategies for improvement in reporting and conducting SRs especially in the domains of appraisal, synthesis, and documentation (i.e., protocol development). Finally, we present guiding questions for HCI researchers and practitioners for reporting SRs, as well as suggestions for best practices.