A generic non-invasive neuromotor interface for human-computer interaction
Patrick Kaifosh, Thomas R. Reardon, CTRL-labs at Reality Labs, Brian D. Allen, Chris Anderson, Sacha Arnoud, Rahul Arora, Mridu Atray, Lana Awad, Francisco Ayerbe, Christopher Baker, Nicholas Baker, Alexandre Barachant, Philip Bard, Wilman Pimentel Beltran, Adam Berenzweig, Rohin Bhasin, Joe Bienkowski, Sean Bittner, Luke Boegner, Anu Bolarinwa, Don Bosley, Matthew Bracaglia, Mario Bräcklein, Maclyn Brandwein, Joe Bravate, Matt Butler, Adam J. Calhoun, Chia-Jung Chang, Daniel Chenet, Joshua Chester, Rudi Chiarito, Rohan Chitnis, John Choi, Won Chun, Jeremiah Chung, James Connors, Jota Costa, Mark Cramer, Raven Cunningham, William F. Cusack, Nathan Danielson, Thomas J. Davidson, Bruno De Araujo, Bob DiMaiolo, Scott Draves, Alan Du, Zaina Edelson, Phina Enemuo, Mina Fahmi, Nariman Farsad, Ali Farshchian, Randy Feliz, Jake Fine, Emanuele Formento, Dustin Freeman, Jianing Fu, Jean-Christophe Gagnon-Audet, Rupesh Gajurel, Jonathan Gamutan, Sida Gao, Jonateal Garcia, Nathalie Therese Helene Gayraud, Minha Ghani, Sayan Ghosh, Vickram Gidwani, Danny Giebisch, Greg Gimler, Alexandre Gramfort, Lauren Grosberg, Bryn Gunther, Ning Guo, Chetan Gupta, Sinem Guven Kaya, Austin Ha, Katarina Hadjer, Carlos Xavier Hernández, Stav Hertz, Carl Hewitt, Daniel N. Hill, Kirak Hong, Lillian Hong, Helen Hou, Stepan Hruda, Alex Hsieh, Vivian Hsiung, Rongqing Huang, Yue Hui, Hazel Hulet, Shaker Islam, Vinay Jayaram, Connie Jiang, Xiaodong Jiang, Brooke Juarez, James Jaeyoon Jun, Na Young Jun, Nirag Kadakia, Nishant Kakar, Ajay Kamdar, Ta-Chu Kao
Abstract
Since the advent of computing, humans have sought computer input technologies that are expressive, intuitive and universal. While diverse modalities have been developed, including keyboards, mice and touchscreens, they require interaction with a device that can be limiting, especially in on-the-go scenarios. Gesture-based systems use cameras or inertial sensors to avoid an intermediary device, but tend to perform well only for unobscured movements. By contrast, brain–computer or neuromotor interfaces that directly interface with the body’s electrical signalling have been imagined to solve the interface problem1, but high-bandwidth communication has been demonstrated only using invasive interfaces with bespoke decoders designed for single individuals2–4. Here, we describe the development of a generic non-invasive neuromotor interface that enables computer input decoded from surface electromyography (sEMG). We developed a highly sensitive, easily donned sEMG wristband and a scalable infrastructure for collecting training data from thousands of consenting participants. Together, these data enabled us to develop generic sEMG decoding models that generalize across people. Test users demonstrate a closed-loop median performance of gesture decoding of 0.66 target acquisitions per second in a continuous navigation task, 0.88 gesture detections per second in a discrete-gesture task and handwriting at 20.9 words per minute. We demonstrate that the decoding performance of handwriting models can be further improved by 16% by personalizing sEMG decoding models. To our knowledge, this is the first high-bandwidth neuromotor interface with performant out-of-the-box generalization across people. A high-bandwidth neuromotor interface offers performant out-of-the-box generalization across people.