Open science: Considerations and issues for <scp>TESOL</scp> research

Abstract

Open science (OS; also known as “open research” and “open scholarship”) refers to various practices to make scientific knowledge openly available, accessible, and reusable. The core purpose of such practices is to open the process of scientific knowledge creation, evaluation, and communication to societal actors within and beyond the traditional scientific community (UNESCO, 2021). Looking across the different areas within TESOL and applied linguistics more broadly, it is clear that OS practices have become more common on the part of individual researchers and journals. For example, while Marsden, Morgan-Short, Thompson, and Abugaber (2018) and Marsden, Morgan-Short, Trofimovich, and Ellis (2018) noted generally low prevalence of replication studies, the number of replications identified from 2010 to 2015 (n = 20) was larger than all located in the period of 1973–1999 (n = 17). Open data and materials have become more common, too, as seen in the widespread use of the instruments and data for research in language studies (IRIS) database (iris-database.org). OS badges now frequently adorn articles in several journals, and journals such as Language Learning have been recognized for adopting a range of support for OS practices, as seen in TOP (Transparency and Openness Promotion) Factor scores (https://topfactor.org, based on the TOP Guidelines, TOP Guidelines Committee, 2015). Given this momentum, we feel that it is time for TESOL researchers to seriously consider the benefits, and potential challenges, of more active, consistent engagement in OS practices. In this article, we focus on four aspects of OS: transparency, preregistration, data and participant protection, and open access. We begin by discussing the issue of transparency, exploring initiatives within and in related areas of our field, examining the benefits of enhancing transparency and challenges underlying certain practices. Next, we explore affordances, challenges, and common perceptions surrounding the practices of preregistration and open data sharing, emphasizing that these are not all-or-nothing endeavors but instead can consist of specific decisions made by researchers depending on their particular situation. Following this, we consider barriers to accessibility of scholarship and discuss grassroots initiatives supporting open access research as a goal underlying OS. Finally, we reflect on potential (mis)conceptions of OS in TESOL, and rather than calling for a universal standard for OS practices in our field, we encourage individual scholars to consider employing contextually appropriate OS practices in their work. Overall, we agree with Liu's (2023) observation that the rich interdisciplinarity and methodological diversity within the field of TESOL in particular and applied linguistics overall has great potential to support “an inclusive understanding of open science” (p. 6). At the forefront of OS topics gaining attention in the field are initiatives to enhance transparency. Here we briefly explore several questions around transparency in TESOL-related research: What is transparency? Why is transparency necessary? What are the main barriers to achieving transparency? What are the costs of limiting transparency? Methodological transparency involves making all aspects of the research process fully available to its academic, institutional, and public stakeholders from the conceptual framing of the research to the design, materials, data collection, analytic methods, and reporting and dissemination. Transparency is intended “to improve the accessibility, visibility, rigour, scrutiny, reproducibility, replicability, and systematicity of research” (Marsden, 2020, p. 26), and it is increasingly seen as coinciding with research quality. Indeed, study design and methods “demand a kind of professional scrutiny that goes directly to the core of what we do and what we know and what we can tell our publics that we know” (Byrnes, 2013, p. 825). One of the core principles of methodological transparency is that the validity of a research claim depends not on the reputation of those making the claim, the venue in which the claim is disseminated, or the novelty of the findings, but rather on the empirical evidence supported by the underlying materials, data, and methods, all of which are made accessible to fellow researchers and the wider public (Nosek et al., 2015). There is an increasing array of empirically demonstrated benefits of methodological transparency (e.g., Gennetian, Tamis-LeMonda, & Frank, 2020; Miguel, 2021; Schroeder, Gaeta, El Amin, Chow, & Borders, 2023). Transparency creates a more complete understanding of the connected network of materials underlying our research and empirical claims. Peer reviewers benefit when they are better able to critically evaluate research. Journals benefit when they have more accurate information about the research that underlies their publications. Professional associations benefit from more complete access to research that is the basis for conclusions made available to practitioners, fellow researchers, and policymakers. In this way, transparency builds confidence in the knowledge we share and apply in practice. Transparency also promotes exchange and collaboration with underrepresented stakeholders, thus enhancing the diversity and equity of our work. Transparency increases the impact and widens the reach of our work as we benefit through increased credibility or understanding of our results, and the potential for more reuse, citation, and broader recognition of our outputs. Overall, we can see that transparency is an important investment in our field's future and advances our ethical imperative in generating knowledge (Ortega, 2005). There are many initiatives for increased transparency both within our field and beyond. For example, IRIS, the open repository for multilingual research instruments and materials, aims to promote the sharing and reuse of research data and materials and make study methods more transparent, down to the individual questions and stimuli used in studies (Marsden, Mackey, & Plonsky, 2016; Marsden, Thompson, & Plonsky, 2017). Open Accessible Summaries in Language Studies (OASIS) and TESOLgraphics, for their part, share the common aim to enhance accessibility of research findings to practitioners and to the wider public. These forms of sharing support access which can help address issues related to trust and research integrity. Contributor Role Taxonomy (CRediT) statements (Brand, Allen, Altman, Hlava, & Scott, 2015) are a tool used to provide a transparent and nuanced understanding of authors' diverse responsibilities and contributions to research projects. TOP Guidelines provide transparency standards in multiple domains of research (e.g., study design, materials, data analysis, and methods) that are used by journals and scholarly associations to model, endorse, and reward practices for making transparent disclosure the default in research publication (see https://topfactor.org). On this view, it is interesting to note that Language Learning has a TOP Factor of 11, Applied Psycholinguistics has a score of 8, The Modern Language Journal has a score of 2, while TESOL Quarterly has a score of 0. A salient question here is: Can we achieve transparency without mandating these procedures? In an effort to promote transparency, TESOL Quarterly will soon formally recognize the adoption of OS practices, including sharing research instruments and materials, making data publicly available, and preregistering research plans. While these practices will not be a requirement for publishing with TESOL Quarterly, they are intended to encourage authors to adopt OS practices where appropriate and increase the transparency of their work. It should come as no surprise that there are other, seemingly less effortful, ways of achieving transparency in the life cycle of research by relying on robust reporting practices. Included here are things such as conflict of interest disclosures, reflexivity and/or positionality statements, and the many considerations and disclosures that are part of research projects beginning with designing a study, gaining ethical approval, choosing and implementing materials and data elicitation instruments, collecting and analyzing data, through to reporting on the results. Readers of research no doubt expect to find many of these details in published manuscripts. However, limits to researchers' available time and effort and inherent space restrictions in such venues may lead to certain omissions and compromises in transparency. Reviews of TESOL-related research in certain domains indeed show evidence of low uptake of such reporting practices (Isbell & Kim, 2023). In this regard, Marsden (2020) cautions that relying primarily on robust reporting challenges methodological transparency because “the inevitable lack of standardisation and the organic nature of reporting standards” (p. 21) leads many to craft a good/sanitized methodological story rather than to tell the whole/real story no matter how imperfect. Seen in this light, achieving methodological transparency in our “everyday” practice should not be taken for granted. Methodological transparency can manifest in many ways, and because not all OS practices are transposable across all research paradigms, the different designs, methods, and epistemologies across TESOL-related research are accompanied by different challenges for achieving methodological transparency (Marsden & Morgan-Short, 2023). Chief among the challenges to increasing methodological transparency are systemic barriers including “inertia and the comfortable embrace of the status quo” (Center for Open Science, 2015). At a field-wide level, for instance, there are formal incentives and a publication infrastructure that make transparency not the obvious default. Given this reality, Marsden (2020, p. 25) argues that there is a collective need for stronger “unified directives and incentives from professional associations, promotion systems, funders, and journals” to encourage, incentivize, and support the field in making our research transparent by default. For individuals or research teams, there are other barriers to making our research transparent by default. These include concerns about ethical protections and confidentiality for research participants and their data (which we examine at length below), the desire to maintain a perceived competitive advantage, or intellectual property concerns for certain materials. There are also legitimate technical and resource limits with certain methodologies and with large and a lack of widespread in methods or practices that enhance transparency in research (e.g., & in limiting transparency can have costs and by many with in it within domains of the of and and the of across studies and within transparency also researchers and science from it barriers to of research findings and equity and in knowledge and sharing (Marsden & Morgan-Short, 2023). not and there are to initiatives for transparency in TESOL-related research. One is the for transparency that it researchers and limits engagement in such practices (see & & 2023). is the low that it not have a researchers in TESOL, we are not to be as we our research. We are not to all the make or use the and methods that We to be and transparent in our research about what we and Methodological transparency, is the for research (Center for Open Science, 2015) and is to the trust that TESOL stakeholders and in our research and the evidence it for and practice. this, we to practice that can promote methodological transparency. is the OS practice of a research a public repository data has We will now at the benefits, and potential of In the preregistration and a specific of preregistration a are The purpose of preregistration is to increase transparency by a public of the methodological decisions and that by researchers and those from and is for research studies (Nosek & preregistration has also been to be for research studies & of the preregistration is that by a public of the methodological decisions that made in of data and analysis, we can more research practices such as are or & p. multiple to which may the and of our work. a publicly available can the in with the and in published researchers are with preregistration at the research we to for our our or we have a that information on study design and The main for preregistration is that this information is in a public The or of in a preregistration can range from to a and methods The important of preregistration is that it to the decisions made in of the study and to help researchers to the For instance, the for Open has a of information related to study preregistration and also a and space for researchers to their work an example, see and a of preregistration the a and the of at of the main of preregistration to from the practice a more see in Here we focus on common preregistration preregistration is and preregistration (see & et al., & The preregistration because the research is in a authors their or is a and a important a study not that the is to all and preregistering a study that an a or the should be and the should be as The are related to other in that they both preregistration for or et to the is for claim by that it that authors not have their studies on the without a requirement to do also be for reviewers there is an that reviewers on the of to preregistration plans. preregistration is an interesting empirical and authors (e.g., & have that this be a One specific of preregistration, reflect a to the publication process in that they a to data (see Marsden, Morgan-Short, Thompson, & Marsden, Morgan-Short, Trofimovich, & have been as a potential to publication and incentives for both authors and reviewers In the instead of a to a for authors the and for data have been the of preregistration in that a has been and data A of the process for a be an that the is and/or are the will be published by the of the results. that the reviewers are to evaluate the research questions are interesting and and the and are and of without by the results. journals in related to TESOL (e.g., Language and Language and Language and have this including the Journal of Language which an in which authors a of to the the to the question of it should be that the process involves of For example, for data be as it is when the process will be this, it be that the of the of the research process without increasing the overall (see the in in of there are multiple potential benefits to preregistering a study, including increased transparency, research practices, confidence in and a of the study design decisions made in of data Given the of this practice in our field, how as TESOL researchers, to adopt study preregistration, evaluate its and future scholars (see & is to focus in the and Open data sharing is at the of the OS because it is as a practice to increase and of research findings, across research through and increased transparency 2023). as a it should not be taken issues with data sharing include in open data standards (e.g., et al., which are to specific and may be to without of with data sharing (e.g., making open data sharing may lead to sharing information that should be and data and of the materials, can make it for to use in the of open data sharing concerns the of data and Indeed, in our data are “the & p. but is a universal (p. It is that “the data are the et al., p. their data to be and on of For that should be taken to and confidentiality to “the of or individuals from the & p. fully data without can be without at in with multiple be (p. the for studies with et al., is a with data without in the but with a that to study openly sharing data can lead to individuals or et al., p. et a of studies in research and that participants at the of study authors to make in their (e.g., and and their data sharing TESOL and applied linguistics data more are as as those in openly sharing data, without may the of participants and other of their help data, at a the should be OS for data see also & in for a to OS in applied linguistics publishing on sharing research data (e.g., et al., for science data with the (e.g., data and as part of the et al., p. discussing with participants how their data will be and that they can make an the purpose of the study is et al., or access for data that can be et al., et al., and sharing data the data be Plonsky, & The is a the data are from (e.g., and researchers should the of their participants and have understanding of and et al., p. (e.g., and may not open data sharing, and their decisions be sharing data can be more in the of data because such data are to et al., is for in language and should all be their but lead to data In this, what can be to support OS when research be made as in The a language study that was more in with rather than open science” to with data p. to for open data sharing and their and data a of time on the part of the For example, can use to and language is to and use to provide more include or is the communication of a (p. The can be made with the help of a research which has for TESOL research in such Overall, it is to data may be but aspects may be less for research. For instance, rather than sharing or in of information in research on of open data sharing the data and of participants The for range from of their to a research study to to which can be by & 2021). public increases the of that researchers' and their and may trust in research et al., 2021). the of researchers should participants of the and consider implementing data use (e.g., when data are to use the data to their research questions but not the and to or research et al., p. In to the of a to data and participant is to the issue of data et al., et al., data to researchers' journals, or However, researchers consider this to participants data et al., be by a repository of research data that be by participants and be used for various research “the researchers be the to data, not the data and participants have more their data, which make more in the research process et al., p. data and participant is the researchers' ethical and professional Given that participants share their data for the et al., and that there are several of of open data in other noted by et al., researchers in TESOL and in the language are to the matter of open data sharing seriously and data sharing and to making their publicly Open access is with the of the of research (e.g., the that scholars can it without to open a of to practitioners, and the broader in the are from the to to their their to status the in the There are we should open access in of for a to it for we that the is and we to apply this to a apply no matter how the or how it it is a of a the for & One has to what kind of are The is that on their that they open is the publishing model, as a this consider a for or on a all this the and research However, when the this research to a the all the and all the this is not a this is more than and intellectual to it that the to evaluate the of this for In many and it be for a to use are able to with this because and researchers, have not taken to the status For example, authors are to share their and & 2023). The of applied linguistics journals restrictions on sharing and while a have an of The these should things their is that without but authors can without the of it is now more to their open access journals the the a of the Journal of Applied to its and to a open access the Open Journal costs the less than to technical support and in applied linguistics are able to this and their journals. journals that are not by are potential to and their including Language The Modern Language and TESOL the increasing interest in and in TESOL and applied linguistics more broadly, it is time that researchers, and be the to to the field by this open access. The to do will be a to the that will for a time and that of scholars will be that the of OS are There are also that OS is across research to should not be on for the of and it is a for the TESOL research community to consider OS practices are to be and is However, we to address from of the perceived of OS within the broader field of TESOL, or perceived with OS it is that researchers may not see how OS is to the work they The of TESOL Quarterly, and that of the other within the field more broadly, to the that TESOL is and While this is a the of the field and different in which and it also that will not see Open Science, as the is in a scientific primarily research methods (see for an of this OS not to be a for or important a with a to knowledge Language researchers, with materials and data that are not for and/or (see & Kim, may OS practices as and/or broadly, researchers in or research may question aspects of preregistration, which was seen as research. with perceived may be by to promote and adopt from the on published to more to how research and journals are (e.g., the TOP researchers, we need to address these issues no to be of the for research On the note of we the Open is OS adopt more inclusive to its and increasingly inclusive such as the by Open Applied an of Applied research methods, it is that OS is not an all-or-nothing and can be seen as better than not a study, there is from sharing materials and share the data from a study, can share of it and/or share materials. consider publishing open access or sharing a At the we also encourage researchers of all to consider adopting on OS practices. For example, researchers in other have for and to preregistration for (Nosek et al., & and et al., that in both there are benefits to transparency. OS practices do effort on the part of researchers (Marsden & Morgan-Short, 2023). Open access publishing a effort on the part, but open data is a more as to the data have been of to and that can make use of it (see 2021). The effort may be with preregistration, when to a as a and for a research be to a of and through data can it all there are benefits to including making research less effort as when can use instruments and by other researchers, or other to while we do not to the it is that do not have to be For instance, while preregistration is not common in applied as it is to that many applied at all do study in and at These are in the of or they be and as preregistration In in Open more Open is as for TESOL as it is for other areas of applied and at the time not have to be an all-or-nothing It is better to do what we can when we and the effort that it is to be it in of transparency, and benefits of our research. The authors of this are by to reflect and is part of TESOL to with open science and has been by the The of this are based on a at the of the for Applied We are to the many and is an of Language at the for and at the of the of and research include research and is a in Language Studies at research and analytic to examine issues related to investment in language in language and and is an in the of at research the of individual and language with also on in language open and methods for is in the of at the of research in and of the including while and and in is for Open of Applied Language is an in the of Language Studies at the of at where in language and research is of at and a of the of is of for TESOL Quarterly, a of and and was for Language Quarterly is at At the core of research is the of in from applied and methodological and is to research and open science practices.

Topics & Concepts