Gender Neutrality in Robots: An Open Living Review Framework

Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction Gender neutrality in robots: An open living review framework KATIE SEABORN Industrial Engineering & Economics Tokyo Institute of Technology PETER PENNEFATHER gDial, Inc. CITATION: Seaborn, K., & Pennefather, P. (2022). Gender neutrality in robots: An open living review framework. Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’22), 634–638. https://doi.org/10.5555/3523760.3523845 The final publication is available via ACM/IEEE at https://dl.acm.org/doi/10.5555/3523760.3523845.  Gender Neutrality in Robots: An Open Living Review Framework Katie Seaborn Peter Pennefather Department of Industrial gDial, Inc. Engineering and Economics Toronto, Canada Tokyo Institute of Technology 0000-0002-2953-8977 Tokyo, Japan 0000-0002-7812-9096 Abstract—Gender is a primary characteristic by which people Aldebaran-SoftBank’s Pepper robot is a case in point. Although organize themselves. Previous research has shown that people the designers aimed to create a gender-neutral robot in voice and tend to unknowingly ascribe gender to robots based on features of body [10], the official materials clearly ascribe a masculine their embodiment. Yet, robots are not necessarily ascribed the gender to Pepper by using he/his/him pronouns 1 . These same, or any, gender by different people. Indeed, robots may be complexities, tensions, contradictions, and possibilities have ascribed non-human genders or used as “genderless” alternatives. spurned interest on gender neutrality in robots. Stanford’s This underlies the notion of gender neutrality in robots: neither Gendered Innovations group, for instance, includes “genderless masculine nor feminine but somewhere in between or even beyond robots” as a key consideration for roboticists and researchers gender. Responding to calls for gender as a locus of study within who wish to explore gender in robots [6]. Gender neutrality is robotics, we offer a framework for conducting an open living review to be updated periodically as work emerges. Significantly, expected to be a central topic on robot gender going forward. we provide an open, formalized submission process and open As work on this topic emerges, building consensus and access dataset of research on gender neutrality in robots. This identifying points for future work will be essential. The standard novel and timely approach to consensus-building is expected to way to do this is by conducting a systematic review of the pave the way for similar endeavours on other key topics within literature [11]. Systematic reviews are considered the gold human-robot interaction research. standard of evidence in many scientific fields because they do not rely on a single study, instead bringing together all of the Keywords—robots, gender, robot gender, gender neutrality, genderless robots, mechanical genders, living review, open data evidence in a standard, structured, and rigorous way [12]. Given the evolving nature of gender neutrality in robots as a topic of I. INTRODUCTION study, a type of systematic survey called a living review may be particularly appropriate. Living reviews are “alive” in the sense Gender is an important feature of people—and increasingly that they are periodically updated with new articles, findings, of robots, too. Decades of work has shown that people can and and possibly reporting structures [11], [13]. Living reviews are do read human characteristics in the design of robots and other especially appropriate for cutting-edge topics where there is agents, even when anthropomorphic cues are subtle. Nass, uncertainty in the corpus of evidence and a need for direction Brave, Moon, Lee, and colleagues [1], [2] were forerunners in and congruency. This makes the topic of gender neutrality an researching this phenomenon. Their work on computer voice led ideal candidate for a living review. Moreover, living reviews can to the Computers Are Social Actors (CASA) model, a widely be an open science initiative if the procedures are registered and recognized descriptive paradigm that shows how people tend to the datasets are provided in an open access venue. Doing so react to computer agents that have humanlike characteristics as increases transparency and integrity, allows for the latest articles if they are people, usually unthinkingly and in stereotyped ways. to be indexed in a structured way, and creates opportunities for The implications of this early research were vast, spurning community engagement, such as crowdsourcing new articles for studies on gender in robots and other artificial agents [3]–[6]. inclusion. To the best of our knowledge, there is no living review While most researchers have taken a binary approach to work on the topic of gender neutrality in robots, or indeed in gender, i.e., male/female or masculine/feminine or man/woman, HRI generally, as well as no open access dataset based on living an increasing number are calling for and starting to conduct review work and/or community contributions. research on alternative models of gender, including ambiguous, As an open science initiative, we propose an open living non-binary, fluid, mechanical/robotic, and neutral genders [7]– review framework for generating and regularly updating an open [9]. The notion of gender neutrality in robots has been raised for access dataset of research on gender neutrality in HRI. We offer its potential to disrupt negative gendered associations [9] and a structure for reporting on data relevant to gender neutrality: model new ideas of gender, especially for non-human agents [7]. demographics, theory, methodology, results, and reflexivity. We Others have argued that robots, especially humanoid and social provide a formalized means of submitting articles to the dataset. robots, cannot be genderless due to the CASA phenomenon: our Our contributions are threefold: (1) a rigorous approach based tendency to gender robots when humanlike cues are present [8]. on an international standard for continued and sustainable 1 https://www.softbankrobotics.com/emea/en/pepper consensus-building on a topic of emerging significance to the the dominant gender binary model, it is difficult to define what HRI community; (2) the means of carrying out this process, constitutes gender neutral cues or even an absence of gender including a formalized submission process; and (3) an open cues. Moreover, gender cues may not be perceived in the same access dataset with initial data generated by following this way from person to person or culture to culture. Robertson, for approach. We expect this framework, along with the facilitating instance, found that the Wakamaru robot was perceived to be tools and open dataset, to be updated with community feminine by Westerners and masculine by Japanese people, who involvement. If successful and useful, it may be transferable to viewed its form factor according to cultural models of dress [21]. other topics of interest within the HRI community. As Sutton argues for voice assistants [8], it may be better to view robots s gender ambiguous until they are ascribed gender, and II. CONCEPTUALIZING GENDER NEUTRALITY then consider what factors go into how these ascriptions are Gender is a common yet complicated subject. Gender refers made and what the results are, especially any negative ones. to a multidimensional set of features around which people Other genders and ways of gendering may be considered categorize themselves and organize society [14]–[16]. Many within the gender neutral space. Gender fluidity refers to a models exist, both academically and colloquially. Typically, change in gender identity or expression over time [17]. In their gender is distinguished from sex, with gender used to describe case study on social robots, Stanford’s Gendered Innovations sociocultural qualities and constructs, such as identity and roles group suggests that robots could be designed in a gender fluid within society, and sex referring to aspects of biology and way, especially to mitigate negative gender stereotypes [6]. physiology [15], [16]. Research across fields of science has Androgyny refers to a mixture of masculine and feminine traits suggested that sex and gender can overlap [16]. Many societies within one individual, or one robot in this case. Androgyny may rely on a binary model of gender/sex, premised on create a balance or negating effect when gender is viewed from male/masculine/man and female/feminine/woman poles. Yet, a a binary perspective. Agender and gender-free are human range of intersexes and genders within and beyond the binary identity complements to genderless robots. Non-binary, third exist, including transgenders, non-binary genders, gender genders, and genderqueer refer to gender identities that are not fluidity, third genders, and more [16], [17]. Gender and sex often premised on the gender binary. For robots, the complements of align for many people, i.e., people with biological characteristics these may be mechanical or robotic genders [7], in theory. These that fall into the male category also tend to have a male gender may not apply if people ascribe a range of mechanical genders identity. As a social characteristic, gender is also a matter of to robots, which is not yet known. perception, based on mental models within individuals and societies. Indeed, this seems to drive phenomena such as the At present, little is known about how gender neutrality has CASA when it comes to non-human but humanlike agents, been approached in the design of robots and human-robot including robots. In addition, because of its sociocultural interaction (HRI) research. Even so, the work so far points to origins, gendering can induce biased responses that are both diversity and convergence. The next step is to map out the independent of biology [14]. This is especially true when robots state of affairs in an open and accessible way. are serving in roles that have been traditionally gendered. III. DATASET Gender neutrality, at its most basic, refers to the absence of gender/ing [18]. In this sense, it is relational: it exists with The dataset we are contributing is an open access repository respect to at least one other alignment. In the binary model of of research on the topic of gender neutrality in HRI. This dataset will be regularly updated as part of a living review process. As gender, these alignments are female/feminine/woman and male/masculine/man. Gender neutral can refer to an identity that of this writing, the first version of the dataset, generated by a rapid review, is provided. Details of the rapid review will be one has or a characteristic that is ascribed by others. As an umbrella term, it may be used to encapsulate a range of gender discussed later in the paper as a case study in performing a living alternatives: a lack of gender/ing, an avoidance of gender/ing, review cycle. Data analysis will be reported in another paper. negating or downplaying gender/ing, allowing for gender to be We chose the Open Science Framework (OSF), created by ascribed in diverse ways, among others. Gender neutral the Center for Open Science (COS)2, to host our dataset. Our language, for instance, removes all traces of gendering. decision was based on Nature’s recommendations for data Compare “meteorologist” to the male-gendered “weatherman”: repositories3. OSF is a researcher-supported, open access data the former is genderless because no linguistic gender markers, repository that offers version control, persistent storage, and such as “man,” are present. At a high level, gender neutrality permanent identifiers. There are few restrictions on the data, and displaces the notion of gender and the act of gendering. it allows the data providers to choose a license. It also provides Gender neutrality in robots remains an open question. a means of sharing the data anonymously for peer review. Roboticists and researchers may assume that robots are gender The dataset files and structure are as follows: neutral by default. However, research has shown that robots can signal gender and/or elicit gendering through cues in their A. Screening of Abstracts morphology, notably body [19] and voice [8], and in the This spreadsheet represents the first phase of a review cycle. interaction context, typically due to gender stereotyped views It includes all results returned from database searches, about the role or activity that the robot is carrying out [4], [20]. alternative search tools (such as Google Scholar), manual Given our propensity to gender, especially in accordance with additions, and, in our case, submissions from the community 2 3 https://www.cos.io https://www.nature.com/sdata/policies/repositories#general (see IV.A.1). It does not include duplicates or inappropriate 1) Step 1: Data Collection types of records, which should be eliminated first. It includes the The first step is data collection, i.e., gathering articles, using article title, authors, year of publication, abstract, decision on a combination of traditional systematic search methods (i.e., inclusion based on the abstract, reason for exclusion, which searching databases with keywords and queries) and records were checked by a second reviewer (ideally 20% or community-driven crowdsourcing of articles through a Google more of the records excluded by the first reviewer), reason for Form. This form asks a series of questions about the article that disagreement between reviewers, and relevance despite are directly tied to the columns in the dataset. Contributors are exclusion (such as theoretical papers that may be cited). asked to fill out as much detail as possible to ease the screening and data extraction process. The submission form is available 1) Screening of Full Text here: https://forms.gle/bipvG34mhU2hzhiEA This spreadsheet represents the second phase of a review cycle, wherein the full text of the article is reviewed based on 2) Step 2: Screening the Data the eligibility criteria (detailed below). The included or The next step is to screen the data for eligibility, including uncertain records from the previous phase, particularly the checking for duplicates. This is described in more detail below. author, title, year of publication, and abstract details, are included. Additionally, inclusions at this stage, reason for 3) Step 3: Updating the Dataset inclusion or exclusion, new manual additions based on citations After the article has been screened and deemed eligible, the or references, and relevance despite exclusion are included. provided data will be checked. Any blanks will be filled in, where possible, and any incongruencies will be corrected. The 2) Data Extraction dataset will then be updated on OSF. This spreadsheet represents the final included articles. An ID is assigned to all articles that are brough forward from the 4) Step 4: Periodic Publication of Updates previous stage. Author, title, and year of publication are After a significant number of updates to the dataset, included, as well as all data relevant to the topic of gender estimated to be within a 2-year span, we will integrate the new neutrality in HRI research. These data were determined by two data into existing analysis and/or generate new analyses. These researchers with experience in HRI research and literature will be peer-reviewed and published in an appropriate venue. review methodology. They include: demographics (total sample B. Living Review Methodology size, total men, total women, totals for other genders, totals for unstated/unreported, age mean, age range); robot information All Cochrane reviews, including living reviews, require the (robot/platform name/s, voice origin, simulation or not); theory following components and steps to be taken and reported on. (theory name/s, previous work, brief description, citation/s); These components are presented in line with PRISMA [22]. We methods (manipulation check, other methods for establishing describe each in turn and offer details specific to the topic of gender neutrality, gender options for participants, gender gender neutrality in HRI research. In essence, this is a options for the robot/s, assessment of robot body, assessment of customized methodology fit for our purposes; readers are robot voice); results (in support of gender neutrality, against encouraged to refer to the Cochrane materials and/or Elliott et gender neutrality, other relevant results, relationship between al. [11] for full details on the basic living review methodology. robot voice and body); reflexivity (researchers’ reasons for 1) Eligibility Criteria exploring gender neutrality, researcher’s ascriptions of robot Papers are included if they report on human subjects research gender, participants’ ascriptions of robot gender, gender neutral in HRI where gender neutral voice was considered in the design cues, differences between researcher and participant evaluation of robots. Papers also need to be published as a full ascriptions of robot gender); and any additional notes. or short study in a peer-reviewed venue. Papers are excluded for 3) Risk of Bias / Quality Assessment these reasons: No human subjects findings on gender neutral This spreadsheet includes the results of a risk of bias or voice reported; gray literature, unpublished reports, preprints, quality assessment tool for the included articles. All items from non-peer reviewed papers, conference papers and talks without the instrument as well as a final score or decision are included proceedings; not in the languages known by the authors. next to the article’s author, title, and publication year. 2) Information Sources 4) Accessing the Dataset Information sources are typically major academic databases The dataset can be accessed here: https://osf.io/v6fwg/ but can also include alternatives (such as Google Scholar), registries, and experts (who recommend articles for manual IV. LIVING REVIEW FRAMEWORK AND METHODOLOGY inclusion). All information sources should be listed. We will use, at minimum, academic databases representing general and We have adapted the Cochrane living review guidelines as a engineering topics: Scopus, Web of Science, IEEE Xplore, and basis for our living review framework and methodology. First, ACM Digital Library. Search dates need to be reported. we will present our framework, which comprises the steps for Bibliographies of included papers should also be searched. carrying out the living review and generating the dataset. Then, we will briefly outline the methodological details that we have 3) Search Terms customized for the topic of gender neutrality in HRI research. Search terms are the keywords that make up the queries used in searches. They should be focused but comprehensives. For A. Living Review Framework this topic, we use the following search terms: robot* and gender Our living review framework comprises the following steps: neutral* or neutral gender* or gender-neutral or genderless or gender-less or without gender or no gender* or gender ambigu* or mechanical gender* or agender or androgynous. The full peer-reviewed HRI venue; or must have worked in gender queries used for each information source should be reported in studies for 2 or more years; or must have published on gender as an appendix. One researcher can run all queries. a focus of study in a peer-reviewed venue; ideally has experience doing review work. Those interested in being reviewers can 4) Study Selection apply directly here: https://forms.gle/WAMtYRujLubF8S6f9 Selection of articles involves a two-phase screening process with at least two reviewers. First, abstracts of potential papers V. INITIAL DATASET POPULATION: RAPID REVIEW are dual-screened independently by two or more reviewers. The We conducted a rapid review of the literature, following the primary reviewer should screen all records. About 20% of the Cochrane guidelines [24] and PRISMA approach to reporting same records or records excluded by this reviewer should be [22], with modifications to account for non-medical literatures5. assessed by a second reviewer. Conflicts are resolved by Rapid reviews are “a type of knowledge synthesis in which discussion as they arise. Then, the full text of each paper is [systematic review] methods are streamlined and processes are independently assessed for inclusion by the first reviewer based accelerated to complete the review more quickly” [24, p. 14]. on the eligibility criteria. The second reviewer double-checks Four databases representing general and engineering topics were excluded papers. Disagreements are resolved by consensus. searched: Scopus, Web of Science, IEEE Xplore, and ACM 5) Data Extraction Digital Library. Searches were conducted on September 15th and At least two authors extract data for an even portion of the 16th, 2021. This protocol was registered on OSF6 before queries papers. Each checks the extractions of the other. were run on September 10th, 2021. 6) Risk of Bias and Quality Assessment From an initial set of 551 records, we screened the abstracts Given the diverse study designs in HRI research, we will use of 512, screened the full text of 44, and arrived at a final set of the 13-item Quality Assessment for Diverse Studies (QuADS) 18 articles reporting on a total of 19 experiments. All records at tool [23]. Each reviewer independently evaluates the quality of each stage of the process are available in the dataset. In effect, articles included at the full text stage using a 3-point scale. the rapid review process generated the initial records for the Severe disagreements are discussed until consensus is reached. dataset as well as provided a first test of its structure with data from a representative sample of real articles. 7) Data Analysis/Synthesis Data analysis methods should be reported, including any VI. CONCLUSION meta-analyses. Changed or new results should be positioned The field of HRI is maturing even as hot new topics emerge. against the results reported in previous cycles. As a form of science, HRI research can, and arguably, should, 8) Dataset Structure Updates adapt to the trend and standards of science as a discipline. Open Living reviews bring in new work, which may include new science methodologies and systematic review work, in particular factors to be recorded in the dataset. However, the dataset may living reviews that provide an open access dataset, represent the not provide the structure for recording these new factors. That latest initiatives and scholarly rigour. We have chosen to focus cycle of the living review should report on any changes to the on gender neutrality in robots, a challenging and exciting field dataset and ensure that the Google Form is updated as well. of study within HRI that is growing in momentum. We have offered a framework and methodology, including a formalized C. Governance and Transparency submission tool, for conducting a living review going forward. A living review must live on. We, the authors, should not be We have also provided an open access database that has been the sole arbiters; it should be a community effort with shared validated with data from a rapid review, representing the first responsibility. Conflicts of interest may also arise; we should not cycle of the living review process. We hope to engage the HRI necessarily review our own work. We encourage everyone to community, especially those interested in gender and gender view this initiative as an essential effort for the good of the HRI neutrality, to further test and refine this approach, adapt it to community and knowledge production. Research may be carried other topics, and contribute research for building consensus. out study by study, but the gold standard for determining the A. Limitations and Future Work current state of general knowledge is the meta-analysis/synthesis of these individual studies as generated by rigorous review Two researchers determined what data should be extracted work. We should not shy away from critical opinions, outdated from the articles for the dataset, i.e., what relevant features of findings, or negative results. This is our duty as researchers. the research for the topic of gender neutrality in robots should make up the structure of the dataset. Future work will involve Gathering reviewers will be a multi-pronged, ongoing effort. seeking input from other researchers, such as through a Workshops at HRI venues (e.g., RO-MAN, which has a gender conference workshop or panel. Finally, there is a bug on OSF workshop4) can “filter in” experts who have a vested interest in for spreadsheet files: it can only support single headers. We are the topic via self-selection. Reviewer criteria will be based on a currently working with OSF to enable multiple headers. modified form of the HRI2022’s reviewer criteria to start, and be refined later with community involvement: Must have REFERENCES worked in HRI for 2 or more years; or must have published at a [1] C. Nass, Y. Moon, and N. Green, “Are machines gender neutral? Gender- stereotypic responses to computers with voices,” J. Appl. Soc. Psychol., 4 6 https://sites.google.com/view/ro-man21-genr-workshop/home https://osf.io/gcesd 5 For example, we did not use a structured abstract or PICOS.  vol. 27, no. 10, pp. 864–876, 1997, doi: 10.1111/j.1559- [13] Cochrane, “Guidance for the production and publication of Cochrane 1816.1997.tb00275.x. living systematic reviews: Cochrane Reviews in living mode,” Cochrane, [2] C. I. Nass and S. Brave, Wired for Speech: How Voice Activates and Dec. 2019. [Online]. Available: Advances the Human-Computer Relationship. Cambridge, MA: MIT https://community.cochrane.org/sites/default/files/uploads/inline- Press, 2005. files/Transform/201912_LSR_Revised_Guidance.pdf [3] K. Seaborn, N. P. Miyake, P. Pennefather, and M. Otake-Matsuura, [14] M. W. Nielsen et al., “Gender-related variables for health research,” Biol. “Voice in human-agent interaction: A survey,” ACM Comput. Surv. Sex Differ., vol. 12, no. 1, p. 23, Feb. 2021, doi: 10.1186/s13293-021- CSUR, vol. 54, no. 4, p. Article No. 81, 2021, doi: 10.1145/3386867. 00366-3. [4] D. Bryant, J. Borenstein, and A. Howard, “Why should we gender? The [15] C. Tannenbaum, R. P. Ellis, F. Eyssel, J. Zou, and L. Schiebinger, “Sex effect of robot gendering and occupational stereotypes on human trust and and gender analysis improves science and engineering,” Nature, vol. 575, perceived competency,” in Proceedings of the 2020 ACM/IEEE no. 7781, pp. 137–146, Nov. 2019, doi: 10.1038/s41586-019-1657-6. International Conference on Human-Robot Interaction, Cambridge, [16] J. S. Hyde, R. S. Bigler, D. Joel, C. C. Tate, and S. M. van Anders, “The United Kingdom, Mar. 2020, pp. 13–21. doi: 10.1145/3319502.3374778. future of sex and gender in psychology: Five challenges to the gender [5] C. R. Crowell, M. Villanoy, M. Scheutzz, and P. Schermerhornz, binary,” Am. Psychol., vol. 74, no. 2, pp. 171–193, 2019, doi: “Gendered voice and robot entities: Perceptions and reactions of male and 10.1037/amp0000307. female subjects,” in Proceedings of the 2009 IEEE/RSJ International [17] S. L. Katz-Wise, “Gender fluidity: What it means and why support Conference on Intelligent Robots and Systems, St. Louis, MO, Oct. 2009, matters,” Harvard Health, Dec. 03, 2020. pp. 3735–3741. doi: 10.1109/IROS.2009.5354204. https://www.health.harvard.edu/blog/gender-fluidity-what-it-means-and- [6] Gendered Innovations in Science, Medicine, and Engineering, why-support-matters-2020120321544 (accessed Oct. 14, 2021). “Gendering Social Robots: Analyzing Gender and Intersectionality,” [18] J. R. Udry, “The nature of gender,” Demography, vol. 31, no. 4, pp. 561– Gendered Innovations in Science, Medicine, and Engineering, n.d. 573, Nov. 1994, doi: 10.2307/2061790. http://genderedinnovations.stanford.edu/case- [19] G. Trovato, C. Lucho, and R. Paredes, “She’s Electric—The Influence of studies/genderingsocialrobots.html#tabs-2 (accessed Oct. 14, 2021). Body Proportions on Perceived Gender of Robots across Cultures,” [7] R. A. Søraa, “Mechanical genders: How do humans gender robots?,” Robotics, vol. 7, no. 3, Art. no. 3, Sep. 2018, doi: Gend. Technol. Dev., vol. 21, no. 1–2, pp. 99–115, May 2017, doi: 10.3390/robotics7030050. 10.1080/09718524.2017.1385320. [20] F. Eyssel and F. Hegel, “(S)he’s got the look: Gender stereotyping of [8] S. J. Sutton, “Gender Ambiguous, not Genderless: Designing Gender in robots,” J. Appl. Soc. Psychol., vol. 42, no. 9, pp. 2213–2230, 2012, doi: Voice User Interfaces (VUIs) with Sensitivity,” in Proceedings of the 2nd 10.1111/j.1559-1816.2012.00937.x. Conference on Conversational User Interfaces, New York, NY, USA, [21] J. Robertson, “Gendering humanoid robots: Robo-sexism in Japan,” Body Jul. 2020, pp. 1–8. doi: 10.1145/3405755.3406123. Soc., vol. 16, no. 2, pp. 1–36, 2010, doi: 10.1177/1357034X10364767. [9] J. Cambre and C. Kulkarni, “One Voice Fits All? Social Implications and [22] A. Liberati et al., “The PRISMA statement for reporting systematic Research Challenges of Designing Voices for Smart Devices,” Proc. reviews and meta-analyses of studies that evaluate healthcare ACM Hum.-Comput. Interact., vol. 3, no. CSCW, p. 223:1-223:19, Nov. interventions: Explanation and elaboration,” PLoS Med., vol. 6, no. 7, Jul. 2019, doi: 10.1145/3359325. 2009, doi: 10.1371/journal.pmed.1000100. [10] A. K. Pandey and R. Gelin, “A mass-produced sociable humanoid robot: [23] R. Harrison, B. Jones, P. Gardner, and R. Lawton, “Quality assessment Pepper: The first machine of its kind,” IEEE Robot. Autom. Mag., vol. 25, with diverse studies (QuADS): an appraisal tool for methodological and no. 3, pp. 40–48, Sep. 2018, doi: 10.1109/MRA.2018.2833157. reporting quality in systematic reviews of mixed- or multi-method [11] J. H. Elliott et al., “Living systematic review: 1. Introduction—the why, studies,” BMC Health Serv. Res., vol. 21, no. 1, p. 144, Feb. 2021, doi: what, when, and how,” J. Clin. Epidemiol., vol. 91, pp. 23–30, Nov. 2017, 10.1186/s12913-021-06122-y. doi: 10.1016/j.jclinepi.2017.08.010. [24] C. Garritty et al., “Cochrane Rapid Reviews Methods Group offers [12] Z. Munn, C. Stern, E. Aromataris, C. Lockwood, and Z. Jordan, “What evidence-informed guidance to conduct rapid reviews,” J. Clin. kind of systematic review should I conduct? A proposed typology and Epidemiol., vol. 130, pp. 13–22, Feb. 2021, doi: guidance for systematic reviewers in the medical and health sciences,” 10.1016/j.jclinepi.2020.10.007. BMC Med. Res. Methodol., vol. 18, p. 5, Jan. 2018, doi: 10.1186/s12874- 017-0468-4.