Caf--Delphi--Hybridising--World-Caf---and--Delphi-Tech_2021_Social-Sciences-.pdf
- 1. Caf e Delphi: Hybridising ‘World Caf e’ and ‘Delphi Techniques’ for successful remote academic collaboration A. Jolly * , L.S. Caulfield , B. Sojka , S. Iafrati , J. Rees , R. Massie Institute for Community Research and Development (ICRD), Mary Seacole Building, University of Wolverhampton, WV1 1AD, UK A R T I C L E I N F O Keywords: Delphi technique World caf e research Remote working Collaboration Academic writing Interdisciplinary writing A B S T R A C T Developing collaborative and cooperative research across academic disciplines and university administrative boundaries can be a challenge. In an attempt to understand and propose solutions to this challenge, the authors of this paper set out to: test an innovative combination of methods to generate and evaluate ideas and strategies; and to write about the findings using collaborative online methods. During this process universities in the UK moved to online working and so the authors completed this paper through entirely online means. The authors - a team of academic researchers from the University of AAA - came together in sessions designed as a hybrid of World Caf e and Delphi technique approaches to discuss challenges and solutions. The findings were written up drawing on insights from the use of massively authored papers (also known as ‘massively open online papers’, MOOPs), and online tools to enable remote collaboration. Expert consensus was sought in this project within a group of participants (N ¼ 7) in one university setting to create a MOOP. This paper presents details of the process, the findings, and reflections on this collaborative and cooperative exercise. That this paper was written using the methods discussed within it, highlights the value and success of the approach. In light of the current Coronavirus pandemic and the increased need to work remotely, this paper offers academics useful strategies for meaningful and productive online collaboration. 1. Introduction Developing collaborative and cooperative research across academic disciplines and university administrative boundaries can be a major challenge, as evidenced by numerous initiatives established by Higher Education Institutions to encourage greater social interaction, shared research cultures, and shared corporate identities. This paper uses a methodologically innovative combination of methods to generate and evaluate ideas and strategies for developing a cross-faculty research culture at the University of AAA and beyond. The paper draws on the findings from two participatory ‘Research Caf e’ seminars hosted by the Institute BBB at the University of AAA. The first seminar used what can be described as a ‘Caf e Delphi’ approach as it drew on a hybrid of the World Caf e approach and the Delphi technique to discuss challenges and solu- tions to building a cross-faculty research culture (both approaches are described individually in more detail below). This session made use of an online tool called Mentimeter. In the second follow-up seminar, the approach our reflections were written up in real-time using a single shared Google Doc. The final draft was completed using Microsoft Teams group chat and video meeting software to discuss changes made, drawing on insights from the use of massively authored papers (also known as ‘massively open online papers’, MOOPs). This methodology could be applied to a number of purposes in aca- demic life, such as developing bids, drafting departmental strategies, and most significantly, writing academic papers. In the current public health environment of increased social distancing and remote working across the world, the method offers a flexible means of using technology to elicit information, develop consensus and collaborate with colleagues. 2. Literature review This section will introduce the two methodological tools used in this paper - the Delphi technique, and the World Caf e method - before dis- cussing the concept of massively authored online papers, and concludes with a brief explanation of how cross-faculty working is currently utilised in higher education institutions. 2.1. Delphi Technique The Delphi technique is a structured communication technique * Corresponding author. E-mail address: [email protected] (A. Jolly). Contents lists available at ScienceDirect Social Sciences Humanities Open journal homepage: www.elsevier.com/locate/ssaho https://doi.org/10.1016/j.ssaho.2020.100095 Received 30 April 2020; Received in revised form 28 November 2020; Accepted 30 November 2020 Available online 23 December 2020 2590-2911/© 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by- nc-nd/4.0/). Social Sciences Humanities Open 3 (2021) 100095
- 2. whereby a group of people aim to arrive at a consensus through a series of questions and discussions, originally developed, and still often used in, future forecasting. Although sometimes perceived as a quantitative technique, because of the use of questionnaires and anonymous aggre- gation of opinions, its ability to capture group narratives and opinion means it can also be considered a qualitative or mixed method approach. In doing so the method allows us to find the most suitable solution for a problem under study. Although the Delphi technique originated in the 1950s as a project conducted by the United States Air Force, today it is successfully used by researchers that seek a consensus of opinions among research participants. In practice, the Delphi technique employs the use of a series of questionnaires. After the first round of questions, a moderator provides participants with anonymous feedback on the an- swers offered by all research participants. Each subsequent questionnaire is adjusted taking into account information and opinions found in the previous questionnaire. Participants reflect on this feedback and are able to include these reflections as well as their opinions during subsequent questionnaires. This process is repeated until a desired level of consensus is reached among all of the participants. There are four key elements to Delphi surveys: 1) anonymity; 2) iteration; 3) controlled feedback; and 4) statistical aggregation of group responses (Rowe Wright, 1999). The Delphi technique is flexible and can be adjusted by the researchers who can include various methods of selecting participants, types of questions employed, and tools used for the analysis of responses. There are various advantages and disadvantages of the Delphi technique. Disadvantages of the Delphi technique include the lack of opportunity to elaborate on and discuss responses with other individuals; the need for a commitment from individuals to continue to revisit similar questions on multiple occasions and that the answer reached may not be ‘correct’ but rather an indication of an important point for consideration for the group of individuals involved. 2.2. World Caf e method If we acknowledge that successfully addressing challenges requires insight from multiple perspectives, appropriate methods must be devel- oped to allow these perspectives to be shared and heard. Where a project aims to include the voices of a range of partners, the World Caf e approach has been used with some success. The World Caf e approach is a participatory method for engagement. This method is more conversational in approach than other methods, allowing participants to engage in ‘constructive dialogue around critical questions’ (Fouch e Light, 2010). The World Caf e is a relatively simple method, but powerful in ‘fostering constructive dialogue, accessing col- lective intelligence and creating innovative possibilities for action, particularly in groups that are larger than most traditional dialogue ap- proaches are designed to accommodate’ (Brown Isaacs, 2005, p. 3). A number of broad principles underpin the World Caf e approach, although the principles can be adopted and adapted to be responsive to the aims of the project and needs of the participant group. The basic model in- corporates the following five elements: 1) Setting - an informal, caf e style environment 2) Welcome and Introduction - the host puts participants at ease through a warm welcome and overview 3) Small-Group Rounds - three or more 20 min rounds of conversation between small groups of four or five participants 4) Questions - a key question underpins each round, which may build upon one another in subsequent rounds 5) Harvest - each group is invited to share the results of their conver- sations with the wider group, and these are reflected visually (eg. through graphic recording in front of the room) (The World Caf e, 2020). Fig. 1, below, outlines the seven principles for hosting World Caf es. All participants are viewed as experts by their own experience. Diverse views and opinions are sought and there is no pressure to reach a consensus (MacFarlane et al., 2017). The World Caf e method has been applied to a number of disciplines and applied settings, with several authors reflecting on their experiences. Fouch e and Light (2010), for example, report on the design and imple- mentation of the method in social work research; while Macarlane et al. (2017) provide an analysis of the method in primary care settings in Ireland and the USA; and Page and Temple-Malt (2018) reflect on its application in criminological research. As MacFarlane et al. (2017: p.278) conclude, the World Caf e method is generally viewed as ‘a valuable, participatory, flexible method’ and thus suitable for gatherings of groups with different backgrounds, ex- periences, and viewpoints. 2.3. Massively authored papers At the heart of both the Delphi Technique and the World Cafe lie changes in technology that have enabled new ways of working within academic writing and research, allowing more people to easily collabo- rate on research and writing both in person and remotely. Tennant et al. (2019) recognise the opportunity for ‘massively open online papers’ (MOOPs). Significantly, this not only highlights new practical possibil- ities within academic writing, but, importantly, begins to enhance the breadth of ideas, experiences, and contexts contained in pieces of aca- demic writing and research that were previously out of the reach of many academics. Whilst this may pose challenges in terms of continuity, it can also bring significant benefits regarding the comprehensive and collab- orative nature of research. Recognising the threat of organised chaos and a loss of focus, Tennant et al. recognise a need for there to be a clear focus and structure un- derpinning a project that is shared among all participants. For Tennant et al. this is achieved by agreeing a set of clear ‘rules’ for the process, such as recognising editorial control, guidelines for contributors, setting the structure of the paper and the ways by which contributors are ‘invited’ by those involved. As such, despite concerns that the research and writing process could descend into being amorphous and fragmented, abiding to Fig. 1. Principles for hosting World Caf es Reproduced from the World Cafe website. Source: www.theworldcafe.com under creative commons license (CCA4) © 2020 The World Cafe Commu- nity Foundation. A. Jolly et al. Social Sciences Humanities Open 3 (2021) 100095 2
- 3. the rules allows for coherence and relevance. To this end, it is important to invite appropriate people and ensure that the opportunity for expo- nential growth in authorship is balanced with clear focus, direction and control over membership. In effect, far from being a fluid or organic process, the management of MOOPs should feature rigorous controls and management. The use of online writing platforms, such as Google Docs or other comparable platforms, can allow for collective writing, reviewing and recognising a broad context for one’s work. It also allows people to be allocated to focus on specific sections of a writing project, in addition to academics working remotely. Recognising the benefits of massively authored papers, Tennant et al. (2019) see the process as being able to open up academic debate and collaboration beyond the academic confines within which we can often find ourselves operating. This might include working with potential partners from other time zones, other institutions or, importantly, those from outside of our own disciplines. Not only does this push the boundaries of our knowledge and expand analyses, it can also lead to appealing to wider audiences. With this in mind, it might be that the authors reflect different traditions within a particular subject area, such as Social Sciences or Humanities, but may also come from academic disciplines not previously associated with these topics but who are in a position to make critical and important contributions. Similarly, it might also be that authors are invited to take part because of their different experiences and knowledge, which may include professional and/or lived experiences. Therefore, editorial control of the process means that not everyone involved has to take on comparable roles within the paper, produce the same amount of work, or come from the same starting point. Tennant et al. (2019) set out a series of steps and guidelines to enable a smooth process and facilitate productive outcomes. Importantly, they also recognise that research and writing projects never turn out entirely as planned and that flexibility needs to be integral to the process. Further to this, Tomlinson et al. (2012) have explored the practicalities of the process, and usefully examined the feelings of those involved in the process. Recognising the challenge that authors could feel abstracted from the process, feeling that their engagement was of a ‘helicopter’ nature and that they would have liked greater engagement. The risk of author isolation and sporadic engagement can limit feelings of owner- ship. This also highlights the importance of inclusivity, raised by Tennant et al. that can be achieved by identifying structures, roles and ensuring that these are maintained. For Vaish et al. (2017), the process of crowd research allows for a democratisation of research and reporting. Inherent in this is the op- portunity to create a more diverse research team and reporting mecha- nisms. On an instrumental level, such an approach is useful for introducing new and critical voices that can add a degree of validity and insight to research and reporting. However, at a time when there is greater scrutiny of diversity within research and higher education in general, then the democratisation of research can be seen as beneficial. We recognise that the process of working together as presented in the article can pose a challenge associated with the level of technological capacity and technological infrastructure required to write together. This can be especially problematic when working with institutions based in countries with significant technological limitations. The work of scholars from the Global South is underrepresented in European and US journals in general and poor infrastructure is shown as one of the most important causes of that omission (American Political Science Association, 2016). 3. Method The core approach underpinning this project was a hybrid ‘Caf e Delphi’ method, synthesising the approaches outlined in more detail in the literature review above. The project took place across three 2-h ses- sions over three days several weeks apart. In the first, information was elicited using the process outlined in Fig. 2 below. In the second, Google Docs was used to write the article collaboratively in real time. In the final session, Microsoft Teams was used to review and redraft the paper. In this case the method was adopted as a novel approach (participants had not used it together before as a single group) in the hope that it would maximise the generation of ideas and dialogue that could be taken forward and then formally written-up via the ‘massively distributed authorship’ approach illustrated by this paper. In order to conduct the initial piece of ‘research’ and prepare the ‘findings’ reported in this paper, a small and coherent group of researchers, already known to each other in a workplace setting (n ¼ 7), went through a three-stage process, during which the clear, shared intention was to learn and practice the approach in order to apply it in other similar projects, and potentially in other settings. It was therefore embarked on in a shared experimental and ‘open-minded’ spirit. The first step was to conduct an exercise that in effect was a hybrid combination of a ‘real-time’ Delphi panel process (Gordon Pease, 2006) and a World Caf e, which lasted for approximately 1 h in total, and was carried out in the room where the researchers were gathered (all researchers were physically present). In keeping with the World Caf e approach the setting was informal, and food and drink was provided. In its most basic sense, the approach consisted of a group ‘brainstorming’ discussion in order to surface ideas, the outcomes of which were shared on a large wall mounted video screen using Mentimeter; followed by a second stage in which the first round ideas were ranked in perceived order of importance. Mentimeter allows users to ‘vote’ on the options in order to establish the final ranking. The schedule employed was as follows: The group-sourced answers to each question were displayed on the screen (in ‘bubbles’) so that the group could see the emerging ideas, but these were anonymised and unattributable to individuals (unless they chose to highlight that they have supplied a particular answer, which in this case no-one did). At this point there was no ranking or any other method of highlighting or distinguishing the ‘quality’ or popularity of the ideas - they are treated as equal at this stage. The next step of the exercise however is to rank the ideas, an implicit recognition or judgement of the relative merit of each idea. These ‘findings’ were then taken forward into the writing process carried out through massively distributed authorship using Google Docs. In the transition from the ‘Caf e Delphi’ phase, decisions were made as to which answers to prioritise and discuss in more detail. The quantitative ranking suggested that the top four challenges were worth further exploration; and the top five ranked solutions came out well above the following options (see Figs. 3 and 4. In the findings we focus on these top four challenges and top five solutions. For this process, the authors were face to face in the same room and able to communicate directly with one another, although this could be adapted to operate remotely using software such as Microsoft Teams. 4. Findings 4.1. Challenges Reflecting on the role of technology and online collaboration, the group raised 12 potential challenges to building a cross-faculty research Fig. 2. Caf e Delphi process. A. Jolly et al. Social Sciences Humanities Open 3 (2021) 100095 3
- 4. culture. Research leadership and reporting within faculty structures was identified as an issue, where in some instances leadership can become either too dominant or too weak, thus closing down opportunities for a collaborative culture. A lack of opportunities for researchers to spend time together, whether this be ad hoc opportunities or cross-faculty events, was noted as a possible challenge. There is an additional chal- lenge of people attending and prioritising structured sessions and events that are run as and purposely set up to be cross-faculty. Another perceived challenge was the time to prioritise research generally. Disciplinary boundaries were considered another challenge, due to different methodological traditions and disciplinary styles of working (one might even say dogmas and pedagogic and research ideologies); the challenge of knowing how to identify who might be open to and ‘good at’ collaborating in other faculties, and the varying incentive structures/ motivations to engaging with research. Finally, challenges were raised in relation to how to develop shared aims and how to measure success. Similarly, another challenge raised was that in a UK context, the REF (Research Excellence Framework) process and the method by which QR funding (Quality Related - distributed to Universities based on REF per- formance) was distributed (and may be in future) could push research to be conducted in silos, thus not promoting cross-faculty research. When asked to rank the biggest challenges to building a cross-faculty research culture, participants clearly felt that the most important chal- lenge was a lack of opportunities to meet face to face in daily University life (Fig. 3). This almost certainly reflects a feeling that the physical space of the University precluded ad hoc interactions or possibly also that there was a dearth of organised cross-faculty sessions (such as seminars, public events, and social gatherings). The balance between these would be worthy of further qualitative exploration. Second most important (though only chosen half as many times) was ‘getting people to attend organised sessions’ (reflecting perhaps frustration that even when pro- vided, events are not well attended). 4.2. Solutions Nine potential solutions were identified that might contribute to building a cross-faculty research culture. The solutions proposed fit within three broad areas: spaces to interact; funding opportunities; and leadership support. Solutions relating to face to face interaction included: (1) face to face events to address the culture and encourage informal conversations; (2) core research staff attending subject meetings to ensure research remains on the agenda; and (3) a physical shared space (i.e., bar, cafe, pub or Union). Virtual communication (4) was also discussed whereby re- searchers could share news of successes and research related activity via virtual spaces (i.e. newsletters). Three solutions proposed related to funding. These included: (5) the availability of internal funding for cross-disciplinary projects; (6) incentivising collaboration through joint calls/bids/seedcorn proposals to kickstart discussions and projects; and (7) cross-faculty research themes proactively identifying funding opportunities to bring people together. The final two solutions related to senior leadership and support, including (8) Associate Dean’s facilitating thematic collaboration and (9) clear leadership engagement with support for research. The top ranked solution was joint calls, bids and seedcorn funding promoting discussions and projects, followed by cross-faculty research themes and internal funding for cross-disciplinary projects as shown in Fig. 4. 5. Reflections on the process This section discusses, firstly, the authors’ reflections of their involvement in the ‘Caf e Delphi’ approach and secondly, their experi- ences of writing a multi-authored paper using Google Docs. 5.1. ‘Caf e Delphi’ The authors were in agreement that the hybridisation of Delphi Technique and World Caf e research methods could be considered com- plementary methods, whereby expert consensus can work well with group consultation. It was felt that this hybrid method allowed for both defining the discussion and moving forward to propose and rank chal- lenges/solutions in one session. It was good to have participants in the same room for the purpose of sharing opinions and a combined sense of brainstorming, however, the authors believe the method is flexible enough to involve wider participants in discussion and ranking activities from a distance. Bearing in mind the claim in the literature review that clear editorial guidance, membership, and direction are essential to avoid a process sometimes called ‘mission drift’ or the production of an unfo- cused piece of work, the communication was useful. Had we been in different institutions or even different time zones, this element of everyone recognising a shared vision and process may have been slightly more challenging. Geographic or even temporal variation can be recog- nised as a reason why some people may feel ‘helicoptered’ into the process. As such, clear communication and guidance from the start is important if the process is to be effective and focused. Significantly, the potential fluidity and variation of multi-authored work should not be confused with being directionless or without leadership. The method was also suitable for bringing together disparate perspectives, given that all suggestions are considered equal during the discussion phase, before anonymous expert ranking took place. The use of technology (in this instance Mentimeter) was rated highly by the authors, who stated this tool was intuitive; inclusive, allowing ideas and thoughts to be suggested simultaneously in real time; and unfiltered by the facilitator, encouraging equal voice. On a more practical level, the authors felt that future implementation would be easier having now been through the full process. In particular, authors reflected on the need for a clear briefing email to all participants in advance of the first session so that they were fully aware of the process and need to bring a laptop in order to participate fully in the workshop and the subsequent multi-authored paper session. Fig. 3. Ranking of challenges identified. Fig. 4. Ranking of solutions identified. A. Jolly et al. Social Sciences Humanities Open 3 (2021) 100095 4
- 5. 5.2. Multi-authored papers The positive, supportive peer environment, which led to a focussed effort and rapid formation of a draft paper, was considered the main advantage to writing a multi-authored paper. All authors felt a great sense of achievement at the end of a 1 h session. The authors reflected on the positive social pressure of seeing others contributing to the content in real-time and how this helped them maintain their own focus. Having identified a dedicated time for all group members to participate in this shared activity, it was found to be a good corrective to the procrastina- tion and ‘writer’s block’ which can bedevil lone working (and particu- larly lone home working). In particular it was found that writing sections of the paper in small blocks of time, and agreeing a deadline, contributed to the timely completion of writing tasks. Furthermore, the agreed approach was for individuals to draft sections separately, so as to avoid duplication and ‘over-writing’, then to ‘swap’ in the next block of time and to edit or add to other participants’ previously drafted sections. This led to rapid refinement of text and the overall progress of the paper. Overall it was felt that this method of multiple authorship was easily applied to report writing. Despite having a clear report structure in place, one challenge of multi-authored papers is the various writing styles of different authors which can result in a poor flow to the paper and/or discrete sections of the report, though this can clearly be an issue with traditional multi- authored papers. It is really important to outline the style and focus of the report and to have identified an editor who will take responsibility to pull the draft paper together more coherently at the end. Later in the process of completing the paper the Coronavirus crisis intervened and we were forced to distance ourselves physically. We therefore found it invaluable to move to Microsoft Teams in order to discuss changes to the paper in real time and particularly in order to agree wording and final edits. We used Google Docs throughout and recommend this way of working as a way to prepare and complete MOOPs at distance. The findings and reflections presented in this paper should be read in the context of a group of collaborators who were, before embarking on this process, already known to one another. This meant that some of the initial tasks were able to be embarked upon with good knowledge of the group’s strengths and working styles, creating a quickly relaxed atmo- sphere. However, it is worthy of note that although the group were well known to one another, they had not previously embarked on a whole group collaborative project or paper. Furthermore, they did not have previous experience working as a group via only online means. We recommend that where others adopt the methods outlined in this current paper, that if they are unknown to one another, more time at the beginning might be spent on the introductions and early small group discussion rounds. This is likely to be particularly important where all sessions are conducted online. We would welcome insights from groups working in that way. 5.3. Limitations The method relied on social networking technology both in devel- oping and collaboratively writing the paper. Although other tools are available, for the purposes of this paper we used ‘google docs’ and ‘mentimeter’. These were chosen because they are either free to use, or have free versions. They are therefore more accessible to those in con- texts without institutional access to academic networking software - such as those outside of higher education, and using a common, free to use platform, can encourage Global north - Global South partnerships and collaborations. However, it is recognised that in contexts without a reliable wi-fi infrastructure, this method would be of less utility. As discussed above, the method also relies on ‘real-time’ editing and collaboration which could act as a barrier to collaboration across time zones. We also used Microsoft Teams for part of the write up process, which was convenient for collaboration within our institution, but would have been a barrier for those outside of the institution who did not have paid access to Microsoft Office. The reliance on proprietary technology by multinational corporations such as Google and Microsoft, while conve- nient due to the network effects of their common use, raises concerns about the impact if a particular programme or website is removed or becomes inaccessible (for instance the difficulty of accessing Google for users in mainland China), and privacy concerns around the security and use of sensitive data by technology corporations (Larson, 2020). 6. Conclusion This paper, we hope, does a number of things. Firstly, it is an attempt to put into practice the collaborative writing of a ‘massive open online paper’ (MOOP), in this case among a small group of academics who ordinarily work together quite closely; but in so doing we hope to demonstrate that this is an approach that can work well for looser groups of academics to collaborate at distance. This is particularly useful at a time when people are being required to work at considerable physical distance from each other as a result of the Coronavirus pandemic. Underpinning this particular process for producing a finished paper, we employed a hybrid combination of the World Caf e approach and the Delphi technique to brainstorm an issue and to reach a consensus. In this instance we used these techniques to discuss challenges and solutions to building a cross-faculty research culture, a topic that seemed particularly fitting given the methods we had adopted. Given that this approach was quite novel for us both individually and as a collective, we also saw it as a good opportunity to think more reflectively about what we had learnt. Thus we feel able to make a number of recommendations: ● Encourage some time to become familiar with each other’s commu- nication styles and the relevant technology ● Select software solutions that work for you and your collaborators, in this case we used Mentimeter, Google Docs, and MS Teams ● Selecting a time when every member of the team is ‘present’ and free from distractions is important ● Working in short timed bursts on sections of the paper, and then rotating in turn to edit each other’s writing was found to be partic- ularly effective ● Identifying a coordinator/editor to provide structure, guidance, and to refine the final voice and flow of the written paper ● Further Delphi panels for each ‘solution’ to explore and unpack in greater detail 7. Weblinks Here are links to the online tools we used in the work described above. Other similar tools are available. Mentimeter https://www.mentimeter.com. Google Docs https://docs.google.com. Microsoft Teams https://products.office.com/en-gb/microsoft -teams/group-chat-software. Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. CRediT authorship contribution statement A. Jolly: Conceptualization, Methodology, Investigation, Writing - original draft, Writing - review editing, Visualization. L.S. Caulfield: Investigation, Writing - original draft, Writing - review editing, Visu- alization. B. Sojka: Investigation, Writing - original draft, Writing - re- view editing, Visualization. S. Iafrati: Investigation, Writing - original draft, Writing - review editing, Visualization. J. Rees: Investigation, A. Jolly et al. Social Sciences Humanities Open 3 (2021) 100095 5
- 6. Writing - original draft, Writing - review editing, Visualization. R. Massie: Investigation, Writing - original draft, Visualization. Declaration of competing interest The authors whose names are listed immediately below certify that they have NO affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consul- tancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript. References Brown, J., Isaacs, D. (2005). The World Caf e: Shaping Our Futures Through Conversations that Matter. San Francisco: Berrett-Koehler Publishers. Fouch e, C., Light, G. (2010). An invitation to dialogue. ‘The World Caf e’ Soc. Work Res. Qualit. Soc. Work: Res. Pract., 10(1), 28–48. https://doi.org/10.1177/ 1473325010376016, 26th November 2019. Gordon, T., Pease, A. (2006). RT Delphi: An efficient,“round-less” almost real time Delphi method. Technological Forecasting and Social Change, 73(4), 321–333. Larson, R. (2020). Bit Tyrants: The Political Economy of Silicone Valley. Chicago: Haymarket Books. Macfarlane, A., Galvin, R., O’sullivan, M., Mcinerney, C., Meagher, E., Burke, D., Lemaster, J. W. (2017). Participatory methods for research prioritisation in primary care: An analysis of the world caf e approach in Ireland and USA. Family Practice, 34(3), 278–284. https://doi.org/10.1093/fampra/cmw104, 26th November 2019. Page, S. J., Temple-Malt, E. (2018). ‘World caf e: A participatory research tool for the criminologist engaged in seeking world views for transformation: British Society of Criminology Conference papers (Vol. 18, pp. 5–19). Rowe, G., Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4), 353–375. Tennant, J. P., Bielczyk, N., Tzovaras, B. G., Masuzzo, P., Steine, T. (2019). Introducing Massively Open Online Papers (MOOPs). https://doi.org/10.31222/osf.io/et8ak The World Caf e. (2020). World Caf e Method Greenbrae CA. World Caf e Foundation. http://www.theworldcafe.com/key-concepts-resources/world-cafe-method/, 21st January 2020. Tomlinson, B., Ross, J., Andr e, P., Baumer, E. P. S., Patterson, D. J., Corneli, J., Mahaux, M., Nobarany, S., Sambasivan, N., Lazzari, M., Penzenstadler, B., Torrance, A. W., Callele, D. J., Olson, G. M., Silberman, S., St€ ander, M., Romancini, F., Palamedi, F., Salah, A. A., … Saper, C. (2012). Massively distributed authorship of academic papers. In International Conference: International Conference on Human Factors in Computing Systems (CHI) at: Austin, TX. https://upcommons.upc.edu/bitstre am/handle/2117/171295/Academicþpapers.pdf;jsessionid¼F972A0C5EA1 A8ED455393824945A836B?sequence¼1, 26th November 2019. Vaish, R., Gaikwad, S. N. S., Kovacs, G., Veit, A., Krishna, R., Ibarra, I. A., Simoiu, C., Wilber, M., Belongie, S., Goel, S., Davis, J., Bernstein, M. S. (2017). Crowd research: Open and scalable university laboratories. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, October 2017 (pp. 829–843). https://doi.org/10.1145/3126594.3126648 A. Jolly et al. Social Sciences Humanities Open 3 (2021) 100095 6