1 Introduction

The internet is now firmly embedded at the heart of the student learning experience with ‘e-learning’, web-enabled-based module materials and virtual learning environments (VLEs) changing how students interact with information. Given this, in this paper we investigate how students gather information to answer a coursework question using the internet. Our focus is on the process of undertaking research rather than the actual question being asked, allowing our findings to be used across all academic disciplines. While our research was undertaken before the explosion of AI in higher education over the past eighteen months it provides pointers on how AI use may develop.

Our initial belief, in line with the observations of Goldman et al. (2018), is that the internet could lead to surface-level text analysis. Students are drawn to internet sites that contain relevant keywords and new/different information, but students are failing to locate, decipher and critically evaluate the higher level, detailed information. This belief fits with the early work of Entwistle (1988), and the areas of strategic, surface and deep learning approaches. This distinction between deep and surface learning emerged from the investigations conducted by Marton and Säljö in 1976.

Students can employ either deep or surface strategies, or a blend of both, during their information gathering. The design of courses and assessments, as well as the teaching methods, significantly influence the cultivation of deep, surface, and strategic learning (see Bennett, 2011 or Guest, 2019). This may lead to a deficiency in student engagement, discovery, integration, and application. Of note is that students’ internet-based information-seeking behaviour may lead to different levels of engagement with source quality, reflecting potential challenges in balancing quick discovery with critical evaluation. This underscores the importance of fostering digital fluency, beyond digital literacy skills, to ensure higher-level analysis and comprehension.

The literature in this area considers several theoretical approaches. These include Information Problem Solving IPS-I, including the Big6 (see Brand-Gruwel et al., 2009, Walraven et al., 2008a and 2008b, 2013 or Wopereis et al. 2011, Eisenberg & Berkowitz, 1990); information literacy (see Miller & Bartlett, 2012 or Kiili et al., 2008); the new literacies perspective (Coiro et al., 2015); Epistemic thinking (Barzilai et al. 2012); Sourcing (Brante & Strømsø, 2018); and Multi-text comprehension theory (Goldman et al., 2012 or Wiley et al., 2009). These approaches, especially those in information problem-solving utilising the internet, have tended to examine user attributes, evaluation and selection of multiple sources, and how individuals validate source material.

Our focus is to examine how far undergraduate students go in their internet information search to complete a task (in this case, a module coursework assignment). We investigate whether students surface search, referencing the immediate sites (those that come up on page one of a Google search), or whether students continue to search for a more refined, in-depth analysis. The paper has three research objectives that guide our investigation: first, to assess successful completion rates as assessment task complexity increases; second, to evaluate the relationship between the quantity and quality of bibliographic references and the use of higher quality scholarly references on academic performance; and third, to investigate the effects that student engagement with the provided module eLearning materials has on literature review methods.

We first present the literature review, synthesising relevant theoretical frameworks on internet-based information searches, multiple source comprehension, digital fluency, and cognitive load. At the end of the literature review, we consolidate our hypotheses. Following that, we describe our Methodology, including our conceptual framework, data collection, and empirical methods, and then present our Results. We conclude with a discussion and conclusions.

1.1 Coursework completion, synthesising from multiple sources and the internet

It remains common in social science undergraduate assessments to set coursework with a maximum word count (say 3,000). Such coursework usually asks students to investigate and synthesise a response to a chosen topic to ensure learning at the higher ends of Bloom’s Taxonomy. Students will then utilise the internet to compile their answers.

The literature regarding the impact of ICT changes on how students tackle their assessments, and the output quality is in its infancy. It is, once again, playing catch-up since the development of AI tools (Moustaghfir & Brigui, 2024). Social media and ICT have become integral components of our daily existence, and mobile technologies and social networking platforms offer various advantages to how students work, live, and play, with access to real-time data (Gezgin & Cakir, 2016). Such technologies affect learners and enable people to connect, communicate, and interact with others (see Horovitz, 2012, Green, 2020 or for a thorough study of ICT issues across the globe see Fraillon, 2018; Fraillon et al., 2014 and 2019).

Reports from over a decade ago, when the internet and social media apps were not yet fully established, suggest some optimism about the benefits of pedagogy and research rigour (see Wang & Artero, 2005). This optimism, however, has been tempered by findings that technology is being used in limited ways, with worries emerging amongst module leaders about the utilisation of ICT. Additional concerns have centred around students lacking the necessary skills to maximise the internet’s potential as a research and deep learning tool (Mason, 2014). This suggests that students may lack the ability to utilise ICT to locate, evaluate, and use the appropriate data in their journey to create, make sense of, and discover impactful information (see Janes, 2007; Jenson, 2004; Jones et al., 2008; Kiili et al., 2008 for a review of these issues). For example, Jones et al. (2008) argued that the rise of ICT may lead to ‘surface searching’.

What is certain is that the rise of digital technology has led to more immediate access to multiple information sources, which can be both empowering and bewildering to students in equal measure. For example, in a recent review of how people learn and synthesise from such a vast array of quickly accessed information, Goldman et al. (2018) state:

The twenty-first-century society relies on digital technology for information relevant to all spheres of life. Yet, most citizens lack the knowledge, skills, and practices needed to meet the challenges posed by the ubiquitous array of information and technologies they have at their fingertips. (p. 86)

Corroborating this, back in 2008, Jones et al. found that all surveyed students reported using the internet to search for information to complete their assessments and that 95% used either Google or Yahoo, but only 48% used online encyclopaedias. Similarly, studies consistently reveal that students struggle to critically assess online sources, leading to the consumption and dissemination of misinformation. For instance, Wineburg and McGrew (2020) found that more than 90% of students could not differentiate between credible news articles and false information on social media platforms.

1.2 Information Problem Solving (IPS) models and related perspectives

One of the theoretical starting points in this area was developed by Kuhlthau (1988, 1991, 2004; Kuhlthau, Heinstrom, and Todd, 2008) in the late 1980s and early 1990s under the nomenclature IPS: Information Problem-Solving. This approach examines student behaviours and skills brought to information search, evaluation, and use. Kuhlthau's IPS method describes six steps to undertaking and completing a research task: initiation, topic selection, exploration, formulation, collection, and search closure.

Kuhlthau’s approach represents students beginning their tasks in an unfocused fashion, which eventually becomes more focused as relevant information is sorted and evaluated. Stripling and Pitts (1988) developed this IPS approach into an information search model focusing more on completing a research paper. The Stripling and Pitts model introduced ten stages, with many mirroring those described by Kuhlthau (1988, 1991, 2004): choosing a topic; obtaining an overview; narrowing the subject under investigation; developing the statement; creating the questions; planning research; finding information; evaluating evidence; drawing conclusions; and then writing the final script. These ten stages offer a more detailed view of the steps introduced by Kuhlthau (1988).

A third similar approach is the widely utilised Eisenberg and Berkowitz (1990) six-stage IPS model, now commonly called the Big6 model. The model suggests students use a systematic approach to solve information-gathering problems. As the name suggests, it has six components: task definition, information-seeking strategies, location and access, use of information, synthesis, and evaluation. The Big6 model, developed by Eisenberg and Berkowitz (1990), distinguishes itself by offering a systematic, step-by-step approach to information-gathering’.

Unlike Kuhlthau’s model, which focuses on the cognitive and emotional journey of students, the Big6 model provides clear, actionable steps, such as ‘task definition’ and ‘location and access’. This makes it more practical and easier for students to follow. While models like Kuhlthau’s and Stripling and Pitts emphasize exploration and writing, the Big6 places greater importance on strategic problem-solving, with clear guidance on synthesis and evaluation, crucial for handling multiple sources and critically assessing information.

The rapid development of ICT has led to a proliferation of immediately accessible sources, which has led to the further growth of the Big6 IPS model, taking into account how students’ piece together information using the World Wide Web. For example, Brand-Gruwel et al. (2009) have developed the IPS model into the IPS-I model (with the ‘I’ being ‘internet’—see Goldman et al. (2018) for a brief review). This approach considers the increased complexity of synthesising the seemingly endless amounts of information in a world with no gatekeepers. Gatekeepers in the past being librarians or even a physical publisher giving it a ‘stamp’ of validity to guide search (see Pickard et al., 2014)). Goldman et al. (2018) state that the Brand-Gruwel et al. (2009) model considers the:

...interacting processes, knowledge, and dispositions that are called on when learning and solving problems in multiple source situations, including identifying information needs, locating information sources, evaluating these sources (e.g., in terms of quality, trustworthiness, etc.), extracting, comprehending, and organising the information from each source, and then synthesising across sources to address the task...(p. 88)

The IPS-I model fosters critical thinking by requiring students to evaluate source credibility, synthesize diverse information, and refine their searches. It enhances adaptability, as students must navigate multiple sources, adjust strategies, and integrate findings effectively. This structured yet flexible approach helps them develop more nuanced research skills in an era of abundant but unfiltered information.

The Brand-Gruwel et al. (2009) IPS-I model, examines how students ‘process information’ but also assesses students ‘evaluation skills’. Brand-Gruwel et al. (2009) argue that students should ‘reach a deep understanding of the information’ within the processing information stage. Of concern is that past research indicates that teenagers hardly evaluate results, information, or sources (see also Walraven et al., 2008a, 2008b; Pickard et al., 2014), and that this may have become more acute in the internet age. Indeed, more recently Wineburg and McGrew (2020) found that more than 90% of students could not differentiate between credible news articles and false information on social media platforms. Failing to evaluate sources weakens students’ research quality, leading to misinformation and poor academic arguments. This issue may stem from information overload, reliance on convenience over credibility, and insufficient digital literacy, making it more difficult to navigate complex academic discourse.

1.3 Information literacy, digital fluency, and cognitive load

Interlinking with the IPS models is the literature on information literacy and new information literacy research, which sheds light on students' process and evaluation skills (see Miller & Bartlett, 2012; Kiili et al., 2008; Williams & Rowlands, 2007). Pickard et al. (2014) focus on the requirement in the IPS model of individuals being able to ‘evaluate and use information’ (p. 5, quoting the International Encyclopaedia of Information and Library Science). Pickard et al. (2014) state that there is a common theme established in the literature that:

There is a tendency for young people not to evaluate the information with which they come into contact rigorously. Although many information literacy models stress the need to take a critical approach, the reality of behaviour is often very different. (p. 3)

So, students' lack of evaluation may persist, highlighting a gap between theory and practice.

Further, the form in which the information is presented may well have changed radically, but the incentive for students to cut corners has always been apparent. Notably, the speed of access to an overwhelming amount of information for today's students may well lead to information overload (Hoq, 2016; Simmel, 1950; Tidline, 1999). In addition, information-action latency may push students into less, not more, critical investigation. Cognitive overload is a prevalent issue in digital environments, where students are often presented with more data than they can effectively process. Research by Paas and van Merriënboer (2020) demonstrated that students could only retain and process a limited amount of information, underscoring the need for tools to alleviate the cognitive load. As such, information-action latency may then drive students to prioritise speed over evaluation, and this could be amplified by todays digital environment.

Educational technologies can implement cognitive load reduction techniques, such as content summarisation tools and curation algorithms, that assist students in focusing on essential information without being overwhelmed by irrelevant data. Adaptive learning technologies, which adjust the difficulty and amount of information presented to students based on their progress, effectively ensure that learners remain focused on key content (Plass et al., 2021). This approach not only enhances comprehension but also helps students with the aforementioned information overload, offering a hopeful outlook for the future of digital learning.

The literature has continued to branch off into a variety of interesting approaches. Barzilai and Zohar (2012) developed an approach focused on understanding the challenges and strategies associated with effectively evaluating and integrating information from online sources. Their emphasis is on how students assess the credibility and reliability of online data. Another approach, introduced in the work of Wineburg (1991) and developed recently by Bråten and Strømsø (2018), examined the notion of ‘sourcing’. This essentially refers to critically examining and evaluating the origin and context of historical documents or sources. The sourcing approach believes that students undertaking an assessment analyse the sources’ authorship, purpose, and reliability to determine their credibility and potential biases.

In an era of ‘information glut’ (as Hoq (2016) refers to information overload), students must become digitally literate and fluent. In line with this notion of information glut, the new literacies literature emphasises the ability to evaluate the vast amounts of information on the web. Leu et al. (2015) regard new literacy as a dynamic process that evolves rapidly to account for new applications, ways of accessing data, and other newly emerging technologies. They believe that new technologies and the evolution of instant messaging, social media, blogs, and wikis amplify the need to re-evaluate information and normalise new social practices.

Students need to effectively and confidently use digital tools, technologies, and resources to complete academic tasks (see Jenkins, 2009; Erstad et al., 2016). However, in the context of our paper, digital fluency goes beyond basic digital fluency skills to emphasise the application of technology in the academic context (Martin, 2006). It involves the capacity to navigate and search for relevant information, evaluate the credibility and reliability of online sources, and integrate digital resources into the research process (Erstad et al., 2016). Poorer learners will engage in superficial reading and will struggle to extract relevant information, relying instead on memorisation rather than comprehension. Factors such as domain knowledge and metacognitive skills will influence learners’ performance.

Studies suggest that many students do not conduct effective online research and rely on simplistic queries, often selecting the top results without assessing the credibility of the sources. For instance, Druin et al. (2021) found that students usually rely on the first page of search results. Nagel et al. (2020) reported that 48% of students were unaware that search engines prioritise results based on algorithms that may not always reflect credibility.

Solutions have been forthcoming. Educators have been encouraged to integrate structured guidance on evaluating sources, recognising bias, and identifying credible information into their teaching. This can be achieved by embedding information literacy instruction directly into curricula or using gamified learning environments that engage students in critically evaluating digital information (Balalle, 2024). Scaffolded search techniques can help students refine their queries and critically assess different perspectives. Integrating scaffolded search systems within learning platforms can improve students' online research skills, enabling them to access and use diverse and reliable sources (Reinhold et al., 2024).

Further, collaborative learning platforms can encourage the development of critical analysis skills and foster communication and reasoning. As students must provide evidence for their claims, enhancing the depth of engagement with digital content. Platforms, like shared digital workspaces, forums, or wikis, allow students to interact, critique, and co-construct knowledge. However, recent research suggests that large language models (LLMs) reduce cognitive load but may limit the depth of content engagement; LLM users reported a lower cognitive load than traditional search engine users (Stadler et al., 2024).

Artificial intelligence (AI) for personalisation has vast implications for helping students navigate digital information. AI-driven systems can offer personalised learning experiences, suggest relevant content, and provide real-time feedback. AI-powered intelligent tutoring systems can offer just-in-time hints or direct students to trustworthy sources when they struggle to find reliable information. Similarly, AI can tailor learning pathways based on individual progress, ensuring that students encounter digital content in manageable chunks, thereby reducing cognitive overload (Roll & Wylie, 2016). This makes navigating complex digital environments more efficient and effective for students.

Finally, there is a developing conversation relating to digital privacy and ethical uses of ICTs. In the context of educational technology, it is essential to consider how students access and evaluate information and how their data is collected and used. Ethical considerations around data privacy, algorithmic bias, and the transparency of AI systems are increasingly important in the design and implementation of educational technologies (van Dijck, 2014). To generalise, students may choose to evaluate and move on, with disregard to the depth of their analysis and it is impossible for students to assess what they do not know (see Pickard et al., 2014; Kiili et al., 2008).

In order to be consistent with the notion of digital fluency, we reiterate that ‘digital fluency’ encompasses both basic digital fluency competencies and the advanced capability to integrate, analyse, and critically evaluate information. This consistency informs our analysis of information-seeking behaviours. These models provide a structured for analysing contemporary digital search behaviours. Although we have introduced these multiple approaches, our study leans somewhat on the IPS/IPSi/ Big6 approach to structure our analysis (see Eisenberg and Berkowitz or 1990 Brand-Gruwel et al., 2009). This choice reflects these models well-defined stages. Within these frameworks, we also make us of Nicholas et al’s (2008) and Jones et al’s (2008) notion of ‘surface searching’ and ‘skittering’.

2 Methodology

This section explains how we operationalise our framework, detailing how our data collection and empirical methods are designed to help us assess whether students transition from surface-level searching to deeper critical engagement. Our study is based on assessment outcomes in an Applied Economics module in a UK Higher Education (HE) Institution during the 2017–2020 academic years. Bridging the gap between the theoretical constructs of economics and tangible implications in the real world, the module is designed to enable students to grasp how abstract economic principles can be applied to yield practical policy solutions. The curriculum is based on a thematic approach, encompassing a wide range of topics including conflict, crime, climate crisis, and religion. It aims to provide students with a rich array of economic perspectives, highlighting the varied applications of economics in understanding and addressing societal issues. Our research process is outlined in Fig. 1, illustrating the sequential flow in our study design. We explain each element of the research process in turn.

Fig. 1
figure 1

Research process flow chart

Full size image

2.1 Research model and study design

This section details the research model used to examine how educational interventions influence undergraduate students’ information-seeking behaviours in our Applied Economics module. We primarily chose a cohort-based design to isolate the effects of iterative interventions across distinct groups, each introduced to strategies of increasing intensity. By comparing cohorts, rather than tracking the same students longitudinally, we reduce carryover biases and measure how each tailored intervention affects the group receiving it. As stated, our study is guided by the Information Problem Solving (IPS) models (Eisenberg & Berkowitz, 1990; Kuhlthau, 2004) and the concepts, introduced earlier of ‘surface searching’ and ‘skittering’ (Nicholas et al., 2008). Each intervention targets a different stage (for example, identification, evaluation, synthesis) in the information-seeking process. In summary, a series of educational interventions were implemented over a three-year period, each designed to address specific deficiencies identified in students’ research practices. The interventions evolved based on formative feedback and findings from the previous year, allowing for iterative refinement and targeted support.

A1. Study Design: As stated, our study employs a quantitative, cohort-based design. It observes three distinct groups of undergraduate students enrolled in an Applied Economics module over consecutive academic years (2017–2020). Each cohort receives a progressively intensified set of educational interventions designed to address information-seeking behaviours and research practices. The ratcheting up of interventions allows us to evaluate how increasingly targeted strategies impact different cohorts of students with varying baseline skills and behaviours.

This approach also enables a comparison across cohorts rather than tracking individual student behaviours over time. By varying the intensity and scope of the interventions each year, the study assesses the effectiveness of these changes in improving information-seeking skills and research output. The cohort-based approach is ideal for examining the direct impact of interventions on each distinct group, providing stronger insights into the relationship between specific educational strategies and student outcomes.

A.2 Research Hypotheses: Emerging from the literature we have developed our main research question and hypotheses. Our research question is: How do educational interventions influence undergraduate students' information-seeking behaviour and the quality of their research outputs in an internet-mediated coursework environment?

From this, we develop four specific hypotheses are formulated:

  • H1: Student successful completion rates will significantly decrease as tasks become more advanced, reflecting deficiencies in advanced research methodology skills.

  • H2: An increased quantity of bibliographic references is positively associated with the likelihood of attaining a ‘good honours’ (Upper Second or higher) outcome.

  • H3: The use of higher-quality scholarly references significantly increases the chances of attaining a ‘good honours’ outcome.

  • H4: Enhanced student engagement with the module’s eLearning materials is associated with a higher likelihood of adopting superior literature review methods.

These hypotheses are designed to test specific aspects of students’ information-seeking behaviour, the effectiveness of the educational interventions, and the relationship between research practices and academic performance.

A3. Conceptual Framework: The conceptual framework integrates IPS/Big6 models and the notion of digital fluency to understand how students interact with information in the digital age (Leu et al., 2015). It considers cognitive, affective, and behavioural factors influencing information-seeking, such as students’ prior knowledge, motivation, self-efficacy, and engagement levels. This allows us to examine students’ ability to navigate digital resources, critically evaluate sources, and synthesise information from multiple texts.

Educational interventions aimed at enhancing these skills are expected to improve the quality of students’ research outputs. Our framework also acknowledges the role of external factors, including curriculum design, teaching strategies, and technological tools, in shaping students’ information literacy and research competencies.

2.2 Interventions design

During the three years of our study, a series of educational interventions were implemented, each designed to address specific deficiencies identified in students’ research practices. The interventions evolved based on formative feedback and findings from the previous year, allowing for iterative refinement and targeted changes.

B1. Year 1 Knowledge: In the first year, the intervention focuses on assessing and enhancing students’ competencies in foundational research skills. A workshop was developed to address three key areas:

  • Data Identification: Students were tasked with locating specific datasets related to public opinion on the death penalty in the UK. They were introduced to reputable online databases such as the British Social Attitudes Survey and the UK Data Service. This component assessed their ability to formulate effective search queries, understand database interfaces, and apply filters to retrieve relevant data.

  • Analysis Proficiency: Students were asked to analyse UK homicide trends over the past decade, requiring them to interpret statistical data from sources like the Office for National Statistics. They were encouraged to identify patterns, and discuss potential factors influencing these trends, such as changes in legislation or societal factors. This activity evaluated their critical thinking, data interpretation skills, and ability to contextualize quantitative information.

  • Empirical Source Discovery: Students were challenged to find a key empirical analysis on the impact of the death penalty from scholarly journals. They were guided to use academic databases like JSTOR, Scopus, and Google Scholar. This task tested their ability to differentiate between primary and secondary sources, assess the credibility of publications, and recognise peer-reviewed articles.

Formative assessments included quizzes, peer discussions, and instructor feedback. Workshops aimed to identify existing gaps in research skills, promote self-awareness among students regarding their competencies, and establish a foundation for subsequent interventions.

We propose that students would successfully navigate the first two stages, reflecting their adeptness in leveraging online search tools to gather crucial information. However, assuming there are existing limitations in their research methodology skills, we anticipate a steep decline in successful completion rates between the second and third stages. Students are likely to resort to a quick internet search to locate a widely referenced resource, which often turns out to be a non-scholarly, secondary source. This informs our first hypothesis, H1: Student successful completion rates will significantly decrease as tasks become more advanced, reflecting deficiencies in advanced research methodology skills.

B2. Year 2 Review: Building on the insights from Year 1, the second-year intervention aims to deepen students’ understanding of literature review processes. To test H2 and H3, the instructional approach encompasses several components:

  • Developing Research Questions: Students should learn how to formulate clear, focused, and researchable questions. Workshops emphasised the importance of specificity, feasibility, and relevance. Techniques such as the PICo (Population, Interest, Context) framework were introduced to structure questions effectively.

  • Systematic Literature Searches: Instructions focused on advanced search strategies, including Boolean operators, wildcards, and truncation. Students were trained to use subject-specific databases, manage search results, and employ reference management software like EndNote or Zotero.

  • Critical Evaluation of Sources: Using tools like the CRAAP test (Blakeslee, 2004) and the PROMPT criteria (Provenance, Relevance, Objectivity, Method, Presentation, Timeliness), students assessed the quality and reliability of sources. Emphasis is placed on identifying biases, recognising methodological strengths and weaknesses, and evaluating the significance of findings within the broader literature.

  • Synthesis and Writing Skills: Students were guided on synthesising information from multiple sources and identifying themes, gaps, and contradictions in the literature. Workshops covered academic writing conventions, paraphrasing, avoiding plagiarism, and adhering to referencing styles.

Assessments included submitting a literature review draft, which received formative feedback, and the final literature review as part of their coursework. The intervention sought to enhance students’ ability to conduct rigorous, scholarly research and produce well-structured literature reviews.

B3. Year 3 Critique: The final year’s intervention, to test H4, integrated blended learning and decolonisation strategies to promote critical thinking and deeper engagement with research materials. This included:

  • Diversifying Content through Decolonisation: The curriculum was expanded to include economic case studies from underrepresented countries in Africa, Asia, and Latin America. This exposed students to a broader range of perspectives, challenging Eurocentric narratives and encouraging students to consider global contexts.

  • Blended Learning with eLearning Modules: Customised eLearning materials were developed, offering interactive content on topics like the economic impacts of climate change in the Global South or the role of religion in emerging markets. This non-linear design allowed students to navigate issues according to their interests, fostering autonomy and self-directed learning (Knowles 1975).

  • Enhancing Digital fluency: Workshops and online tutorials on advanced digital skills, such as using data visualisation tools (for example Tableau, Power BI), conducting meta-analyses, and utilising open-access repositories. This aimed to improve students’ ability to handle complex information and leverage technology effectively in their research.

  • Encouraging Reflective Practice: Students maintained research logs or journals to reflect on their information-seeking processes, challenges encountered, and strategies employed. Reflective prompts were provided to facilitate critical self-assessment and continuous improvement.

It is less feasible to rely on superficial sources when studying underrepresented countries, due to the scarcity of generalist websites. As a result, students were ‘nudged’ towards engaging with primary sources, academic journals, and reports from international organisations. This promotes more rigorous research practices and fosters critical engagement and diverse perspectives.

To summarise, each intervention specifically contributed to outcomes by targeting a phase of the IPS/Big6 model: Year 1 (Task Definition, Information-Seeking Strategies), Year 2 (Use of Information, Synthesis), and Year 3 (Evaluation in a global context). This staged approach allowed incremental improvements culminating in higher-level critical thinking and deeper engagement with scholarly materials.

2.3 Data collection

In this section we will describe the participants involved in the study and the instruments and procedures used to collect all our data. With data gathered over three academic years, our aim was to ensure that consistency is observed throughout our study:

C1. Participants: The study involved undergraduate students enrolled in the Applied Economics module over three academic years, totalling 249 participants:

  • Year 1 (2017–2018): 80 students (45 males, 35 females; 60 domestic, 20 international).

  • Year 2 (2018–2019): 80 students (48 males, 32 females; 58 domestic, 22 international).

  • Year 3 (2019–2020): 89 students (50 males, 39 females; 62 domestic, 27 international).

Participants came from diverse socio-economic backgrounds, ethnicities, and academic abilities, reflecting the institution’s commitment to inclusivity and diversity. The international students represented various countries, adding a multicultural dimension to the cohort.

C2. Data Collection Instruments: The data collection process was designed to capture alternative means to measure the impact of educational interventions on students’ research practices. Quantitative data were gathered through assignments, other types of assessments, and academic records, at multiple points throughout each academic year, with key variables aligned to the focus of each intervention.

In Year 1, the intervention targeted foundational data identification and analysis skills. The following variables were collected:

  • Data Identification Score: This measured students’ ability to locate relevant datasets related to the course topic.

  • Analysis Proficiency Score: This assessed students’ ability to analyse and interpret statistical data.

  • Empirical Source Discovery Score: This evaluated students’ ability to find and use primary empirical sources in their research.

In Year 2, the intervention focused on the literature review practices, particularly on the depth and scholarly quality of sources. First, to better understand the nature and credibility of the sources used in students’ bibliographies, each source is classified according to three categories:

  • Government: These included primary data, reports, or publications from government agencies, that offered reliable and comprehensive insights into public policy and socioeconomic conditions.

  • Peer-reviewed: Critically evaluated by field experts, these sources present original research or theoretical discussions, providing a credible, authoritative, and detailed understanding of specific topics.

  • Secondary: These include news articles, blog posts, and textbooks that interpret or summarise primary data. They offer helpful commentary but may lack the depth and rigour of primary or peer-reviewed sources.

Second, the quality of the bibliographies was categorised across four classifications:

  • Sparse & Non-Scholarly: This category had a below-average number of references (i.e., 13 or lower), with at least half of these references of a secondary nature.

  • Sparse & Scholarly: This category included bibliographies with a below-average number of references (that is, 13 or lower), yet less than half of these references are secondary.

  • Abundant & Non-Scholarly: Bibliographies in this category exceeded the average number of references (that is, more than 13), with at least half of these references of a secondary nature.

  • Abundant & Scholarly: This category comprised bibliographies with an above-average number of references (more than 13) but with less than half of these references of a secondary nature.

In Year 3, the intervention further refined the focus on source quality and critical engagement with the literature. The variables collected were:

  • Research Quality Score: This was measured on a 5-point scale, assessing the overall quality of students’ research based on standardised rubrics.

  • UK/US Adopter Variable: This variable identified whether students focused their research on the UK or US. These are both countries with readily accessible but potentially lower-quality sources. This provides a possible proxy for students choosing easier but less rigorous research pathways.

In addition, academic records from previous coursework outcomes were gathered to serve as control variables, allowing us to account for differences in students’ prior academic performance. These data provided a baseline for evaluating the interventions’ impact on research behaviour.

C3. Ethical Considerations: The study adhered to the ethical principles of fairness, transparency, and confidentiality throughout the research process. All students participated in the educational interventions as part of their normal curriculum, ensuring that no group received differential treatment or special advantages. This approach maintained the integrity of the learning experience and avoided introducing any form of bias into the study.

Confidentiality was prioritised by anonymising student data at the point of collection. Each participant was assigned a unique identifier, and personal details were removed before the analysis to protect identities. Data was securely stored to ensure that only authorised research team members had access. This process ensured that students’ privacy was respected throughout the study.

2.4 Empirical methods

This section outlines the statistical methods used to analyse the data collected across the three academic years. Given the quantitative nature of the study, various statistical techniques were employed to test the hypotheses and measure the impact of educational interventions on student information-seeking behaviour and academic performance.

D1. Data Analysis Techniques: To test the study’s hypotheses and evaluate the impact of the interventions, two key statistical techniques were adopted:

  • Binary Probit Regression: This is used to model the relationship between several independent variables (for example, bibliography quantity, bibliography quality) and the binary outcome of achieving a ‘good honours’ grade (≥ 60%). The binary nature of the dependent variable makes probit appropriate for estimating the likelihood of students attaining a certain academic threshold based on our predictor variables. Independent variables, such as the number of references used and their scholarly quality, were included to understand their impact on academic outcomes, while controlling for previous academic performance to isolate the intervention effects.

  • Ordered Probit Regression: An ordered probit regression was employed to analyse Year 3 data, where the dependent variable (research quality score) is measured on a 5-point scale. This model is suited for ordinal dependent variables, allowing the analysis of how factors such as the scholarly quality of sources and the choice of research focus (for example UK/US Adopter variable) influenced the probability of achieving higher research quality scores. The ordered probit approach was chosen because it assumes a latent continuous variable underlying the ordinal responses. This makes it ideal for interpreting the gradation in research quality across different students.

Both models were tested for key assumptions, including the absence of multicollinearity, and verified using Variance Inflation Factors (VIFs). The proportional odds assumption, essential for ordered probit regression, was confirmed to hold using appropriate diagnostic tests, ensuring that the model’s use was valid.

D2. Validity and Reliability: Ensuring the validity and reliability of the instruments and results is essential. Our research used a 5-point scale to assess students’ research quality, with the scale specifically developed for this study to align with our aims. The 5-point scale evaluates the following aspects of students’ work:

  • Source Quality: The scholarly nature of the sources, ranging from superficial web-based references to peer-reviewed journal articles and primary empirical sources.

  • Depth of Analysis: The extent to which students engaged critically with their sources, demonstrating deep understanding and interpretation of the material.

  • Synthesis of Information: How well students integrated multiple sources to build a cohesive argument or narrative, avoiding simple summarisation of individual sources.

  • Originality of Thought: The level of independent thinking and creativity in the student’s approach to the topic, including developing new insights or perspectives.

  • Clarity and Structure: The overall organisation and coherence of the research, ensuring that arguments are clearly articulated and logically presented.

These criteria are designed to measure the key competencies targeted by the educational intervention. While they were all developed specifically for this study, the criteria are based on widely accepted academic standards for evaluating research quality. Thus, aligning with broader educational practices. Expert reviewers, including academic staff, validated the instrument to ensure that it effectively measured the constructs it aimed to assess. This includes information-seeking behaviour and critical engagement with sources.

The reliability of this scale is confirmed through Cronbach’s Alpha, with values exceeding 0.80, indicating a high level of internal consistency. This demonstrates that the scale is a reliable tool for consistently measuring different dimensions of student performance. Inter-rater reliability is ensured by training markers and conducting calibration sessions before the assessments. Cohen’s Kappa, calculated at 0.82, indicates substantial agreement among graders, guaranteeing consistency across the cohorts. This process mitigates potential grading discrepancies, ensuring that student work is evaluated uniformly and contributing to the reliability of the data.

Model fit is also assessed to ensure that the chosen statistical methods are appropriate for the data. The Hosmer–Lemeshow test confirms the goodness of fit for the binary model, while the log-likelihood function and pseudo-R2 measures indicate a firm fit in the ordered probit regression model. These fit measures confirm that the statistical models accurately reflect the relationships between the variables, strengthening the overall validity of the findings.

D3. Controls and Limitations: The statistical models in this study account for three key explanatory variables: socio-economic characteristics; engagement levels; and innate ability, to ensure the educational interventions’ effects are isolated.

  • Socio-economic characteristics, such as gender and domestic or international status, are controlled for as they can influence academic performance. Gender may affect learning approaches, while international students may face challenges like language barriers, impacting their research practices. Including these variables helps ensure the analysis distinguishes the effects of the interventions from those due to demographic factors.

  • Engagement levels are measured through attendance, serving as a proxy for involvement. While attendance indicates participation, it may not fully capture a student’s engagement, including online activity or independent study habits. Despite this, attendance remains a valuable indicator for assessing how student involvement affects their adoption of high-quality research practices.

  • Innate ability is controlled using previous assessment scores, which indicate students' baseline academic performance. This helps to ensure that any improvements in research quality can be attributed to the interventions rather than pre-existing differences in academic aptitude.

However, several limitations exist. Engagement is only measured through attendance, potentially overlooking other forms of involvement. Socio-economic factors beyond gender and international status, such as financial background, were not considered, which may also influence academic outcomes. Additionally, prior academic scores may not perfectly reflect innate ability, as various external factors can impact performance. Finally, the study’s focus on a single module limits the generalisability of the findings to other contexts. While these limitations should be noted, the inclusion of these variables enhances the robustness of the analysis, allowing for clearer conclusions about the interventions’ impact.

3 Results

Table 1 presents the results from a crime-focused workshop. Our findings confirm that students demonstrate high competency in identifying specific information. Yet, when the focus shifted to evaluating the quality of research sources, we observed a substantial decline in student performance. This pattern underscores the challenge students face in comprehending the proper execution of literature review methods, showing a preference for familiar internet search strategies. We accept hypothesis H1.

Table 1 Observing weak research methodology
Full size table

The finding indicates that students struggled when transitioning from Analysis Proficiency to Empirical Source Discovery. This aligns with Entwistle's (1988) assertion that students are prone to rely on less effective, surface-level strategies without robust intrinsic motivation and advanced methodological skills. This drop-off can be attributed to students' challenges when required to engage in deeper, more independent research processes. Regarding Kuhlthau et al.'s (2008) IPS model, Eisenberg and Berkowitz's (1990) Big6 model and Brand-Gruwel et al. (2009) IPS-I model, we find that students encounter difficulties with the exploration and formulation stages of the IPS model, however defined. Between these stages, students may experience uncertainty and doubt in transitioning from gathering initial information to recognising and selecting high-quality primary research sources over secondary ones. As Pickard et al. (2014) state, it is a critical step of the ISP models that students should be able to evaluate and use information. Indeed, Lue et al. (2015) argue that emerging technologies heighten the necessity to reassess and utilise information. In addition, as Erstad et al. (2016) contend, weaker students who engage in superficial reading have difficulty extracting relevant information.

Table 2 outlines the results of our probit analysis regarding Year 2, adopting literature review methods. It demonstrates that the innate ability check, represented by the ‘knowledge’ variable, has a minor yet significant impact on the likelihood of obtaining a report grade above 60%. More notably, the categories delineating the quality and quantity of research sources provide intriguing insights. When the ‘Abundant & Scholarly’ category is set as the reference group (indicating repeated use of high-quality sources), all alternative categories are associated with a lower probability of attaining at least 60%. We, therefore, accept both hypotheses H2 and H3. The coefficient for the ‘Sparse & Scholarly’ category shows that the likelihood of attaining a ‘good honours’ grade decreases by approximately 65.3% compared to those using ‘Abundant & Scholarly’ sources, thus reinforcing the importance of both the quality and quantity of references.

Table 2 Probit analysis of ‘good honours’ outcomes
Full size table

Interestingly, the coefficient for the ‘Abundant & Non-Scholarly’ category is not statistically significant. This suggests that students who compile an extensive bibliography of varying quality can offset the adverse effects of sub-optimal research techniques. Unfortunately, traditional efforts to enhance literature review skills have not curbed this practice.

Our findings can also broadly be contextualised within the discourse on information literacy and research skills as discussed above in the work of Kuhlthau et al. (2008), Stripling and Pitts (1988), Eisenberg and Berkowitz (1990), Brand-Gruwel et al. (2009), and Jones et al. (2008). Kuhlthau’s IPS model emphasises the iterative nature of information seeking, where cognitive and affective aspects are central. The findings align with the IPS model in that students using ‘Abundant & Scholarly’ sources (high-quality, repeated use) likely engage deeply with the information search process, reflecting higher cognitive engagement and affective commitment. As such, the results corroborate Kuhlthau’s emphasis on quality and thoughtful engagement in information seeking. However, the persistence of ‘skimming’ strategies indicates that despite the established benefits of a thorough information search process, some students still resort to superficial methods to achieve satisfactory academic outcomes. This partially undermines the model's prescriptive efficacy.

(1) Analysis of ‘good honours’ outcomes; (2) *, ** and *** denote significance at the 1%, 5% and 10% levels of significance, respectively; (3) Alternative use of a logit methodology did not alter the nature of our findings.

Stripling and Pitts advocate for teaching research skills as integral to critical thinking. Accepting our hypotheses H2 and H3 indicates that structured, scholarly bibliographies contribute significantly to academic success, aligning with Stripling and Pitts’ model regarding strategic and thoughtful research practices. However, the fact that ‘Abundant & Non-Scholarly’ sources can still lead to decent grades suggests that while critical thinking and strategic research are important, practical implementation can sometimes deviate. This indicates the allowance for successful shortcuts, which remains worrying. In addition, the ability of students to attain high grades through non-scholarly sources may indicate a gap in the application of the Big6 skills, specifically in evaluating and synthesising high-quality information. Further suggesting a need for more robust instruction in these areas (see Eisenberg and Berkowitz (1990).

Our findings that high-quality sources correlate with better academic outcomes underscore the importance of proficient information problem-solving skills. The noted efficacy of ‘skimming’ with non-scholarly sources points to the potential inadequacy of current instructional practices in emphasising thorough information processing. Meaning that students may sometimes bypass deeper problem-solving stages while still meeting academic criteria (Brand-Gruwel et al. 2009; Jones et al., 2008). In line with Pickard et al. (2014), this result further indicates that students will often rely on less rigorous methods despite awareness of evaluation principles. The acceptance of ‘Abundant & Non-Scholarly’ sources suggests that even with training, students sometimes choose quantity over quality, reflecting the persistent struggle highlighted in Pickard et al.’s work. Despite this, our finding that ‘Abundant & Scholarly’ sources correlate with better grades supports the argument that deep epistemic engagement leads to higher academic achievement (Barzilai & Zohar, 2012).

These results also support the work of Hoq (2016) regarding information overload with the acceptance of both hypotheses H2 and H3, suggesting that students who manage to filter and prioritise high-quality information achieve better outcomes. However, the efficacy of ‘skimming’ strategies (abundant non-scholarly sources) indicates that students might compensate for information overload by opting for quantity. This approach may mitigate some negative consequences of overload but does not necessarily lead to optimal learning or critical thinking.

Our ordered probit results, detailed in Table 3, reveal valuable insights regarding year 3’s focus on cultivating critical thinking. Although we cannot discern significant differences across our socio-economic control variables, knowledge and non-attendance are notable predictors of literature review methods. More importantly, our hypothesis (H4) is validated, which tested whether increased student engagement in the module's eLearning materials increases the chances of attaining a ‘good honours’ degree. Initially intended to demonstrate the simplicity of constructing a bibliography, lecture content tends to appeal to students prone to sub-optimal research practices. Conversely, those who engage with the eLearning materials and are encouraged to examine these issues through a decolonised lens successfully adopt superior research methodologies.

Table 3 Ordered probit analysis into research quality
Full size table

This result also supports the literature. For example, increased engagement with non-linear, decolonised eLearning materials leads to better research outcomes aligned with Kuhlthau et al’s (2008) IPS model’s focus on deep cognitive engagement. By decolonising the curriculum and encouraging independent exploration through eLearning materials, we find support for the idea that critical engagement and diversified perspectives enhance research quality (Stripling & Pitts, 1988). In addition, this result supports the notion that student engagement with eLearning materials enhances research quality. Our non-linear, decolonised approach indicates that students can better define their research tasks and employ diverse strategies to gather and synthesise information. This further aligns with Erstad et al.’s (2016) assertion that digital competence involves technical skills and the ability to critically engage with digital content, leading to more effective learning outcomes.

To summarise our key findings, Students perform well on basic tasks but struggle with advanced research techniques; effective research methods require both high-quality and sufficient references; and decolonised, interactive eLearning fosters critical engagement and enhances research quality. Table 4 visually encapsulates the key performance patterns, highlighting where each hypothesis was tested and confirmed.

Table 4. Summary of outcomes across three years
Full size table

4 Conclusions

Our analysis examined the evolving landscape of undergraduate students' information-seeking behaviour in the context of internet-mediated coursework research. We analysed three years of students’ research practices within an Applied Economics module at a UK Higher Education institution. This allowed us to examine the evolving dynamics of student internet utilisation to complete a coursework task. The module encompassed diverse topics such as conflict, crime, climate crisis, and religion. This allowed us to investigate the practical application of economic principles to real-world issues and our research to unfold in three distinct phases, each targeting specific aspects of students’ engagement with research, the internet, and its impact on learning outcomes.

In the first year, we focused on appraising research methodology, revealing a significant gap in students’ abilities to critically evaluate and select high-quality empirical sources. Despite data identification and analysis proficiency, students struggled in the crucial phase of practical source discovery. This finding underscored the prevalence of ‘skittering’ behaviour, where students rely on easily accessible but often non-scholarly sources. The second year introduced traditional literature review methods to rectify surface-level search tendencies. However, our results indicated that efforts to enhance literature review skills may not be sufficient to curb students' reliance on quantity over quality in their research sources.

In the final year, a refined approach was adopted, combining face-to-face teaching sessions with eLearning materials. The incorporation of a ‘decolonisation of the curriculum’ strategy aimed to nudge students toward exploring underrepresented countries, fostering critical thinking and rigorous research skills. Our ordered probit analysis revealed that students engaging with eLearning materials and adopting a decolonised perspective demonstrated a greater likelihood of embracing a high-quality literature review.

The findings of our longitudinal study contribute to the broader discourse on pedagogical strategies to enhance students’ critical thinking and research skills in the digital age. Key takeaways are the importance of addressing ‘skittering’ behaviours, blended learning approaches' value, and diverse perspectives' impact on research quality. Our findings also call for ongoing reflection on curriculum design, emphasising the need for dynamic and inclusive pedagogical practices that prepare students for the complexities of a globally interconnected academic landscape. We also confirm an intricate relationship between curriculum design, student engagement, and literature review development.

Our findings affirm the benefits of using high-quality scholarly sources and highlight an educational challenge: students can still achieve satisfactory academic outcomes with less rigorous methods, which may undermine efforts to enhance critical thinking and thorough research practices. This calls for a reinforced focus on teaching strategies that emphasise the use of high-quality sources and discourage superficial research habits, aligning with the frameworks and concerns outlined throughout our study.

In addition, our results highlight the importance of high-quality scholarly sources for academic success while pointing out the pragmatic strategies students use to navigate the digital information landscape. This duality suggests a need for educational frameworks that emphasise the value of high-quality sources and address the realities of information overload and the varied digital fluency skills students employ.

The creation of non-linear, decolonised eLearning resources emphasised the need for digital competence and adaptive reading strategies necessitated by digital environments. Educators are facilitating a learning process that enhances students’ research methodologies and academic outcomes by promoting critical engagement and providing diverse, stand-alone materials.

In summary, the implications of our study highlight the need for tools and strategies that enhance students' digital fluency and their ability to apply technology effectively in academic contexts (Erstad et al., 2016; Martin, 2006). Educational technology can be transformative by incorporating scaffolded search systems, adaptive learning technologies, and cognitive load reduction techniques. Scaffolded search systems, for example, can guide students through advanced search strategies, helping them refine queries and critically assess information sources. Adaptive technologies, such as AI-driven learning platforms, can personalise content and learning paths, enabling students to focus on high-quality, credible sources while reducing cognitive overload. In addition, tools that emphasise source evaluation, such as content curation algorithms and real-time feedback mechanisms, can foster more profound engagement with scholarly materials. These strategies collectively promote more rigorous academic inquiry, enabling students to navigate the complexities of digital information effectively.

Our findings call for ongoing reflection on curriculum design, emphasising the need for dynamic and inclusive pedagogical practices that prepare students for the complexities of a globally interconnected academic landscape. They also highlight an educational challenge: students can still achieve satisfactory academic outcomes with less rigorous methods, which may undermine efforts to enhance critical thinking and thorough research practices.