Online Assessment

Online Assessment

• 1.
  Designing Online Assessment Solutionsthat are rigorous, trusted, flexible and scalable Klaus-Dieter Rossade, José Janssen, Carlton Wood & George Ubachs EADTU-EUSummit, 11 May 2022
• 2.
  Online Assessment SpecialInterest Group • Establishment of SIG January 2020; First meeting March 2021; 5 meetings in 2021-2022 • 18 members institutions from 14 countries • 37 HE practitioners + 18 contributors of good practice • 15 Good Practice case studies • 4 Focus areas • Assessment Futures • Assessment Design • Trust, Privacy and Ethics • Operational processes, technology and support https://online-assessment.eadtu.eu
• 3.
  Assessment Future isalready present • Online Assessment is here to stay • Transition phase from exam hall to online home exams • Distance Teaching sector long history of home-based assessment • EADTU member institutions best placed to lead • Great variety of hybrid solutions to online assessment
• 4.
  Principles for futureassessment • Authentic, i.e. a move away from acquiring knowledge to acquiring transferable skills assessed in a more realistic way; • Accessible i.e. the aspiration to design assessment within an accessibility-first principle and delivered in multiple ways depending on the needs of future learners; • Appropriately automated which includes a mix of automated and human marking and feedback that benefits learners; • Continuous refers to the data and analytics to assess the effectiveness and impact of continuous assessment; and • Secure which should include elements of authoring detection and biometric authentication for identity and remote proctoring JISC (2020). The future of assessment: five principles, five targets for 2025. https://repository.jisc.ac.uk/7733/1/the-future-of-assessment-report.pdf
• 5.
  Features of futureassessment • Assessment remains “most significant prompt for learning” (Boud, 1995) • Authentic assessment that is grounded in realism, provides cognitive challenge and trains authentic evaluative judgement, (Villaroel et. al., 2018) i.e. ability to assess quality within any given discipline • “Encourage and support, not undermine the learning process” (Shute & Kim, 2013) • “stealth assessment” embedded in the learning process – “weaving assessment directly into the learning environments” and ”provide formative information whenever possible” (Shute & Kim, 2013) • Low stakes assessment points, allow frequent repeat, dynamic personalised progression, performance data to contribute to final course results
• 6.
  Assessment Design (Chapter1) • Number of assessment points • Stakes: formative or summative, weighting • Format: open/closed book exams, time limits, • Evidence based implementation with future in mind • Navigate quality standards, student needs, institutional and regulatory context • Balance of existing vs. new technologies • Choice and flexibility as part of the design • Authenticity and personalization by design • Take academic staff along the journey
• 7.
  Good Practice: Flexibilityand Choice Universidade Aberta, Portugal, offers flexibility through its virtual pedagogical model (VPM) by stipulating “a degree of choice” in assessment. Student can opt for continuous assessment via three submissions or a single endpoint assessment, typically a written exam. At TAMK, Finland, flexibility and student choice led to adding weekly assessment to the final exams on a online engineering course. These weekly assignment points were automated and students could retake them as often as they wanted at a time that suited them.
• 8.
  Trust, Privacy andEthics (Chapter 2) Multi-layered concept (Edwards et al., 2018) 1. technology (e.g., functions reliably and as expected; usability) 2. deployment of technology (e.g. transparency, teacher competence) 3. organisation deploying e-assessment (e.g. based on reputation) 4. privacy and personal data processing (e.g., data processed as agreed and intended; informed consent). 5. reliability/fairness process (identity and authorship verification)
• 9.
  Wholistic approaches toenhance trust (1) Build a narrative for stable, sustainable, robust and data-privacy respectful solutions Technology and its deployment chose technology for usability and ease of support. provide careful guidance and opportunities to try out the system Organisation balance academic standards with ethical consideration to student experience of ‘feeling overly scrutinized’ involve students as partners in decision making
• 10.
  Wholistic approaches toenhance trust (2) Build a narrative for stable, sustainable, robust and data-privacy respectful solutions Privacy and personal data processing be transparent about purpose of data processing in relation to each assessment ensure cybersecurity and GDPR compliance of tools used Reliability/fairness process verify identity and authorship build trust in assessor and the assessment of invigilation recordings provide clear protocols for fraud detection and follow up with students
• 11.
  Trust is atwo-way street “Academic integrity is a pedagogical issue and should not be limited to not getting caught during an exam. We need to be able to trust our students that they will not lie about their methods, falsify their results or plagiarise. These are fundamental values of academic work and need to be developed as such.” “We need to teach our students to be critical of the surveillance society in which we live, not force them to accept it.” SIG member Henrietta Carbonell, personal communication, 10-2-2022 “It’s nasty to be treated as a fraud suspect” “To feel you’re being watched is very unpleasant. I even didn’t dare look out of the window for fear of being considered a fraud” Student evaluation, in: Nielissen, 2021
• 12.
  Operational processes, technology andsupport (Chapter 3) • Text Challenges during the pandemic • How did institutions organise themselves to deliver the required operational processes? • Could face-to-face provision be adapted for on-line delivery? • To what extent was technology deployed to assist in the delivery of operational process? • How did Universities support staff and students whilst delivering the process?
• 13.
  Deployment of Technology– Themes • Range of solutions that were employed • Solutions developed in house versus those available commercially • Deployment of technology – proctoring and plagiarism tools • Changes that were introduced could increase concern/anxiety amongst students • Difficulties that were encountered with technology
• 14.
  Online Assessment beyondthe pandemic Meet the challenge of delivering “capable, competent, informed citizens adequate to the challenges of a twenty-first century lifetime” (Ashford, 2014) • Online assessment part of growing demand for shorter and flexible education (Certificates and Diplomas, Microcredentials), short learning programmes (SLP) and Continuing Education and Professional Development (CEPD) • SIG has captured members’ local expertise in how to design and plan institutional online assessment strategies (during pandemic and post-pandemic) • Deliverables: inventory of challenges, practices and new developments, examples of research and innovation in online assessment, identification of policies and frameworks that define the context of assessment for the institutions. • SIG will continue to provide the peer support to help address future challenges, contribute to further community building to develop better assessment without the need to re-invent the wheel every time.
• 15.
  References Boud, D. 1995.“Assessment and Learning: Contradictory or Complementary?” In Assessment for Learning in Higher Education, edited by P. T. Knight, 35–48. London. JISC (2020). The future of assessment: five principles, five targets for 2025. https://repository.jisc.ac.uk/7733/1/the-future-of-assessment-report.pdf Emerge and JISC (2020). Assessment Rebooted. From 2020s quick fixes to future transformation. https://repository.jisc.ac.uk/7854/1/assessment-rebooted-report.pdf Kevin Ashford-Rowe, K, Janice Herrington, J. and Brown, C (2014) Establishing the critical elements that determine authentic assessment, Assessment & Evaluation in Higher Education, 39:2, 205-222, DOI: 10.1080/02602938.2013.819566 Nielissen, G. (2021). Studeren in coronatijd. Thuistentamens en andere aspecten van studeren in de coronaperiode. Open Universiteit. Expertisecentrum Onderwijs (ECO). Shute, V. J., Y. J. Kim. 2013. “Formative and Stealth Assessment.” In Handbook of Research on Educational Communications and Technology, 4th ed., edited by J. M. Spector, D. M. Merrill, J. Elen, and M. J. Bishop, 311- 321. New York: Springer Science Business Media.