Embracing AI In Assessment

CRICOS Provider No: 00300K (NT/VIC) 03286A (NSW) RTO Provider No: 0373 TEQSA Provider ID PRV12069
Embracing AI in Assessment
SISTC
Professor Michael Sankey
Director, Learning Futures and Lead Education Architect
President, Australasian Council on Open Distance and eLearning (ACODE)
Community Fellow, Australasian Society for Computers in Learning in Tertiary Education (ASCILITE)
michael_sankey

Charles Darwin University acknowledges all
First Nations people across the lands on
which we live and work, and we pay our
respects to Elders both past and present.
2

Artificial Insemination
What AI are we talking about?
3

9
https://intelligence.weforum.org/topics/a1Gb0000000pTDREA2?tab=publications

1. Krisp: Krisp's AI removes background voices, noises, and echo from your calls, giving you peace of call: https://krisp.ai/
2. Beatoven: Create unique royalty-free music that elevates your story: https://www.beatoven.ai/
3. Cleanvoice: Automatically edit your podcast episodes: https://cleanvoice.ai/
4. Podcastle: Studio quality recording, right from your computer: https://podcastle.ai/
5. Flair: Design branded content in a flash: https://flair.ai/
6. Illustroke: Create killer vector images from text prompts: https://illustroke.com/
7. Patterned: Generate the exact patterns you need for and design: https://www.patterned.ai/
8. Stockimg: Generate the perfect stock photo you need, every time: https://stockimg.ai/
9. Copy: AI Generated copy, that actually increases conversion: https://www.copy.ai/
10. CopyMonkey: Create Amazon listings in seconds: http://copymonkey.ai/
11. Ocoya: Create and schedule social media content 10x faster: https://www.ocoya.com/
12. Unbounce Smart Copy: Write high-performing cold emails at scale Link: https://unbounce.com/
13. Vidyo: Make short-form vids from long-form content in just a few clicks: https://vidyo.ai/
14. Maverick: Generate personalized videos at scale: https://lnkd.in/dmrkz_ah
15. Quickchat: AI chatbots that automate customer service charts: https://www.quickchat.ai/
16. Puzzle: Build an AI-powered knowledge base for your team and customers: https://www.puzzlelabs.ai/
17. Soundraw: Stop searching for the song you need. Create it: https://soundraw.io/
18. Cleanup: Remove any wanted object, defect, people, or text from your pictures in seconds: https://cleanup.pictures/
19. Resumeworded: Improve your resume and LinkedIn profile: https://lnkd.in/d9EurcnX
20. Looka: Design your own beautiful brand: https://looka.com/
21. theresanaiforthat: Comprehensive database of AIs available for every task: https://lnkd.in/dKhqaaF3
22. Synthesia: Create AI videos by simply typing in text: https://www.synthesia.io/
23. descript: New way to make video and podcasts: https://lnkd.in/d_Kdj35E
24. Otter: Capture and share insights from your meetings: https://otter.ai/
25. Inkforall: AI content (Generation, Optimization, Performance): https://inkforall.com/
26. Thundercontent: Generate Content with AI: https://lnkd.in/djFxMZsZ 10

AI doesn’t always get it right
11

• CEO Sam Altman has said that
OpenAI will devise ways to identify
ChatGPT plagiarism.
• But creating tools that perfectly
detect AI plagiarism is fundamentally
impossible, he said.
• Altman warns schools and policy
makers to avoid relying on plagiarism
detection tools.
ChatGPT
13
https://news.yahoo.com/ceo-chatgpt-maker-responds-schools-174705479.html

• Some treat using AI text-generators as “cheating”
• e.g. public school systems in NSW and Victoria have banned
ChatGPT
• This is hard to enforce. Compared to traditional forms of
plagiarism (where work is compared against source material)
• AI-generated text is hard to detect – and harder still to prove, in
part because ChatGPT generates new responses each time a user
inputs the same prompt
• Some 43% of professionals are using ChatGPT or other AI tools,
often without telling their bosses, according to a survey of nearly
12,000 respondents including from top US-based companies
Cheating bans hard to enforce
14

• Being transparent about decisions we make regarding
when and how AI tools are used in assessment could help
ease anxiety on both sides.
• We will look at 4 approaches:
• Teach students how (not) to use AI text generators
• Continue efforts to design ‘authentic’ assessments
• Balance essays with other types of assessment
• Develop AI-resistant assessments
Addressing the challenges
17

• Students will enter a world where the use of these tools
will be commonplace.
• ChatGPT has already been listed as an author on recent
academic papers, and is utilised by journalists and
authors.
• Its our job to prepare students for a world where tools like
ChatGPT are readily available – and to
• teach them how to use these tools well.
Can’t run and hide
18

• ChatGPT and similar technologies are useful tool, generating
content and ideas to help get words on the page.
• Misinformation is serious. So, we teach students how to use it,
how to understand its limitations, and how to fact-check its
outputs.
• It’s partly about teaching students how to read critically, how to
evaluate and corroborate evidence, and how to distinguish
good arguments from bad.
• However, we do not want to put generative AI at the centre of
education. We should teach students to recognise rubbish
wherever it appears.
Teaching students how to use generative AI
19

• AI can be used to generate assessment-based documents such
as rubrics and learning outcomes, as a launch pad for ideas,
and to automate tasks such as Excel formulae.
• We can use it to generate meeting transcriptions, create
agendas and schedules, summarise long emails and create
templates for student feedback.
• We can use it to challenges our own thinking and accepted
processes of learning.
For teaching staff
20

• AI can provide instant feedback on students’ academic writing,
simplify complex information and to scaffold information on
how to approach a specific task
• Helpful for neurodivergent students or students with English as
a second language
• Students struggling to understand concepts can ask AI to
provide examples to aid their understanding.
• The use of AI by students pivots them from being consumers of
learning materials to creators of their own learning resources.
Students can becoming creators
21

22
https://www.joshcavalier.com

• Choosing to participate in higher education come with a genuine
interest.
• This may mitigate some temptation to outsource their studies to AI,
when the value of completing their studies is clear to them
• The onus is on us to design assessments that are relevant to students’
future careers and to reinforce the message, so the value proposition
is diminished
• By clarifying the purpose of tasks in relation to their development, we
can encourage learners to engage with their learning
• Let’s not fool ourselves students will turn to AI, when things start
getting though, so additional strategies are needed.
Continue to design ‘authentic’ assessments
23

• A key concern about AI text-generation is that students won’t
actually understand what they have submitted.
• So we balance written work with other kinds of assessments
• In-person oral presentations (viva’s) cannot be produced by AI, yet.
• Supplementing essays with other assessments need not come at the
expense of good assessment design.
• There are good reasons to vary written work with other forms of
assessment; e.g. oral communication skills are enormously valuable
across a range of professions and yet are often undervalued in higher
education.
• This does not mean that writing skills are poised to become less
important as AI tools start to more prominence
Balance essays with other types of assessment
24

• Designing assignments where students can demonstrate their
understanding (independent of written work) not just recite knowledge
(think application)
• Pen-and-paper exams are back on the table for some. But at the same
time others are shifting away from exams to more “authentic”
assessments (assessments that evaluate real-world skills that students
will employ in the workplace)
• Few workplaces require their employees to write detailed discussions of
difficult questions by hand, and without a computer.
• Viva’s require students to understand and communicate the ideas
they’ve defended in their essay – regardless if they are outsourced
• Assignments that ask for deep research on recent developments are,
are relatively AI-resistant. But you need to continuously reviewed this.
Develop AI-resistant assessments
25

• Ask students to include their personal experience or
perspectives in their writing.
• Asked them to analyse a class discussions.
• Ask them a complex instruction that involve long texts that do
not fit a typical ChatGPT prompt, or
• Ask them to write about a very recent events (in the last week
or so) that may not be found in the ChatGPT database yet.
• But test it first
Set personalised, complex or topical tasks
26

• #1 Minimise the use of essays
• Provide them the readings that they can and only use.
• Source these readings from Google Scholar or from
University Databases (closed journals)
• Submit a word version or use a common drive with version
history enabled
• Ask them to reflect on what they learned from doing an
activity (3-500 words max)
Essays
27

• Present questions using images, figures, or charts as auxiliary
information, and a nonspecific question as stem.
• For example, ‘which section of the figure below demonstrates. . . ?’
• Present questions with auxiliary visuals as hotspot questions
where the student must click on an area of the image to indicate
the correct answer
• For example, ‘select the area on the image which shows . . .’
• Present questions using a series of images, or a video
accompanied with conditional logic branching questions.
• For example, ‘at this point in the interaction, which question should
you ask the customer?’
Multiple choice?
28

• Present questions that require the student to apply a concept
or principle to an up-to-date scenario or case study.
• For example, ‘the VOICE legislation to hold a referendum went
through parliament this week, but there were those that spoke
against it. What are the implication of the dissenting voices?’
• Use plausible distractors, consistent in content and structure,
and share important information with the correct option.
• The plausibility of all distractors increases the need for the
evaluation of all options to identify the correct answer.
Cont…
29

• Alternatives to essay and quiz assessment need to be
explored. Including,
• Using (impromptu) video presentations for assessments
• Create a podcast
• Us other digital forms, such as animations, mind maps,
infographics, Padlet, Voice Thread
Consider options that are not text-based
30

• Self-assessment or reflective writing, where students can
discuss their thinking process, ‘explain how you came to this
conclusion’
• Peer evaluations/assessments and group member evaluations
• Interactive assessments that engaging students in group
activities, requiring research and analysis, where students are
expected to co-construct knowledge and apply certain skills.
• You might consider emphasizing the process of learning rather
than the outcome. So not just grading a final piece, but ask
them to submit some of their drafts and assess how their
writing, or evolution of their thoughts have improved.
Use students’ thoughts and reflections
31

• Assessments that are formulaic need to be avoided. Instead,
• Consider assessments that are open-ended or encourage students to
demonstrate originality and creativity in their performance (iterative).
• That means formulating original ideas or research questions.
• We must find ways to assess students’ deep, critical-thinking skills.
• So, ask them to critique a paper (or even some text that has been
generated by ChatGPT)
• Build logical arguments, that demonstrate reasoning skills. Maybe
that’s a concept map, diagram, or a set of virtual sticky notes, to
demonstrate an understanding of a topic and its structure.
Assess critical thinking and avoid formulas
32

• ChatGPT doesn’t do well at providing appropriate sources and
quotations in its output.
• We use this as an opportunity to engage students in writing
practices focused on correcting factual errors and locating
accurate data sources (compare and contrast activity)
• Ask students cite and reference the work of others accurately
and properly by using in-text citations or including bibliography.
• You could ask a student to critique a piece of writing generated
by ChatGPT through analysing and interpreting how it conveys
an idea and assessing its strengths and weaknesses in terms of
readability, credibility, comprehensiveness, accuracy and so on.
Take advantage of AI’s shortcomings
33

• Beyond assessing cognitive skills, we can incorporate assessments of
other skills: collaboration, communication, leadership, or role play.
• If we have to use quizzes create questions that are hard for ChatGPT to
respond to, so check yourself first.
• Reweight assessment so that lower order thinking has lower weight.
• Compare, contrast, share activities
• Using an analysis of simulations
• Evidence-based portfolios by students
• Add a self-assessment step in all essays and reports
• Request direct quotes and multiple citations
Other idea
34

“… the power of generative AI tools
requires a deep rethink of approaches
to teaching and learning and
assessment practices and how higher
education institutions are ensuring
that students have attained the skills
and knowledge they need to graduate
with their awards.”
35
TEQSA Chief Commissioner Peter Coaldrake

• Bearman, M., Ajjawi, R., Boud, D., Tai, J. & Dawson, P. (2023). CRADLE Suggests… assessment and genAI. Centre for Research in
Assessment and Digital Learning, Deakin University, Australia. doi:10.6084/m9.figshare.22494178 Available from:
https://figshare.com/articles/online_resource/CRADLE_Suggests_Assessment_and_genAI/22494178
• Dastin, J., & Tong, A. (2023). Google, one of AI’s biggest backers, warns own staff about chatbots. Thomson Reuters Products.
16 June. https://www.reuters.com/technology/google-one-ais-biggest-backers-warns-own-staff-about-chatbots-2023-06-15/
• Gonsalves, C. (2023) On ChatGPT: what promise remains for multiple choice assessment? Journal of Learning Development in
Higher Education. Issue 27 (April). ISSN: 1759-667X
• Koplin, J., Sparrow, R., Hatherley, J., Rivers, N.,& Williams, I. (2023). Tailoring university assessment in the age of ChatGPT.
Monash University. 15 May. Available from: https://lens.monash.edu/@politics-society/2023/05/15/1385696/tailoring-
university-assessment-in-the-age-of-chatgpt
• Lee, J. (2023) Effective assessment practices for a ChatGPT-enabled world. Times Higher Education. 8 May,
https://www.timeshighereducation.com/campus/effective-assessment-practices-chatgptenabled-world
• Liu, D., & Bridgeman, A. (2023)., How can I update assessments to deal with ChatGPT and other generative AI? Assessment,
Educational integrity. Teaching tips University of Sydney. 23 January. Available from: https://educational-
innovation.sydney.edu.au/teaching@sydney/how-can-i-update-assessments-to-deal-with-chatgpt-and-other-generative-ai/
• Lodge, J., Howard, S., & Broadbent, J. (2023). Assessment redesign for generative AI: A taxonomy of options and their viability.
2 May. Available from: https://www.linkedin.com/pulse/assessment-redesign-generative-ai-taxonomy-options-viability-lodge/
• Lodge, J. (2023). Assessing learning processes instead of artefacts won’t be easy. 5 June. Available from:
https://www.linkedin.com/pulse/assessing-learning-processes-instead-artefacts-wont-easy-lodge/
• Sayers D. (2023). A simple hack to ChatGPT-proof assignments using Google Drive. Times Higher Education. 23 May.
https://www.timeshighereducation.com/campus/simple-hack-chatgptproof-assignments-using-google-drive
References
36

Embracing AI In Assessment

More Related Content

What's hot

Similar to Embracing AI In Assessment

More from Charles Darwin University

Recently uploaded

Embracing AI In Assessment