Maharg plenary panel slides
Download as PPTX, PDF1 like1,446 views
This document discusses various tools for assessing legal practice skills. It begins by outlining challenges with the current evidence base for assessment tools, noting a lack of reliable research and systematic reviews. It then discusses how assessment can challenge conventional teaching and assessment models by being more learner-centered, collaborative, and focused on skills and values. The document also explores using simulated clients for assessment, noting various jurisdictions that have implemented this approach successfully. It discusses the potential for online digital simulations and transactional learning. Finally, it proposes new program designs like problem-based learning to better integrate learning and assessment.
Maharg plenary panel slides
- 1. Assessment tools for practice skills Professor Paul Maharg http://paulmaharg.com/slides
- 2. 1. LETR and assessment The problems and what our Report said 2. Challenging hegemonies Why itโs difficult to change; how we can change 3. Adapting from other disciplines โ client-centred assessment Simulated clients 4. Learning from other jurisdictions โ the future of digital sim assessment Online digital sims 5. Extreme law schooling NB: Slides available at http://paulmaharg.com/slides preview 2
- 3. reform anxieties Law Society of Scotland (2006-09) FLSC Task Force (2007-09) LETR (2011-13) (phase 1) ABA Task Force (2012-13) CBA Legal Futures (2012-14) LACC (Australia, 2014-??) CEE Evaluation Report; and Wu Review of Legal Education (Hong Kong, ongoing)
- 4. 1. LETR on assessment
- 5. the problem of weak evidence base Absence of much reliable research: โ Small-scale qualitative studies โ Under-/undefined success criteria โ Few longitudinal studies or follow-ups (recency effects and other biases) โ Few systematic attempts at replication or meta-analysis โ Almost no systematic summaries of research (but see PEARL -- https://pearl.law.anu.edu.au)
- 6. what we said about assessment tools 4.50 โa move to outcomes-based education and training (OBET) โฆ may improve reliability of assessment techniques, rather than the reverseโ 4.111 โExisting assessment strategies tend to focus on conformity as a proxy for consistency 4.123 โThere is a legitimate regulatory interest in ensuring that assessment is robust. It must be capable of assessing that which it sets out to assess (valid); it must produce consistent and replicable results (reliable); and it must assess against the syllabus and learning outcomes that have been set out (fair) 4.123 โStandardisation โฆ can be achieved by extending the number of credible judgments in respect of a task, and by ensuring that those judgments are collected and evaluated systematicallyโ Outcomes, shared standards and standardisation ThanxJulian
- 8. conventional teaching & assessment If learning โฆ then assessment is oftenโฆ 1 Is teacher-focused Teacher-centred, not learner-centred. 2 Follows a transmission model of education Focused only on whatโs supposed to have arrived / been delivered 3 Focuses only on the individual Individual, alienating, where collaborative, peer- review or self-review canโt take place 4 Consists of monolithic & substantive law content Lacking interdisciplinarity, with little assessment of skills, values, attitudes as well as knowledge 5 Sits in the contested relations between practice & academy Problematic, because content & forms of academic assessments canโt transfer well to professional learning and formation of identity
- 9. signature pedagogies assessments Surface structure โข Observable, behavioural features Tacit structure โข Values and dispositions that the behaviour implicitly models Deep structure โข Underlying intentions, rationale or theory that the behaviour models Shadow structure โข The absent assessments, only weakly engaged at best Sullivan, W.M., Colby, A., Wegner, J.W., Bond, L., Shulman, L.S. (2007) Educating Lawyers. Preparation for the Profession of Law, Jossey-Bass, p. 24
- 10. transforming learning and assessment: four key themesExperience ofโฆ โขlaw in the world โขinterdisciplinary trading zones โขcreative, purposeful acts Ethics inโฆ โขan integrated curriculum โขhabitual action โขreclamation of moral spaces in the curriculum Technology forโฆ โขour discipline, our curricula โขlearner-centred control โขtransactional learning Collaboration betweenโฆ โขstudents โขinstitutions โขacademic & professional learning โขopen-access cultures
- 11. 3. Adapting from other disciplines โ client-centred assessment
- 12. standardised client initiative (SCI) We train lay people to simulate clients, and to do two things well: โ Discuss their case with the (trainee) lawyer in a way that is standard across the cohort of students/lawyers that the SC meets โ Assess the client-facing skills of the lawyer.
- 13. SCI aims โข develop a practical and cost-effective method to assess the effectiveness of lawyer-client communication that correlates assessment with the degree of client satisfaction. โข ie answer the following questionsโฆ โ Is our current system of teaching and assessing interviewing skills sufficiently reliable and valid? โ Can the Standardised Patient method be translated successfully to the legal domain? โ Is the method of SC training and assessment cost-effective? โ Is the method of SC training and assessment more reliable, valid and cost-effective than the current system?
- 14. SCI conclusionsโฆ โข Use of SCs is as reliable and valid as tutor assessments โข We make what the client thinks important in the most salient way for the student: a high-stakes assessment where most of the grade is given by the client โข We did not conclude that all aspects of client interviewing can be assessed by SCs โ We focused the assessment instrument on aspects we believe could be accurately evaluated by non-lawyers โ SCs are also trained to give feedback to students โข This has changed the way we enable students to learn interviewingโฆ Barton, K., Cunningham, C.D., Jones, G.T. and Maharg, P. (2006) What clients think: standardized clients and the assessment of communicative competence, Clinical Law Review, 13(1), pp. 1โ65.
- 17. feasibility? cost? impact? Feasibleโฆ? โข Very: lots of experience out there in at least 12 centres. Initial and refresher training needed for SCs, but no high-maintenance. Costโฆ? โข Training of SC trainer + SCs; payment of SCs. โข SC documentation is freely available under Creative Commons Impactโฆ? โข Big: on students, on ethical performance, practice of skills within professional value contexts; formative and high-stakes assessment. โข Also on regulatory bodies, eg Law Society of Scotland, SRA, and others
- 18. who uses SCs? Strathclyde University Law School (DLP) WS Society (Edinburgh) University of New Hampshire (Daniel Webster Scholars program) The Australian National University College of Law (GDLP) Northumbria U Law School, LLB Kwansei Gakuin U Law School, Osaka SRA (Qualifying Lawyersโ Transfer Scheme, QLTS) Law Society of Ireland (CPD) Hong Kong University Faculty of Law (PCLL) University of Adelaide Law School (LLB) The Chinese University of Hong Kong Law School National Skills Centre for Social Care, London
- 19. who uses SCs? Strathclyde University Law School (DLP) WS Society (Edinburgh) University of New Hampshire (Daniel Webster Scholars program) The Australian National University College of Law (GDLP) Northumbria U Law School, LLB Kwansei Gakuin U Law School, Osaka SRA (Qualifying Lawyersโ Transfer Scheme, QLTS) Law Society of Ireland (CPD) Hong Kong University Faculty of Law (PCLL) University of Adelaide Law School (LLB) The Chinese University of Hong Kong Law School National Skills Centre for Social Care, London
- 20. DW program โข Two-year Bar practicum โข Students educated in professional skills and judgment through simulated, clinical and externship settings โข Exposure to numerous fields, including real estate, business, and litigation is offered. โข Instead of a two-day bar exam, the program provides a two-year, comprehensive exam in conjunction with the training received. โข Students who complete the program are certified as having passed the New Hampshire Bar examination, subject only to passing the Multistate Professional Responsibility Examination (MPRE) and the New Hampshire character and fitness requirements.
- 21. DW program evaluation Evaluation of the program found that: โข In focus groups, members of the profession and alumni said they believe that students who graduate from the program are a step ahead of new law school graduates โข When evaluated based on standardized client interviews, students in the program outperformed lawyers who had been admitted to practice within the last two years โข The only significant predictor of standardized client interview performance was whether or not the interviewer participated in the DW Program. Neither LSAT scores nor class rank was significantly predictive of interview performance.
- 22. who uses SCs? Strathclyde University Law School (DLP) WS Society (Edinburgh) University of New Hampshire (Daniel Webster Scholars programme) The Australian National University College of Law (GDLP) Northumbria U Law School, LLB Kwansei Gakuin U Law School, Osaka SRA (Qualifying Lawyersโ Transfer Scheme, QLTS) Law Society of Ireland (CPD) Hong Kong University Faculty of Law (PCLL) University of Adelaide Law School (LLB) The Chinese University of Hong Kong Law School National Skills Centre for Social Care, London
- 23. QLTS assessment regime Assessment consists of: โข The Multiple Choice Test (MCT) โข The Objective Structured Clinical Examination (OSCE) โข The Technical Legal Skills Test (TLST)
- 24. OSCE Tests the oral skills of interviewing and advocacy/oral presentation together with the three content areas of: โ Business โ Civil and criminal litigation โ Property and probate Components of the OSCE Each candidate completes three stations in each content area: โข Station 1: Client interview and completion of attendance note (+ research) โข Station 2: Client interview and completion of attendance note โข Station 3: Advocacy/oral presentation
- 25. test quality of simulated clients? โOverall the test quality is remarkably good for such a new set of assessment procedures and challenging targets for a new high stakes assessment have largely been met.โ Eileen Fry , Jenny Crewe & Richard Wakeford (2012) The Qualified Lawyers Transfer Scheme: innovative assessment methodology and practice in a high stakes professional exam, The Law Teacher, 46:2, 132-145, p.144.
- 26. is it a valid & reliable assessment? โAssessment by standardised clients proved to be very reliable, with the six standardised client assessments conducted for each candidate by a total of 45 different actors having an alpha coefficient of 0.81 and SEm of 5.07% in OSCE #2.โ Eileen Fry , Jenny Crewe & Richard Wakeford (2012) The Qualified Lawyers Transfer Scheme: innovative assessment methodology and practice in a high stakes professional exam, The Law Teacher, 46:2, 132-145, p.144.
- 27. SCs: people as co-producers, co-designers The SC approach challenges: 1. Curriculum methods 2. Ethics of the client encounter 3. The cognitive poverty of conventional law school assessment 4. Law school as a self-regarding, monolithic construct 5. Law school categories of employment 6. The curricular isolation of clinic within law schools 7. Hollowed-out skills rhetoric 8. Conventional forms of regulation by regulatory bodies 9. The role of regulator, as encourager of innovation & radical reformโฆ? 10. Disciplinary boundaries โ what about a SC Unit thatโs interdisciplinary? 11. Local jurisdictional practices: how might such a project work globally?
- 28. 4. Learning from other jurisdictions โ the future of digital sim assessment
- 29. Transactional Learning active learning through performance in authentic transactions involving reflection in & on learning, deep collaborative learning, and holistic or process learning, with relevant professional assessment that includes ethical standards
- 30. what are we assessing in simulation? โข Professionalism and ethical performance โข Skilled performance to benchmarked levels โข Substantive knowledge of law โข Procedural knowledge โข Many other categories of assessable experience โข Purpose of assessment: โ Formative (feedback and feedforward) โ Summative Think of a social space where workspace, learning space & assessment space co-exist, eg, between master & apprentice.
- 31. how are we assessing in sims? 1. Discrete tasks, eg drafting, letter-writing, research (Estate Planning transaction) 2. Whole file + performative skill (Personal Injury transaction) 3. Tasks + whole file (Property or Planning transaction) 4. Tasks + file + performative skill (Litigation transaction)
- 32. โข Set context (or not: let student figure that out โ the clearing in the forestโฆ) โข Set task (but in how much detail? Supported with templates? Guidelines? Commented examples?) โข Design feedforward (but donโt do the task for students) โข Deadline a task (bearing all contextual factors in mind) โข Task completed (and sent by students to staff in role) โข Feedback on task (by staff in role) โข Debrief (either in role or out of role) 1. discrete tasks
- 33. โข Holistic assessment of document chain โข Bodies of evidence generally, but can embed critical points of assessment, eg report to client, speech plan, etc โข Preparation for performative skill, including overlap with other skills โ eg relation of legal research to professional negotiation. 2. whole file + performative skill
- 34. โข Specific tasks are the foreground, eg draft a Completion Certificateโฆ โข โฆ but students must also create entire file. No file, no competence evidenced. โข Tasks may shadow tutorial work or precede tutorial work or neither โข Quaere: How many attempts at each task? 3. tasks + whole file
- 35. โข Most complex, most authentic and most demanding โข Potentially 1-3 plus more โ eg performative skills can be assessed in role. 4. tasks + whole file + performative skill(s)
- 36. Practice Management @ ANU Legal Workshop includes work onโฆ Developing professional identity โข Meeting deadlines, courtesy in communications โข Practice Organisation skills โข Ethical obligations Supporting disruptive pedagogy โข Team work โข Time, file and risk management Enhancing wellbeing โข Identifying responses and feeling comfortable with uncertainty โข Giving Voice to Values (GVV) โ by Mary Gentile, Babson College ThanxAnneka&Liz
- 37. sim creation tools: the transitive power of music notationโฆ Panmure Lute MS (c.1632), 5, no.3. Music for Lute Consort, c.1500
- 38. Narrative event diagram, Personal Injury Transaction: Pursuer โฆ could be used as simulation notation
- 39. Narrative event diagram: Personal Injury Transaction: Pursuer and Defender (Gould et al 2009)
- 40. feasibility? cost? impact? Feasibleโฆ? โข Yes: following the SIMPLE project, ANU Legal Workshop created a Virtual Office Space (VOS) and other environments, and now runs an online Professional Practice Course for over 1,000 students. Now designing a Ready for Practice capstone course. HKU run a version called SMILE. Costโฆ? โข Development of environment + sims; learning support for students โข Staff costs; training costs. Impactโฆ? โข Big: on students, on ethical performance, practice of skills within professional value contexts; formative and high-stakes assessment; transactional learning; with the capacity for Deweyan social justice learning
- 41. 5. Extreme law schooling โthe key to expert problem-solving lies in how knowledge is organized, not the quantitative knowledge acquiredโ (Lung 2008)
- 42. new programme designs โข Eg online PBL JD at ANU in Australian law: โ New 3 year curriculum, part-time over 6 years: qualifying subjects + Masters options. โ Integration into clusters of traditionally separate subjects, eg Torts/Civil โ Focus on collaborative problem-solving using a PBL methodology โ Learning intellectual structures through problem immersion โ Fusing learning and immersive, integrative assessment โ Healing the academic / professional divide, in design and in new forms of employment (adjuncts as trained PBL facilitators) โ Opening up choice of career pathways โ Possibility of global partnerships with other innovative PBL centres.
- 43. PBL design
- 44. approaches to PBL online โข Design approach similar to aspects of the Edinburgh Manifesto โข PBL โ key issue is the relation between distance vs intimate learning โข In conceptualising online learning, dichotomy of f2f campus vs online platform is the norm. โข But campus-based learning could be seen as a platform. And online platform as form of campus.
- 45. Edinburgh University online teaching manifesto https://onlineteachingmanifesto.wordpress.com/
- 46. more depth vs more breadth, or both? On knowledge acquisition, Schmidt et al (2009) noted what many others observed: that PBL students better integrate their knowledge, which resulted in more accurate reasoning; that in the clinical case recall (a measure of expertise) and processing speed (a sign of better understanding) they were superior to the conventionally- educated cohorts (227). In skills acquisition, PBL students demonstrated much better interpersonal skills, and knowledge about skills (a variable closely related to skilled performance โ 236). Student and expert perceptions of the quality of PBL education were higher than the results for the conventionally-educated cohorts, with students commenting positively in particular on their practices in independent study and critical thinking. In passing, Schmidt et al also noted that PBL schools graduate students faster and in larger numbers and retain students better (237). (Maharg 2015, 12-13)
- 47. assessment tools? Integrative assessments are essential: โข Exams that embed course elements โข Skills that fuse with knowledge items โข Integration of SCs and Transactional Learning โข Evaluation where formative becomes the new summative, eg in capstone + e-portfolio โข Ethics & social justice are threaded pervasively through an entire programme, via problems & assessments โข Use of predictive data & analytics โข Potential use of AI & blockchain technologies
- 48. References
- 49. Barton, K., Cunningham, C.D., Jones, G.T. and Maharg, P. (2006) What clients think: standardized clients and the assessment of communicative competence, Clinical Law Review, 13(1), pp. 1โ65 Fry, E. , Crewe, J. & Wakeford, R. (2012) The Qualified Lawyers Transfer Scheme: innovative assessment methodology and practice in a high stakes professional exam, The Law Teacher, 46:2, 132-145 Gould, H., Hughes, M., Maharg, P., Nicol, E. (2009) The narrative event diagram (NED): a tool for designing professional simulations, in Gibson, D. (ed) Digital Simulations for Improving Education: Learning Through Artificial Teaching Environments, IGI Global Books, Hershey, PA. Lung, S. (2008). The problem method: no simple solution. Willamette Law Review, 45, 723 Maharg, P. (2007). Transforming Legal Education: Learning and Teaching the Law in the Early Twenty-first Century. Ashgate Publishing, Aldershot. Maharg, P. (2015). โDemocracy begins in conversationโ: The phenomenology of problem-based learning and legal education. Nottingham Law Journal, 24, 1. Savin-Baden, M. (2007). A Practical Guide to Problem-Based Learning Online. Routledge, London. Schmidt, H.G., Rotgans J.I., and Yew E.H. (2011). The process of problem-based learning: what works and why. Medical Education, 45, 792. Schmidt HG et al. (2009). Constructivist, problem-based learning does work: a meta-analysis of curricular comparisons involving a single medical school. Educational Psychologist 45, 227. Sullivan, W.M., Colby, A., Wegner, J.W., Bond, L., Shulman, L.S. (2007) Educating Lawyers. Preparation for the Profession of Law, Jossey-Bass Webb, J., Ching, J., Maharg, P., Sherr, A. (2013). Setting Standards. The Future of Legal Services Education and Training Regulation in England and Wales. SRA, BSB, IPS. references