Before the Surveys, Spreadsheets, and numbers: Laying a Solid Assessment Foundation

BEFORETHE SURVEYS, SPREADSHEETS, AND
NUMBERS
LAYINGA SOLIDASSESSMENT FOUNDATION
ShaneYoung
GLACUHO Annual Conference
October 22nd 2019
© ShaneYoung at www.sensiblyshane.com

BeforeWe Begin
■ Want a copy of this presentation?
– www.sensiblyshane.com > Presentations > Before the surveys […]
■ Want a copy of the documents referenced in this presentation?
– Email me at syoung@sensiblyshane.com
■ Looking for resources or want to share resources with the region?
– Check out our new GLACUHOToolkit (https://www.glacuho.org/page/Toolkit)
– Or Click on Resources >Toolkit

Pictured above: What people think assessment is.

Pictured above: What people think assessment feels like.
And admittedly how it sometimes is.

ATale ofTwo Paradigms
■ Ewell (2009) points to two assessment paradigms
– Improvement
– Accountability

What is assessment?
■ Banta and Palomba (1999) indicated that “assessment is the systematic collection,
review, and use of information about educational programs undertaken for the
purpose of improving student learning and development.
■ Banta and Palomba (2015), but more specifically Banta’s definition, says that
“assessment is the process of providing credible evidence of resources,
implementation actions, and outcomes undertaken for the purpose of improving the
effectiveness of instruction, programs, and services in higher education.
■ Angelo (as cited in Suskie 2009) defines assessment as “the ongoing process of:
– Establishing clear, measurable expected outcomes of student learning
– Ensuring that students have sufficient opportunities to achieve those outcomes
– Systematically gathering, analyzing, and interpreting evidence to determine how
well student learning matches our expectations
– Using the resulting information to understand and improve student learning

What is assessment, but in red and
underlined!
■ Banta and Palomba (1999) indicated that “assessment is the systematic collection,
review, and use of information about educational programs undertaken for the
purpose of improving student learning and development.
■ Banta and Palomba (2015), but more specifically Banta’s definition, says that
“assessment is the process of providing credible evidence of resources,
implementation actions, and outcomes undertaken for the purpose of improving the
effectiveness of instruction, programs, and services in higher education.
■ Angelo (as cited in Suskie 2009) defines assessment as “the ongoing process of:
– Establishing clear, measurable expected outcomes of student learning
– Ensuring that students have sufficient opportunities to achieve those outcomes
– Systematically gathering, analyzing, and interpreting evidence to determine how
well student learning matches our expectations
– Using the resulting information to understand and improve student learning

Evaluation
■ Scheerens, Glas, &Thomas (2003) define evaluation, generally, as “systematic
information gathering and making some kind of judgment on the basis of this
information” (pg. 3)
■ Suskie (2009) picks three definitions of evaluation
1. Judgement: Did we do the thing we wanted to do?What can we do better?What
did we do well?
2. Matching: Did intent match our impact?
3. Worth: Is our program doing well enough to continue?

Evaluation continued
■ This term comes up often in assessment work and some people use the term
assessment and evaluation interchangeably.
– We’re going to assess whether or not you learned the thing.
– Our evaluation cycle has completed and now we can make some change
■ Theorists and researchers define and refine these terms regularly
■ My advice?

What is the point?Why does assessment
matter?
■ Accreditation
– Accrediting bodies want to know more and more about the full breadth of the student
experience, including what and how they learn outside of the classroom
■ Students are changing
– Millennials are the entry and mid-level workforce
– Gen Z are your “traditional” college students (Hughey, 2019)
■ Budgets are under lock and key
– Why does your program deserve this funding more than X?
■ Improvement
– Even the best of the best needs to solicit feedback and make small adjustments

Why does assessment matter?
■ Residence Life and Housing is trying to sell an experience, a lifestyle
– One that is better than other universities
– One that is better than off campus
■ What’s more compelling?

Made with Canva

-Made with Canva
“Student PNG image with transparent background”
by pngimg.com is licensed under - Creative
Commons 4.0 BY-NC

ASSESSMENT IS ULTIMATELY ABOUT CREATING
A ROBUSTAND EVOLVING STORY OFA
DEPARTMENT, DIVISION, OR OFFICE

I get it… Now when do I do the surveys?
■ There are plenty of things to do before getting to conducting surveys
The Assessment Cycle

NACADA
The Assessment Cycle. [Image]. (n.d.) Retrieved from https://nacada.ksu.edu/Events/Assessment-Institute/The-Assessment-Cycle.aspx

Northwestern
University
The Assessment Cycle. [Image]. (n.d.-b) Retrieved from https://www.northwestern.edu/searle/assessment-of-student-learning/assessment-process/index.html

George
Washington
University
The Assessment Cycle. [Image]. (n.d.-c) Retrieved from https://assessment.gwu.edu/assessment-cycle

Shane’sAssessment Approach
■ Identify Purpose
■ Select Personnel
■ Set Program Goals or Outcomes
■ Develop and Implement Assessment Strategies
■ Review Findings
■ Create Action Plan

PURPOSE
Adams, S. (2018, April). ‘Bad Data’ [Cartoon]. Dilbert. Retrieved
from https://dilbert.com/strip/2018-04-03

Identify Purpose
■ This step is one of the most important steps
■ The purpose of assessment is to create a process in which an organization can improve
– In order to improve an organization it’s important to know what the purpose of
the organization is.
■ The purpose of assessment is not just for you, it’s for the entire department

Where do organizations define purpose?
■ What is the mission of the organization?
■ What is the vision of the organization?
■ What does the organization do?
– How is that action reflected in the mission?
– How is that action reflected in the vision?
■ What does leadership believe is the purpose of the organization?

Select Personnel
■ Assessment is not a one person job?
– Why?
■ One person = one subjective view
■ One person collecting, processing, and presenting data = madness
■ Why is this important to me? –Other employees

(You) Select Personnel
■ Name three positions or people that should be
on an assessment committee
– Who did you pick and why?

Selecting Personnel
■ One of the most important factors when creating an assessment team is to be at least
as representative as the campus population.
■ If the team building the data collection tools or reviewing the data does not represent
your campus population the assumptions made and the conclusions drawn will be
incomplete.
■ If your department is unable to create a committee that it representative of your
campus population… then there needs to be some time set aside to examine why that
is.
■ Based on the last activity and the people/positions you selected – would your
committee be representative of your campus population?

Selecting Personnel
■ Other considerations
– New professionals
■ This is a great opportunity to reprogram them to understanding assessment is more
than crunching data
– Student Staff
■ “They wouldn’t be interested in this.” How do you know?
– Members from other departments
■ Is there a faculty member who specializes in educational assessment?
■ Not everyone has to be a “full member” they can be advisors to good assessment
practices

Set Program Goals
or Outcomes
“Learning outcomes seem pretty
specific for a presentation on
laying a foundation…”
-You all, right now, probably

Outcomes
■ According to Henning (2009) there are three main types of outcomes:
– Learning Outcomes:What will students learn?
■ Students will be able to articulate foundational management and leadership skills,
styles, and/or theories
– Operational Outcomes:What does our organization do?
■ Office of Student Engagement will plan and market weekly health and wellness
programming (Yoga, Zumba, etc.…)
– Program Outcomes: By simply existing what impact do we have
■ Increase in applications to and creation of student leadership roles

Outcome
Alignment
■ Henning in (Allen, Elkins,
Henning, Bayless, & Gordon,
2013) posits that there should
be alignment from the top
(institution mission) to the
bottom (department goals)
■ I believe that this is
important… but is just a
framework for even better
assessment in the future

Develop and ImplementAssessment
Strategies
■ This is step #4, but is often step #1 for a lot of organizations.
■ It is possible to start here and work backwards but that can lead to more frustration
■ Don’t reinvent the wheel (Suskie, 2011)
– Sometimes you already have awesome data collection tools.
Finck, N. (2010, April 9). The Commoditization & Fragmentation of the IA Community. [SlideShare] Retrieved
from https://www.slideshare.net/nickf/the-commoditization-and-fragmentation-of-the-ia-community© ShaneYoung at www.sensiblyshane.com

SURVEYS ARE COOL… BUTTHERE IS OTHER STUFFTOO!

My Recommendations
■ Mission/Vision to Outcomes Matrix
■ Outcomes toTools Matrix
■ Data Collection tool to Outcomes

Review Findings

Don’t be scared
■ The first time is the most intimidating and the most difficult
■ When selecting personnel, did you select persons who were familiar with Excel or
other tools used for sifting through data?
■ If you followed the recommendations in the previous section – then go line by line and
fill with the data that you have.
– Operational Outcomes might be a good place to start
■ Reviewing findings should not be something that is done once a year
– This should be done when it makes sense
■ After a big survey push, for example
This Photo by Unknown
Author is licensed under CC
BY-SA

CreateAction Plan
■ Did you meet the objectives that you created? Why?Why not?
– Were the objectives difficult to measure?
– Were the data collection methods not as effective as they needed to be
■ An annual review to prepare for the next academic year is a good idea, but discussion on
action items for next time should not be pushed off until the annual meeting
■ Do not reinvent the wheel
– Small adjustments over time are manageable and lead to less confusion with staff.
■ What your organization learned and that evidence should be transparent and
understandable to those within the organization at multiple levels, but also outside parties
(Bresciana, 2011)

Barriers to Good Assessment
■ There is a reason that presentations on assessment are always sought
■ There are three main barriers to good assessment that I want to highlight

You
■ Every single person in their
room can be a barrier to
good assessment practice
■ How might you be a barrier
to good assessment?
This Photo by Unknown Author is licensed under CC BY

Other Staff
■ Thoughts other staff may have:
– “I had no part in this… why do I have to do this?”
– “I have no idea what this is… So I’ll just do it my way.” (Suskie, 2011)
– “Why do you need to know how my effective my office’s work is?”
– “Why are we doing this?”
■ Getting buy in from your staff is important in determining purpose, but is equally
important during each step
■ Assessment is a part of everyone’s role - but those leading assessment need to make
sure that others know that

LEADERSHIP
Adams, S. (2011, August 17). ‘Untitled’ [Cartoon]. Dilbert.
Retrieved from https://dilbert.com/strip/2011-08-17

Leadership
■ Leaders have their own
– Goals
– Criteria
– Outcomes
■ Sometimes the goals, criteria, and outcomes of a leader do not align with those of the
personnel chosen to complete assessment
■ Leadership may not be receptive to receiving negative feedback

My Story About Assessment
■ Identify Purpose
■ Select Personnel
■ Set Program Goals or Outcomes
■ Develop and Implement Assessment Strategies
■ Review Findings
■ Create Action Plan

Works Cited
■ Allen, K. R., Elkins, B., Henning, G. W., Bayless, L. A., &Gordon, T. W. (2013). Accreditation and the role of the student affairs professional. Washington, DC: ACPA-College
Student Educators International. Retrieved from http://www.myacpa.org/sites/default/files/accreditationmonograph.pdf
■ Banta, T. W., & Palomba, C. A. (2015). Assessment essentials: Planning, implementing, and improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-
Bass.
■ Bresciani, M. J. (2011, August). Making assessment meaningful: What new student affairs professionals and those new to assessment need to know (NILOA Assessment Brief:
Student Affairs). Urbana, IL: University for Illinois and Indiana University, National Institute for Learning Outcomes Assessment.
■ Ewell, P. T. (2009, November). Assessment, accountability, and improvement: Revisiting the tension (NILOA Occasional Paper No.1). Urbana, IL: University of Illinois and
Indiana University, National Institute for Learning Outcomes Assessment.
■ Finck, N. (2010, April 9). The Commoditization & Fragmentation of the IA Community. [SlideShare] Retrieved from https://www.slideshare.net/nickf/the-commoditization-
and-fragmentation-of-the-ia-community
■ Hughey, A. (2019, September 24). Generation Students: Challenges, Opportunities, and Best Practices [Webinar]. Studentaffairs.com. Retrieved from
https://www.studentaffairs.com/Webinars/generation-z-students-challenges-opportunities-and-best-practices/
■ Scheerens, J., Glas, C. A. W., & Thomas, S., Dr. (2003). Educational evaluation, assessment, and monitoring. [electronic resource] : a systemic approach. Swets & Zeitlinger.
Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=cat02173a&AN=akr.b5764476&site=eds-live
■ Suskie, L. A. (2009). Assessing student learning: A common sense guide. San Francisco, CA: Jossey-Bass.
■ Venaas, M. (2017, October 17). Storytelling in Assessment: Compelling Ways to Share Data. Retrieved from http://www.skyfactor.com/storytelling-in-assessment/
■ The Assessment Cycle. [Image]. (n.d.-a) Retrieved from https://nacada.ksu.edu/Events/Assessment-Institute/The-Assessment-Cycle.aspx
■ The Assessment Cycle. [Image]. (n.d.-b) Retrieved from https://www.northwestern.edu/searle/assessment-of-student-learning/assessment-process/index.html
■ The Assessment Cycle. [Image]. (n.d.-c) Retrieved from https://assessment.gwu.edu/assessment-cycle
■ Henning, G. W. (2009, March). Where are we going? What do we pack? Which road do we take? How will we know we arrived? Presentation at the ACPA annual convention,
Washington, DC.

Before the Surveys, Spreadsheets, and numbers: Laying a Solid Assessment Foundation

More Related Content

Similar to Before the Surveys, Spreadsheets, and numbers: Laying a Solid Assessment Foundation

More from Shane Young

Recently uploaded

Before the Surveys, Spreadsheets, and numbers: Laying a Solid Assessment Foundation

Editor's Notes