Abstract image of a woman sitting on books and studying on a laptop

We Teach - But are they Learning?

Download as PPT, PDF
P
Peter Jeschofnig

This document discusses assessing student learning, especially in distance learning science courses. It presents results comparing student performance in a traditional face-to-face chemistry course with an on-campus laboratory to an online chemistry course using a home-based laboratory kit. Students in both courses took the American Chemical Society standardized exam as a pre-test and post-test. Results found no significant difference in learning gains between the two formats, as both groups improved their scores substantially and performed similarly to national averages. This provides quantitative evidence that online science courses using home laboratory kits can effectively develop student understanding and skills comparable to traditional face-to-face courses.

Education
Related topics:
Distance Learning
1 of 42
Downloaded 12 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
We Teach …We Teach … But are they Learning?But are they Learning? Assessing Student LearningAssessing Student Learning inin D.L. Lab Science CoursesD.L. Lab Science Courses Peter Jeschofnig, Ph.D.Peter Jeschofnig, Ph.D. Colorado Mountain CollegeColorado Mountain College 1212thth Sloan-C International ConferenceSloan-C International Conference Nov 9, 2006Nov 9, 2006
Atlantic MonthlyAtlantic Monthly : Nov 2005: Nov 2005 .. "What Does College"What Does College Teach?Teach? It's time to put an end toIt's time to put an end to 'faith-based' acceptance of'faith-based' acceptance of higher education's quality“higher education's quality“ Richard HershRichard Hersh,, former president of Hobart & William Smith Collegesformer president of Hobart & William Smith Colleges and Trinity Collegeand Trinity College Co-director of the "Collegiate Learning Assessment"Co-director of the "Collegiate Learning Assessment"  Hersh & Merrow – 2005Hersh & Merrow – 2005 Declining by Degrees: Higher Education at RiskDeclining by Degrees: Higher Education at Risk
Institutional Level AssessmentInstitutional Level Assessment All institutions have some form of assessment,All institutions have some form of assessment, often linked to accreditationoften linked to accreditation Of 1,393 public and private institutions surveyedOf 1,393 public and private institutions surveyed in 1999, 82% listed “Excellence inin 1999, 82% listed “Excellence in Undergraduate Education” as part of missionUndergraduate Education” as part of mission statement.statement. However: Direct measures of studentHowever: Direct measures of student learning remain rare!learning remain rare!
Secretary of Education’s Commission onSecretary of Education’s Commission on The Future of Higher EducationThe Future of Higher Education Chairman Charles Miller, believes that collegesChairman Charles Miller, believes that colleges must better measure the skills and knowledgemust better measure the skills and knowledge they impart to students, and openly share thatthey impart to students, and openly share that information with the public.information with the public. ““We need to assure that the American publicWe need to assure that the American public understand through access to sufficientunderstand through access to sufficient information, particularly in the area of studentinformation, particularly in the area of student learning, what they are getting for theirlearning, what they are getting for their investment in a college education.”investment in a college education.” "No College Left Behind?""No College Left Behind?" Inside Higher Ed,Inside Higher Ed, February 15, 2006February 15, 2006
Secretary of Education’s Commission onSecretary of Education’s Commission on The Future of Higher EducationThe Future of Higher Education Feb. 3, 2006 meeting in San Diego:Feb. 3, 2006 meeting in San Diego: Higher education institutions must be more “transparent”Higher education institutions must be more “transparent” in collecting and giving the public useful informationin collecting and giving the public useful information about their activities and their performance.about their activities and their performance. Topics about which colleges should provide more andTopics about which colleges should provide more and better information, the various panels suggested, are onbetter information, the various panels suggested, are on their costs and prices, how their graduates fare in thetheir costs and prices, how their graduates fare in the employment market, and their success in impartingemployment market, and their success in imparting knowledge and, more importantly, skills like criticalknowledge and, more importantly, skills like critical thinking, to their students.thinking, to their students.
National Forum onNational Forum on College-Level Learning (NFCLL)College-Level Learning (NFCLL) "I'm disappointed that NASULGC ("I'm disappointed that NASULGC (National Association ofNational Association of State Universities and Land-Grant Colleges)State Universities and Land-Grant Colleges) seems to beseems to be interested in describing ‘the skills and knowledge thatinterested in describing ‘the skills and knowledge that students bring to college’ but evidently not the ones theystudents bring to college’ but evidently not the ones they leave with.leave with. While it's considering publishing ‘data on graduation rates,While it's considering publishing ‘data on graduation rates, admissions, applications, student demographics andadmissions, applications, student demographics and faculty demographics,’ THE OBVIOUS OMISSION OFfaculty demographics,’ THE OBVIOUS OMISSION OF EVIDENCE OF STUDENT LEARNING SUGGESTS THATEVIDENCE OF STUDENT LEARNING SUGGESTS THAT THEY HAVEN'T HEARD THE CLEAR MESSAGE THAT'STHEY HAVEN'T HEARD THE CLEAR MESSAGE THAT'S BEEN COMING FROM THE STATES AND THE FEDSBEEN COMING FROM THE STATES AND THE FEDS FOR ABOUT 20 YEARS: THAT THEY WANT TO KNOWFOR ABOUT 20 YEARS: THAT THEY WANT TO KNOW WHAT OUR GRADUATES KNOW AND CAN DO.“WHAT OUR GRADUATES KNOW AND CAN DO.“ Margaret (Peg) MillerMargaret (Peg) Miller 2006 Director of NFCLL2006 Director of NFCLL
. . ."We expect transparency and accountability for our tax. . ."We expect transparency and accountability for our tax dollars in almost every area of our government—fromdollars in almost every area of our government—from housing to health care to K-12 education. But in higherhousing to health care to K-12 education. But in higher education, we've invested tens of billions of taxpayereducation, we've invested tens of billions of taxpayer dollars over the years and basically just hoped for thedollars over the years and basically just hoped for the best.“best.“ PRESS RELEASE: Nov. 2, 2006PRESS RELEASE: Nov. 2, 2006 Secretary Spellings Delivered Remarks at NationalSecretary Spellings Delivered Remarks at National Postsecondary Education Cooperative Symposium onPostsecondary Education Cooperative Symposium on Student Success in Washington, D.C.Student Success in Washington, D.C.
Threat of Mandated AssessmentThreat of Mandated Assessment Government mandated assessments have manyGovernment mandated assessments have many negative ramifications that should be avoided.negative ramifications that should be avoided. However, that should not deter academicHowever, that should not deter academic professionals from designing valid and reliableprofessionals from designing valid and reliable assessment of learning.assessment of learning. A pro-active academic approach to assessment isA pro-active academic approach to assessment is surely better than a government mandated one!surely better than a government mandated one!
Assessment ApproachesAssessment Approaches  Actuarial Data and Expert Ratings:Actuarial Data and Expert Ratings: GraduationGraduation rates; research funding; student-teacher ratios, ACT &rates; research funding; student-teacher ratios, ACT & SAT scores; admissions selectivitySAT scores; admissions selectivity  Student Ratings:Student Ratings: Survey of Student EngagementSurvey of Student Engagement (NSSE) Students rate educational experience on(NSSE) Students rate educational experience on quantity and quality of faculty contact; homework,quantity and quality of faculty contact; homework, assignments, etc.; Noel-Levitz Student Satisfactionassignments, etc.; Noel-Levitz Student Satisfaction Inventory;Inventory;  Direct Assessment:Direct Assessment: Via grades & grade pointVia grades & grade point averagesaverages  Nearly useless as learning indicatorsNearly useless as learning indicators  Grade InflationGrade Inflation  Tell nothing about knowledge gained, retained, or ifTell nothing about knowledge gained, retained, or if knowledge can be applied to new situationsknowledge can be applied to new situations  Reflect lower level objectives such as facts and definitionsReflect lower level objectives such as facts and definitions rather than higher objectives & critical thinking skillsrather than higher objectives & critical thinking skills
Assessment Approaches – cont.Assessment Approaches – cont.  pre-professional programs withpre-professional programs with specialized accreditation programs whichspecialized accreditation programs which require evidence of student learningrequire evidence of student learning  Rising Junior examRising Junior exam
A Learning College ApproachA Learning College Approach ““The Learning College and its learningThe Learning College and its learning facilitators succeedfacilitators succeed only whenonly when improvedimproved and expandedand expanded learning can belearning can be documenteddocumented for its learners.”for its learners.” O’Banion,O’Banion, 19971997 The Primary Questions ForThe Primary Questions For Every Learning CollegeEvery Learning College Action:Action:  Does this action improveDoes this action improve or expand learning?or expand learning?  How do we know?How do we know?
Assessment ...Assessment ... ... is the systematic, on-going,... is the systematic, on-going, iterative process of monitoringiterative process of monitoring learning in order to determinelearning in order to determine what we are doing well and whatwhat we are doing well and what we must improve.we must improve.
We Want and Need to Know…We Want and Need to Know… …… and we want students, employers, peers,and we want students, employers, peers, policy makers, and the public to know …policy makers, and the public to know … how well students are able to use thehow well students are able to use the complex knowledge and abilitiescomplex knowledge and abilities articulated as important to their learning.articulated as important to their learning.
Basic Course Assessment TypesBasic Course Assessment Types  Formative Assessment:Formative Assessment: Ongoing assessment used to modifyOngoing assessment used to modify instruction and improve learninginstruction and improve learning  Summative Assessment:Summative Assessment: End of class or program assessment to verifyEnd of class or program assessment to verify that learning objectives have been met. Thisthat learning objectives have been met. This is compared to other classes and institutionsis compared to other classes and institutions to gage the effectiveness of courses andto gage the effectiveness of courses and instructional programs.instructional programs.
Assessment is EspeciallyAssessment is Especially Effective When It ...Effective When It ...  is student centeredis student centered  is congruent with instructional objectivesis congruent with instructional objectives  is relevantis relevant  is comprehensiveis comprehensive  is clear in purpose, directions, and expectationsis clear in purpose, directions, and expectations  is objective and fairis objective and fair  simulates "end" behavior/product/performancesimulates "end" behavior/product/performance  incites active responsesincites active responses  shows progress/development over timeshows progress/development over time
Assessment ChallengesAssessment Challenges  Disagreements on what should be taughtDisagreements on what should be taught  Well-conceived programs take time,Well-conceived programs take time, energy, and moneyenergy, and money  Avoiding the problems experienced in “NoAvoiding the problems experienced in “No child left behind” assessmentchild left behind” assessment  Threat to academic freedom?Threat to academic freedom?  Classes looking too much alike?Classes looking too much alike?
Distance LearningDistance Learning is Here to Stay!is Here to Stay! Online enrollments continue to grow at rates faster than forOnline enrollments continue to grow at rates faster than for the overall student body…the overall student body… Schools expect the rate of growth to further increase andSchools expect the rate of growth to further increase and believe that online learning is critical to their long termbelieve that online learning is critical to their long term strategy…strategy… Three quarters of academic leaders at public colleges andThree quarters of academic leaders at public colleges and universities believe thatuniversities believe that online learning qualityonline learning quality is equal tois equal to or superior to face-to-face instructionor superior to face-to-face instruction. And they expect. And they expect online offerings to continue to get better.online offerings to continue to get better. The Sloan Consortium, Nov. 2004The Sloan Consortium, Nov. 2004 Entering the MainstreamEntering the Mainstream The Quality and Extent ofThe Quality and Extent of Online EducationOnline Education in the United States,in the United States, 2003 and 20042003 and 2004
Specific Assessment ChallengesSpecific Assessment Challenges for Distance Educationfor Distance Education Using Nationally Normed Exams:Using Nationally Normed Exams:  Exam content must remain confidentialExam content must remain confidential  Require proctored examsRequire proctored exams  Cost to institutionCost to institution  Inconvenience to studentInconvenience to student Institution Designed Exams:Institution Designed Exams:  ValidityValidity  ReliabilityReliability
What About Lab Sciences?What About Lab Sciences? If we are to avoid seeing a continuingIf we are to avoid seeing a continuing decline in science literacy in America,decline in science literacy in America, lab sciences MUST be fully included inlab sciences MUST be fully included in the increasing mix of online coursethe increasing mix of online course offerings.offerings. However, there are still many instructorsHowever, there are still many instructors and institutions that do not believe laband institutions that do not believe lab sciences can effectively be taught at asciences can effectively be taught at a distance. Valid and reliable assessmentdistance. Valid and reliable assessment data is required to dispel thisdata is required to dispel this misconception.misconception.
Assessing Online ScienceAssessing Online Science There is ample anecdotal evidence ofThere is ample anecdotal evidence of student learning and satisfaction instudent learning and satisfaction in distance science courses that usedistance science courses that use home-based lab kits to fulfillhome-based lab kits to fulfill laboratory requirements.laboratory requirements. However, there has been littleHowever, there has been little quantitative data to support thisquantitative data to support this positive conclusionpositive conclusion..
Objective of this StudyObjective of this Study To quantitatively assess and compare theTo quantitatively assess and compare the performance of my chemistry studentsperformance of my chemistry students In a face-to-face (F2F) chemistry courseIn a face-to-face (F2F) chemistry course with an on-campus laboratorywith an on-campus laboratory andand In an online chemistry (DL) courseIn an online chemistry (DL) course with a home-based laboratory kitwith a home-based laboratory kit
Process of Assessing OutcomesProcess of Assessing Outcomes Administer and Compare Results for Campus-BasedAdminister and Compare Results for Campus-Based CHE-111 Students and Online CHE-111 Students:CHE-111 Students and Online CHE-111 Students: 1.1. American Chemical Society Standardized ExamAmerican Chemical Society Standardized Exam Pre-test at the beginning of the semester andPre-test at the beginning of the semester and Post-test at the end of the coursePost-test at the end of the course 2. Traditional homework, quiz, and exam grades2. Traditional homework, quiz, and exam grades 3. Laboratory reports graded via a specific rubric3. Laboratory reports graded via a specific rubric
American Chemical SocietyAmerican Chemical Society (ACS) General Chemistry I Exam(ACS) General Chemistry I Exam Scores based on 3007 students in 20 collegesScores based on 3007 students in 20 colleges and universities, including:and universities, including: University of AlabamaUniversity of Alabama Albuquerque Technical CollegeAlbuquerque Technical College Mercer University Kennesaw State UniversityMercer University Kennesaw State University Jamestown CCJamestown CC Florida Southern UniversityFlorida Southern University Miami UniversityMiami University Monroe CC;Monroe CC; University Pittsburgh University of N.CarolinaUniversity Pittsburgh University of N.Carolina
ACS Exam Results: F2FACS Exam Results: F2F CHE 111 - ACS Exam 0 10 20 30 40 50 60 70 pre-test-% post-test-% National mean-% Pre-test & Gain; National Score %Score Series2 Series1
ACS Exam Results: DLACS Exam Results: DL CHE 111 (DL) - ACS Exam 0 10 20 30 40 50 60 70 Pre-test % Post-test % National mean Pre-test, Post-test, National score ACSExamScores Series2 Series1
ACS Exam Results: DL vs. F2FACS Exam Results: DL vs. F2F ACS Score Comparison 0 10 20 30 40 50 60 70 80 90 DL-Pre- Test F2F-Pre- Test DL-Post- Test F2F-Post- Test ACS National DL Final Exam F2F-Final Exa, ExamScores Raw Score %
ACS Score ComparisonsACS Score Comparisons MeanMean S.D.S.D. MedianMedian F2FF2F n=24n=24 40.440.4 11.811.8 4242 DLDL n=15n=15 40.940.9 12.212.2 4141 NationalNational n=3007n=3007 41.741.7 12.312.3 41.241.2
ACS Post-test Scores: F2FACS Post-test Scores: F2F ACS Post Test Scores In-House 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Students Scores Series2
ACS Post-test Scores: DLACS Post-test Scores: DL ACS Post Test Scores - DL 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Students Scores Series2
Final Exam vs. ACS Exam – F2FFinal Exam vs. ACS Exam – F2F Final Exam vs ACS Exam y = 0.5233x + 49.372 R 2 = 0.3756 0 20 40 60 80 100 120 0 20 40 60 80 100 ACS Score FinalExamScore Series1 Linear (Series1)
Final Exam vs. ACS Exam – DLFinal Exam vs. ACS Exam – DL Final Exam vs ACS Exam - DL y = 0.8231x + 32.366 R 2 = 0.5917 0 20 40 60 80 100 120 0 20 40 60 80 100 ACS Exam Scores FinalExamScores Series1 Linear (Series1)
Lab Report Grading RubricLab Report Grading Rubric Title Page -Title Page - 5 points5 points Succinct and descriptive title and experiment number, author’s name, partners’Succinct and descriptive title and experiment number, author’s name, partners’ names, course name and number; date of experiment, date of lab report.names, course name and number; date of experiment, date of lab report. Abstract –Abstract – 10 points10 points A brief one to two paragraph statement of the purpose of the experiment and theA brief one to two paragraph statement of the purpose of the experiment and the results (i.e. relative yield, identification of unknown, etc).results (i.e. relative yield, identification of unknown, etc). 10 points10 points Purpose/Hypothesis -Purpose/Hypothesis - 10 points10 points A detailed statement about the experiment’s purpose and hypothesis (yourA detailed statement about the experiment’s purpose and hypothesis (your predictions). Describe what you think the likely outcome of the experiment will be;predictions). Describe what you think the likely outcome of the experiment will be; what scientific principle or law will be tested; what scientific relationship will be shown.what scientific principle or law will be tested; what scientific relationship will be shown. Hypothesis should include any relationship between variables and should mention theHypothesis should include any relationship between variables and should mention the independent vs. dependent variable.independent vs. dependent variable. Procedure –Procedure – 10 points10 points :: If you followed a detailed procedure from the lab manual, a very short procedureIf you followed a detailed procedure from the lab manual, a very short procedure summary will be enough.summary will be enough. If you used a procedure not detailed in the lab report, youIf you used a procedure not detailed in the lab report, you need to write the procedure in enough detail, using numbered steps, so that someoneneed to write the procedure in enough detail, using numbered steps, so that someone should be able to repeat your procedure.should be able to repeat your procedure. This section should include any alterations or errors you made to your operatingThis section should include any alterations or errors you made to your operating methodology. Be thorough and specific! Another chemist should be able to read yourmethodology. Be thorough and specific! Another chemist should be able to read your procedure and reproduce your experiment with precisionprocedure and reproduce your experiment with precision.. This section should notThis section should not contain any results/data from the exercise. 10contain any results/data from the exercise. 10 pointspoints
Data/ObservationsData/Observations -- 25 points25 points A detailed presentation of all hard data gathered during the experiment, as well as anA detailed presentation of all hard data gathered during the experiment, as well as an organized presentation of all your observations during the lab. Describe not only whatorganized presentation of all your observations during the lab. Describe not only what you see, but what it signifies in chemical terms. What is really happening? Also,you see, but what it signifies in chemical terms. What is really happening? Also, answer any questions the lab poses here.answer any questions the lab poses here. Tables should be used whenever possible. Graphs need to be complete with titles,Tables should be used whenever possible. Graphs need to be complete with titles, axes must be labeled; the independent variable should be on the x-axis, theaxes must be labeled; the independent variable should be on the x-axis, the dependent variable on the y-axis. Any calculations should be in this section as well.dependent variable on the y-axis. Any calculations should be in this section as well. They should be presented clearly and explained. The source of all numbers used inThey should be presented clearly and explained. The source of all numbers used in calculations should be included.calculations should be included. Results/Analysis -Results/Analysis - 2020 pointspoints A comprehensive, thoughtful discussion of what your data means and how it provedA comprehensive, thoughtful discussion of what your data means and how it proved or disproved your hypothesis, including the relationship of the variables as presentedor disproved your hypothesis, including the relationship of the variables as presented by the data. Explain any trends in the data that will be used. You should present theby the data. Explain any trends in the data that will be used. You should present the evidence that will support your conclusions. Error analysis, including percent error,evidence that will support your conclusions. Error analysis, including percent error, should be included here. Are the results consistent within the limitations of equipmentshould be included here. Are the results consistent within the limitations of equipment and random error. All questions should be answered here.and random error. All questions should be answered here. Conclusion -Conclusion - 20 points20 points A detailed discussion of how the lab results compare with your predictions, and whatA detailed discussion of how the lab results compare with your predictions, and what the results mean in a practical, real world sense. How can your discoveries bethe results mean in a practical, real world sense. How can your discoveries be applied? Have you verified your hypothesis? Did you demonstrate the scientificapplied? Have you verified your hypothesis? Did you demonstrate the scientific principle of this experiment adequately?principle of this experiment adequately?
UnsatisfactoryUnsatisfactory BorderlineBorderline SatisfactorySatisfactory ExcellentExcellent ScorScor ee Title PageTitle Page Max = 5 ptsMax = 5 pts Missing more thanMissing more than two items, or title ortwo items, or title or namesnames 0-2 points0-2 points Contains tile &Contains tile & names; two othernames; two other items missingitems missing 3 point3 point Contains tile &Contains tile & names; one othernames; one other item missingitem missing 4 points4 points Contains title, author’sContains title, author’s name, partners’ names,name, partners’ names, course name;course name; experiment number;experiment number; experiment & reportexperiment & report dates.dates. 5 points5 points AbstractAbstract Max = 10Max = 10 ptspts No abstract, orNo abstract, or Incomplete purpose,Incomplete purpose, incompleteincomplete ResultsResults 0-3 points0-3 points Purpose or results,Purpose or results, but not bothbut not both 5 points5 points Contains purposeContains purpose statement andstatement and results, but someresults, but some result detailsresult details missing.missing. 8 pts.8 pts. Contains purposeContains purpose statement and completestatement and complete resultsresults 10 points10 points Purpose/Purpose/ HypothesisHypothesis Max = 10Max = 10 ptspts IncompleteIncomplete purpose/hypothpurpose/hypoth Statement,Statement, Incomplete or missingIncomplete or missing scientific principles orscientific principles or variablesvariables 0-4 points0-4 points Clearly states theClearly states the correct purposecorrect purpose /Hypothesis, Both/Hypothesis, Both scientific principlesscientific principles or vari-ables missingor vari-ables missing 6 points6 points Clearly states theClearly states the correct purpose/correct purpose/ Hypothesis, eitherHypothesis, either scientific principlesscientific principles or variablesor variables missingmissing 8 points8 points Clearly states theClearly states the correctcorrect purpose/hypothesis,purpose/hypothesis, which scientificwhich scientific principles to be tested;principles to be tested; the variables involvedthe variables involved 10 points10 points Grading Rubric PointGrading Rubric Point Allocation for Lab ReportsAllocation for Lab Reports
Results/Results/ AnalysisAnalysis Max = 20Max = 20 ptspts Explanation of data isExplanation of data is missing, inaccurate,missing, inaccurate, or not expressed inor not expressed in complete sentences.complete sentences. Error analysisError analysis incomplete, missingincomplete, missing or incorrector incorrect 0-8 points0-8 points IncompleteIncomplete description of data;description of data; 3 or more important3 or more important observations areobservations are missing. Errormissing. Error analysis incompleteanalysis incomplete or only partiallyor only partially correctcorrect 12 points12 points Results statedResults stated correctly in com-correctly in com- plete sentences.plete sentences. No more than 1-2No more than 1-2 important obser-important obser- vations arevations are missing. . Errormissing. . Error analysis presentanalysis present and correctand correct Complete description ofComplete description of what occurred in thewhat occurred in the experiment is stated inexperiment is stated in complete sentences.complete sentences. Data is used accuratelyData is used accurately in reporting/analyzingin reporting/analyzing the results. Error analy-the results. Error analy- sis present and correctsis present and correct 20 points20 points Data/Data/ ObservationObservation Max = 25Max = 25 ptspts Data is missing,Data is missing, incomplete,incomplete, inaccurate, or hasinaccurate, or has several defects;several defects; No data tables whenNo data tables when appropriate, missingappropriate, missing graphs; most or allgraphs; most or all observations missingobservations missing Incomplete or noIncomplete or no calculations, fewcalculations, few questions answeredquestions answered 0-12 points0-12 points Data presented, butData presented, but not well organized,not well organized, some data inaccu-some data inaccu- rate or missing;rate or missing; graphs are inaccu-graphs are inaccu- rate in data display,rate in data display, incorrectly or notincorrectly or not labeled; poor orlabeled; poor or incomplete observa-incomplete observa- tions Poor & incom-tions Poor & incom- plete calculation;plete calculation; some questionssome questions answered.answered. 16 pts16 pts Data presentedData presented clearly and neatly.clearly and neatly. Most charts, tables,Most charts, tables, diagrams, anddiagrams, and graphs labeled andgraphs labeled and accurate; detailedaccurate; detailed and accurate ob-and accurate ob- servations. Mostservations. Most calculations showncalculations shown and correct. Mostand correct. Most questionsquestions answeredanswered 20 points20 points Data presented clearlyData presented clearly and neatly. All charts,and neatly. All charts, tables, diagrams, andtables, diagrams, and graphs labeled andgraphs labeled and accurate; Appropriateaccurate; Appropriate graphing of data chosengraphing of data chosen (line/bar/pie); detailed(line/bar/pie); detailed and accurate observa-and accurate observa- tions. Calculationstions. Calculations shown and correct. Allshown and correct. All questions answered inquestions answered in this section.this section. 25 points25 points ProcedureProcedure Max = 10Max = 10 ptspts Includes an unclearIncludes an unclear set of instructions.set of instructions. Most steps may be:Most steps may be: Missing. IncompleteMissing. Incomplete disorganized, con-disorganized, con- fusing,,difficult tofusing,,difficult to follow,not sequential,follow,not sequential, 0-4 points0-4 points Includes an nclearIncludes an nclear set of instructions.set of instructions. Some steps may beSome steps may be missing. Incomplete,missing. Incomplete, disorganized,confus-disorganized,confus- ing,difficult to follow,ing,difficult to follow, not sequentialnot sequential 6 pts6 pts Includes a clear setIncludes a clear set of instructions. Aof instructions. A few steps are:few steps are: missing,incomplete,missing,incomplete, disorganized,disorganized, difficult to followdifficult to follow 8 points8 points Directions in clear,Directions in clear, concise sentences in aconcise sentences in a step-by-step format.step-by-step format. Experiment can clearlyExperiment can clearly be replicated. Materialsbe replicated. Materials are included in theare included in the methodmethod 10 points10 points UnsatisfactoryUnsatisfactory BorderlineBorderline SatisfactorySatisfactory ExcellentExcellent ScorScor ee
Cover letterCover letter (when(when applicable)applicable) Max = 20Max = 20 ptspts Hypothesis or goal isHypothesis or goal is severely incompleteseverely incomplete or missing; results ofor missing; results of data interpretationdata interpretation severely incompleteseverely incomplete or missing; relation-or missing; relation- ship to scientificship to scientific principles severelyprinciples severely incomplete or missingincomplete or missing 0-8 points0-8 points Hypothesis or goal isHypothesis or goal is incomplete; resultsincomplete; results of data interpretationof data interpretation incomplete;incomplete; relationship torelationship to scientific principlesscientific principles incompleteincomplete 12 points12 points Hypothesis or goalHypothesis or goal is slightlyis slightly incomplete; resultsincomplete; results of dataof data interpretationinterpretation slightly incomplete;slightly incomplete; relationship torelationship to scientific principlesscientific principles slightly incompleteslightly incomplete 15 points15 points Concise description ofConcise description of hypothesis or goal, veryhypothesis or goal, very brief procedure, generalbrief procedure, general description of data,description of data, results of dataresults of data interpretation,interpretation, relationship to scientificrelationship to scientific principles (rest similar toprinciples (rest similar to conclusions above)conclusions above) 20 points20 points ConclusionsConclusions Max =20 ptsMax =20 pts Conclusion is missingConclusion is missing or does not fullyor does not fully explain the objectivesexplain the objectives of the lab. Relevantof the lab. Relevant vocabulary missing.vocabulary missing. No practical applica-No practical applica- tion given; Discus-tion given; Discus- sion of scientificsion of scientific principle missing.principle missing. Only 1-2 sentencesOnly 1-2 sentences 0-8 points0-8 points Conclusion explainsConclusion explains the objecive, butthe objecive, but data not useddata not used accurately to supportaccurately to support itit Only 2-3 sentencesOnly 2-3 sentences 12 points12 points A paragraph withA paragraph with explanation thatexplanation that includes supportingincludes supporting evidence with data,evidence with data, but missing “bigbut missing “big picture”, scientificpicture”, scientific error and furthererror and further experiments. Vo-experiments. Vo- cabulary usedcabulary used correctly (4-5 sen-correctly (4-5 sen- tences)tences) 15 points15 points A paragraph with aA paragraph with a logical explanationlogical explanation supported by data thatsupported by data that addresses the objec-addresses the objec- tives, scientifictives, scientific principles and ends withprinciples and ends with the ‘big picture” furtherthe ‘big picture” further experiments orexperiments or unanswered questionsunanswered questions (7 + sentences)(7 + sentences) 20 points20 points UnsatisfactoryUnsatisfactory BorderlineBorderline SatisfactorySatisfactory ExcellentExcellent ScorScor ee
Lab Report ScoresLab Report Scores On-Campus vs. DL Lab Grades 0 10 20 30 40 50 60 70 80 90 100 On-Campus DL Scores Series1
Effectiveness of a Web-Based Laboratory Course in Biology Jennifer Herzog, Herkimer County Community College, N.Y RetentionRetention RateRate MedianMedian GradeGrade LabLab AvgAvg ClassClass AvgAvg % Scoring B% Scoring B or Betteror Better OnOn CampusCampus 89.489.4 CC 75.875.8 75.875.8 4343 OnlineOnline 64.464.4 AA 82.582.5 87.587.5 6262
Summary of FindingsSummary of Findings  DL and F2F exam scores were basically equivalentDL and F2F exam scores were basically equivalent  DL, F2F, and national ACS exam scores were basically equivalentDL, F2F, and national ACS exam scores were basically equivalent  DL lab grades averaged 5% and course grades 1% higher thanDL lab grades averaged 5% and course grades 1% higher than F2FF2F  Institution exams were as effective as ACS exams for assessmentInstitution exams were as effective as ACS exams for assessment Conclusions:Conclusions: Student learning in DL science courses with home basedStudent learning in DL science courses with home based lab kits is at least equivalent to and usually a little betterlab kits is at least equivalent to and usually a little better than in face-to-face courses with a campus based lab.than in face-to-face courses with a campus based lab. Valid assessment can be achieved via institution exams.Valid assessment can be achieved via institution exams.
Summary of PresentationSummary of Presentation  Assessment is a Major and Growing Concern ofAssessment is a Major and Growing Concern of Government, Educators, and InstitutionsGovernment, Educators, and Institutions  Assessment, especially DL assessment, presentsAssessment, especially DL assessment, presents challenges which must be metchallenges which must be met  DL Science Lab Courses Require Valid AssessmentDL Science Lab Courses Require Valid Assessment Methods to be Included in Online Course OfferingsMethods to be Included in Online Course Offerings  DL and F2F Science Lab Course Learning Can BeDL and F2F Science Lab Course Learning Can Be Reliably Assessed and ComparedReliably Assessed and Compared  DL Science Lab Course Learning is Equivalent To orDL Science Lab Course Learning is Equivalent To or Better Than That in F2F Courses.Better Than That in F2F Courses.
Future PlansFuture Plans  Design secure online end-of-courseDesign secure online end-of-course assessment toolsassessment tools  Develop lab skills and safety examDevelop lab skills and safety exam  Correlate ACS scores with ACT/SATCorrelate ACS scores with ACT/SAT or placement math scoresor placement math scores Feedback and Suggestions are Welcome!Feedback and Suggestions are Welcome! pjeschofnig@coloradomtn.edupjeschofnig@coloradomtn.edu
Abstract of PresentationAbstract of Presentation  This Colorado Mountain College Chemistry ProfessorThis Colorado Mountain College Chemistry Professor reviews current complaints about college assessmentreviews current complaints about college assessment and discusses the importance of and means of achievingand discusses the importance of and means of achieving valid and reliable assessment tools. He utilizedvalid and reliable assessment tools. He utilized standardized pre and post semester exams, traditionalstandardized pre and post semester exams, traditional testing, and a lab report grading rubric to quantitativelytesting, and a lab report grading rubric to quantitatively assess and compare the learning of students in both hisassess and compare the learning of students in both his face-to-face and DL online CHE-111 courses. Hisface-to-face and DL online CHE-111 courses. His findings reflect that student learning in his DL onlinefindings reflect that student learning in his DL online Chemistry course with home based lab kits is at leastChemistry course with home based lab kits is at least equivalent to and actually a little better than studentequivalent to and actually a little better than student learning in his face-to-face Chemistry course with alearning in his face-to-face Chemistry course with a traditional campus based lab.traditional campus based lab.

More Related Content

PPTX
Education Futures--Case Studies for United Nations SDG#4
Sara Murdock, Ph.D.
 
PDF
AssessingHIPS_TGGrantReport
Ashley Finley
 
PDF
The Value of Fit: Helping First-Generation Students Find Their Path After Hig...
Nicole Hornsby
 
PPTX
NCII Guided Pathways: Urgency and Transformations
VCCS_ASR
 
PDF
IAU_KU_2011_Wainaina
IsabelleDevylderIAU
 
PPTX
Guided Pathways Demystified I and II - June 2018
VCCS_ASR
 
PPT
12th Grade Senior Parent Night 2014
Jenny Gildon
 
PPT
Is The NCLB Leaving Minorities Behind Presentation
Jerald Wilson
 
Education Futures--Case Studies for United Nations SDG#4
Sara Murdock, Ph.D.
 
AssessingHIPS_TGGrantReport
Ashley Finley
 
The Value of Fit: Helping First-Generation Students Find Their Path After Hig...
Nicole Hornsby
 
NCII Guided Pathways: Urgency and Transformations
VCCS_ASR
 
IAU_KU_2011_Wainaina
IsabelleDevylderIAU
 
Guided Pathways Demystified I and II - June 2018
VCCS_ASR
 
12th Grade Senior Parent Night 2014
Jenny Gildon
 
Is The NCLB Leaving Minorities Behind Presentation
Jerald Wilson
 

Similar to We Teach - But are they Learning? (19)

PDF
RealizingPotentialWP
Maggie Snyder
 
PPT
Jan Sadlak: Measuring Work and Performance of the University
ISCN_Secretariat
 
PDF
Expanding College Opportunities
Terra Williams
 
PPT
090917 Voice Hartford Final
Skidmore College
 
PDF
July2016 2
Dr.Craig Follins
 
PDF
July2016 NISOD Papers
Dr.Craig Follins
 
PPT
Student Retention Foothill College Roundtable
Robert Cormia
 
PDF
PACAC Is Your Campus Visit Keystone to Your Recruitment Success?
TargetX
 
PDF
PACAC 50th "Keystone Campus Visit"
TargetX
 
PPT
Bsi leadership for student success what matters_most_2010
harrindl
 
PDF
IELeaders
David Shipley, Ph.D.
 
PDF
Disrupting the Disruption in Higher Education - SXSWedu 2015
Corinne Weisgerber
 
PDF
Dual Enrollment: A Strategy for Career Readiness Webinar
Hobsons
 
PPT
Matchmaker matchmaker make me a great college list
Rebecca Joseph
 
PPS
Rigor & Relevance Presentation
salovell
 
PDF
BEST NC Shaftesbury Society - August 2015 - FINAL DECK
EducationNC
 
DOC
Crdbase
rperezjr
 
PDF
Revirtiendo la tendencia al fracaso
Archivos de Criminología, Seguridad Privada y Criminalística
 
PDF
McKenna - Dissertation.May 2011
Michael McKenna, Ed.D
 
RealizingPotentialWP
Maggie Snyder
 
Jan Sadlak: Measuring Work and Performance of the University
ISCN_Secretariat
 
Expanding College Opportunities
Terra Williams
 
090917 Voice Hartford Final
Skidmore College
 
July2016 2
Dr.Craig Follins
 
July2016 NISOD Papers
Dr.Craig Follins
 
Student Retention Foothill College Roundtable
Robert Cormia
 
PACAC Is Your Campus Visit Keystone to Your Recruitment Success?
TargetX
 
PACAC 50th "Keystone Campus Visit"
TargetX
 
Bsi leadership for student success what matters_most_2010
harrindl
 
IELeaders
David Shipley, Ph.D.
 
Disrupting the Disruption in Higher Education - SXSWedu 2015
Corinne Weisgerber
 
Dual Enrollment: A Strategy for Career Readiness Webinar
Hobsons
 
Matchmaker matchmaker make me a great college list
Rebecca Joseph
 
Rigor & Relevance Presentation
salovell
 
BEST NC Shaftesbury Society - August 2015 - FINAL DECK
EducationNC
 
Crdbase
rperezjr
 
Revirtiendo la tendencia al fracaso
Archivos de Criminología, Seguridad Privada y Criminalística
 
McKenna - Dissertation.May 2011
Michael McKenna, Ed.D
 
Ad

Recently uploaded (20)

PDF
Lecture on Viruses: Structure, Classification, Replication, Effects on Cells,...
Miraj Khan
 
PDF
LIFE & LIVING TRILOGY- PART (1) WHO ARE WE.pdf
sureshkumarsoni26
 
PPTX
UNIT_2-__LIPIDS[1].pptx.................
Akanksha Kotangale
 
PDF
African Communication Research: A review
AgathaMashindano1
 
PPT
REGULATION OF RESPIRATION lecture note 200L [Autosaved]-1-1.ppt
muktarabdulwahab7
 
PDF
The TKT Course. Modules 1, 2, 3.for self study
Aydelina Medina
 
PDF
Horaris_Grups_25-26_Definitiu_15_07_25.pdf
rpujol11
 
PDF
Chevening Scholarship Application and Interview Preparation Guide
Leneka Rhoden
 
PPTX
Thinking Routines and Learning Engagements.pptx
n646vnqyjz
 
PPTX
pharmaceutics-1unit-1-221214121936-550b56aa.pptx
zonuntluangimph
 
PDF
Nurlina - Urban Planner Portfolio (english ver)
Nurlina Y.
 
PDF
LIFE & LIVING TRILOGY - PART - (2) THE PURPOSE OF LIFE.pdf
sureshkumarsoni26
 
PDF
Compact First Student's Book Cambridge Official
TunHng48
 
PDF
Everyday Spelling and Grammar by Kathi Wyldeck
JohanaWulandari2
 
PPTX
Reproductive system-Human anatomy and physiology
ZenyTilwani1
 
PDF
Mucosal Drug Delivery system_NDDS_BPHARMACY__SEM VII_PCI Syllabus.pdf
SabsKhan1
 
PDF
fundamentals-of-heat-and-mass-transfer-6th-edition_incropera.pdf
gloryjsingh
 
PDF
Civil Department's presentation Your score increases as you pick a category
OperaVill
 
PDF
LIFE & LIVING TRILOGY - PART (3) REALITY & MYSTERY.pdf
sureshkumarsoni26
 
PPTX
PLASMA AND ITS CONSTITUENTS 123.pptx
sweet2th18
 
Lecture on Viruses: Structure, Classification, Replication, Effects on Cells,...
Miraj Khan
 
LIFE & LIVING TRILOGY- PART (1) WHO ARE WE.pdf
sureshkumarsoni26
 
UNIT_2-__LIPIDS[1].pptx.................
Akanksha Kotangale
 
African Communication Research: A review
AgathaMashindano1
 
REGULATION OF RESPIRATION lecture note 200L [Autosaved]-1-1.ppt
muktarabdulwahab7
 
The TKT Course. Modules 1, 2, 3.for self study
Aydelina Medina
 
Horaris_Grups_25-26_Definitiu_15_07_25.pdf
rpujol11
 
Chevening Scholarship Application and Interview Preparation Guide
Leneka Rhoden
 
Thinking Routines and Learning Engagements.pptx
n646vnqyjz
 
pharmaceutics-1unit-1-221214121936-550b56aa.pptx
zonuntluangimph
 
Nurlina - Urban Planner Portfolio (english ver)
Nurlina Y.
 
LIFE & LIVING TRILOGY - PART - (2) THE PURPOSE OF LIFE.pdf
sureshkumarsoni26
 
Compact First Student's Book Cambridge Official
TunHng48
 
Everyday Spelling and Grammar by Kathi Wyldeck
JohanaWulandari2
 
Reproductive system-Human anatomy and physiology
ZenyTilwani1
 
Mucosal Drug Delivery system_NDDS_BPHARMACY__SEM VII_PCI Syllabus.pdf
SabsKhan1
 
fundamentals-of-heat-and-mass-transfer-6th-edition_incropera.pdf
gloryjsingh
 
Civil Department's presentation Your score increases as you pick a category
OperaVill
 
LIFE & LIVING TRILOGY - PART (3) REALITY & MYSTERY.pdf
sureshkumarsoni26
 
PLASMA AND ITS CONSTITUENTS 123.pptx
sweet2th18
 
Ad

We Teach - But are they Learning?

  • 1. We Teach …We Teach … But are they Learning?But are they Learning? Assessing Student LearningAssessing Student Learning inin D.L. Lab Science CoursesD.L. Lab Science Courses Peter Jeschofnig, Ph.D.Peter Jeschofnig, Ph.D. Colorado Mountain CollegeColorado Mountain College 1212thth Sloan-C International ConferenceSloan-C International Conference Nov 9, 2006Nov 9, 2006
  • 2. Atlantic MonthlyAtlantic Monthly : Nov 2005: Nov 2005 .. "What Does College"What Does College Teach?Teach? It's time to put an end toIt's time to put an end to 'faith-based' acceptance of'faith-based' acceptance of higher education's quality“higher education's quality“ Richard HershRichard Hersh,, former president of Hobart & William Smith Collegesformer president of Hobart & William Smith Colleges and Trinity Collegeand Trinity College Co-director of the "Collegiate Learning Assessment"Co-director of the "Collegiate Learning Assessment"  Hersh & Merrow – 2005Hersh & Merrow – 2005 Declining by Degrees: Higher Education at RiskDeclining by Degrees: Higher Education at Risk
  • 3. Institutional Level AssessmentInstitutional Level Assessment All institutions have some form of assessment,All institutions have some form of assessment, often linked to accreditationoften linked to accreditation Of 1,393 public and private institutions surveyedOf 1,393 public and private institutions surveyed in 1999, 82% listed “Excellence inin 1999, 82% listed “Excellence in Undergraduate Education” as part of missionUndergraduate Education” as part of mission statement.statement. However: Direct measures of studentHowever: Direct measures of student learning remain rare!learning remain rare!
  • 4. Secretary of Education’s Commission onSecretary of Education’s Commission on The Future of Higher EducationThe Future of Higher Education Chairman Charles Miller, believes that collegesChairman Charles Miller, believes that colleges must better measure the skills and knowledgemust better measure the skills and knowledge they impart to students, and openly share thatthey impart to students, and openly share that information with the public.information with the public. ““We need to assure that the American publicWe need to assure that the American public understand through access to sufficientunderstand through access to sufficient information, particularly in the area of studentinformation, particularly in the area of student learning, what they are getting for theirlearning, what they are getting for their investment in a college education.”investment in a college education.” "No College Left Behind?""No College Left Behind?" Inside Higher Ed,Inside Higher Ed, February 15, 2006February 15, 2006
  • 5. Secretary of Education’s Commission onSecretary of Education’s Commission on The Future of Higher EducationThe Future of Higher Education Feb. 3, 2006 meeting in San Diego:Feb. 3, 2006 meeting in San Diego: Higher education institutions must be more “transparent”Higher education institutions must be more “transparent” in collecting and giving the public useful informationin collecting and giving the public useful information about their activities and their performance.about their activities and their performance. Topics about which colleges should provide more andTopics about which colleges should provide more and better information, the various panels suggested, are onbetter information, the various panels suggested, are on their costs and prices, how their graduates fare in thetheir costs and prices, how their graduates fare in the employment market, and their success in impartingemployment market, and their success in imparting knowledge and, more importantly, skills like criticalknowledge and, more importantly, skills like critical thinking, to their students.thinking, to their students.
  • 6. National Forum onNational Forum on College-Level Learning (NFCLL)College-Level Learning (NFCLL) "I'm disappointed that NASULGC ("I'm disappointed that NASULGC (National Association ofNational Association of State Universities and Land-Grant Colleges)State Universities and Land-Grant Colleges) seems to beseems to be interested in describing ‘the skills and knowledge thatinterested in describing ‘the skills and knowledge that students bring to college’ but evidently not the ones theystudents bring to college’ but evidently not the ones they leave with.leave with. While it's considering publishing ‘data on graduation rates,While it's considering publishing ‘data on graduation rates, admissions, applications, student demographics andadmissions, applications, student demographics and faculty demographics,’ THE OBVIOUS OMISSION OFfaculty demographics,’ THE OBVIOUS OMISSION OF EVIDENCE OF STUDENT LEARNING SUGGESTS THATEVIDENCE OF STUDENT LEARNING SUGGESTS THAT THEY HAVEN'T HEARD THE CLEAR MESSAGE THAT'STHEY HAVEN'T HEARD THE CLEAR MESSAGE THAT'S BEEN COMING FROM THE STATES AND THE FEDSBEEN COMING FROM THE STATES AND THE FEDS FOR ABOUT 20 YEARS: THAT THEY WANT TO KNOWFOR ABOUT 20 YEARS: THAT THEY WANT TO KNOW WHAT OUR GRADUATES KNOW AND CAN DO.“WHAT OUR GRADUATES KNOW AND CAN DO.“ Margaret (Peg) MillerMargaret (Peg) Miller 2006 Director of NFCLL2006 Director of NFCLL
  • 7. . . ."We expect transparency and accountability for our tax. . ."We expect transparency and accountability for our tax dollars in almost every area of our government—fromdollars in almost every area of our government—from housing to health care to K-12 education. But in higherhousing to health care to K-12 education. But in higher education, we've invested tens of billions of taxpayereducation, we've invested tens of billions of taxpayer dollars over the years and basically just hoped for thedollars over the years and basically just hoped for the best.“best.“ PRESS RELEASE: Nov. 2, 2006PRESS RELEASE: Nov. 2, 2006 Secretary Spellings Delivered Remarks at NationalSecretary Spellings Delivered Remarks at National Postsecondary Education Cooperative Symposium onPostsecondary Education Cooperative Symposium on Student Success in Washington, D.C.Student Success in Washington, D.C.
  • 8. Threat of Mandated AssessmentThreat of Mandated Assessment Government mandated assessments have manyGovernment mandated assessments have many negative ramifications that should be avoided.negative ramifications that should be avoided. However, that should not deter academicHowever, that should not deter academic professionals from designing valid and reliableprofessionals from designing valid and reliable assessment of learning.assessment of learning. A pro-active academic approach to assessment isA pro-active academic approach to assessment is surely better than a government mandated one!surely better than a government mandated one!
  • 9. Assessment ApproachesAssessment Approaches  Actuarial Data and Expert Ratings:Actuarial Data and Expert Ratings: GraduationGraduation rates; research funding; student-teacher ratios, ACT &rates; research funding; student-teacher ratios, ACT & SAT scores; admissions selectivitySAT scores; admissions selectivity  Student Ratings:Student Ratings: Survey of Student EngagementSurvey of Student Engagement (NSSE) Students rate educational experience on(NSSE) Students rate educational experience on quantity and quality of faculty contact; homework,quantity and quality of faculty contact; homework, assignments, etc.; Noel-Levitz Student Satisfactionassignments, etc.; Noel-Levitz Student Satisfaction Inventory;Inventory;  Direct Assessment:Direct Assessment: Via grades & grade pointVia grades & grade point averagesaverages  Nearly useless as learning indicatorsNearly useless as learning indicators  Grade InflationGrade Inflation  Tell nothing about knowledge gained, retained, or ifTell nothing about knowledge gained, retained, or if knowledge can be applied to new situationsknowledge can be applied to new situations  Reflect lower level objectives such as facts and definitionsReflect lower level objectives such as facts and definitions rather than higher objectives & critical thinking skillsrather than higher objectives & critical thinking skills
  • 10. Assessment Approaches – cont.Assessment Approaches – cont.  pre-professional programs withpre-professional programs with specialized accreditation programs whichspecialized accreditation programs which require evidence of student learningrequire evidence of student learning  Rising Junior examRising Junior exam
  • 11. A Learning College ApproachA Learning College Approach ““The Learning College and its learningThe Learning College and its learning facilitators succeedfacilitators succeed only whenonly when improvedimproved and expandedand expanded learning can belearning can be documenteddocumented for its learners.”for its learners.” O’Banion,O’Banion, 19971997 The Primary Questions ForThe Primary Questions For Every Learning CollegeEvery Learning College Action:Action:  Does this action improveDoes this action improve or expand learning?or expand learning?  How do we know?How do we know?
  • 12. Assessment ...Assessment ... ... is the systematic, on-going,... is the systematic, on-going, iterative process of monitoringiterative process of monitoring learning in order to determinelearning in order to determine what we are doing well and whatwhat we are doing well and what we must improve.we must improve.
  • 13. We Want and Need to Know…We Want and Need to Know… …… and we want students, employers, peers,and we want students, employers, peers, policy makers, and the public to know …policy makers, and the public to know … how well students are able to use thehow well students are able to use the complex knowledge and abilitiescomplex knowledge and abilities articulated as important to their learning.articulated as important to their learning.
  • 14. Basic Course Assessment TypesBasic Course Assessment Types  Formative Assessment:Formative Assessment: Ongoing assessment used to modifyOngoing assessment used to modify instruction and improve learninginstruction and improve learning  Summative Assessment:Summative Assessment: End of class or program assessment to verifyEnd of class or program assessment to verify that learning objectives have been met. Thisthat learning objectives have been met. This is compared to other classes and institutionsis compared to other classes and institutions to gage the effectiveness of courses andto gage the effectiveness of courses and instructional programs.instructional programs.
  • 15. Assessment is EspeciallyAssessment is Especially Effective When It ...Effective When It ...  is student centeredis student centered  is congruent with instructional objectivesis congruent with instructional objectives  is relevantis relevant  is comprehensiveis comprehensive  is clear in purpose, directions, and expectationsis clear in purpose, directions, and expectations  is objective and fairis objective and fair  simulates "end" behavior/product/performancesimulates "end" behavior/product/performance  incites active responsesincites active responses  shows progress/development over timeshows progress/development over time
  • 16. Assessment ChallengesAssessment Challenges  Disagreements on what should be taughtDisagreements on what should be taught  Well-conceived programs take time,Well-conceived programs take time, energy, and moneyenergy, and money  Avoiding the problems experienced in “NoAvoiding the problems experienced in “No child left behind” assessmentchild left behind” assessment  Threat to academic freedom?Threat to academic freedom?  Classes looking too much alike?Classes looking too much alike?
  • 17. Distance LearningDistance Learning is Here to Stay!is Here to Stay! Online enrollments continue to grow at rates faster than forOnline enrollments continue to grow at rates faster than for the overall student body…the overall student body… Schools expect the rate of growth to further increase andSchools expect the rate of growth to further increase and believe that online learning is critical to their long termbelieve that online learning is critical to their long term strategy…strategy… Three quarters of academic leaders at public colleges andThree quarters of academic leaders at public colleges and universities believe thatuniversities believe that online learning qualityonline learning quality is equal tois equal to or superior to face-to-face instructionor superior to face-to-face instruction. And they expect. And they expect online offerings to continue to get better.online offerings to continue to get better. The Sloan Consortium, Nov. 2004The Sloan Consortium, Nov. 2004 Entering the MainstreamEntering the Mainstream The Quality and Extent ofThe Quality and Extent of Online EducationOnline Education in the United States,in the United States, 2003 and 20042003 and 2004
  • 18. Specific Assessment ChallengesSpecific Assessment Challenges for Distance Educationfor Distance Education Using Nationally Normed Exams:Using Nationally Normed Exams:  Exam content must remain confidentialExam content must remain confidential  Require proctored examsRequire proctored exams  Cost to institutionCost to institution  Inconvenience to studentInconvenience to student Institution Designed Exams:Institution Designed Exams:  ValidityValidity  ReliabilityReliability
  • 19. What About Lab Sciences?What About Lab Sciences? If we are to avoid seeing a continuingIf we are to avoid seeing a continuing decline in science literacy in America,decline in science literacy in America, lab sciences MUST be fully included inlab sciences MUST be fully included in the increasing mix of online coursethe increasing mix of online course offerings.offerings. However, there are still many instructorsHowever, there are still many instructors and institutions that do not believe laband institutions that do not believe lab sciences can effectively be taught at asciences can effectively be taught at a distance. Valid and reliable assessmentdistance. Valid and reliable assessment data is required to dispel thisdata is required to dispel this misconception.misconception.
  • 20. Assessing Online ScienceAssessing Online Science There is ample anecdotal evidence ofThere is ample anecdotal evidence of student learning and satisfaction instudent learning and satisfaction in distance science courses that usedistance science courses that use home-based lab kits to fulfillhome-based lab kits to fulfill laboratory requirements.laboratory requirements. However, there has been littleHowever, there has been little quantitative data to support thisquantitative data to support this positive conclusionpositive conclusion..
  • 21. Objective of this StudyObjective of this Study To quantitatively assess and compare theTo quantitatively assess and compare the performance of my chemistry studentsperformance of my chemistry students In a face-to-face (F2F) chemistry courseIn a face-to-face (F2F) chemistry course with an on-campus laboratorywith an on-campus laboratory andand In an online chemistry (DL) courseIn an online chemistry (DL) course with a home-based laboratory kitwith a home-based laboratory kit
  • 22. Process of Assessing OutcomesProcess of Assessing Outcomes Administer and Compare Results for Campus-BasedAdminister and Compare Results for Campus-Based CHE-111 Students and Online CHE-111 Students:CHE-111 Students and Online CHE-111 Students: 1.1. American Chemical Society Standardized ExamAmerican Chemical Society Standardized Exam Pre-test at the beginning of the semester andPre-test at the beginning of the semester and Post-test at the end of the coursePost-test at the end of the course 2. Traditional homework, quiz, and exam grades2. Traditional homework, quiz, and exam grades 3. Laboratory reports graded via a specific rubric3. Laboratory reports graded via a specific rubric
  • 23. American Chemical SocietyAmerican Chemical Society (ACS) General Chemistry I Exam(ACS) General Chemistry I Exam Scores based on 3007 students in 20 collegesScores based on 3007 students in 20 colleges and universities, including:and universities, including: University of AlabamaUniversity of Alabama Albuquerque Technical CollegeAlbuquerque Technical College Mercer University Kennesaw State UniversityMercer University Kennesaw State University Jamestown CCJamestown CC Florida Southern UniversityFlorida Southern University Miami UniversityMiami University Monroe CC;Monroe CC; University Pittsburgh University of N.CarolinaUniversity Pittsburgh University of N.Carolina
  • 24. ACS Exam Results: F2FACS Exam Results: F2F CHE 111 - ACS Exam 0 10 20 30 40 50 60 70 pre-test-% post-test-% National mean-% Pre-test & Gain; National Score %Score Series2 Series1
  • 25. ACS Exam Results: DLACS Exam Results: DL CHE 111 (DL) - ACS Exam 0 10 20 30 40 50 60 70 Pre-test % Post-test % National mean Pre-test, Post-test, National score ACSExamScores Series2 Series1
  • 26. ACS Exam Results: DL vs. F2FACS Exam Results: DL vs. F2F ACS Score Comparison 0 10 20 30 40 50 60 70 80 90 DL-Pre- Test F2F-Pre- Test DL-Post- Test F2F-Post- Test ACS National DL Final Exam F2F-Final Exa, ExamScores Raw Score %
  • 27. ACS Score ComparisonsACS Score Comparisons MeanMean S.D.S.D. MedianMedian F2FF2F n=24n=24 40.440.4 11.811.8 4242 DLDL n=15n=15 40.940.9 12.212.2 4141 NationalNational n=3007n=3007 41.741.7 12.312.3 41.241.2
  • 28. ACS Post-test Scores: F2FACS Post-test Scores: F2F ACS Post Test Scores In-House 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Students Scores Series2
  • 29. ACS Post-test Scores: DLACS Post-test Scores: DL ACS Post Test Scores - DL 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Students Scores Series2
  • 30. Final Exam vs. ACS Exam – F2FFinal Exam vs. ACS Exam – F2F Final Exam vs ACS Exam y = 0.5233x + 49.372 R 2 = 0.3756 0 20 40 60 80 100 120 0 20 40 60 80 100 ACS Score FinalExamScore Series1 Linear (Series1)
  • 31. Final Exam vs. ACS Exam – DLFinal Exam vs. ACS Exam – DL Final Exam vs ACS Exam - DL y = 0.8231x + 32.366 R 2 = 0.5917 0 20 40 60 80 100 120 0 20 40 60 80 100 ACS Exam Scores FinalExamScores Series1 Linear (Series1)
  • 32. Lab Report Grading RubricLab Report Grading Rubric Title Page -Title Page - 5 points5 points Succinct and descriptive title and experiment number, author’s name, partners’Succinct and descriptive title and experiment number, author’s name, partners’ names, course name and number; date of experiment, date of lab report.names, course name and number; date of experiment, date of lab report. Abstract –Abstract – 10 points10 points A brief one to two paragraph statement of the purpose of the experiment and theA brief one to two paragraph statement of the purpose of the experiment and the results (i.e. relative yield, identification of unknown, etc).results (i.e. relative yield, identification of unknown, etc). 10 points10 points Purpose/Hypothesis -Purpose/Hypothesis - 10 points10 points A detailed statement about the experiment’s purpose and hypothesis (yourA detailed statement about the experiment’s purpose and hypothesis (your predictions). Describe what you think the likely outcome of the experiment will be;predictions). Describe what you think the likely outcome of the experiment will be; what scientific principle or law will be tested; what scientific relationship will be shown.what scientific principle or law will be tested; what scientific relationship will be shown. Hypothesis should include any relationship between variables and should mention theHypothesis should include any relationship between variables and should mention the independent vs. dependent variable.independent vs. dependent variable. Procedure –Procedure – 10 points10 points :: If you followed a detailed procedure from the lab manual, a very short procedureIf you followed a detailed procedure from the lab manual, a very short procedure summary will be enough.summary will be enough. If you used a procedure not detailed in the lab report, youIf you used a procedure not detailed in the lab report, you need to write the procedure in enough detail, using numbered steps, so that someoneneed to write the procedure in enough detail, using numbered steps, so that someone should be able to repeat your procedure.should be able to repeat your procedure. This section should include any alterations or errors you made to your operatingThis section should include any alterations or errors you made to your operating methodology. Be thorough and specific! Another chemist should be able to read yourmethodology. Be thorough and specific! Another chemist should be able to read your procedure and reproduce your experiment with precisionprocedure and reproduce your experiment with precision.. This section should notThis section should not contain any results/data from the exercise. 10contain any results/data from the exercise. 10 pointspoints
  • 33. Data/ObservationsData/Observations -- 25 points25 points A detailed presentation of all hard data gathered during the experiment, as well as anA detailed presentation of all hard data gathered during the experiment, as well as an organized presentation of all your observations during the lab. Describe not only whatorganized presentation of all your observations during the lab. Describe not only what you see, but what it signifies in chemical terms. What is really happening? Also,you see, but what it signifies in chemical terms. What is really happening? Also, answer any questions the lab poses here.answer any questions the lab poses here. Tables should be used whenever possible. Graphs need to be complete with titles,Tables should be used whenever possible. Graphs need to be complete with titles, axes must be labeled; the independent variable should be on the x-axis, theaxes must be labeled; the independent variable should be on the x-axis, the dependent variable on the y-axis. Any calculations should be in this section as well.dependent variable on the y-axis. Any calculations should be in this section as well. They should be presented clearly and explained. The source of all numbers used inThey should be presented clearly and explained. The source of all numbers used in calculations should be included.calculations should be included. Results/Analysis -Results/Analysis - 2020 pointspoints A comprehensive, thoughtful discussion of what your data means and how it provedA comprehensive, thoughtful discussion of what your data means and how it proved or disproved your hypothesis, including the relationship of the variables as presentedor disproved your hypothesis, including the relationship of the variables as presented by the data. Explain any trends in the data that will be used. You should present theby the data. Explain any trends in the data that will be used. You should present the evidence that will support your conclusions. Error analysis, including percent error,evidence that will support your conclusions. Error analysis, including percent error, should be included here. Are the results consistent within the limitations of equipmentshould be included here. Are the results consistent within the limitations of equipment and random error. All questions should be answered here.and random error. All questions should be answered here. Conclusion -Conclusion - 20 points20 points A detailed discussion of how the lab results compare with your predictions, and whatA detailed discussion of how the lab results compare with your predictions, and what the results mean in a practical, real world sense. How can your discoveries bethe results mean in a practical, real world sense. How can your discoveries be applied? Have you verified your hypothesis? Did you demonstrate the scientificapplied? Have you verified your hypothesis? Did you demonstrate the scientific principle of this experiment adequately?principle of this experiment adequately?
  • 34. UnsatisfactoryUnsatisfactory BorderlineBorderline SatisfactorySatisfactory ExcellentExcellent ScorScor ee Title PageTitle Page Max = 5 ptsMax = 5 pts Missing more thanMissing more than two items, or title ortwo items, or title or namesnames 0-2 points0-2 points Contains tile &Contains tile & names; two othernames; two other items missingitems missing 3 point3 point Contains tile &Contains tile & names; one othernames; one other item missingitem missing 4 points4 points Contains title, author’sContains title, author’s name, partners’ names,name, partners’ names, course name;course name; experiment number;experiment number; experiment & reportexperiment & report dates.dates. 5 points5 points AbstractAbstract Max = 10Max = 10 ptspts No abstract, orNo abstract, or Incomplete purpose,Incomplete purpose, incompleteincomplete ResultsResults 0-3 points0-3 points Purpose or results,Purpose or results, but not bothbut not both 5 points5 points Contains purposeContains purpose statement andstatement and results, but someresults, but some result detailsresult details missing.missing. 8 pts.8 pts. Contains purposeContains purpose statement and completestatement and complete resultsresults 10 points10 points Purpose/Purpose/ HypothesisHypothesis Max = 10Max = 10 ptspts IncompleteIncomplete purpose/hypothpurpose/hypoth Statement,Statement, Incomplete or missingIncomplete or missing scientific principles orscientific principles or variablesvariables 0-4 points0-4 points Clearly states theClearly states the correct purposecorrect purpose /Hypothesis, Both/Hypothesis, Both scientific principlesscientific principles or vari-ables missingor vari-ables missing 6 points6 points Clearly states theClearly states the correct purpose/correct purpose/ Hypothesis, eitherHypothesis, either scientific principlesscientific principles or variablesor variables missingmissing 8 points8 points Clearly states theClearly states the correctcorrect purpose/hypothesis,purpose/hypothesis, which scientificwhich scientific principles to be tested;principles to be tested; the variables involvedthe variables involved 10 points10 points Grading Rubric PointGrading Rubric Point Allocation for Lab ReportsAllocation for Lab Reports
  • 35. Results/Results/ AnalysisAnalysis Max = 20Max = 20 ptspts Explanation of data isExplanation of data is missing, inaccurate,missing, inaccurate, or not expressed inor not expressed in complete sentences.complete sentences. Error analysisError analysis incomplete, missingincomplete, missing or incorrector incorrect 0-8 points0-8 points IncompleteIncomplete description of data;description of data; 3 or more important3 or more important observations areobservations are missing. Errormissing. Error analysis incompleteanalysis incomplete or only partiallyor only partially correctcorrect 12 points12 points Results statedResults stated correctly in com-correctly in com- plete sentences.plete sentences. No more than 1-2No more than 1-2 important obser-important obser- vations arevations are missing. . Errormissing. . Error analysis presentanalysis present and correctand correct Complete description ofComplete description of what occurred in thewhat occurred in the experiment is stated inexperiment is stated in complete sentences.complete sentences. Data is used accuratelyData is used accurately in reporting/analyzingin reporting/analyzing the results. Error analy-the results. Error analy- sis present and correctsis present and correct 20 points20 points Data/Data/ ObservationObservation Max = 25Max = 25 ptspts Data is missing,Data is missing, incomplete,incomplete, inaccurate, or hasinaccurate, or has several defects;several defects; No data tables whenNo data tables when appropriate, missingappropriate, missing graphs; most or allgraphs; most or all observations missingobservations missing Incomplete or noIncomplete or no calculations, fewcalculations, few questions answeredquestions answered 0-12 points0-12 points Data presented, butData presented, but not well organized,not well organized, some data inaccu-some data inaccu- rate or missing;rate or missing; graphs are inaccu-graphs are inaccu- rate in data display,rate in data display, incorrectly or notincorrectly or not labeled; poor orlabeled; poor or incomplete observa-incomplete observa- tions Poor & incom-tions Poor & incom- plete calculation;plete calculation; some questionssome questions answered.answered. 16 pts16 pts Data presentedData presented clearly and neatly.clearly and neatly. Most charts, tables,Most charts, tables, diagrams, anddiagrams, and graphs labeled andgraphs labeled and accurate; detailedaccurate; detailed and accurate ob-and accurate ob- servations. Mostservations. Most calculations showncalculations shown and correct. Mostand correct. Most questionsquestions answeredanswered 20 points20 points Data presented clearlyData presented clearly and neatly. All charts,and neatly. All charts, tables, diagrams, andtables, diagrams, and graphs labeled andgraphs labeled and accurate; Appropriateaccurate; Appropriate graphing of data chosengraphing of data chosen (line/bar/pie); detailed(line/bar/pie); detailed and accurate observa-and accurate observa- tions. Calculationstions. Calculations shown and correct. Allshown and correct. All questions answered inquestions answered in this section.this section. 25 points25 points ProcedureProcedure Max = 10Max = 10 ptspts Includes an unclearIncludes an unclear set of instructions.set of instructions. Most steps may be:Most steps may be: Missing. IncompleteMissing. Incomplete disorganized, con-disorganized, con- fusing,,difficult tofusing,,difficult to follow,not sequential,follow,not sequential, 0-4 points0-4 points Includes an nclearIncludes an nclear set of instructions.set of instructions. Some steps may beSome steps may be missing. Incomplete,missing. Incomplete, disorganized,confus-disorganized,confus- ing,difficult to follow,ing,difficult to follow, not sequentialnot sequential 6 pts6 pts Includes a clear setIncludes a clear set of instructions. Aof instructions. A few steps are:few steps are: missing,incomplete,missing,incomplete, disorganized,disorganized, difficult to followdifficult to follow 8 points8 points Directions in clear,Directions in clear, concise sentences in aconcise sentences in a step-by-step format.step-by-step format. Experiment can clearlyExperiment can clearly be replicated. Materialsbe replicated. Materials are included in theare included in the methodmethod 10 points10 points UnsatisfactoryUnsatisfactory BorderlineBorderline SatisfactorySatisfactory ExcellentExcellent ScorScor ee
  • 36. Cover letterCover letter (when(when applicable)applicable) Max = 20Max = 20 ptspts Hypothesis or goal isHypothesis or goal is severely incompleteseverely incomplete or missing; results ofor missing; results of data interpretationdata interpretation severely incompleteseverely incomplete or missing; relation-or missing; relation- ship to scientificship to scientific principles severelyprinciples severely incomplete or missingincomplete or missing 0-8 points0-8 points Hypothesis or goal isHypothesis or goal is incomplete; resultsincomplete; results of data interpretationof data interpretation incomplete;incomplete; relationship torelationship to scientific principlesscientific principles incompleteincomplete 12 points12 points Hypothesis or goalHypothesis or goal is slightlyis slightly incomplete; resultsincomplete; results of dataof data interpretationinterpretation slightly incomplete;slightly incomplete; relationship torelationship to scientific principlesscientific principles slightly incompleteslightly incomplete 15 points15 points Concise description ofConcise description of hypothesis or goal, veryhypothesis or goal, very brief procedure, generalbrief procedure, general description of data,description of data, results of dataresults of data interpretation,interpretation, relationship to scientificrelationship to scientific principles (rest similar toprinciples (rest similar to conclusions above)conclusions above) 20 points20 points ConclusionsConclusions Max =20 ptsMax =20 pts Conclusion is missingConclusion is missing or does not fullyor does not fully explain the objectivesexplain the objectives of the lab. Relevantof the lab. Relevant vocabulary missing.vocabulary missing. No practical applica-No practical applica- tion given; Discus-tion given; Discus- sion of scientificsion of scientific principle missing.principle missing. Only 1-2 sentencesOnly 1-2 sentences 0-8 points0-8 points Conclusion explainsConclusion explains the objecive, butthe objecive, but data not useddata not used accurately to supportaccurately to support itit Only 2-3 sentencesOnly 2-3 sentences 12 points12 points A paragraph withA paragraph with explanation thatexplanation that includes supportingincludes supporting evidence with data,evidence with data, but missing “bigbut missing “big picture”, scientificpicture”, scientific error and furthererror and further experiments. Vo-experiments. Vo- cabulary usedcabulary used correctly (4-5 sen-correctly (4-5 sen- tences)tences) 15 points15 points A paragraph with aA paragraph with a logical explanationlogical explanation supported by data thatsupported by data that addresses the objec-addresses the objec- tives, scientifictives, scientific principles and ends withprinciples and ends with the ‘big picture” furtherthe ‘big picture” further experiments orexperiments or unanswered questionsunanswered questions (7 + sentences)(7 + sentences) 20 points20 points UnsatisfactoryUnsatisfactory BorderlineBorderline SatisfactorySatisfactory ExcellentExcellent ScorScor ee
  • 37. Lab Report ScoresLab Report Scores On-Campus vs. DL Lab Grades 0 10 20 30 40 50 60 70 80 90 100 On-Campus DL Scores Series1
  • 38. Effectiveness of a Web-Based Laboratory Course in Biology Jennifer Herzog, Herkimer County Community College, N.Y RetentionRetention RateRate MedianMedian GradeGrade LabLab AvgAvg ClassClass AvgAvg % Scoring B% Scoring B or Betteror Better OnOn CampusCampus 89.489.4 CC 75.875.8 75.875.8 4343 OnlineOnline 64.464.4 AA 82.582.5 87.587.5 6262
  • 39. Summary of FindingsSummary of Findings  DL and F2F exam scores were basically equivalentDL and F2F exam scores were basically equivalent  DL, F2F, and national ACS exam scores were basically equivalentDL, F2F, and national ACS exam scores were basically equivalent  DL lab grades averaged 5% and course grades 1% higher thanDL lab grades averaged 5% and course grades 1% higher than F2FF2F  Institution exams were as effective as ACS exams for assessmentInstitution exams were as effective as ACS exams for assessment Conclusions:Conclusions: Student learning in DL science courses with home basedStudent learning in DL science courses with home based lab kits is at least equivalent to and usually a little betterlab kits is at least equivalent to and usually a little better than in face-to-face courses with a campus based lab.than in face-to-face courses with a campus based lab. Valid assessment can be achieved via institution exams.Valid assessment can be achieved via institution exams.
  • 40. Summary of PresentationSummary of Presentation  Assessment is a Major and Growing Concern ofAssessment is a Major and Growing Concern of Government, Educators, and InstitutionsGovernment, Educators, and Institutions  Assessment, especially DL assessment, presentsAssessment, especially DL assessment, presents challenges which must be metchallenges which must be met  DL Science Lab Courses Require Valid AssessmentDL Science Lab Courses Require Valid Assessment Methods to be Included in Online Course OfferingsMethods to be Included in Online Course Offerings  DL and F2F Science Lab Course Learning Can BeDL and F2F Science Lab Course Learning Can Be Reliably Assessed and ComparedReliably Assessed and Compared  DL Science Lab Course Learning is Equivalent To orDL Science Lab Course Learning is Equivalent To or Better Than That in F2F Courses.Better Than That in F2F Courses.
  • 41. Future PlansFuture Plans  Design secure online end-of-courseDesign secure online end-of-course assessment toolsassessment tools  Develop lab skills and safety examDevelop lab skills and safety exam  Correlate ACS scores with ACT/SATCorrelate ACS scores with ACT/SAT or placement math scoresor placement math scores Feedback and Suggestions are Welcome!Feedback and Suggestions are Welcome! [email protected]@coloradomtn.edu
  • 42. Abstract of PresentationAbstract of Presentation  This Colorado Mountain College Chemistry ProfessorThis Colorado Mountain College Chemistry Professor reviews current complaints about college assessmentreviews current complaints about college assessment and discusses the importance of and means of achievingand discusses the importance of and means of achieving valid and reliable assessment tools. He utilizedvalid and reliable assessment tools. He utilized standardized pre and post semester exams, traditionalstandardized pre and post semester exams, traditional testing, and a lab report grading rubric to quantitativelytesting, and a lab report grading rubric to quantitatively assess and compare the learning of students in both hisassess and compare the learning of students in both his face-to-face and DL online CHE-111 courses. Hisface-to-face and DL online CHE-111 courses. His findings reflect that student learning in his DL onlinefindings reflect that student learning in his DL online Chemistry course with home based lab kits is at leastChemistry course with home based lab kits is at least equivalent to and actually a little better than studentequivalent to and actually a little better than student learning in his face-to-face Chemistry course with alearning in his face-to-face Chemistry course with a traditional campus based lab.traditional campus based lab.