Sample DevelopinganInstrument
Sample DevelopinganInstrument
© REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
ABSTRACT
This paper focuses on the design and development of an educational survey instrument that
will effectively measure the participants’ self-rating of competence and professional confidence
acquired through a given faculty development program. To develop the instrument, a nine-
months long engineering faculty development program - IUCEE International Engineering
Educator Certification Program (IIEECP) was chosen, and the developed instrument was
implemented on Indian IIEECP certified faculty. The IIEECP program is a specially designed
certification program designed to improve the pedagogical acumen and professional
confidence of Indian engineering educators. For this study a total of 193 participants were
recruited and effort was made to capture as diverse a population as possible. The sample
included 59 percent women and 41 percent men teaching different engineering disciplines in
different types of engineering institutions in India. The survey instrument is designed in three
part that include i) demographic analysis, ii) a 39-item questionnaire related to the achievement
of specified learning outcomes of the IIEECP program, and iii) a set of six qualitative questions
designed to help participants rate their enhanced competencies and professional confidence.
An exploratory factor analysis (EFA) was conducted to examine the factor structure of the
survey instrument under consideration. The EFA revealed six distinct factors each
corresponding the six different modules. The Cronbach’s alpha for the six factors ranged
between 0.82 and 0.87, indicating high internal consistency between the items. The study
serves as an effective measurement tool for faculty, engineering institutions as well as the
IIEECP expert team. For the faculty, it provides a practical tool for self-reflection; for the
institutions it allows to develop criteria for faculty readiness and identify their training needs.
For the IIEECP team it provides invaluable feedback to further refine and reinforce the
program. The designed instrument demonstrates how the efficacy of faculty development
programs can be measured through participants rating of acquired competencies and
confidence. One of the limitations of this work is that the evidence for content validity was not
collected. The instrument will benefit from evidence collected from the expert team teaching
and evaluating each module of IIEECP. Investigating the influence of participants’
demographic variables on participants’ performance and professional confidence is another
direction for future work.
Keywords: effective teaching, exploratory factor analysis, faculty development, international
certification program, survey instrument
Introduction
India is known to be the global hub of engineering education with over 1.5 million engineers
graduating every year. Today, India has over 3,500 engineering institutions that can be
classified in different tiers. Except for the graduates coming from elite, top ranking engineering
institutions like the Indian Institutes of Technology (IITs) or the National Institutes of
Technology (NITs), the un-employability rate amongst Indian engineering graduates is
alarming. Citing the latest report by All India Survey on Higher Education (AISHE, 2019-2020)
issued by the Ministry of Education, India, one commercial magazine claims that nearly 80%
of engineering graduates are unemployable [1]. The information is confirmed in another article
962
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
“80% of Indian engineers not fit for jobs – says survey” by a reputed business magazine
(Business Today, March 25, 2019).
A well-recognized reason for this unhappy situation is the lack of pedagogy-savvy faculty and
the use of outdated teaching practices leading to poor preparation of students for a demanding
workplace. Over time a lot of national and international resources have been spent in faculty
development mainly in the form of short 5-10 days workshops. In 2007, a group of American
engineering educators of Indian origin came together to form a volunteer organization for
improving engineering education in India. The organization initially named as Indo-US
Collaboration for Engineering Education (IUCEE) was soon renamed as the Indo-Universal
Collaboration for Engineering Education when other educators from Singapore and Australia
joined the organization. The IUCEE also started its activities with a series of conventional one-
week long faculty training programs. Over 2008 -2010, more than 2500 faculty from all over
the country were trained. However, it became clear that in order to bring in sustainable change
in the competency and confidence levels of the faculty, a more formally structured training
program needs to be designed which would include theory and a substantial practicum
component.
Faculty development programs (FDPs) for university faculty focused on improving teaching
skills began in the early 1970s internationally and since then there have been numerous FDPs
conducted nationally and internationally [2]. The duration of the faculty development programs
is usually in the form of a day, three-days, five-days, two-weeks, etc. Different research studies
on the effectiveness of FDPs present minimal assessment of the activities of simplistic
measures mostly relying on participant feedback or satisfaction surveys [3]. A few exceptions
exist, for example, researchers assessed the outcomes of the FDP after the completion of the
program by collecting data from observing faculty members teaching and analyzing them [4].
Six faculty members teaching in the clinic and/or in the classroom were observed and
interviews were conducted to collect data. In a study, by Hoffmann-Longtin et al., [5], focused
on understanding the trends on assessment on FDPs, summarized that there is need to shift
the focus on assessing the impact and outcomes of FDPs and the data collection methods for
evaluating the effectiveness of the FDPs must be critically designed. With this as a brief
background and motivation, in this study the authors present a survey instrument designed
following the outcomes of a nine-month long certification program which aimed at assessing
the participants self-assessment of their competency in their confidence on performing
different tasks learned in the certification program.
Per se, a single assessment tool cannot be used to assess different programs as the needs
and defined goals/outcomes of different programs vary. However, the approach presented in
the paper can be used to design survey instruments to measure participants personal
perceptions related to the different outcomes of the faculty development program.
Design Framework for IIEECP
In 2014, the IUCEE invited a reputed education technologist (Dr. Veena Kumar, retired
Professor and Head, education Technology, Indian Institute of Technology, Delhi, India) to
design a comprehensive certification program - the IUCEE International Engineering Educator
Certification Program (IIEECP), inspired by a similar certification program offered by IGIP
(translates in English as the International Society for Engineering Pedagogy), Austria. IGIP is
a renowned European engineering society with over 40-year tradition of making valuable
contribution to engineering pedagogy and faculty development. IGIP has certified over 1500
faculty in 52 countries (Wikipedia).
As the IIEECP program was fully customized to meet Indian education, socio-economic,
cultural ground realities, it was quite different from the certification program offered by IGIP.
However, both programs covered similar theoretical, ethical, and practical issues, and both led
to a valuable certification in engineering pedagogy. The IIEECP was formally launched in
January 2015 with the financial support of Microsoft India. In 2016, IGIP recognized the IIEECP
for joint certification. The underlying philosophy of IIEECP is to focus on developing both
Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Javeed Kittur and Veena
Kumar, 2021.
963
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
professional skills and personal growth (Figure 1). The most challenging of all was to bring in
a new mindset that distinguished a good academician from a good teacher, and conceptually
recognizing that teaching was a skill that needed to be learnt with time and effort.
Reflective Teaching
Managing Managing mind-set Philosophy
Students Course Content Statement
Time
Preparing Students
Management
for Employment
Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Javeed Kittur and Veena
Kumar, 2021.
964
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
Methods
Development of the Survey Instrument
The survey instrument for this study was developed in Spring 2021 by the authors. The
instrument includes six scales each corresponding to each of the six modules. Survey items
were framed based on specific learning outcomes of each module. The instrument is intended
to capture the IIEECP certified faculty members personal assessment of enhancement in their
competencies and confidence level as a result of attending the certification program. The
survey instrument is designed in three part that include i) demographic analysis, ii) a 39-item
questionnaire related to the achievement of specified learning outcomes of the IIEECP
program, and iii) a set of six qualitative questions designed to help participants rate their
enhanced competencies and professional confidence.
Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Javeed Kittur and Veena
Kumar, 2021.
965
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
The survey items were initially written by the first author and were reviewed by the second
author. The survey items were revised based on the feedback from the second author. Table
1 provides the overview of the survey instrument which includes the six scales, description of
each scale, number of items in each scale and sample items for each scale. The faculty
respondents were asked to rate their confidence in accomplishing participants’ tasks related
to skills learned in each of the six modules on a five-point Likert scale with response options
(1) strongly disagree (2) disagree (3) neither agree nor disagree (4) agree (5) strongly agree.
The target population for this study were Indian faculty members who had completed the
IIEECP certification. The survey was distributed to around 900 certified faculty members
across India. A total of 280 faculty members responded to the survey which resulted in a
response rate of approximately 31%. Most of the respondents were from the 2019 & 2020
batches. First the certified faculty members email addresses were collected, and the potential
participants were invited to complete the survey through email during Spring 2021.
As mentioned, a total of 280 responses were received. The participants responses included
six blank responses, three participants responded to less than 50% of the questions, and 78
participants responses were same on all the questions (they strongly agreed or agreed or
disagreed on all the questions in the survey). The final sample after cleaning the data included
193 responses.
Table 2 shows the respondents’ profile and demographic information. The sample included 59
percent women and 41 percent men, 54 percent faculty from the autonomous institution, and
31 percent faculty from affiliated universities. Most faculty completed IIEECP in 2020 (36%)
and in 2019 (29%). About a third of the faculty held assistant professor positions (68%). The
final sample after cleaning the data included 193 responses from ten different states, a pictorial
representation of the participants respondents from different parts of India. Figure 2 shows the
respondents’ representation from different parts of India.
Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Javeed Kittur and Veena
Kumar, 2021.
966
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
967
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
missing data on the survey items was handled using the group mean substitution method. To
ensure significant correlation among the items with one another in each scale, inter-item
correlations were examined. The suitability of the survey items for factor analysis was
determined using the Bartlett’s test for sphericity (p<0.05). To check the meaningful variance
among the extracted factors from the survey items, Kaiser-Meyer-Olkin measure of sampling
adequacy (KMO) was used.
Exploratory Factor Analysis
To investigate the fundamental factor structure of the IIEECP survey instrument and the items
that belong to each scale, exploratory factor analysis (EFA) was conducted. Principal axis
factoring was used to extract the factors and Promax with Kaiser normalization method
(kappa=4) was used as the rotation method. To determine the number of factors to be
extracted from the data, Kaiser’s criterion, parallel analysis, and scree plots were used [6].
Items that had low loadings on all factors (<0.4) or cross loadings on at least two factors (>0.3)
were removed from the factor structure [6]. This process was repeated until there were no low-
or cross-loading items remaining. With the finalized factor structure for the scales of the survey
instrument, Cronbach’s alpha (α>0.8 preferred) was used to calculate the internal consistency
reliability for each scale of the instrument [7]. The final scores for each scale were calculated
by averaging the scores of all items associated with that scale.
Results
The suitability of the IIEECP survey instrument was confirmed by the Bartlett’s test of sphericity
(p<0.001) and the Kaiser-Meyer-Olkin measure of sampling adequacy (KMO) indicated that if
factor analysis was conducted then the extracted factors would account for meaningful
variance (KMO=0.946) [6]. Scree plots, parallel analysis, and Kaiser’s criterion methods
suggested extracting three, five, and six factors respectively from the data. The authors
decided to develop the instrument with six factors as this matched the number of hypothesized
factors. The inter-item correlations for each of the hypothesized scales were significantly
correlated (p<0.01), thereby supporting a six-factor structure of the instrument.
Two items – “I can design my lectures using the ARCS model of motivation” (The teaching-
learning process) and “I can map advantages of including collaborative activities with
promoting graduate attributes” (Collaborative learning) – had factor loadings less than 0.4 on
all the factors and were removed from the data. Eight items cross-loaded (loadings > 0.3) on
two factors and were removed: “I have better clarity about my responsibilities as an engineering
educator” (The teaching-learning process), “The quality of my lectures has improved
substantially” (Course design and delivery), “I can successfully develop good rapport with my
students” (Creating a dynamic classroom), “I can video record lectures and upload them on
you tube” (Harnessing the power of technology), “I can create a course website using free
resources like Canvas, Google Classroom, Edmodo, etc.” (Harnessing the power of
technology), “I can effectively use virtual labs in laboratory courses” (Harnessing the power of
technology), “I can effectively implement collaborative activities with my students”
(Collaborative learning), and “I can deal with unethical practices during assessments”
(Effective assessment). One item cross-loaded on three factors and was removed: “I can
design my course using backward design” (Course design and delivery). Five items were
deleted as they had different focus than most of the items in that factor.
The final factor structure with the list of items in each factor and factor loadings is presented in
Table 4. The items in each factor are sorted in decreasing order of the factor loadings. The
factor loadings for the first factor range from 0.67 to 0.81, the second factor from 0.75 to 0.81,
the third factor from 0.52 to 0.75, the fourth factor from 0.43 to 0.81, the fifth factor from 0.42
to 0.77, and the sixth factor from 0.55 to 0.90. The Cronbach’s α (coefficients of internal
consistency reliability) for the six factors ranged from 0.82 to 0.87. Table 3 shows the mean
and standard deviations of the questions on the survey related to performance and
professional confidence of IIEECP certified faculty members. Like the other survey items, the
participants responded to these questions on a five-point Likert scale. The average values of
Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Javeed Kittur and Veena
Kumar, 2021.
968
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
the self-reported scores by the faculty respondents on all the six prompts presented in Table
3 are more than four (out of five). This indicates that most of the faculty members who
completed the IIEECP has shown improved performance and professional confidence.
Table 4: Final factor loadings of the IIEECP survey instrument
# Category F1 F2 F3 F4 F5 F6
The Teaching-Learning Process (Cronbach’s α=0.85)
1 I have been sensitized about my role in keeping my students
engaged and motivated 0.81
2 I can use major theories of learning in teaching my courses 0.77
3 I can define major theories of learning required in teaching my
courses 0.74
4 I can compose my teaching philosophy statement 0.73
5 I can identify my short and long-term professional goals 0.67
Course Design and Delivery (Cronbach’s α=0.82)
6 I can plan an independent study program which helps me to
complete my course in time 0.81
7 I can implement an independent study program which helps me
to complete my course in time 0.80
8 I can implement a flipped class 0.76
9 I can plan a flipped class 0.75
Creating a Dynamic Classroom (Cronbach’s α=0.85)
10 I can successfully manage students with disruptive behavior 0.75
11 I can predict students with disruptive behaviour 0.61
12 I can plan my office hour effectively for individual and small group
meetings 0.52
Harnessing the Power of Technology (Cronbach’s α=0.87)
13 I can effectively incorporate virtual labs in lectures 0.81
14 I can effectively integrate MOOCs in my courses 0.51
15 I can deliver online classes effectively 0.43
Collaborative Learning (Cronbach’s α=0.85)
16 I can create instruments for evaluating individual performance in
a collaborative activity 0.77
17 I can create instruments for evaluating group performance in a
collaborative activity 0.74
18 I can plan effective collaborative activities for my course 0.42
Effective Assessment (Cronbach’s α=0.84)
19 I can deal with plagiarism practices during assessments 0.90
20 I can create effective rubrics for class assignments 0.83
21 I can create effective rubrics for class projects 0.68
22 I can create good question papers for tests and exams 0.66
23 I can create good open-book tests/exams 0.55
Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Javeed Kittur and Veena
Kumar, 2021.
969
Downloaded from search.informit.org/doi/10.3316/informit.354915168962511. on 07/23/2024 11:06 PM AEST; UTC+10:00. © REES AAEE 2021 conference: Engineering Education Research Capability Development , 2021.
Conclusions
In this paper, a survey instrument was designed to measure the competencies, skills and
professional confidence acquired by Indian engineering faculty through the IIEECP. The
instrument was designed on the basis of the specified outcomes for each of the six modules
of the certification program. An exploratory factor analysis was conducted to determine the
factor structure and it resulted in six factors aligned with the six modules of the certification
program. The internal consistency reliability of the six factors was checked using the
Cronbach’s α.
The instrument was found to be effective in measuring the target skills and professional
confidence. The study brings some valuable outcomes for engineering education in India. To
begin with it provides a framework around which engineering institutions can develop their own
criteria for faculty evaluation. It allows a better understanding of faculty training needs and how
to address them. The instrument will serve as a practical tool for faculty to self-reflect and
assess their own competencies as well as their learning needs. Finally, the survey instrument
provides invaluable feedback to the IIEECP team to assess the strength and weaknesses of
different modules, and to further reinforce the program.
One of the limitations of this work is that the evidence for content validity was not collected.
Hence, in a future version, the instrument can be further improved by collecting evidence from
the expert team teaching and evaluating each module. Investigating the influence of
participants’ demographic variables on all the six modules is another direction for future work.
A follow-up qualitative study is planned to investigate the beliefs and perceptions of the certified
faculty members to understand changes in their personal and professional growth after
completing the certification.
References
1. Roy, Subhashri (July 2020) “80 Per Cent of Indian Engineers Unemployed: AISHE 2019”,
Indian Engineers Employment Trends, CollegeDekho. Retrieved from
https://www.collegedekho.com/news/indian-engineering-graduates-employability-
trends-18148/
2. Fletcher, J. J., & Patrick, S. K. (1998). Not just workshops any more: The role of faculty
development in reframing academic priorities. The International Journal for Academic
Development, 3(1), 39-46.
3. Eckstrom, E., Homer, L., & Bowen, J. L. (2006). Measuring outcomes of a one‐minute
preceptor faculty development workshop. Journal of general internal medicine, 21(5),
410-414.
4. Behar‐Horenstein, L. S., Childs, G. S., & Graff, R. A. (2010). Observation and
assessment of faculty development learning outcomes. Journal of Dental
Education, 74(11), 1245-1254.
5. Hoffmann-Longtin, K., Fassett, K., Zilvinskis, J., & Palmer, M. M. (2019). Measuring
faculty learning: Trends in the assessment of faculty development. Stylus Publishing.
6. McCoach, D. B., Gable, R. K., & Madura, J. P. (2013). Instrument development in the
affective domain. New York, NY: Springer. doi, 10, 978-1.
7. Cronbach, L. J. (1984). How to judge tests. Essentials of Psychological Testing. 4th ed.
New York, Harper & Row.
Copyright statement
Copyright © 2021 Names of authors: Javeed Kittur and Veena Kumar. The authors assign to the Research in Engineering
Education Network (REEN) and the Australasian Association for Engineering Education (AAEE) and educational non-profit
institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is
used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to REEN and AAEE to
publish this document in full on the World Wide Web (prime sites and mirrors), on Memory Sticks, and in printed form within the
REEN AAEE 2021 proceedings. Any other usage is prohibited without the express permission of the authors.
Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Javeed Kittur and Veena
Kumar, 2021.
970