Readers are Leaders Project

Early Grade Reading and Mathematics

Assessment in the Republic of Macedonia:
Study Report

January 2016
USAID Readers are Leaders Project
Study Report May 2015

Table of Contents:

Acknowledgments .................................................................................................................... 10
List of Abbreviations ................................................................................................................ 11
Executive Summary ................................................................................................................. 12
Main Findings and Recommendations ................................................................................. 13
1. Project Background .......................................................................................................... 16
2. EGRA AND EGMA STUDY IN 2015 ............................................................................ 17
2.1 Sample design ............................................................................................................ 18
2.2 Instrument structure ................................................................................................... 20
3. Early Grade Reading Assessment .................................................................................... 21
3.1 The EGRA instrument for Macedonia....................................................................... 22
4. Early Grade Mathematics Assessment ............................................................................. 30
5. Fieldwork and Data Collection Process for EGRA & EGMA ......................................... 36
5.1 Training for Administration of EGRA and EGMA Instruments ............................... 36
5.2 Data collection ........................................................................................................... 37
5.3 Quality Assurance...................................................................................................... 38
5.4 Data analysis and reporting ....................................................................................... 40
6. EGRA & EGMA Baseline Findings ................................................................................ 40
6.1 Characteristics of Sample .......................................................................................... 40
6.2 How Well Are Students Reading in Macedonian and Albanian? ............................. 41
6.2.1 Results according to gender .................................................................................... 43
6.2.2 Results according to education of parents ............................................................... 47
6.2.3 Results according to type of school......................................................................... 49
6.2.4 Results according to attendance of pre-school ........................................................ 51
6.2.5 Results according to availability of books at home................................................. 52
6.2.6 Results according to reading before going to school .............................................. 53
6.2.6 Results according to borrowing library books ........................................................ 55
6.2.7 Results according practicing reading at home......................................................... 56
6.2.8 Results according to EGRA tasks ........................................................................... 58
6.3 How Well Are Pupils Doing Basic Mathematics? .................................................... 60
6.3.1 Overall test results according to gender .................................................................. 61
6.3.2 Results according to education of parents ............................................................... 63
6.3.3 Results according to language of instruction .......................................................... 64
6.3.4 Results according to type of school......................................................................... 66
6.3.5 Results according to attendance of pre-school ........................................................ 67
6.3.6 Results according to availability of books at home................................................. 67
6.3.7 Results according to EGMA tasks .......................................................................... 68
7. EGRA & EGMA Longitudinal Findings ......................................................................... 73
8. FINDINGS FROM THE QUESTIONNAIRES ............................................................... 92
8.1 School Climate .......................................................................................................... 93
8.2 School Resources for Teaching Reading and Mathematics .................................... 107
8.3 Teacher Preparation ................................................................................................. 112
8.4 Classroom Instruction .............................................................................................. 116
8.5 Home Environment Support for Reading Achievement.......................................... 120
9. CONCLUSIONS ............................................................................................................ 134
9.1 Lessons Learned from EGRA AND EGMA application ........................................ 134
2
USAID Readers are Leaders Project
Study Report May 2015

9.2 School Climate ........................................................................................................ 135

9.3 School Resources for Teaching Reading and Mathematics .................................... 136
9.4 Teacher Preparation ................................................................................................. 136
9.5 Classroom Instruction .............................................................................................. 137
9.6 Home Environment Support for Reading Achievement.......................................... 138
1. APPENDICES ................................................................................................................ 140
1.1 Appendix 1: List of schools participating in EGRA and EGMA studies ................ 140
1.2 Appendix 2: EGRA and EGMA longitudinal results .............................................. 142
1.3 Appendix 3: Questionnaire for School Director ...................................................... 152
1.4 Appendix 4: Questionnaire for Teachers ................................................................. 158
1.5 Appendix 5: Questionnaire for Parents ................................................................... 165

List of Tables:

Table 1. Number of students per grade and language of instruction in the baseline sample ... 19
Table 2. Number of students per grade and language of instruction in the longitudinal sample
.................................................................................................................................................. 19
Table 3. Example of Comprehension Questions Task in Macedonian language ..................... 25
Table 4. Example of Comprehension Questions Task in Albanian language .......................... 27
Table 5. Timing and stop rules for EGRA tasks ...................................................................... 29
Table 6. Timing and stop rules for EGMA tasks ..................................................................... 35
Table 7. Number of Grade 2 and 3 students attending pre-school ........................................... 51
Table 8. Number of Grade 2 and 3 students having additional books at home ....................... 52
Table 9. Number of Grade 2 and 3 students reading before starting school ............................ 54
Table 10. Number of Grade 2 and 3 students borrowing library books ................................... 55
Table 11. Number of Grade 2 students reading at home .......................................................... 56
Table 12. Number of Grade 3 students reading at home .......................................................... 57
Table 13. Student results on Letter Identification task ............................................................. 58
Table 14. Student results on Familiar Words Reading task ..................................................... 58
Table 15. Overall student results on Reading Fluency task ..................................................... 59
Table 16. Student results on Reading Fluency per task ........................................................... 59
Table 17. Overall student results on Reading Comprehension task ......................................... 60
Table 18. Student results on Reading Comprehension per task ............................................... 60
Table 19. Number of Grade 2 and 3 students attending pre-school ......................................... 67
Table 20. Number of Grade 2 and 3 students having additional books at home ..................... 67
Table 21. Overall student results on Number Identification task ............................................. 68
Table 22. Overall student results on Number Discrimination task .......................................... 69
Table 23. Overall student results on Missing Numbers task .................................................... 69
Table 24. Overall student results on Addition task .................................................................. 70
Table 25. Overall student results on Subtraction task .............................................................. 71
Table 26. Overall student results on Word Problems task ....................................................... 71
Table 27. Overall student results on Geometric Shapes Recognition task ............................... 72
Table 28. Overall student results on Geometric Patterns Extension task................................. 72
Table 29. Student results on Letter Identification task ............................................................. 74
Table 30. Student results on Familiar Words Reading task ..................................................... 74
Table 31. Overall student results on Reading Fluency task ..................................................... 75
Table 32. Student results on Reading Fluency per task ........................................................... 75
Table 33. Overall student results on Reading Comprehension task ......................................... 76

3
USAID Readers are Leaders Project
Study Report May 2015

Table 34. Student results on Reading Comprehension per task ............................................... 76

Table 35. Overall student results on Number Identification task ............................................. 77
Table 36. Overall student results on Number Discrimination task .......................................... 77
Table 37. Overall student results on Missing Numbers task .................................................... 78
Table 38. Overall student results on Addition task .................................................................. 78
Table 39. Overall student results on Subtraction task .............................................................. 79
Table 40. Overall student results on Word Problems task ....................................................... 79
Table 41. Overall student results on Geometric Shapes Recognition task ............................... 80
Table 42. Overall student results on Geometric Patterns Extension task................................. 80

Table of Figures:

Figure 1. Letter Knowledge task in Macedonian language ...................................................... 23

Figure 2. Letter Knowledge task in Albanian language ........................................................... 23
Figure 3. Familiar Word Reading Task in Macedonian language ........................................... 24
Figure 4. Familiar Word Reading Task in Albanian language ................................................ 24
Figure 5. Example of Reading Fluency Task in Macedonian language ................................... 25
Figure 6. Example of Reading Fluency Task in Albanian language ........................................ 25
Figure 7. Example of Number Identification Task for Grade 2 ............................................... 31
Figure 8. Example of Number Discrimination Task for Grade 2 ............................................. 31
Figure 9. Example of Naming Missing Numbers Task for Grade 2 ........................................ 32
Figure 10. Example of Addition task Level 1 .......................................................................... 32
Figure 11. Example of Addition task Level 2 .......................................................................... 32
Figure 12. Example of Subtraction task Level 1 ...................................................................... 33
Figure 13. Example of Subtraction task Level 2 ...................................................................... 33
Figure 14. Example of Word Problems Tasks in Macedonian and Albanian language for
Grade 2 ..................................................................................................................................... 33
Figure 15. Example of Shape Recognition Task for Grade 2 .................................................. 34
Figure 16. Example of Pattern Extension Task for Grade 2 .................................................... 34
Figure 17. Percentage of students in EGRA sample per grade, language of instruction,
location of the schools and gender ........................................................................................... 41
Figure 18. Percentage of students in EGМA sample per grade, language of instruction,
location of the schools and gender ........................................................................................... 41
Figure 19. Average percentage of students that completed EGRA tasks for Grade 2 ............. 42
Figure 20. Average percentage of students that completed EGRA tasks for Grade 3 ............. 43
Figure 21. Average percentage of Grade 2 students that completed EGRA according to gender
.................................................................................................................................................. 43
Figure 22. Average percentage of Grade 3 students that completed EGRA according to gender
.................................................................................................................................................. 44
Figure 23. Average percentage of Grade 2 students that completed EGRA tasks in
Macedonian language according to gender .............................................................................. 44
Figure 24. Average percentage of Grade 3 students that completed EGRA tasks in
Macedonian language according to gender .............................................................................. 45
Figure 25. Average percentage of Grade 2 students that completed EGRA tasks in Albanian
language according to gender ................................................................................................... 46
Figure 26. Average percentage of Grade 3 students that completed EGRA tasks in Albanian
language according to gender ................................................................................................... 46

4
USAID Readers are Leaders Project
Study Report May 2015

Figure 27. Average percentage of Grade 2 students that completed EGRA in Macedonian
language according to education of parents ............................................................................. 47
Figure 28. Average percentage of Grade 2 students that completed EGRA in Albanian
language according to education of parents ............................................................................. 48
Figure 29. Average percentage of Grade 3 students that completed EGRA in Macedonian
language according to education of parents ............................................................................. 49
Figure 30. Average percentage of Grade 3 students that completed EGRA in Albanian
language according to education of parents ............................................................................. 49
Figure 31. Average percentage of Grade 2 students that completed EGRA according to type of
school ....................................................................................................................................... 50
Figure 32. Average percentage of Grade 3 students that completed EGRA according to type of
school ....................................................................................................................................... 50
Figure 33. Average percentage of Grade 2 students that completed EGRA according pre-
school attendance ..................................................................................................................... 51
Figure 34. Average percentage of Grade 3 students that completed EGRA according pre-
school attendance ..................................................................................................................... 52
Figure 35. Average percentage of Grade 2 students that completed EGRA according to
availability of books at home ................................................................................................... 53
Figure 36. Average percentage of Grade 3 students that completed EGRA according to
availability of books at home ................................................................................................... 53
Figure 37. Average percentage of Grade 2 students that completed EGRA according to
whether they read before going to schools ............................................................................... 54
Figure 38. Average percentage of Grade 3 students that completed EGRA according to
whether they read before going to schools ............................................................................... 55
Figure 39. Average percentage of Grade 2 students that completed EGRA according to
whether they borrow books from library.................................................................................. 56
Figure 40. Average percentage of Grade 3 students that completed EGRA according to
whether they borrow books from library.................................................................................. 56
Figure 41. Average percentage of Grade 2 and 3 students that completed EGRA tasks in
Macedonian and Albanian language according to with whom they are reading at home ........ 58
Figure 42. Average percentage of students that completed EGMA tasks ................................ 61
Figure 43. Average percentage of students that completed EGMA according to gender ........ 62
Figure 44. Average percentage of Grade 2 students that completed EGMA tasks according to
gender ....................................................................................................................................... 62
Figure 45. Average percentage of Grade 3 students that completed EGMA tasks according to
gender ....................................................................................................................................... 63
Figure 46. Average percentage of Grade 2 students that completed EGMA according to
education of parents ................................................................................................................. 64
Figure 47. Average percentage of Grade 3 students that completed EGMA according to
education of parents ................................................................................................................. 64
Figure 48. Average percentage of Grade 2 students that completed EGMA tasks according to
language of instruction ............................................................................................................. 65
Figure 49. Average percentage of Grade 3 students that completed EGMA tasks according to
language of instruction ............................................................................................................. 66
Figure 50. Average percentage of students that completed EGMA according to type of school
.................................................................................................................................................. 66
Figure 51. Average percentage of students that completed EGMA according to pre-school
attendance ................................................................................................................................. 67

5
USAID Readers are Leaders Project
Study Report May 2015

Figure 52. Average percentage of students that completed EGMA according to availability of
books at home........................................................................................................................... 68
Figure 53. Comparison of average percentage of students that completed EGRA tasks in
Macedonian language of instruction ........................................................................................ 73
Figure 54. Comparison of average percentage of students that completed EGRA tasks in
Albanian language of instruction ............................................................................................. 74
Figure 55. Comparison of average percentage of students that completed EGMA tasks ........ 77
Figure 56. Comparison of average percentage of Grade 3 students that completed EGRA in
Macedonian language of instruction in longitudinal and baseline sample............................... 81
Figure 57. Comparison of average percentage of Grade 3 students that completed EGRA in
Albanian language of instruction in longitudinal and baseline sample .................................... 82
Figure 58. Comparison of average percentage of Grade 3 students that completed EGRA in
Macedonian language of instruction per gender in longitudinal and baseline sample ............. 82
Figure 59. Comparison of average percentage of Grade 3 students that completed EGRA in
Albanian language of instruction per gender in longitudinal and baseline sample .................. 83
Figure 60. Comparison of average percentage of Grade 3 students that completed EGRA in
Macedonian language of instruction per school location in longitudinal and baseline sample 83
Figure 61. Comparison of average percentage of Grade 3 students that completed EGRA in
Albanian language of instruction per school location in longitudinal and baseline sample .... 84
Figure 62. Comparison of average percentage of Grade 3 students that completed EGRA in
Macedonian language of instruction per type of school in longitudinal and baseline sample . 85
Figure 63. Comparison of average percentage of Grade 3 students that completed EGRA in
Albanian language of instruction per type of school in longitudinal and baseline sample ...... 85
Figure 64. Comparison of average percentage of Grade 3 students that completed EGRA in
Macedonian language of instruction according to pre-school attendance in longitudinal and
baseline sample ........................................................................................................................ 86
Figure 65. Comparison of average percentage of Grade 3 students that completed EGRA in
Albanian language of instruction according to pre-school attendance in longitudinal and
baseline sample ........................................................................................................................ 86
Figure 66. Comparison of average percentage of Grade 3 students that completed EGRA in
Macedonian language of instruction according to having books at home in longitudinal and
baseline sample ........................................................................................................................ 87
Figure 67. Comparison of average percentage of Grade 3 students that completed EGRA in
Albanian language of instruction according to having books at home in longitudinal and
baseline sample ........................................................................................................................ 88
Figure 68. Comparison of average percentage of Grade 3 students that completed EGМA in
longitudinal and baseline sample ............................................................................................. 88
Figure 69. Comparison of correct number of items of Grade 3 students that completed EGМA
in longitudinal and baseline sample ......................................................................................... 89
Figure 70. Comparison of average percentage of Grade 3 students that completed EGМA per
gender in longitudinal and baseline sample ............................................................................. 89
Figure 71. Comparison of average percentage of Grade 3 students that completed EGМA per
school location in longitudinal and baseline sample ................................................................ 90
Figure 72. Comparison of average percentage of Grade 3 students that completed EGМA per
type of school in longitudinal and baseline sample.................................................................. 90
Figure 73. Comparison of average percentage of Grade 3 students that completed EGМA per
language of instruction in longitudinal and baseline sample ................................................... 91

6
USAID Readers are Leaders Project
Study Report May 2015

Figure 74. Comparison of average percentage of Grade 3 students that completed EGМA
according to pre-school attendance in longitudinal and baseline sample ................................ 91
Figure 75. Comparison of average percentage of Grade 3 students that completed EGМA
according to having books at home in longitudinal and baseline sample ................................ 92
Figure 76: Gender structure of surveyed school directors ....................................................... 94
Figure 77: Gender structure of surveyed early grade teachers ................................................. 94
Figure 78: Ethnic structure of surveyed early grade teachers .................................................. 95
Figure 79: Percentage of schools working in split shifts ......................................................... 95
Figure 80: Representation of schools in the sample according to the language of instruction 96
Figure 81: Representation of early grade teachers in the sample according to the language of
instruction ................................................................................................................................. 96
Figure 82: Number of early grade teachers in sample schools ................................................ 97
Figure 83: Absence of teachers during school year ................................................................. 97
Figure 84: Measures taken by school directors when teachers are late for classes .................. 98
Figure 85: Alternative for students when their teacher is absent ............................................. 98
Figure 86: Absence of students during school year according to teachers .............................. 99
Figure 87: Absence of students during school year according to parents ................................ 99
Figure 88: Practice of monitoring performance of teachers ................................................... 100
Figure 89: Frequency of class observations reported by school directors ............................. 101
Figure 90: Frequency of class observations reported by teachers .......................................... 101
Figure 91: Documentation of class observations by school director in the form of report .... 101
Figure 92: Feedback after class observations by school director ........................................... 102
Figure 93: Most common topics for collaboration among teachers ....................................... 102
Figure 94: Consultation of teachers regarding curriculum needs .......................................... 103
Figure 95: Frequency of class observation by student support service .................................. 103
Figure 96: Feedback after class observation by student support service ............................... 104
Figure 97: Frequency of school visits by BDE advisors ........................................................ 104
Figure 98: School directors’ expectations about student literacy .......................................... 105
Figure 99: Teachers’ expectations about student literacy ...................................................... 105
Figure 100: Ability of children to read reported by parents ................................................... 106
Figure 101: Location where children learn to read ................................................................ 106
Figure 102: Supplied didactic materials over the last three years .......................................... 108
Figure 103: Existence of library in the school ....................................................................... 108
Figure 104: Existence of librarian in the school .................................................................... 109
Figure 105: Opening hours of school library ......................................................................... 109
Figure 106: Books in the library for early grade students ...................................................... 110
Figure 107: Borrowing books from the library by early grade students ................................ 110
Figure 108: Percentage of early grade students borrowing books from the library ............... 111
Figure 109: Children members of a library as reported by parents ........................................ 112
Figure 110: Access to Internet of schools .............................................................................. 112
Figure 111: Level of education of surveyed early grade teachers .......................................... 113
Figure 112: Percentage of teachers that attended training on teaching students to read ........ 113
Figure 113: Percentage of teachers that need additional training on literacy ........................ 114
Figure 114: Percentage of teachers that attended training on teaching math to students....... 114
Figure 115: Percentage of teachers that need additional training in math ............................. 115
Figure 116: Organizations delivering training in literacy and math ...................................... 115
Figure 117: Percentage of early grade teachers that have students with disabilities in class 116
Figure 118: Working with students with disabilities ............................................................. 117

7
USAID Readers are Leaders Project
Study Report May 2015

Figure 119: Measuring of student knowledge ........................................................................ 117

Figure 120: How teachers use student results ........................................................................ 118
Figure 121: Feedback for parents about student results ......................................................... 118
Figure 122: Teaching methods for students with poor results ............................................... 119
Figure 123: Additional classes in literacy and math .............................................................. 119
Figure 124: Books at home as reported by parents ................................................................ 120
Figure 125: Frequency of buying magazines and newspapers as reported by parents .......... 121
Figure 126: Frequency of buying books and story books as reported by parents .................. 121
Figure 127: Frequency of reading together with the child as reported by parents ................. 122
Figure 128: Type of books mostly read at home as reported by parents ............................... 122
Figure 129: Discussing the content after reading as reported by parents ............................... 123
Figure 130: Frequency of child’s independent reading as reported by parents ..................... 123
Figure 131: Literature mostly read by children independently as reported by parents .......... 124
Figure 132: Satisfaction with parents’ participation in monitoring student assignments
according to teachers .............................................................................................................. 124
Figure 133: Engagement in child’s homework assignments as reported by parents ............. 125
Figure 134: Satisfaction with parents’ participation in monitoring student assignments
according to teachers .............................................................................................................. 125
Figure 135: Participation in school activities according to parents ....................................... 126
Figure 136: Ways of participation of parents in school activities .......................................... 126
Figure 137: Participation of parents in classroom activities according to teachers ............... 127
Figure 138: Participation of parents in classroom activities according to school directors ... 127
Figure 139: Frequency of parents involved in classroom activities according to teachers .... 128
Figure 140: Frequency of parents’ request to be involved in school activities ...................... 128
Figure 141: Frequency of parents asking to get involved in school activities as reported by
parents .................................................................................................................................... 129
Figure 142: Membership in the Parents’ Council as reported by parents .............................. 129
Figure 143: Meetings of Parents’ Council in a school year as reported by school directors . 130
Figure 144: Meetings of Parents’ Council in a school year as reported by parents ............... 130
Figure 145: Who initiates meetings of Parents’ Council reported by school director ........... 131
Figure 146: Who initiates meetings of Parents’ Council reported by parents ....................... 131
Figure 147: Suggestions of Parents’ Council for overcoming certain school problems as
reported by parents ................................................................................................................. 132
Figure 148: Suggestions of Parents’ Council for overcoming certain school problems as
reported by school directors ................................................................................................... 132
Figure 149: Satisfaction of school directors with support of Parents’ Council ..................... 133
Figure 150: Satisfaction of parents with support of Parents’ Council ................................... 133
Figure 151. Percentage of students in EGRA sample per grade, language of instruction,
location of the schools and gender ......................................................................................... 142
Figure 152. Percentage of students in EGМA sample per grade, language of instruction,
location of the schools and gender ......................................................................................... 143
Figure 153. Comparison of average percentage of students that completed EGRA tasks in
Macedonian language of instruction ...................................................................................... 143
Figure 154. Comparison of average percentage of students that completed EGRA tasks in
Albanian language of instruction ........................................................................................... 144
Figure 155. Comparison of average percentage of students that completed EGRA according to
gender in Macedonian language of instruction ...................................................................... 145

8
USAID Readers are Leaders Project
Study Report May 2015

Figure 156. Comparison of average percentage of students that completed EGRA according to
gender in Albanian language of instruction ........................................................................... 145
Figure 157. Comparison of average percentage of students that completed EGRA according to
type of school in Macedonian language of instruction .......................................................... 145
Figure 158. Comparison of average percentage of students that completed EGRA according to
type of school in Albanian language of instruction................................................................ 146
Figure 159. Comparison of average percentage of students that completed EGRA according to
pre-school attendance in Macedonian language of instruction .............................................. 146
Figure 160. Comparison of average percentage of students that completed EGRA according to
pre-school attendance in Albanian language of instruction ................................................... 147
Figure 161. Comparison of average percentage of students that completed EGRA according to
access to books at home in Macedonian language of instruction .......................................... 147
Figure 162. Comparison of average percentage of students that completed EGRA according to
access to books at home in Albanian language of instruction................................................ 148
Figure 163. Average percentage of students that completed EGRA tasks according to the
language of instruction and whether third grade students read before going to school ......... 148
Figure 164. Average percentage of students that completed EGRA tasks according to the
language of instruction and whether third grade students borrow library books ................... 149
Figure 165. Average percentage of Grade 3 students that completed EGRA tasks in
Macedonian and Albanian language according to with whom they are reading at home ...... 149
Figure 166. Average percentage of students that completed EGMA according to gender .... 150
Figure 167. Average percentage of students that completed EGMA tasks according to
language of instruction ........................................................................................................... 150
Figure 168. Average percentage of students that completed EGMA according to type of
school ..................................................................................................................................... 151
Figure 169. Average percentage of students that completed EGMA according to pre-school
attendance ............................................................................................................................... 151
Figure 170. Average percentage of students that completed EGMA according to availability
of books at home .................................................................................................................... 152

9
USAID Readers are Leaders Project
Study Report May 2015

ACKNOWLEDGMENTS
The Foundation for Education and Cultural Initiatives “Step by Step” – Macedonia would like
to thank all members of the Work Groups that participated in development and modification
of EGRA and EGMA instruments in Macedonian and Albanian language. The Work Groups
consisted of representatives from the Bureau for Development of Education, the National
Examinations Centre, professors from the Institute of Pedagogy and Faculty of Pedagogy in
Skopje, as well as psychologists, pedagogues and early grade teachers, ensured that all
tasks developed in the instruments were appropriately aligned to the local curriculum and
context.

We would also like to acknowledge our international consultants, who helped us a lot in the
pilot phase of the project: Helen Abadzi, José Noijons, Jehona Xhaferi and David Carroll, as
well as Scott Kipp, who gave us invaluable support in the process of electronic data
collection with Tangerine. Furthermore, we would like to thank Beti Lameva, from the
National Examinations Centre, who was responsible for analysis of data collected in all
studies implemented so far.

Most importantly, this study could not have succeeded without the cooperation and
contributions of the Grade 2 and 3 students, teachers, pedagogues, psychologists, and
directors who welcomed us in the 103 primary schools that participated in the assessment
process in Macedonia. Special thanks goes to the parents, who were willing to fill out our
online and hard copy questionnaires.

Finally, Step by Step Foundation would like to thank USAID Macedonia, especially Natasha
Buleska, for the valuable guidance and support to the development and implementation of
this study.

“This report was made possible with support from the American people through the U.S.
Agency for International Development (USAID). The contents are the responsibility of the
Foundation for Education and Cultural Initiatives “Step by Step” - Macedonia which is
implementing the USAID funded Readers are Leaders Project and do not necessarily reflect
the opinion of USAID or the U.S. Government.”

10
USAID Readers are Leaders Project
Study Report May 2015

LIST OF ABBREVIATIONS

BDE Bureau for Development of Education

clpm correct letters per minute
cwpm correct words per minute
EGMA Early Grade Mathematics Assessment
EGRA Early Grade Reading Assessment
MOES Ministry of Education and Science
NEC National Examinations Center
RTI RTI International (trade name of Research Triangle Institute)
PIRLS Progress in International Reading and Literacy Study
SSME Snapshot of School Management Effectiveness
TIMSS Trends in International Mathematics and Science Study
USAID United States Agency for International Development

11
USAID Readers are Leaders Project
Study Report May 2015

EXECUTIVE SUMMARY

Assessments of student learning in the primary grades, with instruments such as the Early
Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA), as
part of the USAID Readers are Leaders Project, implemented by the Foundation for
Education and Cultural Initiatives “Step by Step” - Macedonia in partnership with the Ministry
of Education and Science continued in 2015 with two parallel studies. A baseline study was
conducted in 61 primary schools to gain insight into student foundational reading and
mathematics skills. Furthermore, to monitor the progress of student achievements from
grade 2 into grade 3, a longitudinal study was conducted in the original 42 primary schools
that first took part in the study in 2014.

To better understand characteristics among schools associated with student performance in

reading and math, questionnaires were also administered to school directors, teachers and
parents from 63 schools.

The main objective of the baseline study was to explore the existing situation in the schools,
and based on the identified needs and findings to plan appropriately the future project
activities in the schools, tailoring the training modules to the needs of teachers for
professional development and involvement of families and local communities in promoting
the reading and mathematics skills in early grade students. The findings will also ensure
ongoing monitoring and evaluation of changes in the schools after the introduction of the
project interventions.

The initial section of this report refers to the oral assessments of 1,901 students with the help
of EGRA and EGMA instruments in 61 schools and explains the design of the various
subtests of the instruments, pointing out how they are related to important characteristics of
early reading and mathematics. The test adaptation process, pretesting, and pilot testing
stages are then described, followed by a description of the sampling and testing procedures.
Afterward, the analysis of results is presented in detail followed by general observations.

The second part presents the findings from the longitudinal study, which helped us monitor
the results of the cohort of 921 students with Macedonian and Albanian language of
instruction from 41 schools. As expected their results are much higher in Grade 3 during the
reassessment compared with their results in Grade 2 during the baseline assessment.

The findings from the background questions are provided in the third part. The
questionnaires were filled out by school directors, teachers and parents to get deeper
understanding of school and classroom practices as well as parental involvement traditionally
associated with student performance.

The report concludes with conclusions and recommendations.

12
USAID Readers are Leaders Project
Study Report May 2015

Main Findings and Recommendations

The main findings and recommendations from this study are summarized in the table below.

Findings Recommendations
Student performance on EGRA
Students continue to have good results in Asking “why” questions not just as part of
naming letters correctly, but their reading the reading instruction, but also in other
comprehension skills are not well areas should promote comprehension and
developed, regardless of the language of the higher level of thinking of students.
instruction.
EGRA results should be used for opening
Main predictors of student success in policy dialogue with education institutions
reading are: for modification to the early grade
 Higher education of parents language curriculum.
 Studying in central or urban
schools Instruction in mother tongue is critically
 Attendance of pre-school important for the effectiveness of literacy,
institution so funding should be provided for
 Having books at home instructional materials and support in
 Borrowing books from the library mother tongue.
 Reading at home (either
independently or with someone
else)

Longitudinally, students have better

scores in Grade 3. The comprehension
scores also increase.

Students who were assessed

longitudinally performed better than
those assessed for the first time this year
in all EGRA tasks.
Student performance on EGMA
The most difficult task is subtraction for Teachers should offer opportunities for
Grade 2 students and word problems for students to use manipulatives to
Grade 3 students. understand the concept of numbers and
develop skills for using numbers in
Male students are slightly better in math practical and problem-solving activities.
as well as students whose parents have
higher education. More time should be given for the new
mathematics curriculum to be rooted and
The modification of EGMA tasks to practiced by the teachers.
comply with the newly introduced
mathematics curriculum has made the
comparison of results difficult.

13
USAID Readers are Leaders Project
Study Report May 2015

Findings Recommendations

In the field of mathematics, the

comparisons show the following:
longitudinal sample has better results in
number identification, number
discrimination and addition, while the
baseline sample has better results in
naming missing numbers, subtraction,
word problems, geometric shape
recognition and geometric pattern
extension.
School Climate
Teachers consider the collaboration with The idea of teacher collegiality and
colleagues to be important in building a collaboration should focus on the idea of
professional community. They collaboration for the purpose of improving
collaborate with other teachers in teaching.
planning and implementation of lessons,
consultations with one another,
Proper selection of mentors is essential so
preparation of joint lesson plans and
that they can effectively support teachers.
dissemination of the training. The results of different mentorship
programs indicate that schools visited
Other teachers and student support frequently are likely to have stronger
services are preferred as the main student performance. Large-scale
source of advice for teachers for instructional improvements, however, are
resolving curriculum issues difficult, as they require face to face time,
practice, and ongoing feedback.
School Resources for Teaching Reading
Resources are crucial for improving Libraries, both within the school and in the
schooling, as the extent and quality of local community, should provide a range
school resources can have an important of reading materials and other resources
impact on the quality of classroom from which teachers can draw to expand
instruction. their instructional approaches, and from
which students can choose books for their
own learning and enjoyment.
Provision of reading and didactical
materials to the classrooms is paramount Investing in a small classroom library is a
in improving student literacy and great option so that children can have
numeracy. ready access to books and magazines as
part of their reading lessons and activities.
Teacher Preparation
Majority of teachers received a training in Training of teachers remains a complex
literacy and math but they still have task but it must be assumed that teachers
expressed needs for additional training. learn best by doing and interacting with
other professionals. This implies that
teacher training should be organized
around modeling and practice, and that
having brief trainings with follow-up and
refresher meetings is more effective than
longer trainings. Regular professional

14
USAID Readers are Leaders Project
Study Report May 2015

Findings Recommendations
development through training and other
activities should fill a demand for
instructional practice and support.
Classroom Instruction
Teachers report using various It is recommended for teachers to provide
instructional activities and strategies, instruction that will interest and engage
including additional and remedial classes students in learning.
for children.

Teachers should tailor the classes based

Student results are mainly used for
on the needs and results of their students.
assessment and less for adjusting and
planning for class activities.
Home Environment Support for Reading Achievement
Parents need to be more involved in Parents should be encouraged to be
developing early literacy skills of their engaged in early literacy activities with
children. their children, such as: reading books,
telling stories, singing songs, playing with
alphabet toys, talking about things done,
talking about things read, playing word
Overall EGRA results showed that both games, writing letters or words, and
students with Macedonian and Albanian reading aloud signs and labels.
language of instruction from both grades
that have books at home performed
Children should have access to different
better than those that did not report
reading materials at home.
having books at home.

15
USAID Readers are Leaders Project
Study Report May 2015

1. PROJECT BACKGROUND

USAID’s Readers are Leaders Project was planned as a 30-month initiative, designed to
improve early-grade students reading and numeracy skills, strengthen teachers’ pedagogical
skills, especially diagnostic and formative assessment skills, and increase overall community
recognition of the value of reading and numeracy skills for students’ intellectual growth.

However, after the implementation of the baseline study conducted in 42 primary schools in
Macedonia in 2014 and the presentation of project activities and results to the
representatives from the Ministry of Education and Science and other education
stakeholders, the Ministry requested from the USAID Macedonia to expand the project
activities, with special emphasis on EGRA and EGMA assessment in all primary schools
across the country and extend the project duration. Based on this request, USAID
Macedonia has granted unfunded extension of the project until November 30, 2017 so that
the project activities can be implemented in all primary schools.

Readers are Leaders Project implements its activities through five components, each having
multiple activities to encourage reading and numeracy skills in early grade children:

 Component 1: Reading and Numeracy Assessment

 Component 2: Professional Development
 Component 3: Learning Communities
 Component 4: Digital Learning Resources
 Component 5: Family and Community Involvement

As part of the Reading and Numeracy Assessment component, the USAID Readers are
Leaders project utilized two international assessment tools1 to collect literacy and numeracy
assessment data and analyze them for diagnostic purposes:
 The Early Grade Reading Assessment (EGRA2) as a one-on-one oral assessment
instrument, providing a simple diagnostic tool that measures individual student
progress in reading.
 The Early Grade Mathematics Assessment (EGMA) measuring student foundational
skills in numeracy and mathematics.
The objectives of the study conducted in May 2015 were twofold:

1. to collect baseline performance data in literacy and math on a sample of 1,000 Grade
2 and 1,000 Grade 3 students from additional 61 schools;
2. to collect progress data on reading and math performance of 1,000 Grade 2 students
first tested with EGRA and EGMA in May 2014 in order to monitor their performance
and compare the results of these students from the schools that have been involved
in project activities since the very beginning (and benefited from several years of

1 Since 2006, EdData II, implemented by RTI International, has developed several instruments including EGRA
and EGMA to capture essential, reliable, and valid education data, which were piloted in multiple countries, with
funding from both USAID and other donors (www.eddataglobal.org).
2 EGRA is the key element that will help us reach USAID’s global target: to improve the reading skills of 100

million children all over the world.

16
USAID Readers are Leaders Project
Study Report May 2015

improved teaching and access to appropriate reading materials) with results of their
peers from schools that entered the project activities later.

This report presents:

 Baseline results of EGRA and EGMA administration in 60 primary schools

 Longitudinal results of EGRA and EGMA administration in 41 primary schools;
 Results of EGRA and EGMA administration in two special primary schools;
 Analysis of questionnaires filled out by the school directors, teachers and parents
from the 60 schools.

All data presented in this report will be used for planning the activities in the forthcoming
period, tailoring the training modules to the needs of teachers for professional development
and involvement of families and local communities in promoting the reading and mathematics
skills in early grade students.

2. EGRA AND EGMA STUDY IN 2015

The EGRA and EGMA instruments, standardized to the local context, measure how well
students are learning basic reading and mathematics skills in second and third grade, identify
the bottlenecks in acquiring these skills and subsequently help developing intervention in
reading and mathematics improvement.
EGRA and EGMA instruments were piloted in December 2013 with 1,762 assessments of
Grade 3 and Grade 4 students from 22 primary schools in both Macedonian and Albanian
languages. Baseline data were collected through 3,895 assessments of Grade 2 and Grade
3 conducted from May 19 to June 3, 2014 in 42 primary schools.
More information on the background of localizing EGRA and EGMA instruments and their
administration in schools to measure student reading and math performance at the end of
Grade 2 and 3 is available in the Baseline study report, available in Macedonian and English
on http://www.stepbystep.org.mk/WEBprostor/RAL_Baseline_report_September_2014.pdf.
With the project extension in all primary schools in Macedonia by May 2016, it was decided
this year to involve 41 new schools in addition to the 62 schools selected through the
application process for participation of the initial phase 1, 2 and 3 of the project.

The plan for May 2015 was to collect baseline performance data in literacy and math in
additional 61 schools on a sample of 1,000 Grade 2 and 1,000 Grade 3 students.
Additionally, reading and math performance of 1,000 Grade 2 students first tested with
EGRA and EGMA in May 2014 were monitored in 2015 (in Grade 3), and will be further
monitored in May 2016 (in Grade 4) and May 2017 (in Grade 5). The longitudinal
assessment of these students for four years in a row (2014 – 2017), until the end of their
early-grade primary education will help us monitor their performance and compare the results
of these students from the schools that have been involved in project activities since the very
beginning (and benefited from several years of improved teaching and access to appropriate
reading materials) with results of their peers from schools that entered the project activities
later.

The remaining primary schools will be involved in EGRA and EGMA assessments in May
2016, when a sample of 5,000 Grade 2 and Grade 3 students will be assessed. These

17
USAID Readers are Leaders Project
Study Report May 2015

results will indicate the baseline performance in reading and math of these students, but on
the other hand they will serve as a control group for comparison with the results of the
students that were first assessed in 2014 and 2015.

A follow-up assessment study in all schools will be conducted in May 2017 on the same
sample of students tested in the 2016 study (now the students will be in Grade 3 and Grade
4 respectively). This, on one hand, will provide progress data for this sample of students and
on the other will allow comparison with the results of students from the schools that were
involved earlier in project activities.

The improvements are expected in schools that started the project activities earlier (the
original 42 schools in which the instruments were piloted and the first baseline study was
contacted). The baseline and follow-up results from the remaining schools will be used for
diagnostic purposes and will be a basis for establishing the reading standards and eventually
revising current curricula with respect to literacy and numeracy.

Having in mind, that national assessment data in early grades are lacking, these large-scale
studies are a step towards obtaining reliable data and using them for comparison in
subsequent years if the National Examinations Center continues the use of the assessment
tools.

2.1 Sample design

Upon the Ministry’s interest for expansion of project activities in all primary schools around
Macedonia and the official request submitted to USAID Macedonia for no-cost extension, it
was decided to involve additional 40 schools to the original 62 schools for the administration
of EGRA and EGMA instruments in May 2015.

To this end, a step by step procedure was drafted on the best way to select these additional
schools, from the pool of applications collected in September 2013. The regional
representation, language of instruction (Macedonian, Albanian, bilingual) and school location
(urban vs. rural) were also taken into consideration.

Step 1: List with final evaluation ranking of school applications was reviewed (total of 129
remaining applications).

Step 2: Schools from the list were grouped according to the region they belong to (eight
statistical regions in the country: Skopje, Eastern, Southeastern, Pelagonija, Southwestern,
Polog, Northeastern and Vardar region).

Step 3: Schools were proportionally selected according to the language of instruction:

monolingual schools (Macedonian or Albanian language of instruction) and bilingual schools
(both Macedonian and Albanian language of instruction) and according to school location
(urban and rural) to correspond to the original sample of 62 selected schools.

Step 4: Schools that were highly ranked from each region and met the above mentioned
requirements were selected in the sample (priority was given to schools from municipalities
that were not selected in the first round of evaluation).

The list of schools participating in EGRA and EGMA longitudinal and baseline study is
presented in Appendix 1.

18
USAID Readers are Leaders Project
Study Report May 2015

The sampling design and procedure were prepared following the model applied in the
previous studies by PhD Beti Lameva from the National Examinations Center.

The target population for collecting baseline data was defined as all students enrolled in
Grade 2 and 3, with Macedonian and Albanian language of instruction in 60 schools. Using
enrollment data from the pedagogues, a sample of students was selected using a stratified
random design with proportional allocation based on school location, language of instruction
and school size to ensure all regions would have a probability of selection equal to their
actual distribution in the country.

The 60 schools were divided into two strata according to the language of instruction
(Macedonian and Albanian). The bi-lingual schools were considered as two separate
schools. In order for the sample of students to reflect as close as possible the features of
selected schools, first the ratio between Macedonian and Albanian students was determined.
Then based on the number of students in Grade 2 and Grade 3, schools were divided into
small, medium and large schools to calculate the number of students to be selected in each
group.

Each selected student in the sample received a unique code comprised of: unique number of
the school, language in which the test would be administered (Macedonian or Albanian),
number of the class and student’s number in the registry.

This year, it was decided to collect baseline data from around 1,000 students in grade 2 and
1,000 students in grade 3 from the 61 schools that participate in EGRA and EGMA study for
the first time.

Table 1 below provides a breakdown of selected students per category.

Table 1. Number of students per grade and language of instruction in the baseline
sample

Grade
Language of instruction Grade 2 Grade 3 Total
Macedonian 724 702 1426
Albanian 269 265 534
Total 993 967 1960

Additionally, longitudinal data were collected from the sample of around 1,000 Grade 3
students from the initial 42 schools. The baseline data for this cohort of students was
collected as part of the baseline study conducted in May 2014.

Table 2. Number of students per grade and language of instruction in the longitudinal
sample

Grade 2 assessed Grade 3 assessed

Language of instruction in 2014 in 2015
Macedonian 731 692
Albanian 250 229
Total 981 921

19
USAID Readers are Leaders Project
Study Report May 2015

Currently, part of the USAID Readers are Leaders Project are two primary schools for
children with disabilities: Maca Ovcarova – Veles and Sv. Kliment Ohridski – Novo Selo. With
the help of the recommendations received from the special educators from the special
primary school Maca Ovcarova – Veles, who have been already partaking in the reading and
math assessment study, EGRA and EGMA instruments were adapted to be used not only to
assess current level of reading and math skills with students from these schools, but also
their potential for developing these skills in the future.

2.2 Instrument structure

The instrument structure remained the same, keeping the same components as the
instruments developed for the purpose of the pilot and baseline study. However, based on
the lessons learned from the baseline study and notes from the EGRA and EGMA assessors
and observers, some modifications were made in the tasks.

Having in mind the international and Kosovo experience with EGRA and the consultations
with NEC expert, Beti Lameva, this year it was decided to develop one reading instrument for
both Grade 2 and Grade 3, so that the performance of students in these two grades can be
compared.

Considering the EGMA results in the Macedonian context, the comments of the assessors
and observers, but also the changes to the early grade math curriculum introduced in
September 2014, additional modifications were made to the mathematics instrument, which
are outlined in the subsequent sections of this report.

Background questions section

The background information about the students was collected only during EGRA
administration and was then linked with EGMA assessment only through the student ID
number.

The tangerine wizard automatically recorded the date (year, month, day) and time of
assessment, the name of school, its location (urban/rural), place, municipality and region.
Each student was assigned a unique ID number during the selection procedure (same for
EGRA and EGMA), which was entered prior to the test administration.

The background questions were slightly modified based on the lessons learned from the
baseline study and included:

 Type of the school the student attends (central or satellite): satellite schools are
usually located in rural areas, so even if the central school is listed as urban, the data
of the students from satellite schools are considered as rural;
 Grade of the student (second or third);
 Gender of the student (male or female);
 Language of instruction (Macedonian, Albanian, Turkish or Serbian);
 Education of the mother (primary or less, secondary or higher and more);
 Education of the father (primary or less, secondary or higher and more)
 Age of the student;
 Language spoken at home (Macedonian, Albanian, Turkish, Romani, Serbian or other
language): in this question it was possible to mark two and more options for students
from multilingual families;

20
USAID Readers are Leaders Project
Study Report May 2015

 Attendance of preschool;
 Availability of additional books at home;
 Reading before starting school;
 Borrowing books from library;
 Habit of reading independently or together with family members (mother, father, both
parents, siblings, grandparents, someone else).

This section of the instrument was also used to start conversation with the students and
establish rapport, which is an important task in oral, one-on-one assessments.

General Instructions
The instructions from the original instrument were carefully reviewed, paying close attention
to the clarity of the instructions for both students and assessors. To the extent possible,
student instructions were made to be consistent (including ways to encourage the child
without helping him with the task, number of examples, etc.) across each of the tasks.

For all EGRA and EGMA tasks, a practice item was introduced, so the students could easily
understand what is required of them and the feedback provided by the assessors could tell
them whether they are on the right track.

Specific instructions
As to the specific instructions in the tasks, the following decisions were made:
 Guiding the children along the task. Most children showed a natural tendency to
point at tasks and numbers themselves. The assessor is to show what the order of
solving the tasks is, but does not need to glide his/her hand along the numbers and
tasks.
 Mistakes made in the example. If children make a mistake doing the example, the
assessor is allowed to ask: Are you sure? If the child makes a mistake again, the
assessor is to say what the correct answer is, but he/she is not to guide the child in
any way to the correct answer.
 Mistakes made doing the tasks. If the child makes a mistake in one of the tasks and
later wishes to revise its answer, the assessor will accept and register the new
answer and code it as correct or incorrect as the case may be, but should not correct
the mistake or guide the student towards the right answer.
 Asking for repetition of the instruction to a task. Notably in the case of word
problems, children may ask for the task to be repeated. In such cases only the
complete task will be repeated and not the key passage(s).
 Children wanting to see/read the task. Assessors must not show the text they are
reading out to the children. This issue may arise in the word problems, when children
want to see/read the text of the word problem. This is not to be allowed.

Timing
Time-limitation of tasks is useful in making the assessment shorter, and is also less stressful
for both child and assessor, as the child does not have to keep trying to do the whole task at
a slow pace. In addition, timing helps to assess automaticity. More details on timing and stop
rules of all tasks is provided below in the EGRA and EGMA sections.

3. EARLY GRADE READING ASSESSMENT

21
USAID Readers are Leaders Project
Study Report May 2015

The Early Grade Reading Assessment (EGRA)3, is an instrument developed to assess the
main skills that are known to predict reading success within the early grades of primary
school (first to third grade).

Prior to this study, modifications were made to the tasks and instructions developed for the
baseline study to address deficiencies revealed in the previous assessments. Instructions for
each task were also reviewed to assure that they are specific and thorough enough but also
understandable for the assessors and especially the children being assessed.

3.1 The EGRA instrument for Macedonia

Based on the international and Kosovo experience with EGRA and the consultations with
NEC expert, Beti Lameva, one EGRA instrument was developed for both grades in
Macedonian and in Albanian, keeping the same components as in the baseline study:

 Task 1 - Letter knowledge (100 letter combinations)

 Task 2 - Familiar word reading (50 words)
 Task 3 and 5 - Reading fluency (story of around 100 to 200 words)
 Task 4 and 6 - Reading comprehension (several comprehension questions).

The work group revised the letter and familiar word reading tasks taking into account the
results from the baseline study and the difficulties students encountered in these tasks. Two
new stories as well as comprehension questions were developed to assess reading fluency,
accuracy and understating.

Task 1 - Letter knowledge

In this task, students were asked to provide the name of listed letters of the alphabet in
Macedonian and Albanian language.

The student sheet included 10 lines with 10 letters on each line. The letters that were not
recognized or were incorrectly identified were marked and entered into the tablet. In the end,
the sum of all correctly identified letters within 60 seconds time-frame was recorded. Time
taken to read all letters was also recorded.

Examples of the letter knowledge task in both languages are presented below.

А Р С и к Е н Ф т А

И у В Х Л З ж Н б Ц

Т д Г И Ш В ч Е О к

Р В п Ј Т л Е н З Б

и К љ м р с Џ О Ѕ А

3 EGRA: Early Grade Reading Assessment (RTI International for U.S. Agency for International Development
[USAID]). For instruments and reports, see: www.eddataglobal.org.

22
USAID Readers are Leaders Project
Study Report May 2015

т Н д Е Х З М ј Р о

С Р м к б А Ж В Ч е

Љ И Ц Д Т з Ф ѕ п Л

ш Л Ж Ј ќ О У Г џ Н

к Ѓ А П в Г ј Р Х о

Figure 1. Letter Knowledge task in Macedonian language

D V j u N k b i S o

f R q A ll Nj m y Z dh

rr X t sh G ë ç L h gj

Th c p M e zh d K n U

s L Xh a r F ll O nj Y

J x T Sh v xh K i g z

V Gj d f Ll o Zh j S Ë

e Q u H Rr nj t m C y

X Dh a N k B r L i H

Ç P v th D g U xh e S

Figure 2. Letter Knowledge task in Albanian language

Task 2 - Familiar word reading

The second task consisted of a list of 50 words for each language (10 lines with 5 words).
During the development of items, special attention was paid, the words in Macedonian and
Albanian to have same number of letters thus equalizing the time required for reading (as
both languages are phonetic).

Each student was asked to read every word as best as s/he could and as reasonably fast,
within 60 seconds. The assessors were instructed to mark as incorrect all those words that
were read in non-acceptable formal pronunciation. If a student read all words in less than
one minute, the time taken to complete the task was also recorded and entered, so as to
calculate the correct words per minute (cwpm).

The sums of a) all words read irrespective of being correct or not (attempted), and b) all
words read correctly within 60 seconds (correct) were also recorded in order to calculate the
accuracy of students in reading familiar words.

23
USAID Readers are Leaders Project
Study Report May 2015

со леб Илир тесла летово

весели трета мила хор на
ќе оди Неда прсти капина
Фанија труба брат цар во
еж син мува брада лисица
дарува Љубен лале нос од
ни прв Едон љубов спомен
именка Линда горд воз на
рог цути басна штурец зборува
подарок ѕвонче топка дада прв

Figure 3. Familiar Word Reading Task in Macedonian language

ti fle pemë letër domate

këpucë Elena luaj sot la
po tre Besa valoj liqeni
gazeta shtylla Andi gjeth ka
ha gur lumi dreri Vojdan
dëgjues qielli trim mur ve
ju çaj verë druri tigani
Jonila llamba bukë thoi pi
ura Dime tigri xhaketa vallëzoj
fletore liqeni unazë pula kjo

Figure 4. Familiar Word Reading Task in Albanian language

Task 3 and 5 - Reading fluency

For the reading fluency, two short stories were developed. Each story was created in such
way to reflect the sentence complexity and vocabulary for the grade level.

The task was timed, so that the correct number of words read per 60 seconds was recorded.
The number of words read incorrectly was also recorded and entered as attempted. If a
student read the passage in less than 60 seconds, the time taken to complete the task was
recorded as well. Typically, oral reading tests are calibrated so as to allow a student reading
at somewhere around 60 words per minute or to read the passage in about one minute.
Еден топол пролетен ден, маjка ми отиде да ја посети тета. Јас и
бато останавме дома со тато. Пред да замине, таа нѐ замоли да си ја
средиме собата.
Се договоривме со бато, секој да си ги среди своите играчки,
книги и облека.
24
Кога мајка ми се врати, ја погледна собата. Таа се насмеа и силно
нѐ прегрна.
USAID Readers are Leaders Project
Study Report May 2015

Figure 5. Example of Reading Fluency Task in Macedonian language

Një ditë të nxehtë të pranverës nëna ime shkoi ta vizitojë tezen. Unë
dhe vëllai ngelëm në shtëpi me babin. Para se të shkojë, ajo na luti ta
rregullojmë dhomën tonë.
U morëm vesh me vëllanë, secili t’i rregullojë lodrat, librat dhe veshjet
e tij.
Kur u kthye nëna ime, e pa dhomën. Ajo buzëqeshi dhe fort na
përqafoi.

Figure 6. Example of Reading Fluency Task in Albanian language

Task 4 and 6 - Comprehension questions

Comprehension questions were designed to correlate with the content of the stories. Given
that the comprehension questions task was based on the text from the reading fluency task,
these two tasks were also linked in the Tangerine, so if students failed to read through the
end of the text, the wizard provided the questions only for the portion of the text that was
read.

The assessors were provided in the tablet with several alternative answers for each of the
comprehension questions, but were also instructed to assess as correct any reasonable
answer, as these alternatives were not exhaustive.

The total number of attempted and correct answers was recorded and as such entered into
the database, to calculate the accuracy.

Below is an example of the reading comprehension questions in both languages.

Table 3. Example of Comprehension Questions Task in Macedonian language

Текст бр. 1 ПРАШАЊА

Еден топол пролетен ден маjка 29 1. Кого посетила мајката?
ми отиде да ја посети тета. - тета
Јас и бато останавме дома со - отишла кај тетката на детето
тато. - сестра си (својата сестра)
Пред да замине, таа нѐ замоли да - мајката ја посетила... (сите можни
си ја средиме собата. горенаведени категории)

2. Со кого останале децата дома?

- со тато
- со татко им
- децата останале дома со татко им

3. Што ги замолила мајката своите

25
USAID Readers are Leaders Project
Study Report May 2015

деца?
- да ја средат (нивната) соба
- да ја наместат (нивната) соба
- да ја исчистат (нивната) соба
- мајката ги замолила децата да си ја
средат/ наместат/ исчистат собата
Се договоривме со бато, секој да 43 4. Што се договориле децата?
си ги среди своите играчки, - да ја средат собата
книги и облека. - да ја послушаат мама
- да ја израдуваат мама
- секој да си ги среди своите играчки,
книги и облека
- да си ги средат играчките, книгите и
облеката
- да си ги средуваат своите работи
- секој сам да си ги средува своите
играчки, книги и облека
- децата се договориле... (сите можни
горенаведени категории)
Кога мајка ми се врати, ја 58 5. Зошто мајката ги прегрнала децата кога
погледна собата. се вратила?
Таа се насмеа и силно нѐ - затоа што ја средиле собата
прегрна. - затоа што ги средиле играчките,
книгите и облеката
- затоа што ја послушале мајката
- затоа што биле одговорни (одговорно
дете)
- затоа што биле добри/ вредни
- затоа што биле исполнителни/уредни
- затоа што биле умни
- затоа што не ја оставиле собата
несредена/ растурена
- затоа што ја завршиле нивната задача
- мајката ги прегрнала децата затоа
што... (сите можни горенаведени
категории)

6. Како се чувствувале децата по

прегратката на мајката?
- среќно
- радосно
- весело
- насмеано
- задоволно
- гордо
- по прегратката на мајката децата се
чувствувале... (сите можни
горенаведени категории)

7. Како се чувствувала мајката кога

видела дека децата ја послушале?
- среќно
26
USAID Readers are Leaders Project
Study Report May 2015

- весело
- радосно
- задоволно
- гордо
- насмеано
- изненадено
- кога видела дека децата ја послушале,
мајката се чувствувала... (сите можни
горенаведени категории)

8. Што мислиш, зошто треба собата да

ти биде средена?
- ќе бидеме поздрави
- мама нема да не кара (нема да ни се
налути)
- за да ми биде убава
- за да ми биде чиста
- за никој да не ме кара
- за да не ми го земат мобилниот
- за да не се усрамиме пред гости
- за да ја израдуваме мама
- за да не се собираат пајаци и инсекти
- за да не се разболиме (од микроби/
бактерии)
- за да немаме прашина
- за да можеме да играме
- за да можеме подобро да спиеме
- за да не се сопнеме и паднеме
- за да учиме подобро
- собата треба да биде средена... (сите
можни горенаведени категории)

Table 4. Example of Comprehension Questions Task in Albanian language

Tregimi nr. 1 Pyetje

Një ditë të nxehtë të pranverës nëna 31 1. Kë vizitoi nëna?
ime shkoi ta vizitojë tezen. - tezen
Unë dhe vëllai ngelëm në shtëpi me - shkoi te tezja e fëmijës
babin. - te motra (motra e saj)
Para se të shkojë, ajo na luti ta - nëna shkoi ta vizitojë ...(të gjithë kategoritë e
rregullojmë dhomën tonë. përmendura më lartë

2. Më kë ngelën fëmijët në shtëpi

- me babin
- me babin e tyre
- fëmijët ngelën në shtëpi me babin e tyre

3. Çka i luti nëna fëmijët?

27
USAID Readers are Leaders Project
Study Report May 2015

- ta rregullojnë dhomën (e tyre)

- ta rendisin dhomën (e tyre )
- ta pastrojnë dhomën (e tyre)
- nëna i luti fëmijët ta rregullojnë/ta rendisin/ ta
pastrojnë dhomën
U morëm vesh me vëllanë, secili t’i 45 4. Çka u morën vesh fëmijët?
rregullojë lodrat, librat dhe veshjet e - ta rregullojnë dhomën
tij. - ta dëgjojnë nënën e tyre
- ta gëzojnë nënën e tyre
- secili t’i rregullojë lodrat, librat dhe veshjet e tij
- t’i rregullojnë lodrat, librat dhe veshjet e tyre
- t’i rregullojnë gjerat e tyre
- secili vet t’i rregullojë lodrat, librat dhe veshjet
e tyre
- fëmijët u morën vesh ... (të gjithë kategoritë e
përmendura më lartë)
Kur u kthye nëna ime, e pa dhomën. 59 5. Pse nëna i përqafoi fëmijët kur u kthye?
Ajo buzëqeshi dhe fort na përqafoi. - sepse e rregulluan dhomën
- sepse i rregulluan lodrat, librat dhe veshjet (e
tyre)
- sepse e kishin dëgjuar nënën
- sepse ishin përgjegjës (fëmijë të përgjegjshëm)
- sepse ishin të mirë/ të zellshëm
-sepse e plotësuan detyrën
- sepse ishin të mençur
- sepse nuk e lënë dhomën e pa rregulluar/ pa
pastruar
- sepse e mbaruan detyrën
- nëna i përqafoi fëmijët sepse... (të gjithë
kategoritë e përmendura më lartë)

6. Si u ndjen fëmijët pas përqafimit të nënës?

- të lumtur
- të gëzuar
- të kënaqur
- të buzëqeshur
- u ndjen krenar
- u ndjen mirë
- fëmijët pas përqafimit të nënës u ndjen...(të
gjithë kategoritë e përmendura më lartë)

7. Si ndjehej nëna kur pa se fëmijët e kanë

dëgjuar?
- e gëzuar
- e lumtur
- e kënaqur

28
USAID Readers are Leaders Project
Study Report May 2015

- u ndie krenare
- e buzëqeshur
- e befasuar
- kur e pa se fëmijët e kanë dëgjuar nëna u ndie
... ...(të gjithë kategoritë e përmendura më lartë)

8. Çka mendon, pse dhomën duhet ta kesh të

rregulluar?
- do të jemi më të shëndoshë
- nëna nuk do të më qorton (nuk do të
hidhërohet)
- të jetë më e bukur
- të jetë më e pastër
- që askush mos të më qortojë
- mos të ma marrin telefonin mobil
- mos të turpërohem para mysafirëve
- ta gëzojë nënën
- mos të mblidhen merimanga dhe insekte
- mos të sëmuremi (nga mikrobet/ bakteret)
- mos të ketë pluhur
- të mundemi të luajmë
të mundemi të flemë më mirë
- mos të pengohemi dhe të lëndohemi
- të mësojmë më mirë
- dhoma duhet të jetë e rregulluar ... ...(të gjithë
kategoritë e përmendura më lartë)

Timing and Stop Rules, by task

Timing of EGRA tasks is very important, as students achieve automaticity on the mechanics‛
of reading – i.e. matching letters and graphemes to sounds to make up words and
sentences- they develop fluency in reading, allowing them to read longer texts and focus on
the meaning of the text. “Automaticity” means fluency in word recognition so that the reader
is no longer aware of or needs to concentrate on the mental effort of translating letters to
sounds and forming sounds into words. At that point, the reader is decoding quickly enough
to be able to focus on comprehension.
Table 2 breaks down the timing and stop rules in place for each of the EGRA tasks applied in
Macedonia in both grades.

Table 5. Timing and stop rules for EGRA tasks

Task Timing Stop Rule

Letter knowledge 60s 3s

Familiar word reading 60s 3s

Reading fluency 60s 3s

29
USAID Readers are Leaders Project
Study Report May 2015

Reading comprehension No 15s

4. EARLY GRADE MATHEMATICS ASSESSMENT

Early Grade Mathematics Assessment (EGMA) is consisted of eight tasks: number

identification, quantity comparison, missing number (number patterns), addition, subtraction,
word problems, shape recognition and pattern extension.

Having in mind that new curriculum in mathematics and sciences introduced in Macedonia
starting from September 2014, the tasks in the EGMA instrument for the follow-up studies
were carefully revised to reflect the changes and to be adjusted to the new curriculum
objectives.

The following modifications to the tasks were made:

 Number identification: the number of items increased to 20 in Grade 2 and 25 in
Grade 3, as well as the level of difficulty of items.
 Number discrimination: more difficult items were included to measure this ability
more accurately. The task was also timed for 60 seconds.
 Missing numbers: the items were reformatted to provide even distribution of the
items that progress in difficulty.
 Addition and subtraction: these tasks were divided into two sets of
addition/subtraction items. The first set consisted of five items that we would expect
the child to be able to calculate mentally and answer fluently. Level 1 assessed for
fluency i.e. whether children were becoming familiar with simple addition/subtraction
problems, with a total of five items to be timed for 60 seconds. All of the numbers
used in this set did not involve the bridging through tens. The second set, also
comprised of five items, involved double-digit numbers as well as bridging through
tens. For the second set, the students will be allowed to use manipulatives (counters)
and/or pencil and paper to perform the calculations. Level 2 was timed for 2 minutes,
however the timing was not for fluency, but for efficiency.
 Problem solving: the number values in the word problems are small because with
this subtest we were not testing the student ability to solve the arithmetic as much as
we were testing their ability to make sense of/interpret a problem statement.
 Shape recognition and pattern extension tasks were not modified.

In order to check whether the new tasks corresponded to the new math curriculum, the
instrument was piloted with students from a primary school in Skopje. Minor changes were
made to the tasks after this piloting.

The instructions for each task were additionally revised to reflect the changes in the tasks.

A sample of mathematics task in each area are provided below:

Task 1 - Number Identification

For the number identification task, students were shown a stimulus page with four rows (five
in Grade 3) and five numbers in each row. Students were asked to point to each number and
tell the assessor the number name. This task was timed for 60 seconds.

Although number identification seemed way too easy even for Grade 2 students, it was kept
as part of the instrument, because it is the first assignment the students get. The easy task

30
USAID Readers are Leaders Project
Study Report May 2015

can make them feel more confident and competent to do the math test. However, the number
and the difficulty of items increased.

2 9 16 11 13
20 15 70 4 18
32 57 75 99 8
100 64 7 51 44

Figure 7. Example of Number Identification Task for Grade 2

Task 2 - Number Discrimination

For the quantity discrimination task, students were shown two numbers at a time and asked
to tell the assessor the number name of the bigger number. The task included seven items
for Grade 2 and ten items for Grade 3.

As the average percentage of students that performed number discrimination task was high,
more difficult items were included to measure this ability more accurately. Given the increase
of difficulty in items and the goal of having students demonstrate some level of efficiency with
this task, this year the task was timed for 60 seconds.

4 7
17 14
89 98
15 13
43 27
19 16
62 63

Figure 8. Example of Number Discrimination Task for Grade 2

Task 3 - Naming Missing Numbers

For the missing number task, students were shown a stimulus page, and asked to tell the
assessor the missing number.

In the baseline study, students in each of the grades tested had difficulty with this task. This
could make it difficult for them to master multiplication and other, more complex, problem-
solving later on. The results showed that students were struggling with counting in steps
other than 1, such as a count-by-two, three or five.. Counting in steps is critical if children are
to be able to decompose and recompose numbers—critical to working flexibly with numbers.

Based on the information collected, this task was reformatted with an even distribution of
items that progressed in difficulty. There were five items in the Grade 2 and seven items in
the Grade 3 instrument.

31
USAID Readers are Leaders Project
Study Report May 2015

5 6 7 [8]
21 24 [27] 30
75 80 85 [90]
[9] 11 13 15
18 16 [14] 12

Figure 9. Example of Naming Missing Numbers Task for Grade 2

Task 4 and 5 - Addition and Subtraction

As mentioned earlier, these two tasks were substantially modified and two level items were
provided: level one with easier items, without use of counters that were timed for 60 seconds
and level two with items with carryover that were timed for efficacy for 120 seconds, in which
use of counters was allowed.

2 + 6 = (8)
12 + 7 = (19)
24 + 6 = (30)
40 + 4 = (44)
40 + 30 = (70)

Figure 10. Example of Addition task Level 1

6 + 7= (13)
8 + 9 = (17)
33 + 5= (38)
48 + 2 = (50)
77 + 7 = (84)

Figure 11. Example of Addition task Level 2

3 - 3 = (0)
10 - 5 = (5)
19 - 2 = (17)
60 - 20 = (30)
56 - 10 = (46)

32
USAID Readers are Leaders Project
Study Report May 2015

Figure 12. Example of Subtraction task Level 1

50 - 25 = (25)
65 - 9 = (56)
42 - 16 = (26)
66 - 36 = (30)
100 - 27 = (73)

Figure 13. Example of Subtraction task Level 2

Task 6 - Word Problems: addition/subtraction & multiplication/division (orally)

In the Grade 2 task, two items tapped addition and two items tapped subtraction. In the
Grade 3 task there was an items for each of the four operations: addition, subtraction,
multiplication and division.

The number values in the word problems were purposefully small because with this subtest
we are not testing the student ability to solve the arithmetic as much as we are testing their
ability to make sense of/interpret a problem statement.

This section was timed at four minutes to prevent exhausting the weaker students and to
keep the test within the time limit of 15-20 minutes.

Учениците од второ одделение собирале книги за училишната библиотека.

Колку вкупно книги собрале ако во месец март донеле 8, а во месец април 7
книги?
Точен одговор: 15

Nxënësit e klasës së dytë mblidhnin libra për bibliotekën e shkollës. Sa libra

gjithsej kanë mbledhur nëse në muajin mars kanë sjellë 8, kurse në muajin prill 7
libra?
Përgjigje e saktë 15

Figure 14. Example of Word Problems Tasks in Macedonian and Albanian language for
Grade 2

Task 7 - Geometry: Shape Recognition

In this section student ability to identify and select specific shapes is assessed (circles,
squares, rectangles and triangles). Four items were given to both grades, not timed.

33
USAID Readers are Leaders Project
Study Report May 2015

Figure 15. Example of Shape Recognition Task for Grade 2

Task 8 - Geometry: Pattern Extension

In this task student ability to identify similarities and differences among objects that make up
a pattern is assessed. There were three items for Grade 2 and four items for Grade 3 in this
untimed section.

Figure 16. Example of Pattern Extension Task for Grade 2

Counting Strategies
The counting strategies used by children were observed during the level 2 of addition and
subtraction as well as word problem tasks. For this purpose, counters or manipulatives as
well as paper and pencil were provided to students for doing the necessary calculations. The
children were also allowed to use their fingers. After each task, the assessor was asked
based on the observations to record the type of strategy used by the student (e.g., counting
fingers, using counters, writing down on paper or mental calculation).

34
USAID Readers are Leaders Project
Study Report May 2015

Timing and Stop Rules, by Task

Timing of some of the tasks is vital to establish fluency, to reduce the time taken to complete
the assessment, and to relieve the stress children might feel in trying to perform a task
unsuccessfully for an indefinite period of time.

To ensure that children do not get fatigued or overwhelmed and to learn of their ability for
each of the tasks, a universal stop rule has been put in place. This stop rule applied to all of
the tasks, timed or with current stop rules. The rule was: If a student gets the first four items
incorrect, one after the other, the assessor should stop the student and move on to the next
task. However, each task must be attempted.

Table 3 breaks down the timing and stop rules in place for each of the EGMA tasks applied
in Macedonia in both grades.

Table 6. Timing and stop rules for EGMA tasks

Task Timed Stop rule

Grade 2 Grade 3 Grade 2 Grade 3
Number identification 60s 60s 5s 5s
Number Discrimination 4 consecutive 4 consecutive
60s 60s errors or 5s errors or 5s
Naming Missing 4 consecutive 4 consecutive
Numbers No No errors or 5s errors or 5s
Addition/ Subtraction
Level 1 60s 60s 10s 10s
Addition/ Subtraction
Level 2 120s 120s 30s 30s
Word Problems No 4min 10s 10s
Geometric Shape
Recognition No No No No
Geometric Pattern
Extension No No No No

Two instruction manuals prepared for the baseline study were modified to serve as a helpful
resource for assessors and observers in the schools.

The first manual for administration of EGRA and EGMA tools presented the two instruments
and their tasks as well as the general procedures for their administration. Furthermore, it
provided specific instructions for EGRA and EGMA tasks for second and third grade
students, how to mark the responses of the students, how to use the student worksheets and
counters for math tests, and how to use the stop rule. The final part of this manual was
focused on the tangerine application and use of the tablet computers, the process of logging
in and out, the basic settings, data gathering, using the automatic stopwatch, saving and
synchronizing the data and other useful recommendations for the testing process.

The second manual was focused on quality assurance during the testing process and was
intended for the observers. It outlined the main responsibilities of observers, their specific
role in the school, the way of recording and reporting their observations.

35
USAID Readers are Leaders Project
Study Report May 2015

5. FIELDWORK AND DATA COLLECTION PROCESS FOR EGRA & EGMA

5.1 Training for Administration of EGRA and EGMA Instruments
Assessors responsible for administration of EGRA and EGMA instruments in the schools
have a substantial influence on the quality of any EGRA and EGMA implementation, so they
should be properly trained for administration of the instruments with the help of tablet
computer. This is why each school director from the participating schools was advised to
nominate for the training a member of the student support staff (pedagogue or psychologist)
and only if this was not possible then to nominate an early grade teacher that is not directly
involved in the instruction process in Grade 2 and 3. Total of 72 persons were nominated to
attend the training. The training was held from May 5 to 7, in Ohrid.

Because the number of participants was larger than planned for this type of workshop, the
training content was adjusted to be delivered in more interactive format, taking special care
to allocate sufficient time to the most important parts – inter-raters’ reliability and practical
work with tablets.

The training started with presentation of project activities to introduce the participants with
everything the project has implemented so far. Then the participants were divided into
several groups and assumed the role of task developers and based on the EGRA and EGMA
methodology developed the tasks for the instruments. In this way, they had hands-on
experience for developing the instruments and understood the logic behind the methodology.

The second day was focused on practical work with tablet computers and use of tangerine
application and by the end of the day their inter-rater reliability was tested. The results from
the inter-rater reliability were discussed the next day and the most common mistakes were
noted in order to pay attention to this during the assessment process.

The training ended with explanation of the procedure for selection of students in the sample,
review of roles and responsibilities of supervisors and assessors, and the logistical
preparations for the study in May. During the final day of training, the participants also
worked in groups and by using the method of Six Thinking Hats, highlighted the benefits and
cautions of using EGRA and EGMA in the schools. With this exercise, the participants
became aware of the advantages of using these instruments, the facts behind EGRA and
EGMA, the creative ways in which they could be applied in the schools, but also addressed
the criticism they may face from the teachers and school staff.

Each participant received one tablet (for the participants from bilingual schools training
tablets were provided from the project office), which proved to be useful, as all participants
simultaneously tried the tests, and worked with the tablets.

According to the participants’ evaluation, they mostly liked the practical work with tablet
computers and the use of EGRA and EGMA on the tangerine application. The participants’
comments were very positive of the organization of the training, suggesting dissemination of
this type of training for the teachers. The participants from schools with Turkish language of
instruction recommended the instruments to be also available in Turkish as well. The only
negative comments of participants were that each assessor should have own tablet in the
school for practice but also for assessment purposes.

From the trainers’ perspective, the following lessons were learned:

36
USAID Readers are Leaders Project
Study Report May 2015

 The group of 72 participants is too large for training of this type. It is recommended
the size of the groups to be reduced to maximum of 50 persons for the future training
sessions;
 Two participants should be trained from each school, so they could exchange their
experience and help each other;
 Each participant should have own tablet during the training to make the work more
productive. In the bilingual schools, the assessor in each language should have own
tablet to make the assessment more effective;
 The internet in the hotel should be of high quality in order to prevent collapse of the
system when more than 70 tablets are trying to download the instruments;
 The participants should work in mixed groups and when presenting their work one
participant in Macedonian and one in Albanian should present interchangeably;
 The presentations should be always provided in two languages and the participants
should be allowed to speak in their native language, while translation to be provided
for the others.

These lessons learned will be applied when organizing the training for administration of
EGRA and EGMA in the remaining primary schools in Macedonia.

5.2 Data collection

Data collection took place between May 18 and June 6, 2015 and was carried out by 158
assessors, comprised of pedagogues, psychologists and/or selected early grade teachers
that were trained to take the role of assessors.

The project team prepared the correct number of laminated student stimuli sheets in
Macedonian and Albanian language that were delivered to the schools through Delivery
Company four days prior to the start of the data collection process. This was also done in
consultation with experts from the National Examinations Center, so that the assessors had
enough time to familiarize with the new content of the tasks and modified instructions.

The assessment process involved a 10- to 20-minute, individual, oral assessment between
the student and the EGRA and EGMA assessor. Depending on the size of the school and the
language of instruction (Macedonian only, Albanian only or bilingual), assessors conducted
the assessment, with around 20 to 120 students per school. The data were recorded
electronically via the Tangerine application and sent by the assessors directly to the central
database.

Selected observers as well as the project team visited the schools regularly to verify the
completeness and clarity of the assessment process.

For successful implementation of the future studies these lessons learned should be taken
into consideration:
 At least two assessors should be trained to administer the instruments in each
language of instruction in each participating school;
 Additional tablets should be provided to the large schools to make the assessment
less time consuming;
 The time period for administration of EGRA and EGMA should be at the beginning of
May at latest, so that the school pedagogues and psychologists can perform the other
duties that are usually carried out in May (enrollment of first-graders, preparation for
external testing, etc.);

37
USAID Readers are Leaders Project
Study Report May 2015

 The internet connection in the schools is very bad, which makes the sending of
assessment data to the central database more difficult;
 The tablets of better quality (Nexus, Lenovo) have better performance, so these types
of tablets should be procured.

Once the data collection process was finalized, the Literacy and Numeracy Assessment
coordinator extracted the data from the cloud, cleaned the missing data and sent them to
Beti Lameva from NEC for analysis.

5.3 Quality Assurance

Quality assurance of data collection process is crucial for obtaining valid and reliable data.
Monitoring is one of the components on how to improve the process of EGRA and EGMA
administration. Because additional 61 schools were included in this year’s assessment study,
more monitoring visits were planned to cover as many schools as possible. Again observers
from the ranks of NEC advisors, professors from Pedagogical faculties’, practitioners from
the Work Groups that developed the instruments and the Step by Step educators were
selected to monitor the administration process. Total of 21 observers conducted 122 school
visits during the assessment process.

Also members of the project team were mobile and provided assistance and technical
support to the school assessors. Additionally a member of the project team administered
EGRA and EGMA instruments in two schools where no Albanian speaking assessor was
trained: “Elpida Karamandi” – Bitola (satellite school in Dolenci village) and “Goce Delcev” –
Aerodrom.

Each observer had to submit report with observation notes and check lists from the
monitoring process. The main findings from the reports are the following:

Testing conditions:

 Sound insulation in classroom, pedagogue/psychologist’s office or school library

where administration of assessments took place was satisfactory. However, the issue
with the noise during the school breaks remains, which negatively affects and
disturbs the administration process;
 Regarding the visual surrounding, in some instances it was not satisfactory,
especially in schools in rural areas (not enough space, lot of technical equipment in
the classroom - computers, laptops etc.);
 Most of the assessors established good rapport and communication with the students
and they had good pedagogical approach with students;
 In most of the schools the materials were ready and in place before each
assessment;
 In some of the schools the assessment process was interrupted on multiple
occasions because school representatives (teachers, director) were entering the
classroom, even though in most cases the classroom/office was labeled with special
sign “Assessment in progress, do not disturb”. Also, in one rural school, parents were
present in the same classroom during the assessment of their child. Some schools
directors and other school representatives were present together with the assessor
during the administration process, which is forbidden and is against the assessment
guidelines. Additionally, the increased load of activities in this period of the school
year has burdened the assessors (especially in those schools where one assessor

38
USAID Readers are Leaders Project
Study Report May 2015

was trained) and there were rare cases in which they failed to complete the
assessment process.

Conduct of assessors:

 Some of the assessors were not familiar enough with the instructions and they did not
follow the manual guidelines. This was especially evident in providing the
manipulatives for almost all math tasks (the guidelines suggested using counters
and/or pencil and paper only for second level of addition/subtraction tasks and word
problems) which prolonged the administration of EGMA;
 In few cases assessors did not turn off their cell phones during the administration
process, so the students were interrupted which negatively affected their focus and
attention;
 Some of the assessors gave feedback when the student had correct answer and
were silent when the answer was wrong (indicating to the student whether s/he was
right or not). Small number of assessors were guiding the students towards the
correct answer;
 Some of the assessors repeated the instructions more times than it was allowed;
 Sometimes the STOP rule was not used consistently;
 In few occasions assessors failed to inform the observers about changing their
schedule for administration of EGRA and EGMA in their schools, so the planned
monitoring visits had to be rearranged.

Although the observers shared their comments with the project team through their reports
and during the regular telephone briefings, a separate meeting was held with most of the
observers, so that they could exchange their experience and give recommendations for the
next year’s study which will include all primary schools in Macedonia.

This is the summary of observers’ recommendations:

 Assessors should stop with administration of the instruments during the student
recess (lunch break) in order to prevent any disturbances of the process;
 To be careful with the translation of words from EGRA instrument in Albanian
language, as some words may be unknown in different areas as they are not
frequently used in everyday communication. This was an issue especially in rural
areas where dialects of the language are used;
 The assessors should have more practice with tablet computers (to be familiar with
the manual guidelines) prior to the beginning of the study;
 In the introductory questions, particularly on education level of parents, the option
unknown should be added for children who have no parents or live in foster care.
Also, the information of whether the child studies in a combined class is relevant for
the EGRA and EGMA results;
 Administration of EGRA and EGMA instruments should be implemented either before
or in the first two weeks of May;
 For the upcoming studies, at least one pedagogue and/or psychologist should be
trained from each school, as the existing experience proves that early grade teachers
have lack of time, as they are also busy with the classes.

At the end of the meeting, the observers concluded that one observer should monitor up to
five schools and spend more time in each school to become familiar with the assessment
skills of the assessors.

39
USAID Readers are Leaders Project
Study Report May 2015

5.4 Data analysis and reporting

The actual data analysis was done by Beti Lameva from the National Examinations Center.

The analysis and reporting of results from EGRA and EGMA present the student
performance in reading and mathematics and examined differences by gender, language of
instruction, location of school and education of parents.
Licensed TiaPlus4 and SPSS software were used for data analysis.

The following outputs were obtained with TiaPlus:

 Frequency distributions for each subgroup-subtest combination – where each entry in
the table contains the frequency or count of the occurrences of values within a
particular group or interval, and in this way, the table summarizes the distribution of
values in the sample.
 P-value, which represents a measure of average 'difficulty' (or 'easiness') for the
(sub)test. It is normally obtained by dividing the average test score by the maximum
possible test score and multiplying this by 100. Can be understood as the proportion
(x 100) of persons selecting the correct item answer. Indicating the difficulty of the
item. The higher the P-value the easier the item.

Based on the student results, individual school reports will be prepared and distributed to
each school. For project purposes, the schools will be ranked by their performance using the
average scores (p-values) for all tasks in Macedonian and Albanian. This section is
developed for internal purposes only to guide the focus of remedial interventions and to
indicate which schools deserve more attention.

6. EGRA & EGMA BASELINE FINDINGS

This section presents summary statistics for all tasks of the EGRA and EGMA baseline
assessment in Macedonia conducted in 60 schools.

6.1 Characteristics of Sample

The breakdown of sample of students taking EGRA by grade, language of instruction,
location of the schools and gender is presented below, showing a balanced sample in both
grades according to all three features.

Approximately 73% of the sampled students attend classes in Macedonian and 27% in
Albanian language of instruction. Majority of students (66%) are from urban areas, while 34%
from rural. The sample is composed of an average of 53% male and 47% female students.

4TiaPlus is a 32 bits Windows computer program for Test and Item Analysis (TIA for short), focused
on "classical" test and item analysis developed by Cito, one of the world’s leading testing and
assessment companies, based in the Netherlands.

40
USAID Readers are Leaders Project
Study Report May 2015

Figure 17. Percentage of students in EGRA sample per grade, language of instruction,
location of the schools and gender

The distribution of students in different categories may vary in EGMA sample a little, because
some students were absent during the testing time and were not able to take the test, but it is
still balanced in both grades according to all three features.

Figure 18. Percentage of students in EGМA sample per grade, language of instruction,
location of the schools and gender

6.2 How Well Are Students Reading in Macedonian and Albanian?

The average percentage of students that completed the EGRA test is based on the average
P-value which is obtained by dividing the average test score by the maximum possible test
score and multiplying this by 100.

The comparison of results between EGRA in Macedonian and in Albanian language in Grade
2 shows that students in Albanian language of instruction perform better than those in
Macedonian language in all tasks except for the reading comprehension task.

41
USAID Readers are Leaders Project
Study Report May 2015

The students in both languages have best results in the letter knowledge task (71% in
Macedonian and 72% in Albanian language complete this task), while the worst results are in
the reading comprehension (41% of students completed this task in Macedonian and 39% in
Albanian language).

Figure 19. Average percentage of students that completed EGRA tasks for Grade 2

In Grade 3, students with both Macedonian and Albanian language of instruction have similar
results in letter knowledge and familiar reading tasks, while students with Macedonian
language of instruction outperform the students in Albanian language of instruction in reading
fluency (72% against 68%) and reading comprehension (65% against 58%). Again the best
results are shown in the letter knowledge (83% in Macedonian and 84% in Albanian
language of instruction) and worst results in reading comprehension (65% in Macedonian
and 58% in Albanian language).

42
USAID Readers are Leaders Project
Study Report May 2015

Figure 20. Average percentage of students that completed EGRA tasks for Grade 3

6.2.1 Results according to gender

In the overall test results, female students tend to perform slightly better than male students
in Macedonian language of instruction (56% for the girls, compared with 55% for the boys),
while in Albanian language of instruction male students have better results (55% for girls
compared with 58% of the boys).

Figure 21. Average percentage of Grade 2 students that completed EGRA according to
gender

In Grade 3, female students are better than male students for both languages of instruction
(78% vs 73% for Macedonian language of instruction and 75% vs 72% for Albanian language
of instruction).

43
USAID Readers are Leaders Project
Study Report May 2015

Figure 22. Average percentage of Grade 3 students that completed EGRA according to
gender

When the test is broken down by tasks, for the students with Macedonian language of
instruction girls are slightly better than boys in all EGRA tasks in Grade 2, but the familiar
word reading. Again the best results are achieved in letter knowledge and worst in reading
comprehension.

Figure 23. Average percentage of Grade 2 students that completed EGRA tasks in
Macedonian language according to gender

44
USAID Readers are Leaders Project
Study Report May 2015

In Grade 3, the same trend continues among the students with Macedonian language of
instruction, where girls score better than boys. Letter knowledge is still the task where best
results are achieved (around 82% for the boys and 85% for the girls) and reading
comprehension is the task with worst results (68% for girls and 61% for boys).

Figure 24. Average percentage of Grade 3 students that completed EGRA tasks in
Macedonian language according to gender

In Albanian language of instruction in Grade 2, male students are better in letter knowledge,
familiar word reading and reading fluency, while female students are better in reading
comprehension task.

45
USAID Readers are Leaders Project
Study Report May 2015

Figure 25. Average percentage of Grade 2 students that completed EGRA tasks in
Albanian language according to gender

In Grade 3, female students show better results than male students in all tasks.

Figure 26. Average percentage of Grade 3 students that completed EGRA tasks in
Albanian language according to gender

46
USAID Readers are Leaders Project
Study Report May 2015

6.2.2 Results according to education of parents

Based on the level of education of parents, results of Grade 2 students improve

progressively as the level of education of parents increases from primary to higher education,
for both mother and father, in Macedonian and Albanian language of instruction.

Around 40% of the students, whose parents have primary education or less are completing
the test in Macedonian language compared with 65% of students whose parents have higher
education or more.

Figure 27. Average percentage of Grade 2 students that completed EGRA in

Macedonian language according to education of parents

For the Albanian language of instruction, around 52% of the students, whose parents have
primary education or less are completing the test compared with 70% of students whose
parents have higher education or more.

47
USAID Readers are Leaders Project
Study Report May 2015

Figure 28. Average percentage of Grade 2 students that completed EGRA in Albanian
language according to education of parents

In Grade 3, the education of parents again seems to be predictor of the student results,
where students whose parents have lower level of education perform worse than students
whose parents have higher level of education.

48
USAID Readers are Leaders Project
Study Report May 2015

Figure 29. Average percentage of Grade 3 students that completed EGRA in

Macedonian language according to education of parents

This trend is evident for students with both Macedonian and Albanian language of instruction
in Grade 3. For Macedonian language of instruction, when parents have primary education or
less, 60% of students complete EGRA compared with 86% of students when their parents
have higher education or more. For Albanian language of instruction, 68% of students whose
parents have primary education complete the test compared with an average of 84% of
students whose parents have higher education.

Figure 30. Average percentage of Grade 3 students that completed EGRA in Albanian
language according to education of parents

6.2.3 Results according to type of school

The comparison of results of Grade 2 students according to the type of school they attend,
showed that students from central schools in both languages of instruction perform better
than students from satellite schools, which are predominantly rural (57% vs 45% for
Macedonian language of instruction and 57% vs 54% for Albanian language of instruction).

49
USAID Readers are Leaders Project
Study Report May 2015

Figure 31. Average percentage of Grade 2 students that completed EGRA according to
type of school

The same trend is also evident in Grade 3, as students from central schools have better
results than those in satellite schools (77% vs 65% for Macedonian language of instruction
and 76% vs 66% for Albanian).

Figure 32. Average percentage of Grade 3 students that completed EGRA according to
type of school

50
USAID Readers are Leaders Project
Study Report May 2015

6.2.4 Results according to attendance of pre-school

When students were asked as part of the background questions whether they attended pre-
school, more than half of students with Macedonian language of instruction (55% in Grade 2
and 54% in Grade 3) said they attended such institution.

Table 7. Number of Grade 2 and 3 students attending pre-school

Pre-school Macedonian Albanian

attendance Grade 2 Grade 3 Grade 2 Grade 3
N % N % N % N %
Yes 387 55,13 368 54,04 50 18,87 42 17,14
No 315 44,87 313 45,96 215 81,13 203 82,86

However, only around 17% of the students with Albanian language of instruction (18.87% in
Grade 2 and 17.14% in Grade 3) attended pre-school.

Pre-school attendance showed its impact on the results on overall EGRA instrument, as
students from both Grade 2 and 3 as well as in both Macedonian and Albanian language of
instruction that attended pre-school had better results than students that did not attend pre-
school.

Figure 33. Average percentage of Grade 2 students that completed EGRA according
pre- school attendance

For Grade 2 students, the results of those who attended pre-school ranged from 61% for
Macedonian language of instruction to 66% for Albanian language of instruction compared
with 49% and 55% who did not attend pre-school, accordingly.

51
USAID Readers are Leaders Project
Study Report May 2015

For Grade 3 students, for Macedonian language of instruction 80% who attended pre-school
completed EGRA compared with 70% of those who did not. For Albanian language of
instruction 84% of students who attended pre-school completed the test compared with 71%
of those who did not attend pre-school.

Figure 34. Average percentage of Grade 3 students that completed EGRA according
pre- school attendance

6.2.5 Results according to availability of books at home

Having books at home was another variable that was measured as part of the background
questions that may have impact on EGRA results.

Table 8. Number of Grade 2 and 3 students having additional books at home

Book Macedonian Albanian

availability Grade 2 Grade 3 Grade 2 Grade 3
N % N % N % N %
Yes 442 65,38 499 74,15 138 52,3 126 51,22
No 234 34,62 174 25,85 126 47,7 120 48,78

Around 70% of Macedonian students reported having books at home (65% in Grade 2 and
74% in Grade 3) compared with around 51% of Albanian students (52% in Grade 2 and 51%
in Grade 3).

52
USAID Readers are Leaders Project
Study Report May 2015

Figure 35. Average percentage of Grade 2 students that completed EGRA according to
availability of books at home

The overall EGRA results showed that both Macedonian and Albanian students from Grade
2 and Grade 3 that have books at home performed better than those that did not report
having books at home.

Figure 36. Average percentage of Grade 3 students that completed EGRA according to
availability of books at home

6.2.6 Results according to reading before going to school

53
USAID Readers are Leaders Project
Study Report May 2015

The early start of reading, primarily at home or in the kindergarten, may be also an indicator
of better reading performance later in school. Majority of sampled students (71%) did not
know to read before starting school.

Table 9. Number of Grade 2 and 3 students reading before starting school

Reading Macedonian Albanian

before Grade 2 Grade 3 Grade 2 Grade 3
going to N % N % N % N %
school
Yes 200 28,69 209 31,05 74 28,03 75 30,49
No 497 71,31 464 68,95 190 71,97 171 69,51

The results of Grade 2 students with both Macedonian and Albanian language of instruction
were better if they knew how to read before starting school.

Figure 37. Average percentage of Grade 2 students that completed EGRA according to
whether they read before going to schools

Better results are also evident among Grade 3 students in both languages of instruction.

54
USAID Readers are Leaders Project
Study Report May 2015

Figure 38. Average percentage of Grade 3 students that completed EGRA according to
whether they read before going to schools

6.2.6 Results according to borrowing library books

Borrowing books from the library is an indicator of practicing reading, so it is expected to
result into better results on EGRA tasks.

The number of students with Macedonian language of instruction that borrow library books
increases from 60% in Grade 2 to 70% in Grade 3. Almost equal percentage (42%) of
students with Albanian language of instruction borrow books in both second and third grade.

Table 10. Number of Grade 2 and 3 students borrowing library books

Borrowing Macedonian Albanian

library Grade 2 Grade 3 Grade 2 Grade 3
books N % N % N % N %
Yes 416 59,34 480 70,38 112 42,59 104 42,62
No 285 40,66 202 29,62 151 57,41 140 57,38

As expected, the results in EGRA tasks are better for the students that borrow library books
in both languages of instruction and both grades.

55
USAID Readers are Leaders Project
Study Report May 2015

Figure 39. Average percentage of Grade 2 students that completed EGRA according to
whether they borrow books from library

Figure 40. Average percentage of Grade 3 students that completed EGRA according to
whether they borrow books from library

6.2.7 Results according practicing reading at home

More than half of the Grade 2 students with Macedonian language of instruction (56%) read
at home alone or with their mother (47%). Students with Albanian language of instruction
prefer to read with their mother (46%) than to read alone (32%).

Table 11. Number of Grade 2 students reading at home

Macedonian Albanian
Reading at home 5
N % N %
Grade 2
Not reading 5 0,71
Alone 398 56,29 86 32,45

5 More than one alternative was allowed so the sum of percentages does not equal 100.

56
USAID Readers are Leaders Project
Study Report May 2015

With the mother 334 47,24 123 46,42

With the father 120 6,97 42 15,85
With siblings 72 10,18 54 20,38
With grandparents 45 6,36 9 3,40
With someone else 7 0,99 3 1,13
Does not know 4 0,57

In Grade 3, the number of students reading alone increases in both languages (73% for
Macedonian and 58% for Albanian language of instruction). Reading with the mother is the
most preferred alternative (34% for both languages).

Table 12. Number of Grade 3 students reading at home

Macedonian Albanian
Reading at home
N % N %
Grade 3
Not reading 7 1,02
Alone 497 72,77 143 58,13
With the mother 234 34,26 85 34,55
With the father 87 2,74 33 13,41
With siblings 68 9,96 41 16,67
With grandparents 35 5,12
With someone else 10 1,46

Students that do not read at home at all have poorest results in EGRA, while those reading
alone independently have best results in both grades and for both languages of instruction.

57
USAID Readers are Leaders Project
Study Report May 2015

Figure 41. Average percentage of Grade 2 and 3 students that completed EGRA tasks
in Macedonian and Albanian language according to with whom they are reading at
home

6.2.8 Results according to EGRA tasks

Letter knowledge

Students are performing equally well in both Macedonian and Albanian languages in both
grades. Students with Albanian language of instruction in both grades are better in naming
correct letters per minute compared with students with Macedonian language of instruction
(77 vs. 73 correct letters per minute in Grade 2 and 88 vs. 86 in Grade 3). The accuracy of
naming the letters is also high, ranging from 96% in Grade 2 to 97% in Grade 3.

Table 13 presents data on attempted and correct letters, the time remaining after completion
of the task and the correct letters read per minute.

Table 13. Student results on Letter Identification task

Grade 2 Grade 3
Language of instruction Mac. Alb. Mac. Alb.
N 707 265 683 246
Task 1 – Letter knowledge
Number of items in task 100 100 100 100
Average P-value 71.43% 72.42% 83.26% 83.89%
Time remaining 0.57 1.93 1.57 2.13
Attempted letters (M) 73.78 75 85.03 85.88
Correct letters (M) 71.43 72.42 83.26 83.89
Correct letters per minute 73 77 86 88
Accuracy (%) 96 96 97 98

Familiar words reading

For the familiar word recognition task, also timed at one minute, Grade 2 students in
Albanian language of instruction were better than students in Macedonian language of
instruction reading 34 correct words per minute compared with 30 cwpm. However, in Grade
3 students performed equally well in Macedonian and in Albanian language (46 correct
words per minute).
Table 14. Student results on Familiar Words Reading task
Grade 2 Grade 3
Language of instruction Mac. Alb. Mac. Alb.
N 707 265 683 246
Task 2 – Familiar words
Number of items in task 50 50 50 50
Average P-value 54,84% 59,68% 77,32% 77,05%
Time remaining 1,55 2,62 5,63 5,66
Attempted words (M) 29,30 32,16 40,23 40,67

58
USAID Readers are Leaders Project
Study Report May 2015

Correct words (M) 27,43 29,88 38,66 38,52

Correct words per minute 30 34 46 46
Accuracy (%) 91 90 95 94

Reading fluency

On a similar task but in a connected text (reading fluency task) the students are performing
better, as they read around 48 correct words per minute in Grade 2 (47 cwpm in Macedonian
and 49 cwpm in Albanian language). This means that students who could read at least one
word correctly took, on average, 1.25 seconds to read each word. In Grade 3, Macedonian
students have better results reading 76 correct words per minute compared with Albanian
students who read 69 cwpm. Automaticity also increases with the grade, reaching 0.8
seconds per word in Macedonian and 0.9 in Albanian.

Table 15. Overall student results on Reading Fluency task

Grade 2 Grade 3
Language of instruction Mac. Alb. Mac. Alb.
N 707 265 683 246
Overall Reading fluency (Task 3 & 5)
Number of items in task 176 153 176 153
Average P-value 48,21% 48,74% 71,59% 67,93%
Time remaining 2,15 2,63 7,27 5,85
Attempted words (M) 43,85 45,03 63,85 61,64
Correct words (M) 42,36 42,65 62,98 59,44
Correct words per minute 47 49 76 69
Accuracy (%) 95 93 98 97

When the results on this task are divided per story, for the Grade 2 students it seems that
their fluency is better in the second story (around 49cwpm) compared with the first story
(around 45cwpm).

For Grade 3 students, the results of Macedonian language students are almost equal for
both stories (76 cwpm), while the results of Albanian language students are better for the
second story (68 cwpm for the first and 71 cwpm for the second story).

Table 16. Student results on Reading Fluency per task

Grade 2 Grade 3
Language of instruction Mac. Alb. Mac. Alb.
N 707 265 683 246
Task 3 – Reading fluency (Story 1)
Number of items in task 58 59 58 59
Average P-value 66,15% 64,52% 80,44% 84,30%
Time remaining 3,83 3,93 13,87 11,07
Attempted words (M) 39,59 39,98 52,09 51,61
Correct words (M) 38,36 38,07 51,42 49,74
Correct words per minute 45 45 76 68
Accuracy (%) 95 93 98 97
Task 5 – Reading fluency (Story 2)
Number of items in task 118 116 118 116
Average P-value 39,46% 40,71% 63,27% 59,60%
Time remaining 0,47 1,33 13,87 0,58
Attempted words (M) 48,12 50,11 52,09 71,74
Correct words (M) 46,36 47,23 51,42 69,14

59
USAID Readers are Leaders Project
Study Report May 2015

Correct words per minute 49 52 76 71

Accuracy (%) 95 93 98 97

Reading Comprehension questions

Students on average answered correctly around 8 questions for Macedonian and 5 for
Albanian language out of 20 possible for each language in Grade 2 and the level of accuracy
is 83% in Macedonian and 74% in Albanian language. The level of comprehension increased
in Grade 3, when students answered correctly 13 questions in Macedonian and 8 in Albanian
language, out of 20 possible questions and the level of accuracy increased to 93% and 82%
accordingly.

Table 17. Overall student results on Reading Comprehension task

Grade 2 Grade 3
Language of instruction Mac. Alb. Mac. Alb.
N 707 265 683 246
Overall Reading Comprehension (Task 4 & 6)
Number of items in task 20 20 20 20
Average P-value 41,07% 39,25% 64,71% 58,19%
Attempted items (M) 9,64 6,71 14,04 9,77
Correct answers (M) 8 5 13 8
Accuracy (%) 83% 74% 93% 82%

When the results are presented per story, Grade 2 students in Macedonian language of
instruction have similar results for both stories (around 4 questions), while the students with
Albanian language of instruction have only one correct answer out of 8 questions for the first
story. Even Grade 3 students in Albanian language of instruction seem to have problem with
answering the questions for the first story (2 out of 6 questions), which may indicate a
problem in the words used in the story, or in phrasing the questions.

Table 18. Student results on Reading Comprehension per task

Grade 2 Grade 3
Language of instruction Mac. Alb. Mac. Alb.
N 707 265 683 246
Task 4 – Reading comprehension questions (Story 1)
Number of items in task 8 8 8 8
Average P-value 53,34% 47,31% 80,44% 70,78%
Attempted items (M) 4,67 1,74 6,73 2,60
Correct answers (M) 4 1 6 2
Task 6 – Reading comprehension questions (Story 2)
Number of items in task 12 12 12 12
Average P-value 32,15% 33,87% 53,27% 49,80%
Attempted items (M) 4,97 4,97 7,31 7,17
Correct answers (M) 4 4 7 6

6.3 How Well Are Pupils Doing Basic Mathematics?

Just as in EGRA, the average percentage of students that completed the EGMA test is
based on the average P-value which is obtained by dividing the average test score by the
maximum possible test score and multiplying this by 100.

60
USAID Readers are Leaders Project
Study Report May 2015

On the overall EGMA test, Grade 3 students are better on all tasks, except for the word
problems task, where Grade 2 students have better overall results. Number identification and
number discrimination are the tasks completed by most students (more than 90%). The most
difficult task is subtraction for Grade 2 students (61%) and word problems for Grade 3 (59%)
students.

Figure 42. Average percentage of students that completed EGMA tasks

6.3.1 Overall test results according to gender

On average, male students are slightly better in math compared with female students in both
grades.

61
USAID Readers are Leaders Project
Study Report May 2015

Figure 43. Average percentage of students that completed EGMA according to gender

When EGMA instrument was broken down in tasks, the results for Grade 2 students showed
that girls are slightly better than boys in number identification, number discrimination,
addition, word problems and geometric pattern extension tasks.

Figure 44. Average percentage of Grade 2 students that completed EGMA tasks
according to gender

62
USAID Readers are Leaders Project
Study Report May 2015

In Grade 3, boys are better than girls in all tasks, except for geometric pattern extension.

Figure 45. Average percentage of Grade 3 students that completed EGMA tasks
according to gender

6.3.2 Results according to education of parents

Grade 2 students score better as the education of parents increases from primary through
secondary to higher education. The average score of students whose parents have
completed primary education is around 70%, increasing to 84% for secondary education and
reaching an average of 88% for students whose parents have higher education.

63
USAID Readers are Leaders Project
Study Report May 2015

Figure 46. Average percentage of Grade 2 students that completed EGMA according to
education of parents

The same trend of better results with the increase in the level of parental education is also
evident among Grade 3 students: average of 77% for students of parents with primary
education or less, through 89% for secondary and up to 93% for higher education and more.

Figure 47. Average percentage of Grade 3 students that completed EGMA according to
education of parents

6.3.3 Results according to language of instruction

64
USAID Readers are Leaders Project
Study Report May 2015

The language of instruction may be a factor influencing the results in mathematics. For
instance, Grade 2 students with Macedonian language of instruction show better results in all
EGMA tasks.

Figure 48. Average percentage of Grade 2 students that completed EGMA tasks
according to language of instruction

In Grade 2, students in both languages of instruction show best results in number

identification (94% in Macedonian and in Albanian language), number discrimination (92% in
Macedonian and 90% in Albanian) and geometric shape recognition (85% in Macedonian
and 83% in Albanian). Again the worst results are scored in subtraction (62% in Macedonian,
57% in Albanian), naming missing numbers (64% in Macedonian, 61% in Albanian) and word
problems (69% in Macedonian, 63% in Albanian).

The results are not so straightforward for Grade 3 students. Students with Macedonian
language of instruction are better in all arithmetic tasks, while students with Albanian
language of instruction are better in geometry tasks.

65
USAID Readers are Leaders Project
Study Report May 2015

Figure 49. Average percentage of Grade 3 students that completed EGMA tasks
according to language of instruction

6.3.4 Results according to type of school

In both Grade 2 and Grade 3, students from central schools show better average results than
students in satellite schools (82% vs. 76% in Grade 2 and 87% vs. 80% in Grade 3).

Figure 50. Average percentage of students that completed EGMA according to type of
school

66
USAID Readers are Leaders Project
Study Report May 2015

6.3.5 Results according to attendance of pre-school

From the overall EGMA sample of Grade 2 students (N=927), around 46% reported
attending pre-school. The number of Grade 3 students that attended pre-school is slightly
lower (44% of 896 students).

Table 19. Number of Grade 2 and 3 students attending pre-school

Pre-school Grade 2 Grade 3

attendance N % N %
Yes 427 46,06 397 44,31
No 500 53,94 499 55,69

Students that attended pre-school tend to have better average results in EGMA both in
Grade 2 (84% vs. 78%) and Grade 3 (90% vs. 83%).

Figure 51. Average percentage of students that completed EGMA according to pre-
school attendance

6.3.6 Results according to availability of books at home

Around 70% of Grade 2 and 74% of Grade 3 students reported having additional books at
home.

Table 20. Number of Grade 2 and 3 students having additional books at home

Book availability Grade 2 Grade 3

N % N %
Yes 260 28,17 277 31,16
No 663 71,83 612 68,84

67
USAID Readers are Leaders Project
Study Report May 2015

Similarly as with the reading assessment, availability of books at home also affects results in
mathematics, because students that reported having books at home have better results in
both assessed grades.

The overall EGMA score for Grade 2 students that have additional books at home is 85%
compared with 79% of students not having books. The average score for Grade 3 students is
90% for student having books at home and 85% for students not having books at home.

Figure 52. Average percentage of students that completed EGMA according to

availability of books at home

6.3.7 Results according to EGMA tasks

Number identification
The number identification task targeted student knowledge and identification of one- and two-
digit numbers. High average P-value of more than 94% in Grade 2 and 95% in Grade 3
shows that this task is an easy one. Based on the time remaining for this task and the
number of correct numbers recognized, the pace of identifying correct numbers per minute is
set at 47 for Grade 2 students and 40 for Grade 3 students. The accuracy is also high.

Table 21. Overall student results on Number Identification task

Grade 2 Grade 3
N 963 947
Task 1 – Number identification
Number of items in task 20 25
Average P-value 94,23% 95,10%
Time remaining 31,30 21,14
Attempted numbers (M) 19,54 24,50
Correct numbers (M) 18,85 23,77
Correct numbers per minute 47,40 40,60
Accuracy (%) 96 97
68
USAID Readers are Leaders Project
Study Report May 2015

Number discrimination
This task measures children’s ability to make judgments about differences by comparing
numbers.

This seemed to be an easy task for the students, as on average 92% of Grade 2 and 94% of
Grade 3 students have completed it and the level of accuracy is beyond 96%.

Table 22. Overall student results on Number Discrimination task

Grade 2 Grade 3
N 963 947
Task 2 – Number discrimination
Number of items in task 7 10
Average P-value 91,74% 93,54%
Attempted items (M) 6,89 9,82
Correct items (M) 6,42 9,35
Accuracy (%) 93 95

Missing numbers
For the missing number task, the student was asked to determine and name the missing
number. This task is used to evaluate children’s familiarity with number sequences.

On average, Grade 2 students have 3 correct items out of 5, while Grade 3 students 5 out of
7. Lower average P-value and lower level of accuracy on this task in comparison with the
previous two shows that students find it more difficult.

Table 23. Overall student results on Missing Numbers task

Grade 2 Grade 3
N 963 947
Task 3 – Missing numbers
Number of items in task 5 7
Average P-value 63,18% 73,89%
Attempted items (M) 4,72 6,80
Correct items (M) 3 5
Accuracy (%) 64 74

Addition
The addition task assesses student procedural competency in basic operations. The items
were divided in two levels to assess for automaticity in level 1 and for efficacy in level 2. The
items progressed in difficulty.

For level 2, items were supplemented with assessor observations of student strategy use
(e.g., counting fingers, using counters or pencil and paper).

Because the new math curricula was introduced simultaneously in both Grade 2 and 3, the
addition and subtraction tasks were the same, so the results can be directly compared.

69
USAID Readers are Leaders Project
Study Report May 2015

The overall task has lower average p-value (76% in Grade 2 and 87% in Grade 3). However,
although level 2 included more difficult items (with carryover), the average p-values are close
enough (77% for level 1 and 75% for level 2 in Grade 2 as well as 88% for level 1 and 86%
for level 2 in Grade 3).

Regarding the counting strategies used, use of fingers is slightly predominant in Grade 2
while mental calculation prevails in Grade 3. However, Grade 2 students are twice more
likely to use manipulatives than Grade 3 students (35% compared with 17%).

Table 24. Overall student results on Addition task

Grade 2 Grade 3
N 963 947
Task 4 – Addition (overall)
Number of items in task 10 10
Average P-value 76,00% 86,99%
Addition – Level 1
Number of items in task 5 5
Average P-value 76,74% 87,94%
Time remaining 14,76 25,88
Attempted items (M) 4,39 4,76
Correct items (M) 3,84 4,40
Correct per minute 6,83 10,92
Addition – Level 2
Number of items in task 5 5
Average P-value 75,26% 86,04%
Time remaining 49,86 69,26
Attempted items (M) 4,72 4,91
Correct items (M) 3,76 4,30
Correct per minute 4,63 7,47
Addition – Use of manipulatives
Use of fingers 62,72% 45,30%
Use of manipulatives 35,41% 17,42%
Use of pencil and paper 15,26% 18,69%
Mental calculation 61,68% 77,72%

Subtraction
The subtraction task also assessed student procedural competency, exploring two different
levels of difficulty (е. g., counting fingers, using counters).

On the subtraction task, the students show worse results compared with addition. The
average p-value on the overall task is lower for both grades, 61% of Grade 2 students (which
is lowest for the entire EGMA instrument) and 75% of Grade 3 students.

The low number of correct items in level 2, indicates that students have more problem with
subtracting larger numbers that include carryover.

70
USAID Readers are Leaders Project
Study Report May 2015

Again the use of counters, especially in Grade 3, is low in resolving the subtraction problems.
Most students opt for doing the calculations mentally in Grade 3 and use of fingers together
with mental calculations in Grade 2.

Table 25. Overall student results on Subtraction task

Grade 2 Grade 3
N 963 947
Task 4 – Subtraction (overall)
Number of items in task 10 10
Average P-value 61,10% 75,12%
Subtraction – Level 1
Number of items in task 5 5
Average P-value 77,82% 89,02%
Time remaining 15,54 27,88
Attempted items (M) 4,57 4,81
Correct items (M) 3,89 4,45
Correct per minute 6,95 12,27
Subtraction – Level 2
Number of items in task 5 5
Average P-value 44,38% 61,22%
Time remaining 20,29 30,21
Attempted items (M) 4,15 4,58
Correct items (M) 2,22 3,06
Correct per minute 1,55 2,60
Subtraction – Use of manipulatives
Use of fingers 54,62% 41,18%
Use of manipulatives 36,34% 21,22%
Use of pencil and paper 24,51% 41,50%
Mental calculation 58,57% 72,33%

Word Problems
Student informal concepts of addition and subtraction (Grade 2) and of addition, subtraction,
multiplication and division (Grade 3) are assessed through contextual problems, assessing
children strategies and flexibility in solving problems.

From all tasks in EGMA instrument, Grade 3 students have lowest results in word problems
(59.24%), probably as it involves multiplication and division, operations that are less
practiced.

When it comes to using different strategies for solving the word problems, Grade 2 children
tend to use their fingers (52%), while Grade 3 students use pencil and paper for this task
(75%). Majority of children are still trying to do the calculations mentally without using any
other means (around 64%).

Table 26. Overall student results on Word Problems task

Grade 2 Grade 3
N 963 947
Task 6 – Word Problems

71
USAID Readers are Leaders Project
Study Report May 2015

Number of items in task 4 4

Average P-value 67,11% 59,24%
Attempted items (M) 3,65 3,68
Correct items (M) 2,68 2,37
Word Problems – Use of manipulatives
Use of fingers 52,44% 32,42%
Use of manipulatives 29,49% 14,78%
Use of pencil and paper 30,22% 74,55%
Mental calculation 63,24% 64,20%

Geometric Shapes Recognition

This task assesses student ability to identify and point out specific shapes from a variety of
given shapes.

As the items in this task were the same both for Grade 2 and Grade 3 students, the
comparisons between the two grades are more meaningful. The average p-value, the
number of correct items and the level of accuracy are similar in both grades, with Grade 3
students having slightly better results.

Table 27. Overall student results on Geometric Shapes Recognition task

Grade 2 Grade 3
N 963 947
Task 7 – Geometric Shapes Recognition
Number of items in task 4 4
Average P-value 84,45% 85,77%
Attempted items (M) 3,92 3,97
Correct items (M) 3,38 3,43

Geometric Patterns Extension

For the pattern extension task, students were shown a series of shapes and told to select
one of the items below the series to finish the pattern.

The students from Grade 3 have better results on this task, although the items are more
difficult in this grade.

Table 28. Overall student results on Geometric Patterns Extension task

Grade 2 Grade 3
N 963 947
Task 8 – Geometric Patterns Extension
Number of items in task 3 4
Average P-value 82,66% 86,54%
Attempted items (M) 2,88 3,93
Correct items (M) 2,48 3,46

72
USAID Readers are Leaders Project
Study Report May 2015

7. EGRA & EGMA LONGITUDINAL FINDINGS

This section presents summary statistics for EGRA and EGMA longitudinal assessment in
Macedonia conducted in 41 schools.

The cohort of Grade 2 students with Macedonian and Albanian language of instruction from
41 schools that was assessed in May 2014, was reassessed again in Grade 3 in May 2015.

The average scores of students with Macedonian language of instruction on EGRA

instrument show that as expected their results are much higher in Grade 3 compared with
Grade 2. The analysis of the tasks indicates that again the best results are achieved in the
letter knowledge task and worst in reading comprehension task. However, the
comprehension has increased from 46% in Grade 2 in 2014 to 68% in Grade 3 in 2015.

Figure 53. Comparison of average percentage of students that completed EGRA tasks
in Macedonian language of instruction

The same applies for the students with Albanian language of instruction. Their results in
Grade 3 are also higher in every area, with best scores in letter knowledge and worst in
reading comprehension.

73
 USAID Readers are Leaders Project
Study Report May 2015

Figure 54. Comparison of average percentage of students that completed EGRA tasks
in Albanian language of instruction

The comparison of results per task shows that in letter identification task, the students not
only have higher p-values, but also more time remains, which means the students can read
more letters in less time. The number of correct letters per minute has increased from 74 to
87 for the Macedonian language of instruction and from 83 to 108 for the Albanian language
of instruction.

Table 29. Student results on Letter Identification task

Language of instruction Macedonian Albanian
Grade Grade 2 (2014) Grade 3 (2015) Grade 2 (2014) Grade 3 (2015)
N 731 692 250 229
Task 1 – Letter knowledge
Number of items in task 100 100 100 100
Average P-value 73,25% 86,59% 74,65% 88,86%
Time remaining 0,73 1,74 2,38 4,41
Attempted letters (M) 77,94 86,21 77,24 90,64
Correct letters (M) 72,94 79,56 74,06 74,32
Correct letters per minute 74 87 83 108

In the familiar word reading task, the number of correct words per minute progresses from 31
to 49 for the Macedonian language of instruction and from 30 to 53 for Albanian language of
instruction.

Table 30. Student results on Familiar Words Reading task

74
 USAID Readers are Leaders Project
Study Report May 2015

Language of instruction Macedonian Albanian

Grade Grade 2 (2014) Grade 3 (2015) Grade 2 (2014) Grade 3 (2015)
N 731 692 250 229
Task 2 – Familiar words
Number of items in task 50 50 50 50
Average P-value 58,45% 81,12% 54,40% 81,34%
Time remaining 1,52 6,52 3,16 6,97
Attempted words (M) 31,43 41,99 30,22 42,76
Correct words (M) 29,10 40,57 27,09 40,67
Correct words per minute 31 49 30 53
Accuracy (%) 93 97 90 95

Grade 3 students are much faster readers compared with the time when they were Grade 2,
reading 81 correct words per minute in Macedonian and 77 correct words per minute in
Albanian language.

Table 31. Overall student results on Reading Fluency task

Language of instruction Macedonian Albanian
Grade Grade 2 (2014) Grade 3 (2015) Grade 2 (2014) Grade 3 (2015)
N 731 692 250 229
Overall Reading fluency (Task 3 & 5)
Number of items in task 144 176 153 175
Average P-value 54,60% 75,81% 49,96% 71,69%
Time remaining 1,84 8,24 2,61 6,70
Attempted words (M) 40,90 67,14 41,07 63,39
Correct words (M) 39,21 66,48 37,82 62,35
Correct words per minute 41 81 40 77
Accuracy (%) 94 96 99 94

Table 32. Student results on Reading Fluency per task

Language of instruction Macedonian Albanian
Grade Grade 2 (2014) Grade 3 (2015) Grade 2 (2014) Grade 3 (2015)
N 731 692 250 229
Task 3 – Reading fluency (Story 1)
Number of items in task 66 58 72 59
Average P-value 62,66% 91,29% 53,86% 87,20%
Time remaining 2,32 15,68 3,24 1,99
Attempted words (M) 43,07 53,58 41,56 40,57
Correct words (M) 41,24 52,95 38,62 37,01
Correct words per minute 43,86 79,80 41,57 38,61
Accuracy (%) 94,30 98 91,32 93,35
Task 5 – Reading fluency (Story 2)
Number of items in task 78 118 81 116
Average P-value 47,79% 68,26% 46,06% 63,80%

75
 USAID Readers are Leaders Project
Study Report May 2015

Time remaining 1,35 0,76 12,43 1,00

Attempted words (M) 38,72 80,77 53,19 73,56
Correct words (M) 37,17 80,02 51,45 73,26
Correct words per minute 38,37 83,07 76 78
Accuracy (%) 94 94 97 92

The reading comprehension has also significantly increased, as the accuracy of correctly
answered questions from the attempted rose from 48% to 91% in Macedonian language and
from 42% to 80% in Albanian language.

Table 33. Overall student results on Reading Comprehension task

Language of instruction Macedonian Albanian
Grade Grade 2 (2014) Grade 3 (2015) Grade 2 (2014) Grade 3 (2015)
N 731 692 250 229
Overall Reading Comprehension (Task 4 & 6)
Number of items in task 15 20 15 20
Average P-value 46,36% 68,22% 36,05% 57,84%
Attempted items (M) 14,42 15,01 13,21 14,30
Correct answers (M) 6,95 13,71 5,55 11,51
Accuracy (%) 48% 91% 42% 80%

Table 34. Student results on Reading Comprehension per task

Language of instruction Macedonian Albanian
Grade Grade 2 (2014) Grade 3 (2015) Grade 2 (2014) Grade 3 (2015)
N 731 692 250 229
Task 4 – Reading comprehension questions (Story 1)
Number of items in task 7 8 7 8
Average P-value 47,76% 83,33% 37,49% 69,76%
Attempted items (M) 6,72 7,06 6,15 6,69
Correct answers (M) 3,34 6,64 2,72 5,56
Accuracy (%) 50% 94% 44% 83%
Task 6 – Reading comprehension questions (Story 2)
Number of items in task 8 12 8 12
Average P-value 45,13% 57,23% 34,60% 49,89%
Attempted items (M) 7,70 7,95 7,11 7,61
Correct answers (M) 3,61 7,07 2,85 5,95
Accuracy (%) 47% 89% 40% 78%

The comparison of results between the same cohort of students in Grade 2 and Grade 3 in
EGMA is not that straightforward. One of the explanation is the modification of EGMA tasks
in order to comply with the newly introduced mathematics curriculum. So, in 2014 in Grade 2
students learned the numbers and calculations only up to 20, in 2015 in Grade 3 the students
had to learn the number not up to 100 (as it was in the previous curriculum) but they had to
make a leap and learn them up to 1,000.

76
USAID Readers are Leaders Project
Study Report May 2015

This may be a reason why the results on number identification and number discrimination
tasks, which were considered as two of the easiest, in Grade 3 the students had slightly
worse results.

Grade 3 students had a steep plunge in their success in doing the word problems, as their
scores went down from 71% in Grade 2 in 2014 to 54% in Grade 3 in 2015.

Figure 55. Comparison of average percentage of students that completed EGMA tasks

In the number identification task, the number of correct numbers per minute is lower, but the
accuracy is roughly the same.

Table 35. Overall student results on Number Identification task

Grade 2 (2014) Grade 3 (2015)
N 968 847
Task 1 – Number identification
Number of items in task 20 25
Average P-value 97,27% 95,89%
Time remaining 49,30 22,47
Attempted numbers (M) 9,94 24,64
Correct numbers (M) 9,73 23,97
Correct numbers per minute 65,71 44,18
Accuracy (%) 98 97

In the number discrimination, Grade 3 students have good score and the accuracy is 96%.

Table 36. Overall student results on Number Discrimination task

77
USAID Readers are Leaders Project
Study Report May 2015

Grade 2 (2014) Grade 3 (2015)

N 968 847
Task 2 – Number discrimination
Number of items in task 5 10
Average P-value 95,70% 94,62%
Attempted items (M) 4,98 9,84
Correct items (M) 4,79 9,46
Accuracy (%) 96 96

The accuracy increases in the missing number task, indicating it becomes easier as the
children grow up.

Table 37. Overall student results on Missing Numbers task

Grade 2 (2014) Grade 3 (2015)
N 968 847
Task 3 – Missing numbers
Number of items in task 5 7
Average P-value 66,67% 72,37%
Attempted items (M) 4,98 6,69
Correct items (M) 3,34 5,07
Accuracy (%) 67 76

In the addition task, as the task in Grade 3 is divided into two levels of difficulty, the results
cannot be compared straightforwardly. However, the strategy use shows that in Grade 3,
students are more inclined to use paper and pencil or mental calculations instead of using
fingers or counters.

Table 38. Overall student results on Addition task

Grade 2 (2014) Grade 3 (2015)
N 968 847
Task 4 – Addition (overall)
Number of items in task 10 10
Average P-value 77,55% 87,38%
Addition – Level 1
Number of items in task 5
Average P-value 88,81%
Time remaining 29,25
Attempted items (M) 4,85
Correct items (M) 4,44
Correct per minute 12,59
Accuracy (%) 91
Addition – Level 2
Number of items in task 5
Average P-value 85,95%
Time remaining 72,93
Attempted items (M) 4,97
Correct items (M) 4,30
Correct per minute 8,36
Accuracy (%) 86
Addition – Use of manipulatives
Use of fingers 55,37% 41,56%
Use of manipulatives 17,36% 7,79%

78
USAID Readers are Leaders Project
Study Report May 2015

Use of pencil and paper 4,96% 17,47%

Mental calculation 79,13% 90,20%

The same applies for the subtraction. In this task children tend even more to use pencil and
paper to do the calculations.

Table 39. Overall student results on Subtraction task

Grade 2 (2014) Grade 3 (2015)
N 968 847
Task 4 – Subtraction (overall)
Number of items in task 10 10
Average P-value 65,76% 71,62%
Subtraction – Level 1
Number of items in task 5
Average P-value 87,65%
Time remaining 31,05
Attempted items (M) 4,90
Correct items (M) 4,38
Correct per minute 13,48
Accuracy (%) 89
Subtraction – Level 2
Number of items in task 5
Average P-value 55,58%
Time remaining 39,24
Attempted items (M) 4,80
Correct items (M) 2,78
Correct per minute 3,12
Accuracy (%) 58
Subtraction – Use of manipulatives
Use of fingers 59,19% 38,84%
Use of manipulatives 19,83% 10,04%
Use of pencil and paper 5,37% 33,77%
Mental calculation 81,92% 84,53%

In the word problem task, the students have more correct items and the accuracy has
improved from 44% to 61%, but the average p-value is lower, making this task more difficult.

Table 40. Overall student results on Word Problems task

Grade 2 (2014) Grade 3 (2015)
N 968 847
Task 6 – Word Problems
Number of items in task 4 4
Average P-value 71,72% 53,75%
Attempted items (M) 3,99 3,54
Correct items (M) 1,77 2,15
Accuracy (%) 44 61
Word Problems – Use of manipulatives
Use of fingers 55,99% 34,12%
Use of manipulatives 19,63% 7,91%
Use of pencil and paper 20,97% 66,23%

79
USAID Readers are Leaders Project
Study Report May 2015

Mental calculation 78,62% 77,80%

The results in both geometric tasks are similar, as they students in Grade 3 have slightly
better results.

Table 41. Overall student results on Geometric Shapes Recognition task

Grade 2 (2014) Grade 3 (2015)
N 968 847
Task 7 – Geometric Shapes Recognition
Number of items in task 4 4
Average P-value 69,21% 71,55%
Attempted items (M) 3,98 3,91
Correct items (M) 2,79 2,86
Accuracy (%) 70 73

Table 42. Overall student results on Geometric Patterns Extension task

Grade 2 (2014) Grade 3 (2015)
N 968 847
Task 8 – Geometric Patterns Extension
Number of items in task 3 4
Average P-value 84,61% 86,39%
Attempted items (M) 2,99 3,89
Correct items (M) 2,56 3,46
Accuracy (%) 86 89

The detailed breakdown of student achievements in all EGRA and EGMA tasks according to
different variables are presented in Appendix 2.

Additionally, comparisons can also be made regarding the results of Grade 3 students from
the schools in the longitudinal sample (for whom this is a reassessment or follow-up study as
they were also assessed in Grade 2 in 2014) and Grade 3 students from the new
participating schools, in order to see whether any improvement can be observed in the
results of students who had already participated in some project activities.

When we compare the Grade 3 student achievements in EGRA in Macedonian language of

instruction from the baseline and longitudinal sample, students from the longitudinal sample
have slightly better results in all tasks from letter knowledge (87% in longitudinal and 83% in
baseline) to reading comprehension (68% in longitudinal and 65% in baseline sample).

80
USAID Readers are Leaders Project
Study Report May 2015

Figure 56. Comparison of average percentage of Grade 3 students that completed

EGRA in Macedonian language of instruction in longitudinal and baseline sample

For the Albanian language of instruction, again the students from longitudinal sample have
better results in all tasks, but reading comprehension, where students from the baseline
study have slightly better results (58.19% compared with 57.84% from the longitudinal
study).

81
USAID Readers are Leaders Project
Study Report May 2015

Figure 57. Comparison of average percentage of Grade 3 students that completed

EGRA in Albanian language of instruction in longitudinal and baseline sample

The comparison of results according to gender also shows that students from longitudinal
sample, regardless of whether they are boys or girls have better results than those from the
baseline sample.

Figure 58. Comparison of average percentage of Grade 3 students that completed

EGRA in Macedonian language of instruction per gender in longitudinal and baseline
sample

This is evident for both Macedonian and Albanian language of instruction.

82
USAID Readers are Leaders Project
Study Report May 2015

Figure 59. Comparison of average percentage of Grade 3 students that completed

EGRA in Albanian language of instruction per gender in longitudinal and baseline
sample

The school location is also a variable that affects student performance. Again students
assessed longitudinally with Macedonian language of instruction have better results,
particularly those from the urban schools.

Figure 60. Comparison of average percentage of Grade 3 students that completed

EGRA in Macedonian language of instruction per school location in longitudinal and
baseline sample

83
USAID Readers are Leaders Project
Study Report May 2015

For the Albanian language of instruction, the longitudinal sample of Grade 3 students have
better results in both urban and rural schools compared with those from the baseline sample.

Figure 61. Comparison of average percentage of Grade 3 students that completed

EGRA in Albanian language of instruction per school location in longitudinal and
baseline sample

When the type of school is in question, again students from longitudinal sample have better
results than those in baseline sample in both satellite and central schools with Macedonian
language of instruction.

84
USAID Readers are Leaders Project
Study Report May 2015

Figure 62. Comparison of average percentage of Grade 3 students that completed

EGRA in Macedonian language of instruction per type of school in longitudinal and
baseline sample

The same trend is evident in schools with Albanian language of instruction, where the
difference between the longitudinal and baseline sample is particularly obvious in the satellite
schools.

Figure 63. Comparison of average percentage of Grade 3 students that completed

EGRA in Albanian language of instruction per type of school in longitudinal and
baseline sample

Pre-school attendance is also variable that makes a difference between longitudinal and
baseline sample, with students in Macedonian language of instruction who were assessed
longitudinally performing better than those assessed for the first time this year.

85
USAID Readers are Leaders Project
Study Report May 2015

Figure 64. Comparison of average percentage of Grade 3 students that completed

EGRA in Macedonian language of instruction according to pre-school attendance in
longitudinal and baseline sample

However, in the schools with Albanian language of instruction, the students from the baseline
schools that attended pre-school had better results than those from the longitudinal schools.

Figure 65. Comparison of average percentage of Grade 3 students that completed

EGRA in Albanian language of instruction according to pre-school attendance in
longitudinal and baseline sample

86
USAID Readers are Leaders Project
Study Report May 2015

When it comes to having books at home, those students reporting having books at home
from the longitudinal sample have better results, but of those students not having additional
books at home students from the baseline sample have better results.

Figure 66. Comparison of average percentage of Grade 3 students that completed

EGRA in Macedonian language of instruction according to having books at home in
longitudinal and baseline sample

For the Albanian language of instruction, students from the longitudinal sample having
additional books at home have better results than those from the baseline sample.

87
USAID Readers are Leaders Project
Study Report May 2015

Figure 67. Comparison of average percentage of Grade 3 students that completed

EGRA in Albanian language of instruction according to having books at home in
longitudinal and baseline sample

In the field of mathematics, the comparisons show the following: longitudinal sample has
better results in number identification, number discrimination and addition, while the baseline
sample has better results in naming missing numbers, subtraction, word problems, geometric
shape recognition and geometric pattern extension.

Figure 68. Comparison of average percentage of Grade 3 students that completed

EGМA in longitudinal and baseline sample

The comparison of correct items in each task shows that in the timed tasked the longitudinal
sample had better results.

88
USAID Readers are Leaders Project
Study Report May 2015

Figure 69. Comparison of correct number of items of Grade 3 students that completed
EGМA in longitudinal and baseline sample

The gender comparisons show that students from the baseline sample have better results
than those from the longitudinal.

Figure 70. Comparison of average percentage of Grade 3 students that completed

EGМA per gender in longitudinal and baseline sample

89
USAID Readers are Leaders Project
Study Report May 2015

The comparisons of results based on the type of school (central vs. satellite) and location
(town vs. village) are not straightforward between the longitudinal and baseline sample.
However, the results of students in central and urban schools are higher in both samples.

Figure 71. Comparison of average percentage of Grade 3 students that completed

EGМA per school location in longitudinal and baseline sample

Figure 72. Comparison of average percentage of Grade 3 students that completed

EGМA per type of school in longitudinal and baseline sample

90
USAID Readers are Leaders Project
Study Report May 2015

According to the language of instruction, the students from the baseline sample had better
results in EGMA in both Macedonian and Albanian language.

Figure 73. Comparison of average percentage of Grade 3 students that completed

EGМA per language of instruction in longitudinal and baseline sample

The students that attended pre-primary education in both longitudinal and baseline sample
have better results in EGMA.

Figure 74. Comparison of average percentage of Grade 3 students that completed

EGМA according to pre-school attendance in longitudinal and baseline sample

91
USAID Readers are Leaders Project
Study Report May 2015

Having books at home is an important variable as the comparisons of results show that those
students that have books at home in both longitudinal and baseline sample have better
results.

Figure 75. Comparison of average percentage of Grade 3 students that completed

EGМA according to having books at home in longitudinal and baseline sample

8. FINDINGS FROM THE QUESTIONNAIRES

The main instruments for collecting baseline data from 60 schools were three types of
questionnaires, based on the Snapshot for School Management Effectiveness, or SSME, an
instrument developed by RTI that has been applied in conjunction with Early Grade Reading
Assessments (EGRA) and/or Early Grade Mathematics Assessments (EGMA).

Three online questionnaires were developed, available on Google docs for:

 School Director/ Assistant School Director;
 Early grade teachers from first to fifth grade;
 Parents.

The questionnaires this year were reviewed by an education inspector and early grade
teacher, so that their content could be more relevant for the context in our country.

The aim of the questionnaires was to gather data about the perceptions and expectations of
the different stakeholders (school management, teachers, and parents) about the reading
and mathematics in early grade education. In the preparation of the questionnaires, special
attention was paid some questions to overlap, in order to ensure comparability between the
answers of the school directors, teachers and the parents as much as possible.

Responses were received from 48 school directors/ assistant directors, 486 teachers, and
624 parents.

92
USAID Readers are Leaders Project
Study Report May 2015

In order to get more responses from the parents as well as not to put at disadvantage those
that do not have access to internet, hard copy questionnaires were distributed in September
to the additional 60 schools that joined the baseline study in May 2015.

The questionnaires in English language are provided in Appendices 3-5 of this report.

The section below presents the main findings from the baseline questionnaires, grouped in
several areas relevant for the project and based on 2011 Progress in International Reading
Literacy Study6, where these areas were thoroughly studied and comparative data are
available:
 School Climate
 School Resources for Teaching Reading
 Teacher Preparation
 Classroom Instruction
 Home Environment Support for Reading Achievement

8.1 School Climate

PIRLS study showed that students with the highest reading achievement typically attended
schools that emphasized academic success, high educational values reflected by the
teachers, school leadership, the students themselves, and their parents, a positive
atmosphere toward high achievement, effective teachers, students that desired to do well,
and parental support. The positive and negative aspects of the atmosphere in schools may
affect the student achievements. A positive school atmosphere emphasizing academic
achievement can even overcome socioeconomic disadvantages.

Some variables measured with the questionnaires indicated the school climate in the primary
schools in Macedonia.

The structure of the school leadership and particularly of the early grade teachers in
Macedonia’s schools is predominantly in favor of women.

Of the 48 school directors that filled out the questionnaire, 48% were male and 52% female.

6Mullis, I.V.S., Martin, M.O., Foy, P., & Drucker, K.T. (2012). Chestnut Hill, MA: TIMSS & PIRLS International
Study Center, Boston College.

93
USAID Readers are Leaders Project
Study Report May 2015

Figure 76: Gender structure of surveyed school directors

The surveyed 486 early grade teachers, included 90% female and 10% male teachers.
According to the ethnic structure, 84% were Macedonians, 14.6% Albanians and 1.4% were
of other ethnicity (Turks, Serbs and others).

Figure 77: Gender structure of surveyed early grade teachers

94
USAID Readers are Leaders Project
Study Report May 2015

Figure 78: Ethnic structure of surveyed early grade teachers

The analysis showed that almost three quarters of the schools (73%) provide split-shift
schooling, i.e. students attend classes in the mornings or afternoons.

Figure 79: Percentage of schools working in split shifts

According to the language of instruction, 64% of the surveyed schools provided instruction
only in Macedonian language, 10% only in Albanian language, 17% in both Macedonian and
Albanian language, 6% in Macedonian, Albanian and Turkish and 4% in Macedonian and
Turkish.

95
USAID Readers are Leaders Project
Study Report May 2015

Figure 80: Representation of schools in the sample according to the language of

instruction

Most (83%) of the early grade teachers that responded to the questionnaire delivered
instruction in Macedonian language, 15% in Albanian language and 2% in Turkish and
Serbian language.

Figure 81: Representation of early grade teachers in the sample according to the
language of instruction

Depending on the size, the number of early grade teachers varied in the schools. Most of the
schools (44%) have more than 20 early grade teachers, while 33% between 11 and 20
teachers. Total of 21% of the schools in the sample have up to 10 early grade teachers and
2% less than five early grade teachers.

96
USAID Readers are Leaders Project
Study Report May 2015

Figure 82: Number of early grade teachers in sample schools

Based on school director reports, 98% of teachers are seldom absent (up to five working
days per year or 3% of teaching time) and only 2% reported that teachers are often absent
for up to 10 days per year, which is 5% of the total teaching time.

Figure 83: Absence of teachers during school year

In cases when teachers are late for classes, 69% of the school directors warn the teachers to
keep an eye on their punctuality, 6% leave it to their consciousness, while only 2% take strict
legal measures.

97
USAID Readers are Leaders Project
Study Report May 2015

Figure 84: Measures taken by school directors when teachers are late for classes

In case when the teacher is absent from school for the day, majority of school directors
appoint appropriate substitute (69%). In few cases, teacher from another class (17%) is
appointed or substitute teacher is hired (14%).

Figure 85: Alternative for students when their teacher is absent

Regarding the absence of students, teachers report that majority of students (54%) are
seldom absent from school. However, more than one third (37%) of students are absent from
school due to sickness, with additional 9% of students that are frequently absent.

98
USAID Readers are Leaders Project
Study Report May 2015

Figure 86: Absence of students during school year according to teachers

According to the parents, the children are seldom absent from schools (82%). Only 2%
reported that their children were absent often, from three to ten days.

Figure 87: Absence of students during school year according to parents

There was agreement across the parents in all focus groups that children rarely miss classes
and when they do its due to illness.

In Macedonian education system, school directors and student support services (pedagogue,
psychologist, and/or sociologist) are supposed to be doing internal quality control, while BDE
advisors and education inspectors the external monitoring and evaluation.

99
USAID Readers are Leaders Project
Study Report May 2015

The extent to which school directors are or are not involved with the day-to-day work of their
teaching staff can be indicative of the management and oversight capabilities of school
directors, the level of accountability felt by teachers, and the working atmosphere for staff.

When asked how they monitor teachers’ performance by being able select more than one
alternative, 96% said they monitored teachers directly through class observation. More than
two-thirds (67%) rely on the quarterly reports on student progress, while around half of them
also rely on external evidence (results of the integral evaluation, feedback from parents or
external testing). One third take into account the feedback from BDE advisors.

Figure 88: Practice of monitoring performance of teachers

Furthermore, when the frequency of class observations was explored, most of the school
directors reported doing it once a quarter (47%), followed by once a term (32%) and once a
month (15%). Only 4% observe the classes once a school year, while 2% never do this.

100
USAID Readers are Leaders Project
Study Report May 2015

Figure 89: Frequency of class observations reported by school directors

However, when teachers were asked to report how often the school directors observed their
classes most of them (48%) said once a term, followed by 27% who said once a quarter.
Only 8% of teachers said their classes were observed once a month, while 14% of teachers
said school directors observe their classes once a year. Almost same percentage of teachers
(3%) said school directors never observed the classes.

Figure 90: Frequency of class observations reported by teachers

When school directors observe the classes, 94% of them said they were documenting the
observations in a form of report.

Figure 91: Documentation of class observations by school director in the form of

report

101
USAID Readers are Leaders Project
Study Report May 2015

After the class observations, 88% of the teachers reported receiving feedback and trying to
implement the given suggestions in practice. As much as 10% of the teachers said they did
not receive the feedback, while 2% said provided feedback was not useful at all.

Figure 92: Feedback after class observations by school director

The collaboration among teachers is high where majority (77%) of them exchange
experience regarding the planning and implementation of lessons, 72% consult one another,
63% prepare joint lesson plans and 52% disseminate the training.

Figure 93: Most common topics for collaboration among teachers

The good cooperation is also confirmed when teachers need advice regarding the
curriculum. Total of 72% said when they needed assistance, they discussed the issue with
other teachers during formal or informal (63%) meetings. Majority (64%) seek advice from
the student support services, 36% from the school director, and 21% from the BDE advisor.

102
USAID Readers are Leaders Project
Study Report May 2015

Figure 94: Consultation of teachers regarding curriculum needs

When the teachers were asked about the frequency of class observations by the student
support staff, their responses indicated that such visits were infrequent, with the majority of
responses reporting visits once a term (45%) (42%) or once a quarter (33%).

Figure 95: Frequency of class observation by student support service

103
USAID Readers are Leaders Project
Study Report May 2015

Figure 96: Feedback after class observation by student support service

Teachers were also asked about visits of BDE advisors. Most teachers (42%) reported to
have been visited once a year, 24% once a term, and 12% once a quarter.

Figure 97: Frequency of school visits by BDE advisors

School directors and teachers that support and trust in student achievements is also
important for creating positive climate.

When asked about their expectations from students to read in their native language, roughly
the same (81% of school directors and 79% of teachers) expect them to read by the end of
the second grade.

104
USAID Readers are Leaders Project
Study Report May 2015

Figure 98: School directors’ expectations about student literacy

Figure 99: Teachers’ expectations about student literacy

When parents were asked whether their children were able to read, 14% said they were still
not able to read. However, this may be the case as 9% of parents responding to the
questionnaires were having a first grader, and 23% were parents of second graders (the hard
copy questionnaires were administered at the start of the school year, when Grade 2
students start learning the letters).

105
USAID Readers are Leaders Project
Study Report May 2015

Figure 100: Ability of children to read reported by parents

The focus group interviews conducted with the parents from Macedonia revealed that before
starting school children were able to recognize the letters and read simple words such as
their names. However, it is only after starting school that the children learned to connect the
letters into meaningful words or they were still learning it (second graders).

Figure 101: Location where children learn to read

According to the parents, most of the children learned to read while in kindergarten (60%),
while 28% at home with the help of their parents. Only 2% of the parents said their children
learned to read in school.

Summary of results:
- Gender structure of schools in Macedonia is predominantly female, both in terms of
school leadership and teachers;
- Almost three quarters of schools provide split-shift classes;

106
USAID Readers are Leaders Project
Study Report May 2015

- Few teachers are absent or late for classes in schools and school directors rarely
take legal measures against teachers’ tardiness;
- When the teacher is absent from school for the day, mainly appropriate substitute
appoint is appointed or in few cases, teacher from another class is appointed or even
substitute teacher is hired;
- Based on teachers’ account, 37% of students are often absent from school due to
sickness;
- Majority of school directors monitor teacher’s performance directly through class
observation, usually once a quarter. The process is documented;
- Teachers report being monitored by school directors and student support services
once a term and only 10% never received feedback;
- Teachers collaborate with other teachers in planning and implementation of lessons,
consultations with one another, preparation of joint lesson plans and dissemination of
the training;
- School directors and BDE advisors are not the main source of advice for teachers for
resolving curriculum issues. Other teachers and student support services are
preferred;
- BDE advisors usually visit schools once a year;
- Most teachers and school directors expect students to learn to read and write at the
end of the second grade.

8.2 School Resources for Teaching Reading and Mathematics

The learning environment of the school can be a positive influence, encouraging a positive
attitude toward academic excellence and facilitating classroom instruction. Considerable
research has shown that higher levels of school resources are associated with higher
achievement.

Studies have shown that resources are crucial for improving schooling, as the extent and
quality of school resources can have an important impact on the quality of classroom
instruction. For example, the presence of a library or multimedia center may be particularly
relevant for developing reading literacy
(http://timssandpirls.bc.edu/pirls2011/downloads/PIRLS2011_Framework-Chapter3.pdf).

To provide information on the extent to which school resources are available to support
reading and mathematics instruction, the school directors were asked about the didactical
materials supplied over the past three years. Apparently, almost half of the schools (46%)
have supplied such materials, which may be related to the introduction of new mathematics
curriculum and the requirement for providing more manipulatives for the early grades.

107
USAID Readers are Leaders Project
Study Report May 2015

Figure 102: Supplied didactic materials over the last three years

Libraries, both within the school and in the local community, provide a range of reading
materials and other resources from which teachers can draw to expand their instructional
approaches, and from which students can choose books for their own learning and
enjoyment.

According to the school directors, most of the schools (92%) have their own library.

Figure 103: Existence of library in the school

In 72% of the cases, the schools also have a person who is hired and works as librarian.

108
USAID Readers are Leaders Project
Study Report May 2015

Figure 104: Existence of librarian in the school

In most of the schools (79%) the school library is open throughout the day, while in some
schools it is open either during the morning or the afternoon shift (17% and 4% accordingly).

Figure 105: Opening hours of school library

More than half of school directors reported that their libraries had books suitable for early
grade students, apart from the compulsory reading series.

109
USAID Readers are Leaders Project
Study Report May 2015

Figure 106: Books in the library for early grade students

Majority of early grade students (87%), according to the school directors, are borrowing
books from the school library.

Figure 107: Borrowing books from the library by early grade students

The school directors were also asked according to the number of issued books to estimate
the percentage of students from the first three grades that borrow books from the library:
58% said that almost half of the students borrow book, 23% said almost everyone, while 19%
said that only few students borrowed books.

110
USAID Readers are Leaders Project
Study Report May 2015

Figure 108: Percentage of early grade students borrowing books from the library

The children participating in the focus group discussions said they borrowed books from the
library, but only interesting books (the librarian told them briefly the content of the books and
then they chose which book to borrow).

When parents were asked, whether their child was a library member, more than half (54%)
said of the school library, 18% of the city or municipal library. Only 4% of children were
members of both school and municipal library, while 24% were not members of the library,
although in 18% of the cases a library existed in the school and/or municipality.

In the focus group discussions, parents said younger children were members of the school
library and the older children to the municipality library as well – although some children
preferred reading unused books.

111
USAID Readers are Leaders Project
Study Report May 2015

Figure 109: Children members of a library as reported by parents

As the internet is an indispensable resource in the modern schools, the directors were asked
to assess the access to the internet in their schools. Total of 65% of the school directors said
they had internet access, but were not satisfied with the speed and its quality.

Figure 110: Access to Internet of schools

Summary of results:

- More than half of schools have not supplied any didactical materials in the past three
years;
- Almost all schools have school library, which is fully accessible throughout the day for
early grade students with books other than those from the compulsory reading list. In
most schools, a librarian is hired;
- Early grade students borrow books from the school library;
- Schools have internet, but two thirds are not satisfied with the speed and quality.

8.3 Teacher Preparation

A well prepared teaching force is of great importance to an effective education system.
PIRLS results showed that higher average reading achievements of students were
associated with specialized education of their teachers in language or reading, as well as to
teachers’ having more experience and being satisfied with their careers. This section
provides information about teachers’ education, and professional development.

According to the level of education of early grade teachers, majority (82.7%) had university
degree. Only 4.3% had master or doctoral degree, while 13% still have post-secondary
education (two-years of teacher training).

112
USAID Readers are Leaders Project
Study Report May 2015

Figure 111: Level of education of surveyed early grade teachers

Based on the teachers’ responses in the questionnaires, almost two-thirds of teachers (65%)
have attended a training workshop on literacy, more specifically on teaching students how to
read, while 35% have never attended such workshop.

Figure 112: Percentage of teachers that attended training on teaching students to read

However, when asked whether they need additional training in early literacy, 45% of teachers
said they do.

113
USAID Readers are Leaders Project
Study Report May 2015

Figure 113: Percentage of teachers that need additional training on literacy

The situation is bit different for training in mathematics, where 86% of surveyed teachers
answered that they have attended workshop for teaching math, which is mainly result of the
extensive training provided by the BDE for introducing the new math and science curriculum
in the early grades.

Figure 114: Percentage of teachers that attended training on teaching math to

students

Despite all the training workshops, high 44% of the teachers said they needed additional
training in math.

114
USAID Readers are Leaders Project
Study Report May 2015

Figure 115: Percentage of teachers that need additional training in math

The survey also asked the teachers to name the organizations delivering training in literacy
and math. Majority of teachers (89.5%) pointed out the BDE for the new mathematics
curriculum, followed by training provided by USAID/PEP (44.4%) and UNICEF (12.1%).

Figure 116: Organizations delivering training in literacy and math

Summary of results:
- Teaching staff is highly educated, with only 14% of them with two-years of teacher
training;
- One third of teachers never attended literacy training and 14% never attended math
training. In both areas 45% of teachers said they needed additional training;
- Major training provider in the country is the Bureau for Development of Education,
followed by projects financed by international donors.

115
USAID Readers are Leaders Project
Study Report May 2015

8.4 Classroom Instruction

Classroom instruction and students’ day-to-day classroom activities is at the core of student
learning and are likely to have a considerable direct impact on their reading and mathematics
development.

Engaging instruction is highly related to higher achievement. Unfortunately, some teachers

reported limiting instruction because about one-third (29%) of the students in their class
suffered from some type of disability. 15% of them said they themselves have identified the
student with disabilities, 10% said these students had medical certificates, in 2% of the cases
the special education teacher has indicated this, while for 2% they are not sure. Total of 71%
of the teachers said they did not have students with disabilities.

Figure 117: Percentage of early grade teachers that have students with disabilities in
class

However, when the teachers were asked how they work with students with disabilities, only
54% reported they did not have such students. Most of the teachers (25%) pay more
attention to these students, 16% have individual education plans for these students, 10%
have assistance from the parents, 9% have assistance from a professional (school or
municipal special education teacher), 6% have no special treatment for these students and
5.6% have assistance from a professional (special education teacher who is privately hired).

116
USAID Readers are Leaders Project
Study Report May 2015

Figure 118: Working with students with disabilities

In terms of monitoring student progress, teachers reported applying a number of direct and
indirect approaches to evaluate how students were doing. Most common approaches
included direct observation (92%), reliance on written assessment results (90%) and
questioning of pupils (88%), review of student portfolio and other projects (85%) as well as
work sheets (82%), homework (76%) and mid-term exams (71%).

Figure 119: Measuring of student knowledge

Most teachers (94%) use the student results for assessment of understanding. Less teachers
use these results to adjust (63%) and better plan (50%) for class activities.

117
USAID Readers are Leaders Project
Study Report May 2015

Figure 120: How teachers use student results

When parents were asked how they received feedback from the school about the
achievements of their children, half of them (52%) said directly from the teachers, almost one
third (29%) at parents’ meetings. Smaller percentage of parents use the technology to get
reports, 11% via electronic school registry, while 8% via text messages.

Figure 121: Feedback for parents about student results

In the focus group discussions, parents reported cooperating closely with the teachers and
they were named as the first point of contact when a problem has arisen, while the school
pedagogue and psychologist were rarely communicated. They said they received information
about the progress of their children via various means: in individual meetings with the
teacher, teacher-parent meetings, by notes written on tests or text books, or by their children
themselves.

118
USAID Readers are Leaders Project
Study Report May 2015

When early grade teachers were asked how they deal with student that have poor results,
majority (80.5%) said they gave them more encouragement, 80% paid them more attention,
provide them with more didactical materials and 47% use the students with better results to
share their knowledge with their fellow students.

Figure 122: Teaching methods for students with poor results

Three-fourths of the teachers (77%) organize after-class activities for early grade students,
including both supplementary and remedial classes, 15% only remedial classes, while 2%
only supplementary classes for students with high achievements.

Figure 123: Additional classes in literacy and math

Summary of results:

119
USAID Readers are Leaders Project
Study Report May 2015

- One-third of teachers had students with special needs in their class, half of them have
identified themselves the students with disabilities. The work with these students is
mainly played down to paying more attention to them;
- Majority of teachers evaluate student knowledge through direct observation, written
assessments, questioning of pupils as well as review of student portfolio and other
projects;
- Student results are mainly used for assessment and less for adjusting and planning
for class activities;
- To student with poor results, teachers tend to give more encouragement and
attention;
- Not all teachers organize after-class activities for early grade students, but when they
do they include both supplementary and remedial classes.

8.5 Home Environment Support for Reading Achievement

PIRLS study revealed that students had higher reading achievement if their parents reported
that they themselves liked reading, often engaged in early literacy activities with their
children, had more home resources for learning, and that their children had attended
preprimary education. Children also had higher achievement if their parents reported that
their children started school able to do early literacy tasks (e.g., read sentences and write
some words).

One third of the parents (33%) reported having over 100 books at home and almost two
thirds (62%) had only few books. Only 5% of the parents said they had no books at home.

Figure 124: Books at home as reported by parents

The parents were also asked how often they bought magazines, newspapers, books and
story books. Only 8% of parents never buy magazines, while 36% do not buy them, but read
them electronically. Of those that buy magazines and newspapers, most of them (24%) buy
them weekly, 17% monthly and 15% daily.

120
USAID Readers are Leaders Project
Study Report May 2015

Figure 125: Frequency of buying magazines and newspapers as reported by parents

Regarding buying books and story books, only 2% never buy them, while most of them
(41%) buy them once a month.

Figure 126: Frequency of buying books and story books as reported by parents

In the questionnaires, more than half of parents (53%) reported reading together with their
child every day and 26% read on a weekly basis. Only 2% of parents reported never reading
with their child.

In the focus groups, parents said that their children read to them aloud every day, for more
than an hour, but parents said they needed new texts to practice because children learned
the ones they read by heart.

121
USAID Readers are Leaders Project
Study Report May 2015

Figure 127: Frequency of reading together with the child as reported by parents

Almost equal percentage of parents reported reading story books (45%) and books from the
mandatory reading list (42%). Only 13% reported reading books from the library.

According to the parents involved in focus group discussions, the books on the reading list
were not engaging and children were reluctant to read them, that is why parents would like
their children to read books with more content (unlike “Onomono Donomono”). Parents
believed that the chances students would understand a text were greater if parents read it to
them.

Figure 128: Type of books mostly read at home as reported by parents

High 97% of the surveyed parents said they discussed the content after reading with their
children, of which 75% discussed it always and 22% discussed it sometimes.

122
USAID Readers are Leaders Project
Study Report May 2015

In the focus groups, parents explained that they ensured children understood the text by
asking them to retell, by asking them questions, by repeating, explaining, drawing, showing
video on the internet. All interviewed parents said they had enough children’s literature at
home such as children’s stories, newspapers, child magazines, picture books, children’s
Bible, children’s books from when parents were young, encyclopedias and factopedias.

Figure 129: Discussing the content after reading as reported by parents

Apart from reading together with their parents, 75% of parents reported that their children
also read independently every day, while additional 12% read once a week.

Figure 130: Frequency of child’s independent reading as reported by parents

Children mostly read independently books from the mandatory list (45%) and story books
(40%) and less books from the library (15%).

123
USAID Readers are Leaders Project
Study Report May 2015

Figure 131: Literature mostly read by children independently as reported by parents

When teachers were asked about the number of parents checking the homework of their
children, majority of teachers (59%) said many parents checked the homework of their
children, 33% said few of the parents did, while only 7% said that all parents checked the
assignments.

Figure 132: Satisfaction with parents’ participation in monitoring student assignments

according to teachers

Around one third of parents (35%) said they were always involved in their child’s home
assignment, 23% were sometimes involved, while 33% were involved only if the child was
unable to complete the homework alone. Only 9% of parents never helped their children with
the homework.

124
USAID Readers are Leaders Project
Study Report May 2015

Figure 133: Engagement in child’s homework assignments as reported by parents

Almost, two thirds (63%) of the teachers are satisfied with the help parents provide to their
children with school assignments. However, more than one third or 36% are not satisfied.

Figure 134: Satisfaction with parents’ participation in monitoring student assignments

according to teachers

Parents were also asked how often they participate in school activities. Most of them (48%)
reported participation once or twice a school year, 23% once a month, 13% several times a
month, while 11% said never.

125
USAID Readers are Leaders Project
Study Report May 2015

Figure 135: Participation in school activities according to parents

Furthermore, most of the parents (36%) said they helped in decorating the classrooms, 27%
in organizing visits, 25 in providing financial assistance for supplying school materials and
12% provide expert assistance for implementation of curriculum.

Figure 136: Ways of participation of parents in school activities

When teachers were asked in what way parents participated in classroom activities, half of
them said they helped in organizing visits, 46% helped decorating the classroom, 35%
provided expert assistance on implementing some curriculum topics, while 27% provided
financial assistance for supplying school materials.

126
USAID Readers are Leaders Project
Study Report May 2015

Figure 137: Participation of parents in classroom activities according to teachers

To the same question, school directors said parents mostly help in decorating the classroom
(73%), helped in organizing visits (54%), provided expert assistance (50%) and provided
financial assistance for supplying school materials (35%).

Figure 138: Participation of parents in classroom activities according to school

directors

When it comes to the frequency of cooperation between parents and schools, more than half
of the teachers said they engaged them once or twice a school year, 32% said once a
month, while only 13% engaged them several times a month

127
USAID Readers are Leaders Project
Study Report May 2015

Figure 139: Frequency of parents involved in classroom activities according to

teachers

To the question, whether parents requested involvement in school activities, 61% of the
teachers said yes, but seldom, and 26% said no. Only 13% of the parents wanted to be
engaged often.

Figure 140: Frequency of parents’ request to be involved in school activities

To the same question, 47% of the parents said they seldom asked to be involved in school
activities, while 14% said often. One third of parents (34%) admitted of never asking to get
involved in school activities.

128
USAID Readers are Leaders Project
Study Report May 2015

Figure 141: Frequency of parents asking to get involved in school activities as

reported by parents

In the focus group discussions, all parents agreed that they were willing to cooperate when a
teacher asked them to, for example for Easter and New Year they handcrafted and
participated in various bazars. However, they were not willing to take initiative themselves.

Parents also have legal obligation to participate in the school activities through the work of
Parents’ Council. According to the parents responding to the questionnaires, 30% of them
reported being members in the Parents’ Council.

Figure 142: Membership in the Parents’ Council as reported by parents

According to school directors, 60% of parents‘ councils convened from 4 to 7 times last
school year, 25% up to 3 times and 15% more than 8 times.

129
USAID Readers are Leaders Project
Study Report May 2015

Figure 143: Meetings of Parents’ Council in a school year as reported by school

directors

The parents had similar answers to this question, as 24% reported having meeting up to 3
times, 12% more than 8 times, 44% from 4 to 7 times per school year. Around 20% of
parents either did not give response or said the council did not have meetings.

Figure 144: Meetings of Parents’ Council in a school year as reported by parents

The meetings were mainly initiated and the agenda drafted by the President of the Parents’
Council (59%). However, one third of school directors (33% admitted that they have initiated
the Council’s meetings.

130
USAID Readers are Leaders Project
Study Report May 2015

Figure 145: Who initiates meetings of Parents’ Council reported by school director

To the same questions, 29% of the parents said the meetings were mainly initiated and the
agenda drafted by the President of the Parents’ Council, 26% said by the school directors,
while total of 43% either admitted that they did not know (33%) or had no response (10%).

Figure 146: Who initiates meetings of Parents’ Council reported by parents

131
USAID Readers are Leaders Project
Study Report May 2015

Figure 147: Suggestions of Parents’ Council for overcoming certain school problems
as reported by parents

Majority of parents (80%) and school directors (85%) said the Parents’ Council provided
suggestions for overcoming certain school problems.

Figure 148: Suggestions of Parents’ Council for overcoming certain school problems
as reported by school directors

Some of the suggestions of Parents’ Council pointed out by the school director as
instrumental for overcoming certain problems were: regulating day care for early grade
students, later start of classes in winter, student uniforms, organizing excursions and fairs,
infrastructural improvements, landscaping, preventing damages to school property,
organizing student travel, improving the discipline, hygiene and safety.

132
USAID Readers are Leaders Project
Study Report May 2015

High 90% of school directors and 73% of parents said they were satisfied with the level of
support they get from the Parents’ Council.

Figure 149: Satisfaction of school directors with support of Parents’ Council

Figure 150: Satisfaction of parents with support of Parents’ Council

Summary of results:
- Only 7% of teachers are sure that all parents checked the assignments of their
children and total of one third of teachers are not satisfied with the help parents
provide to their children with school assignments;

133
USAID Readers are Leaders Project
Study Report May 2015

- Parents participate in classroom activities, more by helping them in organizing visits

and decorating the classroom, while less by providing expert and/or financial
assistance. School directors agree with the areas in which parents help, but they
believe parental help is greater than what teachers think;
- Only 13% of parents are engaged in school activities more often and at their own
request. The rest get involved seldom, once or twice a year;
- Parents’ Councils are active school bodies and school directors were mostly satisfied
with their work.

9. CONCLUSIONS

9.1 Lessons Learned from EGRA AND EGMA application

The latest EGRA baseline results confirm the results from previous year that the letter
recognition skills of Grade 2 and 3 students are well developed, while reading
comprehension remains the bottleneck in our early grade education.

The main factors affecting the EGRA and EGMA results are the following:
 Higher education of parents
 Studying in central or urban schools
 Attendance of pre-school institution
 Having books at home
 Borrowing books from the library
 Reading at home (either independently or with someone else)

These results highlight some critically important areas in which interventions will be needed.
In almost all tasks urban students outperformed rural students, which could be a red signal
for education policy makers. Poor instruction process, weak supervision from the education
institutions, lower level of education of parents, little exposure to books and other reading
materials, limited support from the community at large could be some of the factors
contributing to these results. Any reading intervention should also target the students from
satellite schools, students that have not attended pre-school and children of parents with
primary or less than primary education.

For students reading the average level, supplemental support should be provided to improve
their reading skills. For these students, progress-monitoring assessments may be
administered more frequently, perhaps once or twice monthly or as often as once per week.
Intensive support might entail:
• delivering instruction in a smaller group,
• providing more instructional time or more practice,
• presenting smaller skill steps in the instructional hierarchy,
• providing more explicit modeling and instruction, and/or
• providing greater scaffolding and practice.

For the results in mathematics, having in mind that new curriculum in mathematics and
sciences was introduced in Macedonia in September 2014, more time is required before
teachers master the new curriculum, so that the student achievements could show whether
this change is for the better.

134
USAID Readers are Leaders Project
Study Report May 2015

The number identification and number discrimination tasks remain to be easy, as even Grade
2 students, have high achievements despite the increase in the number of items, level of
difficulty of items and time limitation.

In the missing numbers task, after reformatting it with an even distribution of 10 items that
progress in difficulty Grade 3 students perform better, probably because they learned this
patterns in the previous grade and with the new curriculum they have more tasks to practice
this skill.

The poor results in addition and subtraction in the previous year, has triggered changes in
these two tasks, by dividing them into two levels. Level 1 consisted of five items assessed for
fluency i.e. whether children are becoming familiar with simple addition/subtraction problems,
timed for 60 seconds. Children were instructed to tell the assessor the first answer that
seems right to them, without using counters. Level 2 was timed for 2 minutes, however the
timing is not for fluency, but for efficiency. Here, children were given the opportunity to use
counters or their fingers to solve the addition/subtraction problems. However, the results did
not prove that if the children could have more time they would be able to do the more difficult
tasks. On contrary, they had more correct items per minute for the timed tasks, which
indicates that if a child has learned how to do the calculation, s/he would do it regardless of
the time.

In the word problems, the results of Grade 2 students are better this year in terms of the
average p-value and the correct items. Increased use of paper and pencil in Grade 2,
increased use of fingers particularly in Grade 3, and increased use of counters in both
grades. This increased use may be a benefit of the new math reform, encouraging teachers
to use more manipulatives in their instruction process.

The results of the EGRA/EGMA already indicate a need to review national policies and their
implementation related to early grade language and mathematics. These include the design
of the curriculum for early grade literacy; a restructuring of the school timetable to increase
time for reading; a priority on providing appropriate and sufficient instructional and reading
materials; teacher training to include using EGRA/EGMA tools for mastery checks in the
classroom, working with students with difficulties, organizing reading corner in the classroom
and providing time for children to read aloud; and increased community and parental
engagement to support early grade reading and mathematics.

9.2 School Climate

Part of creating a school learning environment focused on academic success involves a staff
that collaborates on curricular activities. For example, a study including a comprehensive
theoretical review and a meta-analysis of studies about professional communities indicated a
small but positive effect of professional communities on student achievement7.

Because teachers in Macedonia also considered the collaboration with colleagues to be

important in building a professional community, the idea of teacher collegiality and
collaboration should focus on the idea of collaboration for the purpose of improving teaching.

7Lomos, C., Roelande, H. H., & Bosker, R. J. (2011). Professional communities and student achievement–A
meta analysis. School Effectiveness and School Improvement, 22(2), 121–148.

135
USAID Readers are Leaders Project
Study Report May 2015

Teachers should be also encouraged to interact with other teachers and discuss how to
teach a particular topic, collaborate in planning and preparing instructional materials, share
what they have learned about their teaching experiences, visit another classroom to learn
more about teaching and work together to try out new ideas.

9.3 School Resources for Teaching Reading and Mathematics

A school’s location can have a substantial impact on whether the students attending that
school typically are from economically and educationally advantaged home backgrounds.
Also, the location of the school can provide access to important additional resources (e.g.,
libraries, media centers, or museums) or mean that the school is relatively isolated.

In general, the second and third grade students attending schools in cities had higher
average reading achievement than those in rural areas.

Studies have shown that resources are crucial for improving schooling, as the extent and
quality of school resources can have an important impact on the quality of classroom
instruction. The presence of a library or multimedia center may be particularly relevant for
developing reading literacy.

Libraries, both within the school and in the local community, provide a range of reading
materials and other resources from which teachers can draw to expand their instructional
approaches, and from which students can choose books for their own learning and
enjoyment. Also, with the growing use of technology, libraries should increasingly become
media centers offering a range of materials and Internet access, which is not the case in
Macedonia, as neither school directors nor teachers are satisfied with the speed of the
internet. School libraries are usually old and dusty places with outdated books from
Yugoslavia’s time. Perhaps if school libraries had books that interested students, more of
these students would become readers, improve their reading skills, and find a new enjoyable
pastime.

On the other hand, well-resourced classroom libraries rather than a larger school library, may
provide children with access to a variety of books. The distribution of 20 titles of picture
books in Macedonian and Albanian in every classroom from first to third grade by the
Readers are Leaders project is an effort in establishing these classroom libraries and
promoting the reading skills in children.

It remains books to be provided in Turkish and Serbian language (the other two official
languages of instruction in Macedonia) as well as in Romani for the children from Roma
community, who seem to be further disadvantaged for not being able to study in their mother
tongue.

9.4 Teacher Preparation

There is growing evidence that teacher preparation is a powerful predictor of students’

achievement, perhaps even overcoming socioeconomic and language background factors8.
Higher average reading achievement was associated with specialized education in language

8Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence.
Education Policy Analysis Archives, 8(1). Retrieved from http://epaa.asu.edu/epaa/v10n12/

136
USAID Readers are Leaders Project
Study Report May 2015

or reading. Achievement also was related to teachers’ having more experience and being
satisfied with their careers.

Recent research showed a positive relationship between teacher professional development

and student literacy achievement9 and that the amount of professional development (more
than 14 hours) was an important factor. Although the teachers in Macedonia has undergone
several cycles of training in literacy and math, they still do not feel confident and require
more training. However, it seems that the value of training as a professional development
activity aimed at enhancing their knowledge and skills is not yet seen, as teachers often look
at it as an instrument for obtaining certificate that would help them improve their professional
portfolio into the eyes of education inspectors.

Training of teachers remains a complex task but it must be assumed that teachers learn best
by doing and interacting with other professionals. This implies that teacher training should be
organized around modeling and practice, and that having brief trainings with follow-up and
refresher meetings is more effective than longer trainings. Regular professional development
through training and other activities should fill a demand for instructional practice and
support.

The collaboration of teachers within the learning communities and visits to schools by
mentors are critical for supporting teachers and improving student outcomes. Proper
selection of mentors is essential so that they can effectively support teachers. The results of
different mentorship programs indicate that schools visited frequently are likely to have
stronger student performance. Large-scale instructional improvements, however, are difficult,
as they require face to face time, practice, and ongoing feedback.

9.5 Classroom Instruction

The learning environment of the classroom itself, the instructional approaches and materials
used in the classroom are clearly important to establishing teaching and learning patterns.

Although teachers report using various instructional activities and strategies, including
additional and remedial classes for children, they still need to provide instruction that will
interest and engage students in learning.

The results for the Engaging Students in Learning scale in PIRLS show that six items are
related to teachers’ instructional practices intended to reinforce learning: summarizing the
lesson’s learning goals, relating the lesson to students’ daily lives, questioning to elicit
reasons and explanations, encouraging students to show improvement, praising students for
good effort, and bringing interesting things to class.

In addition for the purpose of developing the reading comprehension skills, the teachers
should ask the students to locate information within the text, identify the main ideas of what
they have read, explain or support their understanding of what they have read, to compare
what they have read with their own experiences or make generalizations and draw
inferences, to make predictions about what will happen next in the text.

9 Biancarosa, G., Bryk, A. S., & Dexter, E. R. (2010). Assessing the value-added effects of literacy collaborative
professional development on student learning. The Elementary School Journal, 111(1), 7–34.
Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how
teacher professional development affects student achievement Retrieved from http://ies.ed.gov/ncee/edlabs

137
USAID Readers are Leaders Project
Study Report May 2015

The skills and strategies of making comparisons, generalizations, inferences, and predictions
are also important reading comprehension processes in the PIRLS Framework, and have
been learned by the fourth grade students in the highest achieving countries.

In general, teachers emphasize on retrieving information and identifying main ideas in texts,
but the emphasis on more complex reading comprehension strategies is still low. Asking
“why” questions not just as part of the reading instruction, but also in other areas should
promote the higher level of thinking of students.

Apart from using textbooks and worksheets as the basis for reading instruction, teachers
should be encouraged to use supplementary resources such as a variety of children’s books
and computer applications.

Having students read books and a variety of different types of materials is fundamental to
developing their reading comprehension skills and strategies, so investing in a small
classroom library is a great option so that children can have ready access to books and
magazines as part of their reading lessons and activities.

9.6 Home Environment Support for Reading Achievement

A supportive home environment and an early start are crucial in shaping children’s reading
literacy. It is important for parents to help their children develop the habit of reading at a
young age. For most children, the home provides modeling and direct guidance in effective
literacy practices. Young children who see adults and older children reading or using texts in
different ways are learning to appreciate and use printed materials. There is a strong positive
relationship between students’ reading achievement and home experiences that foster
literacy learning.

As the research show, students had higher reading achievement if their parents often
engaged in early literacy activities with their children, had more home resources for learning
and their children had attended preprimary education.

Children also had higher reading achievement if they reported that they started school able
to do early literacy tasks (e.g., read some sentences and write some words).

Throughout a child’s development, the time devoted to literacy related activities remains
essential to the acquisition of reading literacy skills and the effects can be long lasting (Levy,
Gong, Hessels, Evans, & Jared, 2006).

A large study in England recently found that a composite variable of seven home activities—
being read to, going to the library, playing with numbers, painting and drawing, being taught
letters, being taught numbers, and singing or reciting songs/ poems/rhymes—had greater
predictive power for literacy and numeracy achievement than any other variables studied,
including the social-economic status, parents’ education, and household income (Melhuish et
al., 2008).

PIRLS has also consistently shown a positive relationship between early reading skills and
average reading achievement at the fourth grade.

As the time devoted to literacy-related activities is essential to the acquisition of reading

literacy skills, the parents should be encouraged to be engaged in early literacy activities with

138
USAID Readers are Leaders Project
Study Report May 2015

their children, such as: reading books, telling stories, singing songs, playing with alphabet
toys, talking about things done, talking about things read, playing word games, writing letters
or words, and reading aloud signs and labels.

Home resources play an important role in acquiring reading literacy skills, including parents’
education, and books in the home.

Preprimary education, in the form of preschool, kindergarten, or an early childhood education

program, plays an important role in preparing children for primary school. Besides giving
students an early start in school and life, preprimary education provides an avenue for
overcoming children’s disadvantages and can help to break the generational cycles of
poverty and low achievement.

139
USAID Readers are Leaders Project
Study Report May 2015

1. APPENDICES

1.1 Appendix 1: List of schools participating in EGRA and EGMA studies

List of schools participating in longitudinal study

No. Name of school Place Municipality

1 Stiv Naumov Skopje Gazi Baba
2 Naum Naumovski - Borche Skopje Gazi Baba
3 11 Oktomvri Skopje Centar
4 Blazhe Koneski Skopje Aerodrom
5 Petar Zdravkovski - Penko Skopje Butel
6 Goce Delchev Mogila Mogila
7 Kocho Racin Ivanjevci Mogila
8 Sv. Kiril I Metodij Bitola Bitola
9 Elpida Karamandi Bitola Bitola
10 Kiril I Metodij Kochani Kochani
11 Slavcho Stojmenski Vinica Vinica
12 Goce Delchev Konche Konche
13 Sv. Kliment Ohridski Ohrid Ohrid
14 Naim Frasheri s.Jagol Dolenci Kichevo
15 Sande Shterjovski Kichevo Kichevo
16 Simche Nastovski Vratnica Jegunovce
17 Bratsvo Migjeni Tetovo Tetovo
18 Hristijan Todorovski Karposh s.Dragomance Staro Nagorichane
19 Ilinden Kriva Palanka Kriva Palanka
20 Liria s.Jabolchishte Chashka
21 Blazhe Koneski Veles Veles
22 Mirche Acev Skopje Gjorche Petrov
23 Strasho Pindzur Skopje Gjorche Petrov
24 Goce Delchev Gorno Lisiche Aerodrom
25 Brakja Miladinovci Skopje Aerodrom
26 Drita s.Rashche Saraj
27 Goce Delchev Prilep Prilep
28 Sv. Kiril I Metodij s.Buchin Krushevo
29 Manchu Matak Krivogashtani Krivogashtani
30 Blazhe Koneski Tochila Prilep
31 Nikola Petrov Rusinski s.Rusinovo Berovo
32 Vancho Kitanov Pehchevo Pehchevo
33 Josip Broz Tito Valandovo Valandovo
34 Sv. Kliment Ohridski Draslajca Struga
35 Grigor Prlichev Ohrid Ohrid
36 Andreja Savevski Kikish Tetovo Tetovo
37 Prparimi Chegrane Gostivar
38 Faik Konica s.Slupchane Lipkovo
39 Vasil Glavinov Veles Veles
40 Jordan Hadzi Konstantinov - Dzinot Veles Veles
41 Kiril I Metodij Sveti Nikole Sveti Nikole
42 Maca Ovcarova Veles Veles

140
USAID Readers are Leaders Project
Study Report May 2015

List of schools participating in baseline study

No. Name of school Place Municipality

1 Vasil Glavinov Skopje Chair
2 Kuzman Shapkarev Drachevo Kisela Voda
3 Avram Pisevski s.Bardovci Karposh
4 Bratstvo Taftalidze Karposh
5 Vojdan Chernodrinski Skopje Karposh
6 Mite Bogoevski Resen Resen
7 Rampo Levkata Prilep Prilep
8 Pere Toshev - Dupjachani Dupjachani - Desovo Dolneni
9 Slavko Lumbarkovski Novaci Novaci
10 Tosho Arsov Shtip Shtip
11 Nikola Karev Probishtip Probishtip
12 Vlado Kantardziev Gevgelija Gevgelija
13 Bratstvo - Edinstvo Ohrid Ohrid
14 Strasho Pindzur Vevchani Vevchani
15 Pashko Vasa Grupchin Zhelino
16 Dervish Cara Dolno Palchishte Bogovinje
17 Hristijan Karposh Kumanovo Kumanovo
18 Tode Hadzi - Tefov Kavadarci Kavadarci
19 Tosho Velkov Pepeto Kavadarci Kavadarci
20 Pere Toshev Rosoman Rosoman
21 Gjorgjija Pulevski Skopje Aerodrom
22 Brakja Miladinovci v. Miladinovci Ilinden
23 Jan Amos Komenski Skopje Karposh
24 25 Maj Hasanbeg Gazi Baba
25 Krum Toshev v. Trubarevo Gazi Baba
26 Njegosh v. Idrizovo Gazi Baba
27 Naim Frasheri v. Studenichani Studenichani
28 Draga Stojanoska v. Rakotinci Sopishte
29 Goce Delchev Shtip Shtip
30 Vancho Prke Delchevo Delchevo
31 Metodi Mitevski Brico v. Lozovo Lozovo
32 Krste Petkov Misirkov v. Orizari Kochani
33 Kiril i Metodij v. Oraovica Radovish
34 Kiril i Metodij v. Stojakovо Bogdanci
35 Petar Musev Bogdanci Bogdanci
36 Vidoe Podgorec Strumica Strumica
37 Krume Volnaroski v. Topolchani Prilep
38 Sv. Kliment Ohridski Bitola Bitola
39 Brakja Miladinovci v. Zhvan Demir Hisar
40 Peco Daskalot v. Dolneni Dolneni
41 Slavejko Arsov v. Podmochani Resen
42 Dobre Jovanoski Prilep Prilep
43 Vladimir Polezinovski Kichevo Kichevo
44 Goce Delchev v. Podgorci Struga
45 Ashim Agushi v. Radolishta Struga
46 Bratsvo - Edinstvo Debar Debar
47 Luigj Gurakuqi v. Zhelino Zhelino

141
USAID Readers are Leaders Project
Study Report May 2015

48 Aleksandar Zdravkovski v. Jegunovce Jegunovce

49 Asdreni v. Glogji Tearce
50 Ismail Qemali Gostivar Gostivar
51 11 Oktomvri Kumanovo Kumanovo
52 Hristijan Todorovski Karposh v. Mlado Nagorichane Staro Nagorichane
53 Kiril I Metodij v. Romanovce Kumanovo
54 Toli Zordumis Kumanovo Kumanovo
55 Petre Pop Arsov v. Bogomila Chaska
56 Goce Delchev Kavadarci Kavadarci
57 Rajko Zhinzifov v. Dolno Orizari Veles
58 Strasho Pindzur Negotino Negotino
59 Dame Gruev v. Gradsko Gradsko
60 Dame Gruev v. Erdzelija Sveti Nikole

1.2 Appendix 2: EGRA and EGMA longitudinal results

Characteristics of Sample
The breakdown of sample of students taking EGRA by grade, language of instruction,
location of the schools and gender is presented below, showing a balanced sample in both
grades according to all three features.

Approximately 73% of the sampled students attend classes in Macedonian and 27% in
Albanian language of instruction. Majority of students (66%) are from urban areas, while 34%
from rural. The sample is composed of an average of 53% male and 47% female students.

Figure 151. Percentage of students in EGRA sample per grade, language of

instruction, location of the schools and gender

The distribution of students in different categories may vary in EGMA sample a little, because
some students were absent during the testing time and were not able to take the test, but it is
still balanced in both grades according to all three features.

142
USAID Readers are Leaders Project
Study Report May 2015

Figure 152. Percentage of students in EGМA sample per grade, language of

instruction, location of the schools and gender

EGRA results

Figure 153. Comparison of average percentage of students that completed EGRA

tasks in Macedonian language of instruction

143
USAID Readers are Leaders Project
Study Report May 2015

Figure 154. Comparison of average percentage of students that completed EGRA

tasks in Albanian language of instruction

144
USAID Readers are Leaders Project
Study Report May 2015

Figure 155. Comparison of average percentage of students that completed EGRA

according to gender in Macedonian language of instruction

Figure 156. Comparison of average percentage of students that completed EGRA

according to gender in Albanian language of instruction

Figure 157. Comparison of average percentage of students that completed EGRA

according to type of school in Macedonian language of instruction

145
USAID Readers are Leaders Project
Study Report May 2015

Figure 158. Comparison of average percentage of students that completed EGRA

according to type of school in Albanian language of instruction

Figure 159. Comparison of average percentage of students that completed EGRA

according to pre-school attendance in Macedonian language of instruction

146
USAID Readers are Leaders Project
Study Report May 2015

Figure 160. Comparison of average percentage of students that completed EGRA

according to pre-school attendance in Albanian language of instruction

Figure 161. Comparison of average percentage of students that completed EGRA

according to access to books at home in Macedonian language of instruction

147
USAID Readers are Leaders Project
Study Report May 2015

Figure 162. Comparison of average percentage of students that completed EGRA

according to access to books at home in Albanian language of instruction

Figure 163. Average percentage of students that completed EGRA tasks according to
the language of instruction and whether third grade students read before going to
school

148
USAID Readers are Leaders Project
Study Report May 2015

Figure 164. Average percentage of students that completed EGRA tasks according to
the language of instruction and whether third grade students borrow library books

Figure 165. Average percentage of Grade 3 students that completed EGRA tasks in
Macedonian and Albanian language according to with whom they are reading at home

EGMA results

149
USAID Readers are Leaders Project
Study Report May 2015

Figure 166. Average percentage of students that completed EGMA according to

gender

Figure 167. Average percentage of students that completed EGMA tasks according to
language of instruction

150
USAID Readers are Leaders Project
Study Report May 2015

Figure 168. Average percentage of students that completed EGMA according to type
of school

Figure 169. Average percentage of students that completed EGMA according to pre-
school attendance

151
USAID Readers are Leaders Project
Study Report May 2015

Figure 170. Average percentage of students that completed EGMA according to

availability of books at home

1.3 Appendix 3: Questionnaire for School Director

1. What is your work position in the school?

ᴑ Principal

ᴑ Assistant Principal

2. Gender

ᴑ Male

ᴑ Female

3. Do students in your school attend classes in shifts?

ᴑ Yes

ᴑ No

4. What is the language of instruction in your school?

ᴑ Macedonian

ᴑ Albanian

152
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Turkish

ᴑ Serbian

5. How many early grade school teachers are employed in the school?

ᴑ Up to 5

ᴑ 5-10

ᴑ 11-20

ᴑ Over 20

6. How often are teachers absent from work (excluding absence due to training)?

ᴑ Seldom (up to 5 days a year)

ᴑ Often (up to 10 days a year)

ᴑ Very often (over 10 days a year)

7. What measures do you take when teachers are late for classes?

ᴑ I take strict legal measures

ᴑ I tell them to keep an eye on their punctuality

ᴑ I leave it to their consciousness

ᴑ I do not do anything in order not to argue with them

ᴑ Teachers are not late in our school

8. What alternatives do you have for students in the absence of their teacher?

ᴑ Teacher from another class is appointed to be in charge of the class

ᴑ Appropriate substitute (pedagogue /psychologist /sociologist) is in charge of the

class

ᴑ Substitute teacher is hired

ᴑ All students join another class

ᴑ Students are transferred in other classes

153
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Students spend school hours in the school playground

ᴑ Classes are dismissed and children can go home

9. How do you monitor teachers’ performance?

ᴑ Class observation

ᴑ Monitoring student results at tests prepared by teachers

ᴑ Quarterly student progress reports submitted by teachers

ᴑ Feedback from parents

ᴑ Feedback from advisors from the Bureau for Development of Education

ᴑ Results of external testing

ᴑ Results of integral evaluation of the school

10. How often do you as director/ assistant director observe teacher classes?

ᴑ Once a school year

ᴑ Once a term

ᴑ Once a quarter

ᴑ Once a month

ᴑ Never

11. Do you document the class observations in the form of report?

ᴑ Yes

ᴑ No

12. Did the school have sufficient textbooks for its students in accordance with the
Ministry of Education and Science procedures at the beginning of this school year?

ᴑ Yes

ᴑ No

13. Does the school have a library?

154
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Yes

ᴑ No

14. Do you have a person hired as librarian?

ᴑ Yes

ᴑ No

15. Do students from early grades borrow books from the library on a regular basis?

ᴑ Yes

ᴑ No

16. Based on the number of borrowed books, what percentage of students from Grade 1
to 3 borrow books from the library?

ᴑ Few students (around 10%)

ᴑ Half of the students (around 50%)

ᴑ Almost everyone (90-100%)

17. How many hours per day is the school library open?

ᴑ Only during the first shift

ᴑ Only during the second shift

ᴑ Throughout the entire school day

18. Does your library have books and story books for early grade students (besides the
mandatory reading lists)?

ᴑ Yes

ᴑ No

19. Has your school supplied didactic materials for early grade mathematics over the last
three years?

ᴑ Yes

155
USAID Readers are Leaders Project
Study Report May 2015

ᴑ No

20. In which grade do you expect students to be able to read in their first language?

ᴑ First

ᴑ Second

ᴑ Third

ᴑ Fourth or higher

21. In which grade do you expect students to be able to write in their first language?

ᴑ First

ᴑ Second

ᴑ Third

ᴑ Fourth or higher

22. How many times in a school year does the Parents Council have meetings in your
school?

ᴑ More than 8 times

ᴑ From 4 to 7 times

ᴑ Up to 3 times

ᴑ Does not have meetings

23. Who initiates the meeting of the Parents Council and drafts the agenda?

ᴑ School director

ᴑ President of the Parents Council

ᴑ Other: ___________________

24. Does the Parents Council gives suggestions for overcoming certain problems in the
school?

ᴑ Yes

156
USAID Readers are Leaders Project
Study Report May 2015

ᴑ No

If the answer is Yes, please list some of them:

ᴑ ___________________

ᴑ ___________________

ᴑ ___________________

25. In general, are you satisfied with the level of support from the Parents Council?

ᴑ Yes

ᴑ No

26. In which ways do parents participate in school activities?

ᴑ Provide expert assistance on certain topics for implementation of curriculum

ᴑ Provide assistance for organizing visits

ᴑ Provide financial assistance for supplying school materials

ᴑ Provide assistance for decorating the classrooms

27. How often do you ask the parents to get involved in school activities?

ᴑ Several times a month

ᴑ Once a month

ᴑ Once or twice a school year

ᴑ Never

28. Do the parents ask to get involved in school activities?

ᴑ Yes, very often

ᴑ Yes, but seldom

ᴑ No

29. Does your school have access to the internet?

ᴑ Yes and we are satisfied with the speed and quality

157
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Yes, but we are not satisfied with the speed and quality

ᴑ No

1.4 Appendix 4: Questionnaire for Teachers

1. Gender

ᴑ Male

ᴑ Female

2. What is your first language?

ᴑ Macedonian

ᴑ Albanian

ᴑ Turkish

ᴑ Serbian

ᴑ Other

3. What language do you teach in?

ᴑ Macedonian

ᴑ Albanian

ᴑ Turkish

ᴑ Serbian

ᴑ Other

4. What is your highest level of education?

ᴑ Post-secondary

ᴑ University

ᴑ Master

ᴑ Doctoral

158
USAID Readers are Leaders Project
Study Report May 2015

5. Does your class have special needs students?

ᴑ Yes, and those students have medical certificate

ᴑ Yes, the special education teacher has indicated this

ᴑ Yes, I as a teacher have observed this

ᴑ No

ᴑ Do not know/ I am not sure

6. How do you work with the students with special education needs in your class?

ᴑ I have no special treatment for these students

ᴑ I pay more attention to these students

ᴑ I have individual education plans for these students

ᴑ I have assistance from a professional (special education teacher who is privately

hired)

ᴑ I have assistance from a professional (school or municipal special education

teacher)

ᴑ I have assistance from the parents

7. While working as a teacher, have you participated in a training course for teaching
students how to read?

ᴑ Yes

ᴑ No

8. Do you think you need additional training for early grade literacy?

ᴑ Yes

ᴑ No

9. While working as a teacher, have you participated in a training course for teaching
math?

ᴑ Yes

ᴑ No

159
USAID Readers are Leaders Project
Study Report May 2015

10. Do you think you need additional training for early grade mathematics?

ᴑ Yes

ᴑ No

11. Please tell us what organization delivered the early grade literacy and mathematics
training?

ᴑ BDE for the new mathematics curriculum

ᴑ UNICEF

ᴑ USAID/PEP

ᴑ Other (please state)

12. Do students from higher grades also have classes in your classroom?

ᴑ Yes

ᴑ No

13. How often are students from your class absent on a regular basis?

ᴑ Some students are frequently absent

ᴑ Some students are absent due to sickness

ᴑ Students are rarely absent

14. How often does the Principal/Assistant Principal observe your classes?

ᴑ Once a school year

ᴑ Once a term

ᴑ Once a quarter

ᴑ Once a month

ᴑ Never

15. Do you get feedback after the class observation?

ᴑ Yes and I try to implement the given suggestions

160
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Yes, but I do not find it useful

ᴑ No

16. What are the most common topics for collaboration with your colleagues in regards to
planning and implementation of lessons?

ᴑ We prepare joint lesson plans

ᴑ We exchange experience regarding planning and implementation of lessons

ᴑ We disseminate the training

ᴑ We consult with one another

17. Who do you consult when you need advice in regards to curriculum?

ᴑ Other teachers during formal meetings

ᴑ Other teachers during informal conversations

ᴑ Student support services

ᴑ School director

ᴑ Assistant school director

ᴑ Advisor from the Bureau for Development of Education

ᴑ There is no adequate person for consultation

ᴑ I never need advice

18. How often does the student support service (psychologist/pedagogue) observe your
classes during school year?

ᴑ Once a school year

ᴑ Once a term

ᴑ Once a quarter

ᴑ Once a month

ᴑ Never

19. Do you get feedback from the student support service after the class observation?

161
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Yes and I try to implement the given suggestions

ᴑ Yes, but I do not find it useful

ᴑ Never

20. How often do advisors from the Bureau for Development of Education visit your
school?

ᴑ Once a school year

ᴑ Once a term

ᴑ Once a quarter

ᴑ Once a month

ᴑ Never

21. How do you measure student knowledge?

ᴑ Written exams

ᴑ Oral exams

ᴑ Portfolio and other projects

ᴑ Homework

ᴑ Class activities

ᴑ Work sheets

ᴑ Mid-term exams

22. How do you use student results of oral and written exams in your class?

ᴑ To grade the student

ᴑ To evaluate how much the student understands the material

ᴑ To plan class activities

ᴑ To adapt class activities to better suit student needs

23. How many of the parents/guardians check your student homework?

162
USAID Readers are Leaders Project
Study Report May 2015

ᴑ All

ᴑ Most of them

ᴑ Few of them

ᴑ None

24. Are you satisfied with the level of parents’ participation in helping their children with
school assignments/work?

ᴑ Yes

ᴑ No

25. In which ways do parents participate in school activities?

ᴑ Provide expert assistance on certain topics for implementation of curriculum

ᴑ Provide assistance for organizing visits

ᴑ Provide financial assistance for supplying school materials

ᴑ Provide assistance for decorating the classrooms

26. How often do you ask the parents to get involved in school activities?

ᴑ Several times a month

ᴑ Once a month

ᴑ Once or twice a school year

ᴑ Never

27. Do the parents ask to get involved in school activities?

ᴑ Yes, very often

ᴑ Yes, but seldom

ᴑ No

28. In which grade do you expect students to be able to read in their first language?

163
USAID Readers are Leaders Project
Study Report May 2015

ᴑ First grade

ᴑ Second grade

ᴑ Third grade

ᴑ Fourth grade or higher

29. In which grade do you expect students to be able to write in their first language?

ᴑ First

ᴑ Second

ᴑ Third

ᴑ Fourth or higher

30. How do you teach students in your class with poor results?

ᴑ I do not have special methods for such students

ᴑ I pay more attention to students with poor results

ᴑ I assign them more homework

ᴑ I instigate/encourage them

ᴑ I talk with their parents more frequently

ᴑ Students with better results help those with poor ones

ᴑ I provide them with more didactic materials

31. Do you organize additional classes in literacy and math?

ᴑ Yes, supplementary classes for the students with high achievements

ᴑ Yes, remedial classes for the students with poor results

ᴑ Yes, both supplementary and remedial classes

ᴑ No

164
USAID Readers are Leaders Project
Study Report May 2015

1.5 Appendix 5: Questionnaire for Parents

1. Gender

ᴑ Male

ᴑ Female

2. What is your first language?

ᴑ Macedonian

ᴑ Albanian

ᴑ Turkish

ᴑ Serbian

ᴑ Other

3. What language does your child attend classes in?

ᴑ Macedonian

ᴑ Albanian

ᴑ Turkish

ᴑ Serbian

4. What is your highest level of education?

ᴑ Elementary

ᴑ Secondary

ᴑ Higher and above

5. Did your child go to kindergarten?

ᴑ Yes

ᴑ No

165
USAID Readers are Leaders Project
Study Report May 2015

6. Which grade is your child in?

ᴑ First

ᴑ Second

ᴑ Third

ᴑ Fourth

ᴑ Fifth

7. How often is your child absent from school?

ᴑ Seldom, up to two days a year

ᴑ Often, from three to ten days a year

ᴑ Very often, more than ten days a year

8. Does your child know how to read?

ᴑ Yes

ᴑ No

9. Where did your child learn how to read?

ᴑ In school

ᴑ In kindergarten

ᴑ At home with the help of parents

ᴑ At home with the help of grandmother/grandfather, brother/sister

10. How do you get feedback from the school for your child’s achievements?

ᴑ Directly from teachers

ᴑ At parents meetings

ᴑ Via text messages

ᴑ Via electronic school register

166
USAID Readers are Leaders Project
Study Report May 2015

11. Are you engaged in your child’s homework assignments?

ᴑ Yes, always

ᴑ Yes, sometimes

ᴑ Yes, whenever my child is not able to do the homework independently

ᴑ No, my child does homework independently

12. Do you have books at your home?

ᴑ Yes, a few

ᴑ Yes, many (over 100)

ᴑ No

13. How often do you buy magazines and newspapers?

ᴑ Every day

ᴑ Once a week

ᴑ Once a month

ᴑ I do not buy them, but I read them electronically

ᴑ I never buy them

14. How often do you buy books and story books to your child?

ᴑ Once a week

ᴑ Once a month

ᴑ Once every few months

ᴑ Seldom

ᴑ I never buy them

15. Is your child a member of a library?

ᴑ Yes, the school library

167
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Yes, the city or municipal library

ᴑ No, there is no library neither in the school or in the municipality

ᴑ No, although there is library in the school or in the municipality

16. How often do you read together with your child?

ᴑ Every day

ᴑ Once a week

ᴑ Once a month

ᴑ Seldom

ᴑ Never

17. What do you mostly read with your child?

ᴑ Story books

ᴑ Books from the mandatory reading lists

ᴑ Books from library

18. Do you and your child discuss the content after reading?

ᴑ Always

ᴑ Sometimes

ᴑ Never

19. How often does your child read independently?

ᴑ Every day

ᴑ Once a week

ᴑ Once a month

ᴑ Seldom

ᴑ Never

20. What does your child mostly read independently?

168
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Story books

ᴑ Books from mandatory reading lists

ᴑ Books from library

21. Are you a member of the Parents Council in the school?

ᴑ Yes

ᴑ No

22. How many times in a school year does the Parents Council have meetings in your
school?

ᴑ More than 8 times

ᴑ From 4 to 7 times

ᴑ Up to 3 times

ᴑ Does not have meetings

23. Who initiates the meeting of the Parents Council and drafts the agenda?

ᴑ School director

ᴑ President of the Parents Council

ᴑ Do not know

24. Does the Parents Council gives suggestions for overcoming certain problems in the
school?

ᴑ Yes

ᴑ No

25. In general, are you satisfied with the level of support from the Parents Council?

ᴑ Yes

ᴑ No

26. In which ways do parents participate in school activities?

169
USAID Readers are Leaders Project
Study Report May 2015

ᴑ Provide expert assistance on certain topics for implementation of curriculum

ᴑ Provide assistance for organizing visits

ᴑ Provide financial assistance for supplying school materials

ᴑ Provide assistance for decorating the classrooms

27. How often are parents involved in school activities?

ᴑ Several times a month

ᴑ Once a month

ᴑ Once or twice a school year

ᴑ Never

28. Do you ever ask to get involved in school activities?

ᴑ Yes, very often

ᴑ Yes, but seldom

ᴑ No

170