Trends in International Mathematics and Science Study (TIMSS)

5. DATA QUALITY AND COMPARABILITY

In addition to setting high standards for data quality, the TIMSS International Study Center has tried to ensure the overall quality of the study through a dual strategy of providing support to the national centers and performing quality control checks. However, just as most surveys with complexity similar to TIMSS would have the possibility of error, potential sources of error are next discussed.

Estimates produced using data from TIMSS are subject to two types of error—nonsampling and sampling errors. Nonsampling errors can be due to errors made in collecting and processing data. Sampling errors can occur because the data were collected from a sample rather than a complete census of the population.

Sampling Error

Sampling errors arise when a sample of the population, rather than the whole population, is used to estimate some statistic. Different samples from the same population would likely produce somewhat different estimates of the statistic in question. This fact means that there is a degree of uncertainty associated with statistics estimated from a sample. This uncertainty is referred to as sampling variance and is usually expressed as the standard error of a statistic estimated from sample data. The approach used for calculating standard errors in TIMSS is jackknife repeated replication (JRR). Standard errors can be used as a measure for the precision expected from a particular sample.

Confidence intervals provide a way to make inferences about population statistics in a manner that reflects the sampling error associated with the statistic, using the standard error. Assuming a normal distribution, the population value of this statistic can be inferred to lie within the confidence interval in 95 out of 100 replications of the measurement on different samples drawn from the same population. For example, the average mathematics score for the U.S. eighth-grade students was 518 in 2015, and this statistic had a standard error of 3.1. Therefore, it can be stated with 95 percent confidence that the actual average of U.S. eighth-grade students in 2015 was between 512 and 524 (1.96 x 3.1 = 6.1; confidence interval = 518 plus and minus 6.1).

Nonsampling Error

Nonsampling error is a term used to describe variations in the estimates that may be caused by population coverage limitations, nonresponse bias, and measurement error, as well as data collection, processing, and reporting procedures. The sources of nonsampling errors are typically problems like unit and item nonresponse, differences in respondents' interpretations of the meaning of the survey questions, response differences related to the particular time the survey was conducted, and mistakes in data preparation.

Unit nonresponse. Unit nonresponse error results from nonparticipation of schools and students. Weighted and unweighted response rates are computed for each participating country or other education system by grade, at the school level, and at the student level. Overall response rates (combined school and student response rates) are also computed.

The minimum acceptable school-level response rate for all countries and other education systems, before the use of replacement schools, is set at 85 percent. This criterion is applied to the unweighted school-level response rate. However, both weighted and unweighted school-level response rates are calculated, with and without replacement schools. It is generally the case that weighted and unweighted response rates are similar.

Like the school-level response rate, the minimum acceptable student-level response rate is set at 85 percent for all countries and other education systems. This criterion is applied to the unweighted student-level response rate. However, both weighted and unweighted student-level response rates are calculated. The weighted student-level response rate is the sum of the inverse of the selection probabilities for all participating students divided by the sum of the inverse of the selection probabilities for all eligible students.

Table TIMSS-2 shows the weighted unit level response rates for the U.S. for the data collections of 1995, 1999, 2003, 2007, 2011, 2015 for grades 4 and 8, and 2015 for grade 12 advanced students.

Measurement error. Measurement error is introduced into a survey when its test instruments do not accurately measure the knowledge or aptitude they are intended to assess. The largest potential source of measurement error in TIMSS results from differences in the mathematics and science curricula across participating countries and other education systems. In order to minimize the effects of measurement error, TIMSS carries out a special test called the Test-Curriculum Matching Analysis. Each country or other education system is asked to identify, for each item, whether the topic of the item is in the curriculum of the majority of the students.

Data Comparability

Through a careful process of review, analysis, and refinement, the assessment and questionnaire items are purposefully developed and field tested for similarity and for reliable comparisons between survey years. After careful review of all available data, including a test for reliability between old and new items, the TIMSS assessments are found to be very similar in format, content, and difficulty level across years.

Findings from comparisons between the results of TIMSS, however, cannot be interpreted to indicate the success or failure of mathematics and science reform efforts within a particular country or other education system, such as the United States. International experts develop the TIMSS curriculum frameworks to portray the structure of the intended school mathematics and science curricula from many nations, not specifically the United States. Thus, when interpreting the findings, it is important to take into account the mathematics and science curricula likely encountered by U.S. students in school. TIMSS results are most useful, however, when they are considered in light of knowledge about education systems that include curricula, but also factors in trends in education reform, changes in school-age populations, and societal demands and expectations. (See also NAEP/TIMSS linking study discussion above.)

The ability to compare data across different countries and other education systems constitutes a considerable part of the purpose behind TIMSS. As a result, it is crucial to ensure that items developed for use in one country or other education system are functionally identical to those used in other countries and other education systems. Because questionnaires are originally developed in English and later translated into the language of each of the TIMSS countries and other education systems, some differences do exist in the wording of questions. NRCs from each country or other education system review the national adaptations of individual questionnaire items and submit a report to the IEA Data Processing Center. In addition to the translation verification steps used for all TIMSS test items, a thorough item review process is used to further evaluate any items that are functioning differently in different countries and other education systems according to the international item statistics. In certain cases, items have to be recoded or deleted entirely from the international database as a result of this review process.