fedea

Fundación de Estudios de Economía Aplicada

Publicizing the results of standardized external tests: Does it have an effect on school outcomes? by Brindusa Anghel^*, Antonio Cabrales^** Jorge Sainz^*** and Ismael Sanz^*** Documento de Trabajo 2013-01

SERIE: Talento, Esfuerzo y Mobilidad Social CÁTEDRA Fedea-Banco Sabadell

January 2012

* Fedea. ** Universidad Carlos III de Madrid. *** Universidad Rey Juan Carlos.

Los Documentos de Trabajo se distribuyen gratuitamente a las Universidades e Instituciones de Investigación que lo solicitan. No obstante están disponibles en texto completo a través de Internet: http://www.fedea.es. These Working Paper are distributed free of charge to University Department and other Research Centres. They are also available through Internet: http://www.fedea.es. ISSN: 1696-750X

Brindusa Anghel

FEDEA

Antonio Cabrales

Universidad Carlos III de Madrid

Jorge Sainz

Universidad Rey Juan Carlos

Ismael Sanz

Universidad Rey Juan Carlos

December 2012

Abstract

We study the effect of standardized external tests on students’ academic outcomes. We exploit the fact that only one of the 17 Spanish regions started doing and publishing the results of standardized tests in 2005 to apply a difference-in-difference methodology, using outcomes of the PISA study from 2000 to 2009. We later confirm our results using synthetic control methods. Using data from a single country allows us to minimize biases arising from differences in legal frameworks, social or cultural environments. Our econometric analysis lends plausibility to the hypothesis that this type of test significantly improves student outcomes. A key novelty is that our exams do not have academic consequences for the students, so that effects have to come directly from the impact on teachers and administrators.

Keywords: external and standardized tests, PISA, difference-in-difference, synthetic control methods

^* We gratefully acknowledge the help of Javier Garcia Crespo and Guillermo Gil to obtain the data and the support from the Ministry of Science and Technology from grants ECO2012-31985 (Anghel).

1. Introduction

External standardized tests allow the administration to better monitor the education process and outcome of the schools. In most of the countries that have these tests, the results of the exam are public and can be used by parents to make decisions.¹ This closer monitoring by parents and administrators provides an additional motivation for teachers and principals to improve the education results of their students. This potential for improvement has encouraged an increasing number of countries to use external examinations as a tool to increase accountability. The Programme for International Student Assessment (PISA) report (OECD, 2010) documents the fact that 22 out of the 34 OECD countries have introduced standards-based external examinations in a majority of their schools. Two more countries, Germany and the US, have this type of tests only in some of their Ländern and states. All in all, two thirds of the 15 years old OECD students attend schools in which there is an external and standardized test.

The existing empirical evidence is supportive of the hypothesis that countries with external exit-exam systems have a better performance in international student achievement tests. The first evidence for this was given by Bishop (1997) for students doing the 1991 IAEP math, science, and geography tests and Bishop (2006) with the PISA 2000 results. Overall, the existing cross-country evidence suggests that the effect of external exit exams on student achievement may well be half or more of a grade-level equivalent, or between 20 and 40 percent of a standard deviation of the respective international tests (OECD, 2010 and 2012 and Hanushek and Woessmann, 2011).

This evidence has been criticized on two grounds. First, these studies use crosssectional data and therefore the adoption of testing by a country is endogenous, and unobserved heterogeneity could bias the results. Second, the introduction of external tests may lead to “teaching to the test”. However, some studies have found the same positive association between central exams and student achievement within countries where some regions have external exam systems and others do not have them.² This evidence rules out the possibility that unobserved national-level factors correlated with the existence of tests drive the observed positive correlation between those tests and students’ outcomes. In addition, students in countries with national external exams have been found to achieve better results in other international tests such as PISA, PIRLS or TIMMS. To the extent that those tests are different in nature from national ones, this may rule out that “teaching to the test” is a main factor driving the better outcomes of students in countries or regions with national external exams.

¹ Even in countries where school zones comprise a single school, concerned parents can decide where to live using school quality as an input to their choice.

² See, e.g. Bishop (1997) for Canadian provinces, Jürges et al. (2005) and Woessman (2010) for Germany, and Bishop et al. (2001), for US states.

A different difficulty of earlier studies is that they are not very clear on what are the channels through which exit exams are effective. This is because for the most part these exams have academic consequences for the students, thereby providing reasons for improvement both to the professionals and to the students. The present study uses a special feature of the Spanish education system to tease out school and student incentives, while at the same time controlling for biases arising from unobserved national-level heterogeneity and arguably also “teaching to the test”.

The special feature to which we refer earlier is that the main Spanish education law (Ley Orgánica de la Educación, LOE 2006) allows the Regions to conduct education system assessments as long as the results are not used for grading students or ranking schools (article 140). That means Spanish exams are not “Curriculum-Based External Exit Examination (CBEEE)” as defined by Bishop (1997), because such examinations should “offer signals of student accomplishments that have real consequences for the student and define achievement relative to an external standard, not relative to other students in the classroom or the school”. This means that the effects of such exams in Spain, if any, have to come directly only from changes in incentives for schools, although in the end those can, and probably will, have an impact on the students’ efforts.

The Region of Madrid introduced a standard external test, called “prueba de Conocimientos y Destrezas Indispensables” (also known for short as CDI test) which means “Indispensable Knowledge and Skills exam” in the academic year 2004/05. The grade achieved by the student in this exam does not have “real academic consequences” for most students, so it cannot be considered a CBEEE.³ So the effects of this initiative will necessarily go directly only through changes in teacher motivation. The Region of Madrid is also the only one that publishes and makes available the average results of each of the schools in the external test to the public opinion.⁴

³ A student with good grades in compulsory secondary schooling and a good mark in the CDI test obtains a certification with Merit or with Distinction, rather just a Certificate, but this has no implications for admissions to schools beyond the compulsory schooling, or for grants, nor is there evidence that employers look at those distinctions. For students with really extraordinary grades (only 25 a year in a region with over 50,000 students in the last year of compulsory secondary schooling) they can obtain an Extraordinary Award yielding a cash prize of 1,000 Euros and a trip to a “cultural destination”.

⁴ Other regions have external standardized exams where all schools are tested, but Madrid is the only one publishing the results.

All the regions in Spain operate under the same legal framework regulating the principles, objectives, and organization of the different school levels (pre-primary, primary, compulsory secondary, post-compulsory secondary), as well as up to 65% (55% in historical regions) of the contents and subjects studied. Hence, along with the amount of public financing of schools, for which we can control, the other main observable difference in education between Spanish regions is the appearance in the period of study of this standardized external exam in Madrid whose results are published.

This feature allows us to conduct a difference-in-difference (diff-in-diff) analysis comparing the PISA results of the treated region (Madrid) before and after the CDI test was introduced with the rest of Spanish regions before and after the treatment. This diff-indiff approach allows us to control for the unobservable time-invariant factors affecting Madrid. Dealing with regions of the same country we also exclude some unobservable effects that appear in cross-country studies with different legislations and cultures.

The fact that we are analyzing a single country also allows us to apply the new inferential methods of synthetic control for comparative case studies proposed by Abadie and Gardeazabal (2003) and Abadie et al. (2010). We use a combination of other Spanish regions to construct a synthetic control region, which resembles relevant education characteristics of Madrid before the introduction of the CDI test. The subsequent education outcome evolution of this “counterfactual” Madrid without CDI is compared to the actual experience of Madrid. The idea behind the synthetic control approach is that a combination of units often provides a better comparison for the unit exposed to the intervention than any single unit alone. Transparency and safeguard against extrapolation are two attractive features of the synthetic control method relative to traditional regression methods.

Our results are also more protected than others from the critique that they are achieved by “teaching to the test”. This is because our measure of outcome, namely, the results in the PISA exam, have somewhat distinct objectives and measure different things than the CDI exam in whose effect we are interested. The Madrid CDI exam questions evaluate knowledge and they are directly related to material seen in Language and Mathematics classes during the academic year. In contrast, the PISA exam questions (called stimulus) are more related to cognitive processes (access and retrieve; integrate and interpret; reflect and evaluate) and on how to use knowledge in particular contexts. That is the PISA evaluation is more related to competencies whereas the Madrid CDI is more related to knowledge.

The paper is organized as follows. Section 2 describes in some detail the institutional setup and the CDI external and standard test. Section 3 discusses the data. Section 4 discusses the econometric methodology and it contains the main results of the paper. Section 5 shows the results of the synthetic control methods. Section 6 concludes.

2. Institutional Setup

The Madrid regional government has been conducting since the academic year 2004/05 a standardized external exam for all 6th grade students in the region, who are hence in the final year of primary school (around 11-12 years old). Three years later, the region introduced another standardized and external exam in the 9th grade (the third year of the secondary school, which is the last common academic year for the students). These exams are compulsory for all primary and secondary schools (public or private). The exam measures what the authorities consider basic knowledge in mathematics (exercises and problems) and language (dictation, reading, general knowledge and questions related to a text).

Our aim is to test whether the introduction of these exams has improved the academic outcomes of the students in Madrid. We use as a measure of student achievement the scores of the exams conducted for the OECD Programme for International Student Assessment (PISA). PISA analyses the key competencies in reading, mathematics and science of 15-year-old students in OECD member countries and partner countries/economies through its triennial surveys. The metric for the overall reading scale is based on a mean for OECD countries set at 500 in PISA 2000, with a standard deviation of 100. PISA conducted its first tests in 2000 covering reading as a major assessment area, and providing a summary profile of the skills of mathematics and science. In 2003, mathematics was the main focus and in 2006 it was science. In 2009, PISA started another cycle, focusing on reading again. When an area is the main focus of the exam two thirds of the exam time is devoted to this area, allowing for its deeper analysis. Since both PISA 2000 and PISA 2009 focused on reading, it is possible to obtain very detailed comparisons of how student performance in that area changed over that period. Comparisons over time in the areas of mathematics and science are somewhat more limited.

In the PISA test, each participating student spends two hours carrying out penciland-paper tasks in reading, mathematics and science. The assessment includes tasks requiring students to construct their own answers as well as multiple-choice questions. In addition, students also answer a survey that takes about 30 minutes to complete and that includes questions about their personal background.

3. Description of the Data

The first CDI exam took place in the academic year 2004/05, so we consider this as the year in which the treatment (the introduction of a standardized exam) was first implemented. For this reason, we compare the results of students of the region of Madrid in (i) Reading, using PISA 2000 and PISA 2009 and (ii) in Mathematics, using PISA 2003 and PISA 2009.⁵

Our first methodology for analysis will be a diff-in-diff regression approach. We construct the treatment and the control groups in the following way: the treatment group before the treatment (the introduction of the CDI exam) is the group of students from the region of Madrid who took the PISA exam in 2000 for Reading or 2003 for Mathematics, the treatment group after the treatment is the group of students who took the PISA exam in 2009, and the control group is formed by students from the other regions of Spain before (PISA 2000 or 2003) and after the treatment (PISA 2009).

The PISA questionnaire allows us to control for various student, family and school characteristics. The student and family characteristics are: gender, age, nationality (immigrant or Spanish), parents’ nationality, languages other than Spanish spoken at home, structure of the family (single parent family, nuclear family, mixed family), learning time in hours per week in Reading and Mathematics (hours per week in Language or Mathematics courses), the index of economic, social and cultural status (ESCS index) calculated by OECD.⁶ The school characteristics are the type of school (public, charter or private), the location of the school (village, small town, town, city or large city), student/teacher ratio, school size, whether the school uses assessments to compare to district/national performance, whether the school uses assessments to make judgments about teacher’s effectiveness, the proportion of girls in the school, the school average of ESCS index, the percentage of immigrant students in school, and school average learning time in Reading and Mathematics.

5 We will not use the PISA scores in Science, since the first year Science was the main focus was 2006 and this is after our treatment was applied.

⁶ The PISA index of economic, social and cultural status (ESCS) was derived from the following three indices: highest occupational status of parents, highest educational level of parents in years of education according to ISCED and home possessions (OECD, 2010).

The tables below contain the descriptive statistics of these four groups, for the most relevant characteristics of students and schools. Table 1 describes the treatment and the control groups in PISA 2000 and in PISA 2009 in Reading and Table 2 describes the two groups in PISA 2003 and in PISA 2009 in Mathematics.

Table 1 – Descriptives statistics of students and schools in PISA 2000 and PISA 2009 (Reading)

Table 2: Descriptive statistics of students and schools in PISA 2003 and PISA 2009 (Mathematics)

The two tables show a very similar evolution of the characteristics of students and schools when we compare PISA 2000 and PISA 2009 in Reading and when we compare PISA 2003 and PISA 2009 in Mathematics.

If we compare the treatment group and the control group before and after the change in Reading we can see patterns that are very similar across both groups: the proportion of girls and students coming from single parent family decreases slightly whereas age, immigrants, learning time, and the Index of Economic, Social and Cultural Status (ESCS) increases. This is consistent with the fact that Spain has experienced a large inflow of immigrants in the last decade and had converged towards the EU and OECD GDP per capita, a process that has, since 2009, reversed.⁷ Nevertheless, the rise in the share of immigrants between 2000 and 2009 was higher in the region of Madrid (in Reading, from 3% to 16%) than in the rest of regions (in Reading, from 2% to 9%). In addition, the share of students speaking foreign languages other than Spanish increased in Madrid over the period (from 1% to 5%) whereas it remained constant in the control group (17% versus 16%).

If we look at the school characteristics, we observe a decrease in the number of private schools and an increase in the number of charter schools over the period 2000- 2009. This could be due to the fact that some private schools have demanded and achieved from the public administration their transformation into charter schools, thus lowering the fees to be paid by the student’s families and avoiding losing enrolment. Nevertheless, the official data from the Statistical Office of the Spanish Ministry of Education, Culture and Sports shows that the rise in the students of charter schools has come from a reduction in the number of students in the public schools. This is in contrast of the PISA sample, which shows that the rise of the students in charter schools come from a reduction in the private schools. That is, the PISA coverage of private schools decreased from 2000 to 2009 whereas the coverage of public schools increased. This could be explained by the fact that the sample of schools in cities or large cities in 2009 decreased whereas those in towns and villages increased.

Student/teacher and school size ratio decreased along the two periods. We also observe that the percentage of schools, which declare that they carry out assessments used to compare the school to district/national performance or assessments used to make judgements about teacher’s effectiveness increased during the two periods and in both the control and the treatment group. In summary, the descriptive statistics show that the trends that we observe when we compare the treatment and control group are similar.

⁷ Spain went from less than 1% of immigrants in the population to almost 10% during this period. The Spanish GDP per capita in PPS terms increased from 97% of that of the EU-27 in 2000 to 103% in 2009 (Source: Eurostat).

The control group and the treatment group have also similar patterns in the PISA 2003 and PISA 2009, the years we are using for the Mathematics analysis. The only exceptions are the proportion of girls and, above all, the ESCS index. In both cases, the indicator of the Region of Madrid increased whereas the one of the control group slightly decreased.

4. Econometric Methodology and Results

In order to estimate the impact of the introduction of a standardized exam in the region of Madrid on students’ outcomes, we propose a diff-in-diff approach. We use as the outcome for student performance, the PISA scores of students. These are calculated using imputation methods, denoted plausible values (OECD, 2009). Thus, for a given year, the score of student i in school j is given by:

\[y _ {i j} = \alpha_ {0} + \alpha_ {1} \text {Madrid} _ {\mathrm{j}} + \alpha_ {2} \text {PISA2009} + \delta \text {Madrid} _ {\mathrm{j}} * \text {PISA2009} + \beta_ {\mathrm{i}} x _ {i} + \beta_ {\mathrm{j}} x _ {j} + v _ {j} + u _ {i} + \varepsilon_ {i j}\]

where are observable characteristics of students and their families described above, are observable characteristics of schools, is a dummy variable for the schools located in the region of Madrid (i.e. it takes the value 1 for the treated group), is a dummy variables for students who took the PISA exam in 2009 (after the introduction of the standardized exam in the region of Madrid), indicates whether school j is in the region of Madrid and participated in PISA exam 2009 (i.e. it takes the value 1 for the treated group after the treatment), are unobservable characteristics of students, such as effort or ability, are unobservable characteristics of school, like quality of the Principal and teachers, and is a random shock.

Our parameter of interest is corresponding to the variable 2 which coincides with the introduction of a standardized exam (the CDI exam) in the region of Madrid.

Tables 3 and 4 below show the results of the diff-in-diff estimation for Reading, using the samples of students from Spain in PISA 2000 and PISA 2009, and for Mathematics, using the samples of students from Spain in PISA 2003 and PISA 2009. Since PISA database provides five plausible values, which are allocated to each student, we use the methodology proposed by the OECD for the computation of regression coefficients and their respective standard errors. According to OECD (2009), statistical analyses should be performed independently on each of these five plausible values and results should be aggregated to obtain the final estimates of the statistics and their respective standard errors.

Table 3: Difference in difference estimations for PISA 2000 and PISA 2009 in Reading

Notes: 1. Standard errors in parentheses. * significant at 10%; ** significant at 5%; *** significant at 1% 2. Base categories for dummies: male, student Spain, Mother Spain, Father Spain, Nuclear family, Public school, School in village

Table 4: Difference in difference estimations for PISA 2003 and PISA 2009 in Mathematics

Notes: 1. Standard errors in parentheses. * significant at 10%; ** significant at 5%; *** significant at 1% 2. Base categories for dummies: male, student Spain, Mother Spain, Father Spain, Nuclear family, Public school, School in village

The first column of the tables shows the estimation results without any control variables. This would be the raw average effect of our treatment. The second column includes individual characteristics of the students and the third and the forth columns add gradually school characteristics.

When we estimate the diff-in-diff without any covariates, the coefficient for the treatment is not statistically significant for both Reading and Mathematics.

However, results of the diff-in-diff estimation for Reading in Table 3, columns (2)- (4) show a positive and statistically significant effect of our treatment on the PISA scores. In the second column, when we control for individual characteristics of students, the coefficient of the treatment variable is positive and significant.⁸ The inclusion of school characteristics in columns (3) and (4) does not change this result. We find a relative improvement in PISA scores in Reading in the region of Madrid between 2000 and 2009 that cannot be explained by observable variables of a magnitude of 14 to 17 PISA points. In 2009, Spain was significantly below the OECD average in Reading by 12 points. If the results are totally explained by the introduction and publication of external exams, this could imply that generalizing those exams would raise the level of Spain in Reading above the OECD average.

In Table 4 we run the same estimations, but for Mathematics and using the scores in PISA 2003 and PISA 2009. Here, we do not find any impact of our treatment on students’ performance. The coefficient switches from positive to negative from one specification to the other and it is not statistically significant in any of them. The lack of significance could be partly explained by the fact that we are using as student outcome PISA scores in a subject, which was the focus of the PISA exam only in 2003 and not in 2009, so comparisons are not fully valid.

⁸ In an additional specification, we dropped the ESCS index, which is an aggregated index of the socioeconomic background of the students, and we controlled separately for the labour market situation and the level of education of the mother and the father. Our results did not change significantly. These results are available upon request.

The different result for Reading and Mathematics could be explained by the fact that Reading was the main focus of the PISA test in 2000 and 2009, whereas Mathematics was the main focus in 2003 but not in 2009. Being the focus of the study means, as stated in the Institutional setup, that two thirds of the study is devoted to this competency whereas the other two thirds are equally divided for the other two areas. This more detailed analysis on one of the competencies allows for disaggregating the students’ outcomes by subscales in the chosen competency. In 2009, PISA detailed the levels of student proficiency in various aspects of reading, such as students’ ability to access and retrieve, integrate and interpret, and reflect and evaluate the information they obtain through reading. It also examined students’ ability to read and understand continuous and noncontinuous texts. In contrast, PISA 2009 only computed an aggregated outcome for Mathematics and Science. Thus, the Reading outcome is a more precise estimation than the Mathematics outcome in 2009 whereas in 2012 it will be the other way around.

Nevertheless, the data seem to indicate that something differential has happened in Madrid between 2000 and 2009 with respect to other Spanish regions. A natural hypothesis in this context is that the introduction and publication of the results of standardized exam played a major role in this change. It is very hard to provide a definitive proof with these data, but we can discard some alternative explanations.

Public spending in education per pupil affects to some extent students’ outcomes (OECD, 2010). The Spanish Ministry of Education provides data on public spending on education per pupil by regions starting from 2004. During the period 2004-2009, Madrid increased public education spending per pupil by 21%, less than the Spanish average of 33%. More importantly, Madrid has been the region increasing less the education spending per pupil among all the 17 Spanish regions. So, education expenditure cannot explain the better behavior of Madrid PISA scores.

Spain received a large amount of immigrants between 2000 and 2009 (immigrants went from 2% to 12% of the population over the period, Instituto Nacional de Estadística, INE), and Madrid was a major place of destination (it has about 18% of the immigrants and about 13% of the population). But our data can identify whether the student is immigrant and the number of immigrants vary enough between schools so that their effect is probably captured at the school level. This was also a period of rapid economic growth, which was not identical between regions, but the ESCS index has enough information about this variable at the individual level to properly control for the effect of economic data. Some other factors affect schools more directly. Madrid has a larger number of charter schools than other regions. Madrid has also increased the share of charter schools but this trend has been similar to the rest of Regions, if anything a little bit smaller. In any case since the identity of the schools is observable its effect can be controlled.

The only other important institutional reform in Madrid school in this period, beyond the introduction and publication of external exams, is the introduction of bilingual schools in the region, where English is a medium of instruction for at least one third of the school time.⁹ Although this is clearly an important reform, it has been implemented only gradually starting from first grade, and the oldest students exposed to the program are now 13 years old. In addition, Anghel, Cabrales and Carro (2012) have not found significant effects of the program in either language or mathematics, and possibly a negative effect on natural and social science (the subjects taught in English).

5. Synthetic control method

In this section, we use the methodology proposed by Abadie and Gardeazabal (2003) and Abadie et al. (2010), which applies synthetic control methods to comparative case studies. Their methodology is motivated by the fact that, in comparative case studies, the researcher is usually forced to find similarities between treated and non-treated units using observable characteristics, something that it is often difficult in practice. To solve this problem they propose to construct a combination of units for comparison purposes, since the combination will typically resemble the treated unit much better than any single unit alone.

In our case we construct a combination of Spanish regions that resembles the region of Madrid in terms of various characteristics before the treatment and we observe the evolution of this combination in the absence of treatment. This combination is called a synthetic control group. It is constructed by searching for a weighted combination of the untreated Spanish regions, in terms of various predictor variables, which are averaged over the entire pre-intervention period. According to Abadie et al. (2010) “because the choice of a synthetic control does not require access to post-intervention outcomes, the synthetic control method allows researchers to decide on study design without knowing how those decisions will affect the conclusions of their studies”.

⁹ Students not only study English as a foreign language but also some subjects (at least Science, History and Geography) are taught in English. Spanish and Mathematics are taught only in Spanish.

In order to construct the synthetic control group (the synthetic Madrid), we have to aggregate the data at school level and then at region level. The year the CDI standardized exam was launched in the region of Madrid was 20004/05, therefore we have two years of pre-treatment data in PISA (PISA 2000 and 2003). PISA 2006 and PISA 2009 will be our post-treatment period. The synthetic Madrid is constructed as a weighted average of the pool of untreated regions. Our donor pool includes 15 regions.¹⁰ The weights are chosen so that the resulting synthetic Madrid resembles the real Madrid as closely as possible in terms of the values of a set of predictors of students’ performance before the introduction of the CDI exam, that is, before the treatment.

We include in the list of predictor variables for calculating the weights the following variables: teacher/student ratio, ESCS school index, proportion of immigrants in the school, proportion of public schools in the region, proportion of private schools in the region, proportion of charter schools in the region, proportion of schools in cities (between 100.000 to about 1 million people), proportion of mixed families (students living with a mother and a guardian, with a father and a guardian or with two guardians), average age in the school, average learning time (in hours/week). All variables are averaged at region level and over the pre-intervention period (2000 and 2003).

Using these predictor variables we construct the synthetic Madrid as the convex combination of regions, which most closely resembles the region of Madrid in the pretreatment period. Table 5 shows the characteristics of the real Madrid region, the synthetic Madrid region and the donor pool (the average of the 15 regions which form the donor pool), in terms of the control variables. The figures prove that the constructed synthetic Madrid is more similar to the real one, in both Reading and Mathematics, than the simple average of the regions that form the donor pool. In Reading, the student/teacher ratio in the real Madrid is 16.18 and in the synthetic Madrid it is 14.25 (the average of the control group is 14.03). The school average of ESCS index is -0.16 in the real Madrid and -0.18 in the synthetic Madrid (the average of the control group is -0.31). The proportion of public schools in the real Madrid is 50% and in the synthetic Madrid is 50.55%, while in the donor pool is 57.14%. Likewise, when we look at the proportion of private/charter schools we observe proportions more similar between the real and the synthetic Madrid than between the real Madrid and the average of 15 regions. The same applies to the proportion of mixed families and the average learning time. Finally, there is a substantial difference in the percentage of immigrant students between the real Madrid and the synthetic Madrid. Furthermore, we find affinities between the synthetic and the real Madrid in students’ PISA outcomes as well.

¹⁰ There are 17 regions (including Madrid region) and two autonomous cities (Ceuta and Melilla) in Spain. We had to drop Baleares and Ceuta and Melilla because of missing data, so this leaves us with 15 regions.

Table 5: Predictor Means for PISA Student Performance

Note: All variables, except PISA scores, are averaged at region level for the pre-treatment period (2000 and 2003). Learning time in Reading for average of 15 regions is for 2000 (there is no data for 2003).

Table 6 displays the weights of the 15 regions from the donor pool in the synthetic Madrid. It shows that the students’ performance in the region of Madrid is best approximated by a combination of Aragón, Asturias, Canary Islands, Cataluña, La Rioja, Murcia y País Vasco for Reading and Aragón, Asturias, Canary Islands, Cataluña, Murcia y País Vasco for Mathematics. The rest of the regions in the donor pool are assigned zero weights.

Table 6: Region weights in the Synthetic Madrid

The next two graphs show the evolution of the real Madrid and the synthetic Madrid in 2000 and 2003 (the pre-intervention years) and in 2006 and 2009 (the postintervention years), separately for Reading and for Mathematics.

Figure 1: The evolution of the real Madrid and the synthetic Madrid in Reading in 2000, Figure 2: The evolution of the real Madrid and the synthetic Madrid in Mathematics in 2000, 2003, 2006 and 2009

For Reading, the graph shows that the synthetic Madrid approximates very well the evolution of the real Madrid in the pre-treatment period. After the treatment, which we take to be the introduction of the CDI standardized exam in the region of Madrid, in 2006 and 2009, the evolution of the synthetic control group is different than the one of the treatment group. In particular we see that, even if both real Madrid and synthetic Madrid experience an increasing trend after 2006, synthetic Madrid is doing worse that the real Madrid: in 2009, the difference in performance is 11,41 PISA points in favour of real Madrid. This difference could be attributed to the introduction of a standardized exam in the region of Madrid, at least with the information that we can observe. This confirms, even quantitatively, the results we obtained previously with the diff-in-diff methodology, where we found that controlling for school characteristics, the region of Madrid improved its performance relative to other regions of Spain in the period between 2000 and 2009, between 14 and 17 PISA points. The flagship education publication of the OECD, Education at a Glance, arrives to a similar conclusion in the latest 2012 edition: “students in school systems that use standards-based external examinations score 16 points higher, on average across OECD countries, than students in school systems that do not use these examinations (Education at a Glance, 2012, page 527). Our estimation is a little bit lower than the range found in the literature by Hanushek and Woessmann (2011) of 20% to 40% of the standard deviation (20 to 40 points in PISA).

For Mathematics, however, the synthetic control group methodology does not work so well. The synthetic Madrid does not approximate very well the evolution of the real Madrid in 2000 and 2003, the pre-treatment period. In the post-treatment period, the synthetic Madrid is performing slightly better than the real Madrid. Nevertheless, the diffin-diff estimation showed no statistical impact in Mathematics.

We are aware of the limitations of our data in performing the estimation by using synthetic control methods. One of them is that since the PISA study started in 2000 and it is carried out each three years, we only have two years of pre-intervention data (2000 and 2003), which complicates the calculation of the region weights for the synthetic control group. The result in Mathematics, where the synthetic Madrid is not so similar to the real Madrid in the years before the treatment can be partly explained by this fact.

6. Conclusions

This paper attempts to identify whether the implementation and publication of the results of external and standardized tests could have any impact on the performance of students. We use the fact that in the region of Madrid a standardized exam was first administered (and its results published) in 2004/05 to all grade primary students, while in the other regions of Spain, no such exam existed. Using a diff-in-diff strategy we find a positive effect in Reading of the order of 14 to 17 PISA points. The synthetic control method yields an effect that is very close in quantitative terms. Our results are in line with previous research in the area, but our study provides one important innovation, since the external exams in Madrid have no consequences for the students, so the effect has to come from the impact on teachers and school principals.

We have identified a possible effect in language, but not in mathematics. This is slightly surprising since many educational programs have observed effects that are larger in mathematics than in language (see e.g. Abdulkadiroglu et al., 2011). A possible explanation may come from the different emphasis of the curricula of primary school education degrees in Spain with respect to other countries, but this question deserves a more thorough investigation, which we defer to further research.

References

Abadie, Alberto and Javier Gardeazabal (2003), “The Economic Costs of Conflict: A

Case Study of the Basque Country”, The American Economic Review, 93 (1), 113-132

Abadie, Alberto, Alexis Diamond and Jens Hainmueller (2010), “Synthetic Control

Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco

Control Program”, Journal of the American Statistical Association, 105 (490), 493-505

Abdulkadiroglu, Attila, Joshua Angrist, Susan M. Dynarski, Thomas J. Kane, and

Parag A. Pathak (2011), “Accountability and Flexibility in Public Schools: Evidence from

Boston's Charters and Pilots,” The Quarterly Journal of Economics, vol. 126, pp. 699–748.

Anghel, Brindusa and Antonio Cabrales (2010), “The Determinants of Success in

Primary Education in Spain,” Working Paper FEDEA, No. 2010-20.

Anghel, Brindusa, Antonio Cabrales and Jesús M. Carro (2012), “Evaluating a

bilingual education program in Spain: the impact beyond foreign language learning”, CEPR

Working Paper, No. 8995.

Bishop, John H. (1997), “The Effect of National Standards and Curriculum-Based

Examinations on Achievement”, American Economic Review, vol. 87 (2), pp. 260-264.

Bishop John H. (2006), “Drinking from the fountain of knowledge: student incentive to

study and learn,” In: Hanushek EA, Welch F (eds) Handbook of the economics of education.

North-Holland, Amsterdam

Bishop, John H., Ferran Mane, Michael Bishop and Joan Moriarty (2001), “The Role

of End-of-Course Exams and Minimum Competency Exams in Standards-Based

Reforms,” Brookings Papers on Education Policy vol. 4, pp. 267-345.

Hanushek, Eric and Ludger Woessmann (2011): “The Economics of International

Differences in Educational Achievement”, in Hanushek, E, Machin, S. and Woessmann, L.

(Eds.) Handbook of the Economics of Education, vol. 3, pp. 89-200, Amsterdam: North Holland

Jürges, Hendrik, Kerstin Schneider and Felix Büchel (2005), "The Effect Of Central Exit Examinations On Student Achievement: Quasi-Experimental Evidence From TIMSS Germany", Journal of the European Economic Association, vol. 3(5), pp. 1134-1155, 09. OECD (2009), “PISA Data Analysis Manual: SPSS Second Edition”. OECD (2010), “PISA 2009 Results: What Makes a School Successful?” Volume IV. Wößmann, Ludger (2010), "Institutional Determinants of School Efficiency and Equity: German States as a Microcosm for OECD Countries", Jahrbücher für Nationalökonomie und Statistik vol. 230 (2), pp. 234–270

1. 2013-01: “Publicizing the results of standardized external tests: Does it have an effect on school outcomes?, Brindusa Anghel, Antonio Cabrales, Jorge Sainz y Ismael Sanz.
2. 2012-12: “Visa Policies, Networks and the Cliff at the Border”, Simone Bertoli, Jesús Fernández-Huertas Moraga.
3. 2012-11: “Intergenerational and Socioeconomic Gradients of Child Obesity”, Joan Costa-Fonta y Joan Gil.
4. 2012-10: “Subsidies for resident passengers in air transport markets”, Jorge Valido, M. Pilar Socorro, Aday Hernández y Ofelia Betancor.
5. 2012-09: “Dual Labour Markets and the Tenure Distribution: Reducing Severance Pay or Introducing a Single Contract?”, J. Ignacio García Pérez y Victoria Osuna.
6. 2012-08: “The Influence of BMI, Obesity and Overweight on Medical Costs: A Panel Data Approach”, Toni Mora, Joan Gil y Antoni Sicras-Mainar.
7. 2012-07: “Strategic behavior in regressions: an experimental”, Javier Perote, Juan Perote-Peña y Marc Vorsatz.
8. 2012-06: “Access pricing, infrastructure investment and intermodal competition”, Ginés de Rus y M. Pilar Socorro.
9. 2012-05: “Trade-offs between environmental regulation and market competition: airlines, emission trading systems and entry deterrence”, Cristina Barbot, Ofelia Betancor, M. Pilar Socorro y M. Fernanda Viecens.
10. 2012-04: “Labor Income and the Design of Default Portfolios in Mandatory Pension Systems: An Application to Chile”, A. Sánchez Martín, S. Jiménez Martín, D. Robalino y F. Todeschini.
11. 2012-03: “Spain 2011 Pension Reform”, J. Ignacio Conde-Ruiz y Clara I. Gonzalez.
12. 2012-02: “Study Time and Scholarly Achievement in PISA”, Zöe Kuehn y Pedro Landeras.
13. 2012-01: “Reforming an Insider-Outsider Labor Market: The Spanish Experience”, Samuel Bentolila, Juan J. Dolado y Juan F. Jimeno.
14. 2011-13: “Infrastructure investment and incentives with supranational funding”, Ginés de Rus y M. Pilar Socorro.
15. 2011-12: “The BCA of HSR. Should the Government Invest in High Speed Rail Infrastructure?”, Ginés de Rus.
16. 2011-11: “La rentabilidad privada y fiscal de la educación en España y sus regiones”, Angel de la Fuente y Juan Francisco Jimeno.
17. 2011-10: “Tradable Immigration Quotas”, Jesús Fernández-Huertas Moraga y Hillel Rapoport.
18. 2011-09: “The Effects of Employment Uncertainty and Wealth Shocks on the Labor Supply and Claiming Behavior of Older American Workers”, Hugo Benítez-Silva, J. Ignacio García-Pérez y Sergi Jiménez-Martín.
19. 2011-08: “The Effect of Public Sector Employment on Women’s Labour Martket Outcomes”, Brindusa Anghel, Sara de la Rica y Juan J. Dolado.
20. 2011-07: “The peer group effect and the optimality properties of head and income taxes”, Francisco Martínez-Mora.
21. 2011-06: “Public Preferences for Climate Change Policies: Evidence from Spain”, Michael Hanemann, Xavier Labandeira y María L. Loureiro.
22. 2011-05: “A Matter of Weight? Hours of Work of Married Men and Women and Their Relative Physical Attractiveness”, Sonia Oreffice y Climent Quintana-Domeque.
23. 2011-04: “Multilateral Resistance to Migration”, Simone Bertoli y Jesús Fernández-Huertas Moraga.
24. 2011-03: “On the Utility Representation of Asymmetric Single-Peaked Preferences”, Francisco Martínez Mora y M. Socorro Puy.
25. 2011-02: “Strategic Behaviour of Exporting and Importing Countries of a Non-Renewable Natural Resource: Taxation and Capturing Rents”, Emilio Cerdá y Xiral López-Otero.
26. 2011-01: “Politicians' Luck of the Draw: Evidence from the Spanish Christmas Lottery”, Manuel F. Bagues y Berta Esteve-Volart.
27. 2010-31: “The Effect of Family Background on Student Effort”, Pedro Landeras.
28. 2010-29: “Random–Walk–Based Segregation Measures”, Coralio Ballester y Marc Vorsatz.
29. 2010-28: “Incentives, resources and the organization of the school system”, Facundo Albornoz, Samue Berlinski y Antonio Cabrales.
30. 2010-27: “Retirement incentives, individual heterogeneity and labour transitions of employed and unemployed workers”, J. Ignacio García Pérez, Sergi Jimenez-Martín y Alfonso R. Sánchez-Martín.
31. 2010-26: “Social Security and the job search behavior of workers approaching retirement”, J. Ignacio García Pérez y Alfonso R. Sánchez Martín.
32. 2010-25: “A double sample selection model for unmet needs, formal care and informal caregiving hours of dependent people in Spain”, Sergi Jiménez-Martín y Cristina Vilaplana Prieto.
33. 2010-24: “Health, disability and pathways into retirement in Spain”, Pilar García-Gómez, Sergi Jiménez-Martín y Judit Vall Castelló.
34. 2010-23: Do we agree? Measuring the cohesiveness of preferences”, Jorge Alcalde-Unzu y Marc Vorsatz.