Fedea Policy Papers - 2016/20

No student left behind? Evidence from the Programme for School Guidance in Spain

J. Ignacio García-Pérez (Universidad Pablo de Olavide & FEDEA)

Marisa Hidalgo-Hidalgo

(Universidad Pablo de Olavide)

fedea

RESUMEN (NON TECHNICAL SUMMARY)

Existe evidencia creciente que muestra que la desigualdad ha aumentado durante las últimas décadas en muchos países desarrollados. Además de la crisis económica mundial esta evidencia podría reflejar también el hecho de que tanto los trabajadores poco cualificados como los estudiantes de bajo rendimiento se están quedando atrás como consecuencia de los rápidos cambios tecnológicos en una economía mundial cada vez más globalizada. De hecho los estudiantes con pobres resultados educativos son más propensos al abandono escolar lo que tiene terribles efectos negativos a largo plazo al aumentar el riesgo de exclusión social y la pobreza.

Por ello, la mejora de la educación y las habilidades de los trabajadores se sitúa como una prioridad para los responsables políticos, obligándoles a hacer frente a la pobreza y la exclusión de forma paralela a la promoción del crecimiento económico. De hecho, uno de los principales objetivos de la Unión Europea en el ámbito educativo para el año 2020 es reducir las tasas de abandono escolar. Pero, ¿cómo podemos hacer de la educación un éxito para los estudiantes en situación de desventaja dentro del sistema educativo?

Los programas de educación compensatoria están diseñados para ayudar a los estudiantes de bajo rendimiento a alcanzar unos estándares académicos mínimos. Esto se hace generalmente por medio de un aumento de las horas de clase y una atención más personalizada a través de grupos de estudio reducidos. Este tipo de intervenciones son de creciente interés en la actualidad. Si bien este tipo de programas están bastante generalizados en los Estados Unidos, hay menos tradición en la aplicación de dichos programas dentro de la Unión Europea.

Por otra parte, la evidencia sobre la eficacia de este tipo de programas es escasa. Este es precisamente el objetivo de nuestro trabajo: evaluar los efectos de un programa plurianual de educación compensatoria para alumnos de educación secundaria implantado en España entre 2005 y 2012, que ofrecía clases extra para alumnos de bajo rendimiento y entorno socioeconómico pobre. Esto fue el Programa de Acompañamiento Escolar (PAE). En particular, tratamos de responder a las siguientes preguntas relativas a los posibles efectos de este programa sobre el rendimiento educativo de los alumnos afectados por el mismo: ¿Sirvió para reducir el número de alumnos que se quedan atrás del progreso general de la clase? ¿Mejoró las puntuaciones medias de los estudiantes que recibieron dicho programa? En el trabajo comprobamos si la intervención tuvo éxito en la consecución de estos dos objetivos, en dos contextos diferentes. En primer lugar, analizamos el impacto del programa mientras se está realizando la intervención. Y en segundo lugar, analizamos si el programa es más eficaz cuanto mayor es el tiempo que el centro escolar ha participado en el mismo. Para realizar este análisis utilizamos evaluaciones externas a los centros, en concreto, los exámenes PISA realizados en el año 2012 a los alumnos de 15 años que cursaban en ese momento estudios de Enseñanza Secundaria Obligatoria.

Estos programas son a menudo muy difíciles de evaluar debido a problemas de selección muestral. Las características individuales y socioeconómicas de los estudiantes y/o centros que reciben el tratamiento afectan tanto a su probabilidad de ser seleccionados para el tratamiento como al éxito o no del mismo y, además, el mecanismo de selección no es completamente observable. Por fortuna, la riqueza de nuestros datos, combinados con el acceso a datos de los colegios tratados en un momento anterior al tratamiento (el curso 2009-2010) nos permite controlar una gran variedad de características observables y no observables que podrían afectar a la selección de las escuelas por el programa PAE así como a los resultados del mismo. Nuestra primera estrategia de estimación compara las notas de lectura en PISA 2012 de los alumnos que asistieron a las escuelas que participaron en el programa PAE con el resultado hipotético que estos mismos estudiantes habrían obtenido si no hubieran asistido a escuelas tratadas. La puntuación de lectura hipotética se infiere del uso de un grupo de control compuesto por los estudiantes en escuelas que no se incorporaron al programa PAE, pero que son similares a las que sí lo hicieron y a la vez participaron en PISA 2012. Para garantizar que los grupos de tratamiento y control son comparables en las características observables, los estudiantes en el grupo de control son re-ponderados asignando relativamente más peso a aquellos alumnos que tienen características individuales, familiares y escolares que son más similares a las de la media del grupo tratado. Como segunda estrategia de estimación se propone utilizar Propensity Score Matching para examinar el impacto del programa PAE.

Por otra parte también estimamos el papel de las variables no observables en la decisión de las escuelas de participar en el programa PAE. La disponibilidad de información sobre el rendimiento de los estudiantes en las escuelas antes de que éstas se unieran al programa nos permite examinar la existencia de sesgo de selección. De hecho, esta es una de las aportaciones de este trabajo. Concretamente, estimamos el sesgo de selección mediante la combinación, por una parte, de la información disponible en PISA 2009 con, por otra, la información respecto a la participación en el programa de PAE uno, dos o tres años más tarde. Para ello, identificamos en la muestra de PISA 2009 las escuelas que participaron en el programa PAE sólo después del año 2009. En esta muestra por tanto, cualquier diferencia en el rendimiento de lectura entre los estudiantes en las escuelas que participaron en el programa PAE sólo después de 2009 y los de las escuelas que no participaron en el programa de PAE puede atribuirse únicamente a la existencia de sesgo de selección. No encontramos ningún efecto significativo en esta comparación y por tanto concluimos que no existe un sesgo de selección significativo en nuestra muestra de estimación. Una posible explicación a la inexistencia de sesgos es que, como el programa se inició en el curso académico 2005/06, ya en el curso 2009/10 la existencia del programa estaba suficientemente extendida entre la comunidad educativa para que los centros que decidieron participar en el mismo no fueran muy distintos a los que no participaron en dicho curso (de hecho, la tasa de participación en el programa supera el 45% en algunas regiones en dicho curso).

Nuestro resultado principal indica que este programa tuvo un considerable efecto positivo sobre el rendimiento académico de los alumnos: la probabilidad de caer por debajo del percentil 25 en la distribución de notas dentro del colegio se reduce aproximadamente un 5% para los alumnos tratados y la media de las puntuaciones de lectura aumentó en aproximadamente 12 puntos (un 14% de una desviación estándar) para dichos alumnos. También encontramos que una mayor exposición al programa mejora las cualificaciones de los estudiantes: mientras que los estudiantes en las escuelas que participaron en el programa durante un máximo de dos años no experimentan ningún efecto estadísticamente significativo y positivo, los de las escuelas que participaron durante al menos tres años sí que lo hicieron. Por otra parte, el programa redujo significativamente la probabilidad de pertenecer a la parte inferior de la distribución de notas (alrededor de un 7,5%) y mejoró las puntuaciones medias en esta parte de la distribución (aproximadamente 16 puntos o lo que es lo mismo en casi el 19% de una desviación estándar). De hecho, encontramos que el efecto más fuerte del programa se concentra entre los alumnos que están entre los percentiles 15 y 30 de la distribución, esto es, entre el colectivo objetivo del mismo. Por último, nos encontramos con que el impacto del programa es mucho más fuerte y positivo para los estudiantes en las escuelas rurales (municipios de menos de 15.000 habitantes) que para los estudiantes en las escuelas urbanas.

Creemos que nuestros resultados son de gran valor y contribuyen a una literatura relativamente escasa sobre los programas de educación compensatoria y su impacto en adolescentes con pobres resultados educativos en toda Europa. En este trabajo, encontramos evidencia que respalda el uso de políticas educativas consistentes en proporcionar horas escolares adicionales destinadas a mejorar los resultados de estudiantes con bajo rendimiento.

J. Ignacio García-Pérez Universidad Pablo de Olavide & FEDEA

Marisa Hidalgo-Hidalgo Universidad Pablo de Olavide

September 12, 2016

Abstract

This paper evaluates the e¤ects of a remedial education programme implemented in Spain between 2005 and 2012 that o¤ered after-school classes for underperforming students from poor socioeconomic backgrounds. We use two di¤erent estimation strategies, re-weighting estimators and propensity score matching, and address the existence of selection bias. We …nd that this programme had a substantial positive e¤ect on children’s academic achievement: the probability of falling behind the general progress of the group declined by approximately 5% and mean reading scores increased by approximately 10% of one standard deviation. We also …nd that a larger exposure to the programme improves students’ scores: whereas students in schools that participated in the programme for at most two years do not experience any signi…cant positive e¤ect, those in schools that participated for at least three years did. The programme signi…- cantly reduced the probability of belonging to the bottom part of the distribution (by approximately 7.5%) and improved mean scores (by approximately 18% of one standard deviation). Finally, we …nd that the impact of the programme is much stronger for students in rural schools than for students in urban schools.

Keywords: Remedial education, PAE, programme evaluation, PISA, selection bias JEL Classi…cation: H52,I23,I28,J24

¤We would like to thank Laura Hospido and Ernesto Villanueva for their very helpful comments and suggestions. We also thank seminar participants at Alicante, Salamanca and the 40 Simposio de Análisis Económico. We gratefully acknowledge Ismael Sanz and Francisco Javier García Crespo (INEE) for their help with the data used in this article. We are also grateful for the assistance with the dataset o¤ered by Angelica Martínez Zarzuelo (INEE). Financial support from the Spanish Ministry of Innovation and Science [ECO2013-43526-R and ECO2014-57413] and Junta de Andalucia [SEJ-5980; SEJ-426, P09-SEJ6882] is gratefully acknowledged. The usual disclaimer applies.

1 Introduction

Growing evidence shows that inequality has increased in many developed countries in recent decades.¹ Recent OECD data (OECD, 2013) indicate that the global economic crisis reduced incomes and that this reduction is not shared evenly across the income distribution, as there are larger reductions in the bottom, thus suggesting further increases in inequality and poverty. In addition to the global crisis, this evidence might also re‡ect the fact that both low-skilled workers and low-achieving students are being left behind by rapid technological change in a globalized world economy (see Freeman, 2008 or Kanbur, 2014). Indeed, poor-achieving students are more likely to be early school leavers, which has long-run negative e¤ects, increasing the risk of social exclusion and poverty.² This recent evidence has arguably made improving the education and skills of the workforce a priority, impelling pol icy makers to address poverty and exclusion and promote growth. Indeed, one of the EU’s education targets for 2020 is to reduce the rates of young people leaving early education and training. In addition, the European Union’s 2013 Social Investment Package focusses on policies designed to strengthen people’s skills and capacities, including education and childcare, as well as active labour market policies (see European Commission 2013a and 2013b). These developments leave us with the following question: how do we make education a success for disadvantaged students in developed countries?

Remedial education programmes are designed to help poor-performing students to satisfy minimum academic standards. This is usually achieved by means of a targeted increase in instruction time combined with after-school individualized instruction in small study groups. Therefore, these types of interventions are currently subject to increasing interest. While remedial education is quite widespread in the U.S., there is less of a tradition in Europe.³ Moreover, the evidence on the e¤ectiveness of such programmes is scarce. Providing such evidence is precisely the goal of this paper. Namely, our objective is to evaluate the e¤ects of a multiyear programme implemented in Spain between 2005 and 2012 that o¤ered remedial education for underperforming students from poor socioeconomic backgrounds. This remedial programme is the Programme for School Guidance (PAE, which is the Spanish acronym for Programa de Acompañamiento Escolar). In particular, we attempt to address the fol lowing two questions: does the programme reduce the number of students left behind the general progress of the group? Does the programme improve students’ mean scores? We assess whether the intervention succeeded in achieving these two goals while it was being implemented (we refer to this as the PAE-Immediacy treatment). In addition, we analyse whether the programme was more e¤ective in achieving both objectives the longer a school participated in it. To do so, we use external evaluations of the schools: the PISA 2012

¹See, among others, Atkinson, 2010 for the EU and Atkinson et al, 2011 for the US.

²See Brunello and De Paola (2014) and references therein for a review of the private and social cost of early school leaving in Europe.

³See the European Commission (2013) for a review of remedial programmes in Europe. Among several examples of remedial interventions at school in the U.S., see, for example, the Bell After-School Instructional Curriculum (BASICs) or that promoted by the 21st Century Community Learning Centers (U.S. Department of Education, 2003).

exams.⁴

Our main results suggest that the PAE had a substantial positive e¤ect on students academic achievement. It reduced the probability of falling behind into the bottom part of the reading score distribution by approximately 5% (nearly 10% of one standard deviation). The estimated e¤ect on mean reading scores is above 12 PISA points (more than 14% of one standard deviation). We also …nd that a larger exposure to the programme improved students’ scores: whereas students in schools that participated in the programme for at most two years do not experience any signi…cant positive e¤ect, those in schools that participated in the programme for at least three years did. The PAE signi…cantly reduced the probability of belonging to the bottom part of the distribution (by approximately 7.5%) and improved mean scores (by approximately 18% of one standard deviation). Furthermore, our evidence suggests that there is heterogeneity in the impact of the programme across school types, namely, urban versus rural. In particular, we …nd that the impact of the programme is much larger among students attending rural schools than students attending urban schools (according to several indicators, the impact in rural schools is more than twice that for urban schools).

Remedial programmes are often very di¢cult to evaluate due to sample selection. Students’ individual and socioeconomic characteristics a¤ect both their probability of being selected for the programme and its success, as the selection mechanism is not completely observable. Fortunately, the richness of our data, combined with access to schools’ performance in 2009 (before a group of schools joined the programme) and in 2012 (after joining it) allows us to control for a variety of observable student characteristics and address unobservables that might a¤ect the selection of schools for the PAE and their outcomes. Our …rst estimation strategy compares the PISA 2012 reading scores of those students that attended schools that participated in the PAE with the hypothetical outcome that these same students would have obtained had they not attended PAE schools. The counterfactual reading score is inferred using a control group composed of students in schools that did not join the PAE but participated in PISA 2012. To ensure that treatment and control groups are comparable on observables, students in the control group are re-weighted by assigning relatively more weight to those students whose individual, family and school characteristics are similar to those of the means of the treated group. As a second estimation strategy, we propose using propensity score matching to examine the impact of the PAE. In addition, we estimate the role of unobservable variables in the schools’ decision to volunteer for the PAE. The availabil ity of information on student performance in schools before joining the programme allows us to examine the existence of selection bias. This is one of the contributions of this paper.⁵ We estimate the selection bias by combining, on the one hand, the information available in PISA 2009 exams with, on the other, the information regarding participation in the PAE one, two or three years later. We identify in the PISA 2009 sample those schools that volunteered for the PAE only after 2009. In this sample, any di¤erence in reading performance among students in schools that volunteered for the PAE only after 2009 and those in schools that never participated in the PAE can be attributed solely to the existence of selection bias. We do not …nd any signi…cant selection bias. A possible explanation is that, as the programme began during the 2005/06 academic year, by the 2009/10 academic year, and afterwards, the existence of the programme was quite widespread in the education community (the rate of programme participation exceeds 45% in some regions).

⁴PISA is the Programme for International Student Assessment. It measures students’ skills in three areas: mathematics, reading and science.

⁵See also Hospido et al. (2015) who employ a similar approach to examine the impact of a …nancial education programme on students’ scores.

Our paper contributes to the relatively scarce literature on the evaluation of remedial education programmes for teenage students in developed countries.⁶ Only a few works address the identi…cation problem and obtain evidence regarding the e¤ectiveness of these programmes in the short run. Jacob and Lefgren (2004) analyse the e¤ect of summer schools on the performance of 9-12 year-old students in Chicago and …nd that the net e¤ect of these programmes was to substantially increase academic achievement among third-graders but not sixth-graders. Lavy and Schlosser (2005) evaluate the short-term e¤ects of the Bagrut 2001 programme, a remedial intervention very close in spirit to that evaluated in this study, which provided additional instruction to underperforming high school students in Israel. Their study shows that it was more cost e¤ective than alternatives based on …nancial incentives for pupils and teachers. Holmlund and Silva (2014) study a remedial education programme targeting English secondary school pupils at risk of school exclusion that, instead of targeting standard cognitive skills (as does the PAE and other programmes mentioned above), targeted students’ non-cognitive skills, …nding little evidence that the programme signi…cantly helped treated youths to improve their age-16 test outcomes. A recent contribution is Battaglia and Lebedinski (2015), who analyse the impact of the Roma Teaching Assistant Programme in Serbia. However, their work di¤ers somewhat from our study, as it is focused on a stigmatized ethnic group, Roma pupils. Thus, one of the contributions of this paper is that it is among the …rst to analyse the impact of a remedial education programme on students academic achievement within the European context, which is crucial considering the current debate over the increasing inequality and poverty in Europe.⁷ Therefore, our insights might be highly relevant from a policy perspective.

The present paper is organized as follows. Section 2 summarizes the PAE and presents the data and descriptive statistics used in the paper. Section 3 describes the methodology. Section 4 reports the results. In Section 5, we examine the existence of selection bias. Section 6 provides a robustness check of the main results. Finally, Section 7 concludes.

⁶Evidence on the impact of remedial and analogous programme in developing countries is more common. See, for example, Banerjee et al. (2007), who evaluate the Balsakhi Programme in India or, more recently, Kremer et al (2013) for a review of the existing evidence on programme impact in developing countries.

⁷A number of recent papers have focused on remedial programmes in tertiary education in Europe and the U.S. For example, De Paola and Scoppa (2014) and De Paola and Scoppa (2015) analyse the impact of remedial courses on the achievement of college students in Italy. Bettinger and Long (2009) and Calcagno and Long (2008) study the causal e¤ect of remediation on the outcomes of college students in Ohio and Florida, respectively.

2 The PAE

The Spanish education system is organized into three levels: primary (grades 1-6), secondary (grades 7-10) and pre-college (grades 11-12). The …rst two levels are compulsory (a student can choose to leave school at age 16). Most schools provide either primary or secondary and pre-college education.⁸ All students born in the same calendar year must enter school in the same academic year, with 10th grade being the reference grade for 15-year-old students (who are the students in our sample).

The PAE is a programme targeting public primary and secondary schools. It was implemented during the period 2005-2012. The PAE is an example of a set policies implemented in Spain to improve poor educational outcomes: the early drop-out rate was over 30% in 2004 and 2008 (see Spanish Ministry of Education, 2016). The PAE provides support to public schools with a signi…cant number of students from disadvantaged backgrounds.⁹ The aim of this intervention was to enhance the learning abilities and academic returns of underperforming students with poor socioeconomic backgrounds. This was pursued by stimulating reading habits, providing students with study organization techniques, and improving their social abilities. It consisted of providing support (at least 4 hours per week) during afterschool hours to those students with special needs and learning di¢culties. This support was provided after-school by instructors or teachers from the students’ own schools who worked with these students in small groups (5-10 students). Students were selected by both their tutor and the rest of the teachers and could be in any grade within secondary school. They were chosen based on their poor academic results, general motivation and prospects, although there was no single quanti…able and explicit selection rule. During the remedial classes, the students engaged in guided reading and worked on the subjects that presented particular di¢culties for them. Instructors o¤ered clari…cation, provided additional material, assisted students with work organization techniques, etc.¹⁰ Although the PAE was implemented in both primary and secondary schools, we focus our analysis on secondary schools. The reason is that we use PISA 2012 exam results as the means of evaluating the PAE, and this exam is taken by 15-year-old students, with 10th grade being the reference grade for them). Finally, as the programme was implemented only in public schools, we exclude from the PISA database both private and private but publicly …nanced schools.

Figure 1 displays the percentage of public secondary schools in which the PAE was implemented in each region during the full period that the programme was implemented, that is, from the 2005/06 until the 2011/12 academic year. We distinguish …ve sub-periods, displayed in the …ve panels in Figure 1. In panels 2 to 5, we have the four academic years that the student attended the same secondary school where she took the PISA exams, that is, 2008/09, 2009/10, 2010/11 and 2011/12, meaning grades 7 to 10. In panel 1, we include the preceding years, 2005 through 2008, that is, the period in which the PAE might also have been implemented at that school but, as the students in our sample are 15-years-old ones, they did not bene…t from it since they were attending primary courses at a di¤erent school during that period.¹¹

⁸Only a very small sample of schools (most of them private) provide the three levels. See Spanish Ministry of Education (2016).

⁹Schools volunteered for the programme and committed themselves to improving their students’ outcomes by providing after-school instruction to those students with special needs.

¹⁰For additional details on the PAE, see (only in Spanish) http://www.mecd.gob.es/educacionmecd/areas-educacion/comunidades-autonomas/programas-cooperacion/plan-proa/acompanamientoescolar-secundaria.html

The …gure indicates that the PAE was progressively introduced throughout this period. The percentage of schools participating in the PAE was very low during the …rst three academic years (below 1% in most regions). However, during the period analysed in this paper from 2008 until 2012, there was a gradual implementation of the programme in most regions (the proportion of schools with the PAE was above 40% in several regions).¹²

¹¹Some secondary schools also teach primary education levels (see Footnote 8). Moreover, these schools (and, thus, the students there) might have participated in the PAE at primary level during this period (from 2005 until 2008). Nevertheless, as we cannot identify these schools in our database, we will simply consider as treated students those attending secondary schools that joined that programme at that level.

¹²As explained before, schools volunteered for the programme. They received funding and had to manage

Next, we analyse how the PAE was introduced in the schools in our sample during the 2005-2012 period. We consider the same …ve sub-periods mentioned above: 2005/08, 2008/09, 2010/11, 2010/11 and 2011/12. Thus, depending on whether the school implemented the PAE in each of these sub-periods, we may have 28 di¤erent types of schools. Table 1 below reports the number of schools of each type. It also shows some descriptive statistics for the schools (their mean reading PISA scores and an index of economic, social and cultural status, ESCS). For example, the …rst thirteen rows show the number of schools where the PAE was implemented at least during the last academic year we consider, 2011/12, regardless of whether it was implemented before.

Several comments can be made. Most schools in our sample, more than 60%, did not implement the PAE. Among those that did, the majority implemented the programme throughout the period considered. For example, the …rst row indicates that more than 10% of the schools in our sample participated in the PAE during every academic year (from 2005 until 2012). Moreover, once a school joins the programme, it is very likely to continue participat ing in it. For example, only 7 schools in the sample participated in the PAE from the very beginning (2005/06 academic year) but dropped out during the last academic year. Finally, schools where the PAE was implemented for a longer period do not seem to di¤er from other schools in terms of mean reading achievement or the socioeconomic index. For example, both the mean reading score and ESCS of schools where the PAE was implemented throughout the full period are not statistically di¤erent from the corresponding means for the full sample. This suggests that schools joined the programme in no particular order. In Section 5 below, we examine this hypothesis.

As noted above, we use external evaluations of schools, speci…cally, PISA 2012 scores for the regions with enlarged samples.¹³ There are at most 35 students per school participating in PISA. These students are selected based on a two-stage sample design developed by the PISA programme organizers. This selection ensured representation of the full target population of 15-year-old students in the participating countries.¹⁴ Table 2 below shows, …rst, the number of secondary schools that participated PISA in 2012 per region (column 1). It also shows the number of secondary schools where the PAE was implemented in a particular academic year, regardless of whether it was also implemented in other academic years (see, for example, column 2 for the …gures corresponding to the 2005/06 academic year). Finally, it shows the number of schools where the PAE was implemented and that also took PISA 2012 (see column 3, for example, for the …gures corresponding again to the 2005/06 academic year).

programme implementation. The criteria to distribute funds for the programme among regions included the number of public schools, the number of students attending public schools and the number of early school leavers or dropouts. Apparently, the guidelines to distribute funds among schools within regions resemble the previous iterations: as we will note below, both a school’s size and its proportion of dropouts increase the probability of joining the programme.

¹³The regions with a representative (enlarged) sample are Andalusia, Aragon, Asturias, Balearic Islands, Canary Islands, Cantabria, Castile Leon, Catalonia, Extremadura, Galicia, La Rioja, Madrid, Murcia, Navarre, and Basque Country. The 2012 edition of PISA focused on science. Following the OECD’s recommended methodology, we use the 5 plausible values in the PISA Technical Report to calculate each student’s educational outcome.

¹⁴Only in a few cases, and with proper justi…cation, PISA national project managers can exclude certain schools (e.g., in a remote geographical region) or students (e.g., special needs students). Nevertheless, the guidelines explicitly state that students must not to be excluded solely because of poor academic performance or normal discipline problems. See the PISA 2012 Technical Report for further details.

Table 1: PAE program implementation

Note: X(respectively, -) indicates the school participated (respectively, did not participate) in the PAE program in the corresponding academic course. Source: INEE (National Institute for Educational Evaluation) and PISA 2012.

As can be observed, more than 10% of the schools where the PAE was implemented during 2011/12 were also evaluated in PISA 2012 (see columns 14 and 15).

The PISA 2012 database provides individual-level information on demographics (e.g., gender, immigration status, month of birth), socioeconomic background (parental education and occupation), school-level variables and achievement test scores in three disciplines: science, maths and reading. We focus on test scores in reading, as the PAE focussed primarily on improving learning abilities by stimulating reading habits, as noted above. Nevertheless, we also assess the impact of the PAE on science and maths scores. In addition, as the main goal of the PAE was to improve poor educational outcomes among students from disadvantaged backgrounds, we concentrate our analysis on the performance of that speci…c group of students. In particular, we de…ne as our main outcome variable the probability of falling behind the general progress of the group or being a low achiever. In doing so, we use the score in the …rst quartile for reading to de…ne the group of “lowest achievers”. Additionally, we also consider as an outcome variable the student’s reading score.

Table 2: Schools with PAE in PISA 2012

Source: INEE (National Institute for Educational Evaluation) and PISA 2012

Finally, we do not consider in the analysis schools that joined other remedial programmes.¹⁵ Our …nal sample consists of 11,747 individuals from 417 schools. We refer to this as our evaluation sample. Table 3 reports the main descriptive statistics of a set of individual, socioeconomic and school-level variables for the evaluation sample (in column 1) and for all public schools in the PISA sample (column 2), that is, schools that joined other remedial programmes, in particular PAR (see Footnote 14 for a description of PAR and Appendix 1 for a detailed de…nition of the variables in the paper):

¹⁵The PAE is part of a larger remedial programme: PROA (which is the Spanish acronym for Plan de Refuerzo, Orientación y Apoyo, literally, Plan for Reinforcement, Guidance and Support). In addition to the PAE, some schools also participated in another PROA-related programme: PAR (which is the Spanish acronym for Programa de Apoyo y Refuerzo, literally, Programme for Reinforcement and Support). It consists of providing additional resources to schools. We focus on the PAE because both the target population and the intervention are more clearly de…ned: the target population of the PAE is students with poor academic results, whereas in PAR, it is not only students but also their parents and the school in general. Second, the intervention in the PAE is similar across schools (providing students with additional classes), whereas under PAR, this was not always the case (improving school infrastructure, follow families more closely, etc.). To provide cleaner results, we drop from the analysis those schools that, in addition to the PAE, also joined the PAR programme. The total budget for the PROA Programme in 2005 was 8.5 million euros, whereas it was in excess of 400 million euros in 2012, the last year it was implemented. See the Spanish Ministry of Education Website at http://www.mecd.gob.es/educacion-mecd/areas-educacion/comunidades-autonomas/programascooperacion/plan-proa.html

Table 3: Summary Statistics

Note: Evaluation sample: PISA sample excluding students in private schools and schools which joined other remedial programs. See the text for details. Source: PISA 2012

As Table 3 indicates, the mean reading score for students in the evaluation sample is higher than that for all public schools. The proportion of immigrants and repeaters is lower in the evaluation sample.¹⁶ However, there is no di¤erence in the proportion of girls or in the proportion of students who attended pre-primary schools for more than one year. Regarding socioeconomic characteristics, our evaluation sample have a slightly smaller proportion of students from disadvantaged families: both the proportion of students with an educated father or mother and the index of educational items in the home are higher than in the full sample of public schools. Finally, we observe that the socioeconomic composition of the schools in the evaluation sample is quite similar to the full sample of public schools: the proportion of students with educated parents and the mean socioeconomic index at the school level are very similar in both samples. The proportion of dropouts in the schools in the evaluation sample is lower than in the full sample of public schools. In addition, students in the evaluation sample are in smaller schools and more likely to be in rural areas.

¹⁶See García-Pérez et al. (2014) for a detailed analysis of the impact of being a repeater on student achievement.

Next, we comment on the design of the programme evaluation. The programme was implemented for several years, and thus we can consider many di¤erent treatment de…nitions (see Table 1). Most students in the sample attended the same school for at least the most recent four academic years prior to taking the PISA exam in 2012, that is, 2008/09, 2009/10, 2010/11 and 2011/12. Therefore, they could be treated in any of these academic years. We focus here on the primary or initial e¤ect of that programme. Thus, we consider as treated students those at schools that participated in the PAE during the same academic year in which PISA exams were taken, namely, 2011/12, regardless of whether the school joined the programme before (that is, in any academic year between 2005/06 and 2010/11). We consider as controls students in schools where the PAE was not implemented at all (that is, in any academic year between 2005/06 and 2011/12). We drop from the analysis students in schools where the PAE was implemented during any academic year between 2005/06 and 2010/11 but not thereafter, i.e., during 2011/12. We refer to this treatment as PAE-Immediacy.

Table 4: Treatments definitions

Note: X(respectively, -) indicates the school participated (respectively, did not participate) in the PAE program in the corresponding academic course. 1(resp., 0) indicates whether the schools participating in PAE in the academic courses shown in that row with an X are consider as treated (resp., controls) according to the different treatment definitions. · means that these schools are dropped from the analysis.

In addition, we also assess whether the impact of the programme is stronger the more years it was implemented. To do so, we de…ne two di¤erent treatments and compare their results. We …rst consider as treated students those at schools where the PAE was implemented for only one or two of the last four academic years. We refer to this treatment as PAE-Intensity

1-2 years. Second, we consider as treated students those at schools where the PAE was implemented for three or four of the last four academic years. We refer to this treatment as PAE-Intensity 3-4 years. Similar to PAE-Immediacy, as controls in the previous two treatments, we employ students in schools where the PAE was not implemented at all (that is, in any academic year between 2005/06 and 2011/12). We refer to the comparison between the results of these two treatments as PAE-Intensity. See Table 4 below for a summary of the several treatment de…nitions.

Table 5 reports the number of treated and control schools in the sample according to each of the treatments de…ned above. The number of control schools (and students) is the same in the three treatments previously de…ned. The high survival rate of the PAE might explain why the number of treated schools in the 3-4 year treatment is larger than that in the 1-2 year treatment.

Table 5: Treated and control: schools and students

Note: The number of schools and students corresponds to the Evaluation Sample. Source: PISA 2012

Table 6 reports descriptive statistics for the treated and control groups and balancing tests corresponding to our main treatment de…nition, PAE-Immediacy: treated students (column 1), control (column 2) and the di¤erence between the two (column 3).

Mean reading test scores are lower among students in treated schools than among students in control schools. In Table 6, we also report the percentage of treated and non-treated students whose reading scores are below the …rst quartile (P25) in the corresponding score distribution (Reading25). The percentage of low-performing students is larger in the treatment group. There are not signi…cant di¤erences with respect to gender composition between the two groups. However, students in PAE schools di¤er from those in schools that did not join the programme: control students are less likely to be immigrants and are 4 points less likely to have repeated a grade. In addition, the proportion of educated parents (mother and/or father), the index of educational materials and the mean socioeconomic index are lower among treated students, suggesting that treated schools have a higher proportion of students from disadvantaged backgrounds. Finally, treated students came from larger sized schools and exhibited a larger proportion of dropouts. Conversely, students in the control sample are from schools with a higher student-teacher ratio and with principals that more frequently work to enhance the school’s reputation in the community. In the analysis below, we comment on weighted treated and control students in columns (4) and (5) of Table 6.

3 Empirical Strategy

We study the e¤ects of the PAE on the students’ probability of falling behind the general progress of the group (having a score in the …rst quartile in the reading score distribution) and students’ reading score, considering the students as the unit of analysis. By selecting the student as the unit of observation, we are aware that, to the extent that we cannot observe whether a particular student actually received the treatment, we can only consider them potentially treated, and thus, the e¤ect we study in this case is the potential e¤ect of the PAE. Nevertheless, we address this point below and attempt to provide a cleaner estimate of the true e¤ect of the PAE by decomposing our evaluation sample.¹⁷

In the evaluation literature, data often come from non-randomized studies. The main assumption in this literature is that individuals’ participation in the policy intervention to be studied can be considered a random event or, at least, independent of treated and control individuals’ characteristics (see Myoung-Jae Lee, 2005). However, selection into the treatment is not independent of treated and control individuals’ characteristics. In their seminal work, Rosenbaum and Rubin (1983) proposed propensity score matching as a method to reduce the bias in the estimation of treatment e¤ects when using such datasets. This method consists of performing a matching between individuals (students) in the treatment group and individuals in the control group who are as similar as possible with respect to observables (individual, socioeconomic and school variables).¹⁸ This implies dividing the sample into cells containing very similar individuals. However, if the vector of observable characteristics is too large, it is possible that we may lack su¢cient observations from treated and control individuals with exactly the same values for every control variable. That is, there is not a positive number of observations within each cell. Propensity score matching is a way to “correct” this problem. The propensity score is de…ned by Rosenbaum and Rubin (1984) as the probability of being treated considering those variables included in the set of regressors.¹⁹ This method proposes to summarize the pre-treatment characteristics of each subject into a single-index variable (the propensity score) that makes the matching feasible. This index is built based on the estimation of the probability of being treated, , where denote the vector of pre-treatment characteristics. If denote a binary variable that indicates exposure

¹⁷ In addition, we study the impact of the programme while considering the school to be the treatment unit as a robustness check and obtain results that are qualitatively unchanged and very similar in size (see Section 6).

¹⁸Heckman et al. (1998) proposed three factors that contribute to reduce selection bias in a evaluation study. First, we need pre-treatment variables. Second, all of the information should come from the same data source. Third, both populations (treated and non-treated) must be in the same geographical area. Ou study satis…es the …rst two conditions due to the speci…c characteristics of our dataset: academic scores come from PISA, and data on school participation in the PAE come from both regional and Ministry of Education registers. We believe that our study satis…es the third condition due to the low mobility within regions in Spain.

¹⁹Any standard probability model can be used to estimate the propensity score. The dependent variable is a binary variable equal to 1 if the individual has been treated or 0 if he has not been treated. Either a logistic distribution (logit model) or a normal distribution (probit model) may be used.

to the treatment:

\[D _ {i} = \left\{ \begin{array}{l l} 1 & \text { if treated } \\ 0 & \text { otherwise. } \end{array} \right.\tag{1}\]

and, as mentioned above, the de…nition of the treated group depends on the speci…c treatment considered: PAE-Immediacy or PAE-Intensity (1-2 years or 3-4 years),²⁰ the propensity score is de…ned as the conditional probability of PAE “participation” given pre-treatment characteristics, :

\[p (X _ {i}) \equiv P r (D _ {i} = 1 | X) = E (D | X)\tag{2}\]

3.1 Re-weighting estimates

Now, let denote the potential outcome (PISA reading score or probability of falling into the …rst quartile of the reading distribution) that student  would have obtained had she received the PAE treatment and had she not received the PAE treatment. We denote by the PISA outcome, and thus, . Therefore, the average e¤ect we are interested in estimating when evaluating the PAE is

\[\tau = E (Y _ {i} ^ {1} / D = 1, X) - E (Y _ {i} ^ {0} / D = 1, X),\tag{3}\]

where the second term is the counterfactual outcome in the absence of the treatment and, thus, is unobservable and must be estimated. This is achieved using the outcomes of control students (that is, those in schools where the PAE was not implemented at all). This requires that the characteristics of the control and treatment group be as similar as possible. However, as previously mentioned, treated and control students di¤er with respect not only to their demographic characteristics, but they also di¤er in socioeconomic background and attend di¤erent schools (see Table 6). To solve this problem, we use the rich information on demographic, parental and school characteristics in the PISA 2012 database to re-weight the sample of controls such that they can provide a counterfactual to the PISA scores of the treated students. Formally, under the standard assumptions of conditional independence or unconfoundedness:

\[(Y _ {i} ^ {1}, Y _ {i} ^ {0}) \perp D _ {i} \mid X _ {i}\tag{4}\]

that is, within each cell de…ned by , treatment is random, or similarly, the selection into treatment depends only on the observables and common support:

\[p (X _ {i}) \in (0, 1)\tag{5}\]

we have that:

\[E (Y _ {i} ^ {0} / D = 1, X) \equiv E (\omega (x _ {i}) Y _ {i} / D = 0, X)\tag{6}\]

where and

This expression indicates that we can identify the mean impact on treated individuals were they to have not received the treatment (recall, this impact is not observable), , by re-weighting the sample of controls. Observe that the weights, , increase the relevance in the control sample of those individuals who are very similar to treated students, where similarity is de…ned here by the predicted probability of “participation” in a logit that explains participation given pre-treatment characteristics, that is, by the propensity score, . This allows us to compute the inverse probability weighting estimator. This estimator is achieved by regressing the outcome variable (either the PISA score or the probability of falling behind the lowest quartile) on the treatment, where each observation is weighted by . Thus, we are estimating a model for the outcome variable using the propensity score to weight our controls in the sample. By doing so, we obtain estimates of the average treatment e¤ect for the treated (ATT), that is, the average e¤ect for those students who attended PAE schools.²¹ As there is a control for all covariates, in this estimation, through the consideration of the propensity score in the weighing procedure, there is no need to include them in the estimation. In any case, we may also include the covariates, , in this regression as a robustness check.

²⁰Here we consider as controls students in schools where the PAE was not implemented at all, i.e., between 2005 and 2012 (see Table 4 above for details).

Finally, we comment on the validity of the previous two assumptions. The second, the common support, can be tested by comparing the propensity score densities of the treated and control groups. We check this assumption graphically in the next section. However, the unconfoundedness assumption is di¢cult to validate. If it is not satis…ed, this means that programme participation could be due, among other reasons, to special interest by parents, teachers or school principals. If these variables are positively correlated with the distribution of potential outcomes (i.e., more interested parents or teachers are also more likely to yield better student reading scores), then our estimates of the impact of the PAE would be biased; in particular, they would be overestimating the true impact of the programme. This assumption is therefore crucial. We attempt to address it by including a set of variables that might capture these parent, teacher and school principal characteristics (particularly whether parents exert pressure at the school and whether the principal is concerned with the school’s reputation). In addition, in Section 5 below, we use the PISA 2009 scores to detect possible selection bias among schools participating in the PAE.

3.2 PAE participation

We estimate the predicted probability of participation in the remedial education programme (PAE) as a function of a set of characteristics of the students, parents and schools, i.e., the propensity score, . The set of variables included in was chosen according to the di¤erences in mean covariates in Table 6. We include indicators for female students, immigrant students, whether the student repeated a grade once or for more than one academic year, and whether the student attended pre-primary education. Regarding socioeconomic variables, we included whether the mother is highly educated and the index of educational materials at home. Finally, we also included a set of school characteristics, including its mean socioeconomic index value, the student-teacher ratio, its size, the proportion of dropouts, whether the school is above the 75th percentile in the distribution of the proportion of dropouts, whether it is a rural or urban school and whether the principal works to enhance the school’s reputation. We then augment the basic logit model by including interactions that were statistically di¤erent from zero according to a two-sided t-test. The …nal speci…cation is shown in Table 6. The …rst column presents the estimates of the propensity score for the PAE-Immediacy treatment. Columns 2 and 3 present the estimates of the propensity score for the PAE-Intensity treatment (PAE-Intensity 1-2 years and PAE-Intensity 3-4 years, respectively). As can be observed, the speci…cations of the three propensity scores are the same. This allows us to obtain comparable results across the di¤erent treatments.

²¹See Hospido et al. (2015) for a similar approach and Hirano et al. (2003) or Busso et at. (2014) for methodological details.

The estimates in the …rst column con…rm the results of Table 6. Treated and control students are similarly likely to be girls. In addition, treated students are more likely to be immigrants and to have repeated at least one grade. However, once a complete set of control variables is considered, the mothers of treated and control students are similarly educated. The index of educational materials in the home also exhibits comparable values between treatments and controls. Regarding school variables, compared to control students, the schools of treated students are more likely to have a lower socioeconomic index value, a larger size, a larger proportion of dropouts, a lower teacher-student ratio, and principals who are less interested in enhancing the school’s reputation. Finally, observe that the results of the propensity score for the three treatments are very similar, in particular regarding the school variables.

Figure 2 illustrates the densities of the predicted probabilities of participation in the PAE for the treated and control groups. Although the two distributions di¤er in form, the …gure shows how similar the control and treatment samples are. First, the support of the values of the propensity score of treated students (solid line) and that of the control (dotted line) are the same: both range from 0 to approximately 0.8 (PAE-Immediacy), 0.6 (PAE-Intensity 1-2 years) or 0.8 (PAE-Intensity 3-4 years). Therefore, the common support assumption seems to hold in our sample. In addition, there is no concentration of predicted values around zero or one (which would mean that there are no comparable control students for some treated students).

Finally, columns (4) and (5) of Table 6 present the means of the treated and control sample once the latter is re-weighted by . First, although column (5) and column (1) should be exactly the same, as treated students receive a weight of 1, they do not coincide due to the existence of missing values in the weight variable (observe the lower number of observations in this column). The last column in Table 6 reports the di¤erences in characteristics between treated and re-weighted controls. As can be observed, these are not statistically di¤erent from one another, particularly for the set of controls considered in the propensity score estimation (i.e., the balancing property is satis…ed). Finally, note that the sample is also similar along characteristics that we do not include in the propensity score (class size, rural, etc.). An exception are the Student Admission and Sta¤ Decision variables (the former being larger in the treatment group and the latter being larger in the control group). The similar composition of treated and re-weighted control groups even in characteristics omitted from the propensity score reinforces the credibility of the assumption that treated and re-weighted control students would have performed similarly had the treated students not been treated.²²

Figure 2: Propensity score support

PAE-Intensity: 1-2 years

Note: Density of the probability of participation (propensity score) for treated (solid line) and control groups (dotted line). The figure in the upper part corresponds to the propensity score computed for the PAE-Immediacy treatment (Table 7 column 1). The figures in the center and bottom part correspond to the propensity score computed for the PAE-Intensity, 1-2 years and 3-4 years, respectively (Table 7 columns 2 and 3).

²²See Lavy and Schlosser, 2005 or Hospido et al. 2015 for a similar test.

4 Main results

In this section, we comment on the impact of the two treatments considered in our analysis: PAE-Immediacy and PAE-Intensity.

4.1 PAE-Immediacy and PAE-Intensity

The estimated e¤ect of the PAE-Immediacy treatment is reported in the …rst column of Table 8. The …rst row in Panel A shows the re-weighting estimate without covariates (IPWEnc). Hence, this result can also be inferred from the …rst row in Table 6. The proportion of treated students in the …rst quartile in the reading score distribution is equal to 0.231, while that of the re-weighted control group is equal to 0.264. The -0.033 di¤erence is the observed impact of the programme. The standard error accounts for arbitrary correlation at the school level and is equal to 0.019; thus, the estimate is only statistically signi…cant at the 10% con…dence level. The e¤ect is quite similar when we include all of the variables considered in the logit model used to obtain the weights; speci…cally, it is equal to -0.030 and statistically signi…cant at the 5% con…dence level (IPWEwc, see row 2). The robustness of this result suggests that the speci…cation of the model that predicts PAE participation is appropriate. Nevertheless, we go further and compare each treated student with her most similar associated control counterparts and thus provide results using several nearest neighbour propensity score estimators. In particular, we provide estimators by varying the number of nearest neighbours considered in the estimation from 2 to 8 (NNPS(2) to NNPS(6) in row 3 to row 6). As can be observed, the results are quite similar to those obtained by using the inverse probability weighting estimator. In particular, the larger the number of nearest neighbours used, the more similar the results are to the estimation without covariates. To summarize, we …nd that the probability of falling behind into the bottom part of the distribution is reduced by between 3% and 6% from receiving remedial education under the PAE.

Panel B in Table 8 shows the results regarding the e¤ect of the programme on the mean reading score. Again, the result in the …rst row can also be inferred from the second row in Table 6. As can be observed, the estimate obtained when we do not include all of the covariates is not statistically signi…cant. Nevertheless, as commented above, the e¤ect is much more precise when we hold constant all variables included in the conditional model. By doing so, we …nd that the estimated e¤ect is equal to 5.53, which amounts to 6.4% of one standard deviation (=5.93/86.61). The point estimate when we use a nearest neighbour propensity score matching estimator is larger, 12.34, which equals 14.2% of one standard deviation.²³

²³The PAE had also a strong impact on maths and science outcomes. For example, according to the NNPS(2) estimator, students in schools than joined the programme at least during the 2011/12 year (PAE-Immediacy) signi…cantly reduced their probability of belonging to the low-achievers group in maths and science by approximately 3.9% and 3.2%, respectively. It also improves their mean maths and science scores by approximately 9.7 and 7.8 PISA points, respectively. All estimates are statistically di¤erent from zero at the 5% con…dence level or better. The complete results are not reported in the paper but are available from the authors upon request.

Table 8: The impact of PAE

Note: The first two rows in Panel A and Panel B report Inverse Probability Weighting Estimator without covariates (IPWEnc) or with covariates (IPWEwc). The covariates included in the estimation are the ones used for the propensity score (Table 7); Rows 3 to 6 in Panel A and Panel B report nearest neighbour propensity score estimators using two neighbours NNPS(2), four NNPS(4), etc. *** p<0.01, ** p<0.05, * p<0.1. Robust standard standard errors in parenthesis. Standard errors for the IPWE are corrected for clustering at the school level. Source: PISA 2012.

Table 9 illustrates how the PAE changes the overall distribution of reading scores. We present the estimated Cumulative Distribution Function (CDF) of the reading score for certain percentiles (see column 1 for the speci…c percentiles computed and column 2 for the corresponding value of the reading score distribution for the complete sample including all public schools). In columns 3-5 we present the values of the three CDFs: the CDF of reading scores among control students, the CDF of reading scores among re-weighted controls and the CDF of reading scores among the treated, for the PAE-Immediacy, PAE-Intensity 1-2 year and PAE-Intensity 3-4 year treatments in Panels A, B and C, respectively. Finally, in column 6, we present the di¤erence between the last two (this column shows a rate equal to the CDF treated/CDF weighted controls minus one).

Table 9: Estimated CDF reading scores

Note: Columns 3, 4 and 5 show the CDF of reading scores among control students, re-weighted controls students and among treated students, respectively. Column 6 presents the difference between columns 4 and 5 (rate equal to the CDF treated/CDF weighted controls minus one).

As can be observed in the three panels, for each percentile, there is a lower fraction of students below that reading score among the control sample than among the treated sample. In addition, when comparing treated and re-weighted controls, we observed that the fraction of students below any score in the distribution among the treated sample is lower than among the re-weighted control sample (except for the PAE-Intensity 1-2 year treatment). As the re-weighted sample is, under our assumptions, the distribution of the scores that treated students would have achieved in the absence of the programme, that pattern suggests an overall increase in the distribution of reading scores. Finally, observe that the group of students who receive the larger impact from the PAE (both the PAE-Immediacy and the PAE-Intensity 3-4 year treatments) are those whose reading scores are between the 15th and 30th percentiles of the distribution, that is, precisely those students whose outcomes are among the main targets of the programme. Next, we analyse these students’ performance in detail.

As previously noted, the results for the full sample presented above might not precisely capture the true impact of the PAE but merely its potential e¤ect. On the one hand, we are assuming that all of the students in schools with the PAE are treated. However, some of them might not have received remedial education at all. Observe that by doing so, we are underestimating the impact of the PAE. On the other hand, by considering all of the students in the PAE school as treated, we might well be capturing peer e¤ects of treated on non-treated students. This assumption might induce an overestimation of the impact of the PAE on treated students. To argue that the e¤ect analysed is closer to the actual e¤ect of the intervention on treated students, we focus our main analysis on two sub-samples of our evaluation sample. First we consider students whose reading score is below the median value of the distribution.²⁴ This sub-sample consists of 5,427 individuals. By considering students with poor academic results, we increase the likelihood that they actually participated in the programme. Second, we consider students whose reading score is above the median value of the distribution. This sub-sample consists of 6,320 individuals. By considering students with high academic results, we reduce the likelihood that they actually participated in the programme and were subject to positive spillover e¤ects from treated students.

Table 10 reports the main …ndings of this analysis. The …rst two columns provide results for the PAE-Immediacy treatment. Column 1 provides the results for the sub-sample of students below the median. It reports the impact on the probability of being in the …rst quartile and the impact on the mean reading score. The second column reports the results for the sub-sample of students above the median. It shows the probability of being above the third quartile and the impact on the mean reading score.

Table 10: The impact of the PAE: sub-samples

We …nd that the probability of falling behind into the bottom part of the distribution is reduced by approximately 4% to 5% for those students in the sub-sample below the median. Therefore, by considering the full sample of students at the school, we came close to estimating the true impact of the PAE on moving students out of low-achiever status, which is the main objective of the programme. We also …nd that, as expected, the programme had no e¤ect on the probability of becoming a high achiever, that is, on the probability of belonging to the third quartile. We do not …nd evidence of spillover e¤ects of potentially treated students on non-treated students (see Lavy and Schlosser, 2004 for a similar result). Finally, observe that the impact of the PAE on mean reading scores is smaller for both the sub-sample of students below and above the median than for students in the full sample. This might be due to the fact that by censoring the sample using the median reading score, we are not considering those cases of treated students who as a result of having received the PAE are above the median but who in the absence of the treatment would have remained below it.

²⁴The median for the PISA sample for all public schools (that is, including schools that might have participated in other remedial programmes) is 482.19.

The estimated e¤ect of the PAE-Intensity treatment is reported in the second and third columns of Table 8. For those students in schools where the PAE was implemented for at most two of the last four years, the probability of falling behind into the bottom part of the distribution declines by (when statistically signi…cant) between 2.8% and 3.9% relative to students in schools where the PAE was not implemented at all. However, for those students in schools where the PAE was implemented for at least three of the last four years, that probability declines by between 3.2% and 7.5%, relative to students in schools where the PAE was not implemented. Therefore, we can conclude that the PAE has an intensity e¤ect: the larger the number of academic years for which it is implemented in a school, the more likely students are to leave the low-achievers’ group. The bottom part of Table 8 reports the results regarding the possible intensity e¤ect of the programme on the mean reading score. As can be observed, most estimates obtained for the e¤ect of the 1-2 year treatment are not statistically signi…cant. However, most estimates for the 3-4 year treatment are signi…cant. Thus, we conclude that the PAE also has an intensity e¤ect on mean reading scores. In particular, by receiving the PAE for at least three years, mean reading scores increase by between 10.7 and 16.2 PISA points, that is, between 12.3% and 18.7% of one standard deviation (10.7/86.61 and 16.2/86.61, respectively).

We next decompose the overall e¤ect into the e¤ect on the sub-samples of students below and above the median. These results can be found in Table 10, columns 3 to 6. We …nd that implementing the PAE for just one or two years has, if any, an impact on the sub-sample of students below the median. When signi…cant, we …nd that it reduces the probability of falling behind into the bottom quartile among such students by approximately 4.6%. However, it has no impact on those students above the median: it does not signi…cantly increase the probability of becoming part of the third quartile for these students. In addition, implementing the PAE for just 1 or 2 years has no e¤ect on the reading scores of either subsample. However, implementing the programme for three or four years has an impact. In particular, the probability of falling behind into the bottom part of the distribution declines by between 4.2% and 4.6% for those students in the sub-sample below the median. In contrast to the results for the PAE-Immediacy treatment, we now …nd that the programme had an e¤ect on the probability of becoming a high achiever, that is, on the probability of belonging to the third quartile. In addition, and similar to the PAE-Immediacy treatment, observe that the impact of implementing the PAE for three or four years on mean reading scores for both the sub-samples of students below and above the median is smaller than for students the full sample.

4.2 Heterogenous e¤ects: rural vs. urban schools

As mentioned above, the PAE consisted of providing support (4 hours per week) to students with special needs and learning di¢culties. This support was provided by after-school instructors or teachers from the student’s own school who work with these students in smal groups. These remedial classes were held during after-school hours (see Footnote 9 for additional details on programme implementation). Therefore, both teachers and students had to return to the school for the programme, which might be more di¢cult for teachers in urban schools than those in rural schools, as the former do not necessarily live close to the school. Therefore, we would expect gradual attrition in PAE participation among teachers in urban schools that, as a result, might reduce the e¤ectiveness of the programme for students there. To assess whether there is heterogeneity in the impact of the PAE, we examine its impact on the previous reading outcomes by school type: rural or urban.

We de…ne a rural school as one located in a community of fewer than 15,000 persons (i.e., a village or a small town) and an urban school as one located in a community of 15,000 or more persons (i.e., a town, city or large city). There are 220 urban schools (with 6,456 students) and 175 rural schools (with 4,669 students) in our sample. Table A.1 in Appendix 2 compares the characteristics of treated and control students in urban and rural schools. Students in urban and in rural schools di¤er in several dimensions. Reading outcomes (both the probability of belonging to the …rst quartile and reading scores) are better among students in urban schools than in rural schools. Moreover, the proportion of immigrant students is larger among urban schools. However, the proportion of students with an educated father or mother is lower among rural schools. In addition, the mean socioeconomic index exhibits much higher values among urban schools. Finally, urban schools are larger in size than rural schools.

The di¤erence between treated and control students also di¤ers between urban and rural schools. For instance, whereas control students in urban schools have better outcomes than treated students, the reverse occurs in rural schools. The distribution of socioeconomic characteristics also di¤ers: in urban schools, parents of control students have higher schooling levels than their counterparts among treated students. Conversely, in rural schools, the proportion of educated parents (fathers) is larger among treated students than among controls.

To estimate the impact of the PAE on urban versus rural schools, we proceed as in the previous section. We …rst estimate the probability of participating in the PAE separately for students in urban and rural schools, considering individual, family and school characteristics, that is, the propensity score. Second, we use the estimated propensity score to construct the re-weighted sample of controls in urban and rural schools.²⁵ Finally, we use the previous results to compute the inverse probability weighting estimator (with and without covariates) and the nearest neighbour matching estimator. Table 11 compares the average outcomes of treated students in urban schools to students in rural schools. Panel A provides results for the impact of the programme on the probability of belonging to the lower achiever group (Reading25), and Panel B provides results for the impact on mean reading scores.

²⁵Columns (3) and (6) in Table A.1 show that treated and control samples, in both urban and rural schools, are comparable once re-weighted.

Table 11: PAE Impact: urban vs. rural schools

The results for the PAE-Immediacy treatment (columns 1 and 2 for students in urban and rural schools, respectively) indicate that the impact is much larger in rural schools. The probability of falling into the …rst quartile reduces by twice as much for students in rural schools than for students in urban schools (7.5% and 3.5%, respectively). The increase in mean reading scores is also larger among students from rural schools. The results for the PAE-Intensity treatment suggest several similar …ndings. First, again, the impact of the PAE on mean reading scores is larger for students in rural schools than students in rural schools, regardless of whether the school joined the programme for at most two years or more than two years. Second, similar to the results for the full evaluation sample, the PAE has an intensity e¤ect in rural schools: the larger the number of academic years it is implemented in a school, the higher the probability of students leaving the low-achievers’ group and the higher the increase in mean reading achievement. However, the PAE has no intensity e¤ect for students in urban schools.

To conclude, the impact of the PAE is, in general, much larger for students in rural than in urban schools. The information reported above regarding the implementation of the programme might provide a possible explanation for the sources of these di¤erent results for students in urban versus rural schools, without attaching any causal interpretation: namely, o¤ering remedial classes in after-school hours might be more di¢cult to implement for teachers in urban schools. As a result, it could be the case that some of them do not teach the total number of remedial classes or even abandon the programme.

5 Selection bias: are PAE schools di¤erent from the rest?

As previously noted, our results above can be called into question based on the argument that treated schools volunteer for the programme, while control schools did not. Therefore, it is possible that principals who decide to participate in the PAE have unobserved characteristics that correlate with students’ characteristics and with their outcomes. Similarly, students in treated schools may have unobserved characteristics that correlate with the decision of the principals to join the PAE and with reading scores. If these unobserved school (principal, teacher, etc.) characteristics are positively correlated with students’ outcomes, then our previous results would be overestimating the true impact of the programme. For example, highly motivated and active principals may, in addition to deciding to participate in the PAE, promote various types of activities and initiatives to improve their students’ results. However, these unobserved school (principal, teacher, etc.) characteristics might also be negatively correlated with students’ outcomes, for example, the existence of a di¢cult student body at the school. In that case, then our previous results would be underestimating the true impact of the programme. Thus, it is very di¢cult to establish a priori the sign and magnitude of the bias. Formally, according to Heckman et al. (1998), we can de…ne selection bias as follows. Let …rst consider the linear model:

\[Y _ {t} = X _ {t} ^ {\prime} \beta + u _ {t},\]

where is the student’s outcome at time  and is a set of observables (individual, family and school variables). Now, suppose that one of the school characteristics is PAE participation. Then, the conditional average of the variable given speci…c values of the regressors would be calculated as follows:

\[E (Y _ {t} \mid X _ {t} = x, P A E _ {t} = p a e) = x ^ {\prime} \beta + \delta p a e\tag{7}\]

However, as noted previously, it is very di¢cult to conclude that parameter is capturing the impact of PAE participation due to possible selection bias. We partially addressed this problem by including in the covariates a set of variables capturing principal characteristics that might be both a¤ecting students’ scores and the probability of participating in the PAE. Here, we use the PISA 2009 dataset to characterize possible selection bias under the assumption that the true impact of a non-existent programme is zero. In particular, we replicate the analysis in (7) by replacing with an indicator, , indicating that the school participated in the PAE after the 2008/09 academic year but not before that date:

\[E (Y _ {t} \mid X _ {t} = x, D _ {t + 1} = D) = x ^ {\prime} \beta + \alpha D\tag{8}\]

Observe that the student outcome, , is measured at time  (in this case, the PISA 2009 scores), whereas the treatment, , is measured at time  + 1, as it will occur well after the PISA 2009 scores were measured (treated schools will be those that did not participate in the PAE between the 2005/06 and 2008/09 academic years, but did participate thereafter) Thus, if there is no selection bias, the estimated impact of this “treatment” should be zero. We next estimate its impact following the empirical strategy presented in a previous section. We …rst estimate the predicted probability of participating in the PAE only after the 2008/09 academic year by considering a set of individual, family and school variables. Then, we reweight the control group such that their observable re-weighted characteristics are statistically similar to those of the treatment group. Finally, we estimate the (non-existent) e¤ect of participating in PAE after the 2008/09 academic year for the treated students (using PISA 2009 scores). As above, this allows us to compute the inverse probability weighting estimator (IPWE). This is achieved by regressing the outcome variable (either the PISA 2009 score or the probability of falling into the lowest quartile) on the “treatment”, where each observation is weighted by . We also include the covariates, , in the regression as a robustness check. In addition, we compute the nearest neighbour propensity score (NNPS) estimators after verifying that our estimates of the propensity score ful…l the balancing property. We …nally compare results from following the two empirical strategies.

5.1 The data

Our sample now consists of 4,568 students from 144 schools that participated in both PISA 2009 and PISA 2012.²⁶ Therefore, for those schools, we know whether they participated in the PAE in any academic year since the programme began. In particular, 31 such schools participated in the programme only after the 2008/09 academic year. Thus, the sample consists of 912 “treated” students and 3,656 “control” students.²⁷ Using the rich information from the PISA 2009 database, we can compare them according to individual, parental and school variables. In addition, we can also identify which variables account for the possible selection bias. Table 12 below compares the characteristics of treated and control students in the sample.

Here Table 12: Summary statistics: treated and control. Selection bias

Although there are no signi…cant di¤erences regarding gender composition between the two groups, students in schools that subsequently participated in the PAE di¤er from controls in an important number of characteristics. As can be observed in Table 11, treated students are more likely to be immigrants and are 10 points more likely to have repeated a grade at least once. In addition, the proportion of educated parents (mother and/or father), the index of educational materials and the mean socioeconomic index are lower among treated students, suggesting that treated schools have a higher proportion of students from disadvantaged backgrounds. Finally, treated students came from smaller sized schools where the proportion of educated parents is lower than that for controls. Conversely, students in the control sample are from schools with a larger student-teacher ratio. Finally, treated students performed worse on PISA 2009: the proportion of students with a reading score in the …rst quartile is larger among treated students, and their mean reading score is lower than among controls. Therefore, the previous results suggest that, if any selection into participation in the PAE based on unobservable characteristics exits, then these variables are negatively correlated with students’ outcomes, which implies that our previous estimates are underestimating the true impact of the programme.

5.2 Selection bias estimation

As we know that PAE participation in treated schools occurred well after the PISA 2009 exams took place, the di¤erence in reading outcomes (once we control for student, parent and school characteristics) can only be due to the in‡uence of unobserved variables or selection bias. Next, we examine the possible existence of selection bias. We …rst estimate the probability of participating in the PAE only after the 2008/09 academic year. We present the results in Table 13.

²⁶We also exclude, as in previous exercises, schools that participated in other remedial education programmes between the 2005/06 and 2011/12 academic years.

²⁷We believe that our sample size is large enough to provide relevant results. See, for example, Heckman et al. (1998) who use samples of approximately 200 treated subjects to study the properties of selection biases in employment programmes.

Here Table 13: Determinants of PAE participation only after 2009

The analysis of participation determinants con…rms that the treated group contains a larger proportion of immigrants and repeater students. In addition, having an educated mother (and living outside Basque Country) or a large index of educational materials reduces the probability of participation. Similarly, the results in Table 13 suggest that the schools of the treated students are more likely to have a lower socioeconomic index value, be of a smaller size and are also more likely to be located in urban municipalities. Column (3) of Table 12 shows the average characteristics of the control group once it is re-weighted according to the predicted probability of participation. Observe that, again, the number of observations for re-weighted controls is reduced due to the existence of missing values for the weighting variable.²⁸ It can be seen that the sample of control students, once re-weighted, is similar to that of the treated students in terms of reading outcomes and individual, family and school variables.

Here Table 14: Impact of PAE participation (only after 2009)

Finally, we proceed to estimate the impact of PAE participation only after 2009. The upper part of Table 14 reports the results for the probability of belonging to the …rst quartile of the reading score distribution. The …rst row shows the result of a simple probit estimation. As can be observed, the e¤ect of programme participation after 2009 is zero. The second row shows the re-weighting estimate without covariates. Therefore, this result can also be inferred from the …rst row of Table 12. The 0.002 di¤erence is the observed impact of the programme (see Footnote 25). The standard error accounts for arbitrary correlation at the school level and is equal to 0.0042; thus the estimate is also not signi…cantly di¤erent from zero. The e¤ect is quite similar when we include all variables considered in the logit model used to obtain the weights (third row). In addition, we go further and compare each treated student with her most similar associated control counterparts and thus provide results using two nearest neighbour propensity score estimators. As can be observed, the results are remarkably similar to those obtained using the inverse probability weighting estimator.

The bottom part of Table 14 shows the results regarding the e¤ect of the programme on the mean reading score. Again, the results in the second row can also be inferred from the second row in Table 12. The estimate obtained when we do not include all of the covariates is not statistically signi…cant. Nevertheless, as noted above, the e¤ect is much more precise when we hold constant all of the variables included in the logit model. By doing so, we also …nd that the estimated e¤ect is not statistically di¤erent from zero. The point estimate when we use the nearest neighbour propensity score matching estimator is also negligible, at 2.915,

²⁸Summary statistics for the sample of treated students for which the weighting variable is not missing are not reported here for clarity but are available upon request.

and is not statistically di¤erent from zero.

To conclude, we …nd that the results of the schools that participated in the PAE only after the 2008/09 academic course were not very di¤erent from the rest, suggesting that no selection bias exists. Nevertheless, if any, possible di¤erences can be explained by di¤erences in individual, parental and school characteristics. Accounting for these di¤erences completely attenuates the selection bias.²⁹ Therefore, our results above suggest that it is feasible to obtain estimates of the impact of PAE participation on reading outcomes with no selection bias by re-weighting the sample according to student, family and school characteristics, as we have done above. A possible explanation for the lack of selection is that, as the programme was introduced in the 2005/06 academic year, by the 2009/10 academic year, and thereafter, the existence of the programme was su¢ciently widespread in the education community (indeed, the rate of participation in the programme exceeded 45% in some regions).

6 Robustness analysis

Finally, we want to check whether our previous results when considering the student as the unit of analysis hold when we instead consider the school as the unit of analysis. Recall that to the extent that we cannot observe whether a particular student actually received the treatment, our previous …ndings merely suggest the potential e¤ect of the PAE. By considering the school as the unit of analysis, and similar to Lavy and Schlosser (2005), two problems emerge. First, if there is a small number of treated students at a school it may be very di¢cult to observe any e¤ect. In addition, to claim that the e¤ect analysed is the actual or true e¤ect of the intervention on treated students, we need to assume that the PAE did not generate spillover e¤ects on non-treated students (which appears to be the case in light of our previous results from the sub-sample of students with reading scores above the median).

Table A.2 in Appendix 2 provides the summary statistics of the schools in our evaluation sample and for all public schools. Table A.3 compares the characteristics of treated and control schools, which di¤er in several dimensions: …rst, the mean reading score is higher among treated schools. In addition, the proportions of repeaters, immigrants and dropouts are also larger among treated schools. Conversely, the proportion of educated parents and the socioeconomic index is higher among the control group. Furthermore, the proportion of treated schools where the principal claims that he/she works to enhance the school’s reputation is nearly twice as large relative to control schools.

To estimate the impact of the PAE on schools, we proceed as above. We …rst estimate the probability of participating in the PAE considering only school characteristics, that is, the propensity score.³⁰ Second, we use the estimated propensity score to construct the re-weighted sample of control schools.³¹ Finally, we use the previous results to compute the inverse probability weighting estimator (with and without covariates) and the nearest neighbour matching. Table 15 provides the estimated e¤ect of the PAE-Immediacy treatment (column 1) and the PAE-Intensity treatment (columns 2 and 3). Panel A provides results for the impact of the programme on the probability of belonging to the lower-achiever group (Reading25), and Panel B provides results for the impact on mean reading scores.

²⁹As an additional robustness check, note that neither the 95% con…dence interval for the estimated bias for the probability of being in the …rst quartile of the reading distribution [-0.0583,0.0541] (Table 13, row 4) nor the 95% con…dence interval for the estimated bias for the mean reading score [-5.426, 11.257] (Table 13, row 8) contains the point estimate of the e¤ect of the PAE in Table 8 above.

³⁰Table A.4 in Appendix 2 presents the results for the estimated propensity score for the PAE-Immediacy and PAE-Intensity treatments.

Here Table 15: The impact of the PAE: schools

As can be observed, the results are very similar to those found in Table 8 when considering the student as the unit of analysis. In particular, the impact of the programme now appears to be larger. First, the proportion of students at the school in the …rst quartile of the distribution declines by between 4.9% and 7.6%, depending on the estimator (compared to the 3%- 6% reduction at the student level), in those schools that participated in the PAE at least during the 2011/12 academic year (PAE-Immediacy treatment). The results regarding the e¤ect of the programme on the mean reading score at both the school and the student level are not signi…cantly di¤erent when we use a nearest neighbour propensity score matching estimator, approximately 12.13 PISA points (12.34 in Table 8 above). Nevertheless, the inverse probability weighting estimator produces larger impacts at the school level than at the student level in this case. Finally, we also …nd that the PAE has an intensity e¤ect: the larger the number of academic years for which it is implemented at the school, the larger the proportion of students exiting the low-achiever group. The programme has almost no impact in those schools that participated in the programme for at most two academic years, whereas it has a strong impact among those schools that participated for at least three years: the proportion of students in the low-achiever group declines by between 5% and 8%, and the mean reading score increases by between 10.3 and 19.8 PISA points.

7 Concluding remarks

There is ample evidence of increasing inequality and poverty …gures in developed countries. As a result, addressing early school leaving and improving the education and skills of the workforce are priorities of policy makers in several countries. National governments are currently being encouraged to undertake evidence-based education policies to reduce the adverse e¤ects of the aforementioned facts. Surprisingly, it is di¢cult to …nd empirical evidence regarding the e¤ectiveness of most of these interventions and in particular remedial education programmes. In this paper, we estimate the e¤ects of a remedial programme implemented in Spain between 2005 and 2012 that o¤ered additional instruction time for underperforming students from poor socioeconomic backgrounds: the Programme for School Guidance (PAE). Our main …nding is that this programme had a substantial positive e¤ect on students’ academic achievement. First, our results suggest that it reduced the probability of falling behind into the bottom of the reading score distribution by approximately 5% (nearly 10% of one standard deviation). The estimated e¤ect on mean reading scores is above 12 PISA points (more than 14% of one standard deviation). We also …nd that a larger exposure to the programme improves students’ scores. Furthermore, our evidence suggests that there is heterogeneity in the impact of the programme across types of schools, urban versus rural, with the impact being much larger among students attending rural schools than urban schools.

³¹Column (3) in Table A.3 indicates that treated and control schools are comparable once re-weighted.

This study has an important limitation. Namely, we lack data on whether a particular student actually received the treatment and instead merely observe whether the student attended a school that participated in the programme, implying that the e¤ects we obtain can only be understood as the potential e¤ects of the programme. We address this shortcoming by performing two additional tests. We …rst decompose our evaluation sample into two sub-samples: one with students whose reading scores are below the median value of the distribution, that is, students who were more likely to received the treatment, and other with students whose reading scores are above the median, that is, students who might not have participated in the programme but received positive spillover e¤ects from treated students. In addition, we check whether our previous results hold when we instead consider the school as the unit of analysis. By proceeding with these strategies, we conclude that our previous results are, if anything, underestimating the true impact of the programme on treated students.

Future research should proceed by evaluating the impact of this or similar programmes on a wider range of student outcomes, such as dropout, absenteeism or even on non-cognitive skills, such as study habits (motivation and discipline), self-esteem, and con…dence (see, among many others, Heckman et al., 2006 on the growing literature demonstrating that young students’ non-cognitive skills signi…cantly a¤ect their school achievement and work outcomes). In this study, we examine only short-term e¤ects due to a lack of su¢cient data on schools participating in the programme only well before the PISA 2012 exams. Nevertheless, learning about the long-run e¤ects of the programme is required to fully understand its e¤ectiveness.

We believe that our results are of value and contribute novel, interesting insights to a relatively scarce literature on remedial education programmes and their impact on underperforming teenagers across Europe. In this paper, we …nd support for policies consisting of targeted additional instruction time to improve poor-performing students’ achievement.

References

1. [1] Atkinson A. (2010): “Macerata lectures on European economic policy. Poverty and the EU: the new decade,” Working Paper 24-2010, Macerata University, Department of Studies on Economic Development (DiSSE).
2. [2] Atkinson A., Piketty T., and E. Saez (2011): “Top incomes in the long run of history” Journal of Economic Literature 49, 1: 3-71.
3. [3] Banerjee, A. V., S. Cole, E. Du­o and L. Linden (2007): “Remedying Education: Evidence from Two Randomized Experiments in India” The Quarterly Journal of Economics 122 (3):1235-1264.
4. [4] Battaglia, M. and L. Lebedinski (2015): “Equal Access to Education: An Evaluation of the Roma Teaching Assistant Program in Serbia”, World Development, 76: 62-81
5. [5] Bettinger E. and B. Long (2009): “Addressing the needs of under-prepared college students: does college remediation work?” Journal of Human Resources 44: 736–771
6. [6] Brunello, G. and M. De Paola (2014): “The Costs of Early School Leaving in Europe” IZA Journal of Public Policy 3(22).
7. [7] Busso, M., J. DiNardo, and J. McCrary (2014): “New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators” Review of Economics and Statistics 96 (5): 885–897.
8. [8] Calcagno J.C. and B. T. Long (2008): “The Impact of Postsecondary Remediation Using a Regression Discontinuity Approach: Addressing Endogenous Sorting and Noncompliance” NBER Working Paper No. 14194
9. [9] De Paola, M. and V. Scoppa (2014): “The E¤ectiveness of Remedial Courses in Italy: A Fuzzy Regression Discontinuity Design” Journal of Population Economics, 27(2): 365- 386
10. [10] De Paola, M. and V. Scoppa (2015): “Procrastination, Academic Success and the E¤ectiveness of a Remedial Program” Journal of Economic Behavior and Organization, 115: 217–236
11. [11] European Commission (2013a). Social Investment Package. Brussels.
12. [12] European Commission (2013b). Education and Training in Europe 2020: Responses from the EU Member States. Eurydice Report. Brussels: Eurydice
13. [13] Freeman, R. (2008). “Globalization and Inequality”, in the Oxford Handbook of Economic Inequality, edited by W. Salverda, B. Nolan and T. Smeeding. Oxford: Oxford University Press.
14. [14] García-Pérez, J.I., Hidalgo-Hidalgo M. and J. A. Robles-Zurita (2014): “Does grade retention a¤ect students’ achievement? Some evidence from Spain” Applied Economics 46 (12): 1373–1392.
15. [15] Jacob, B. A. and L. Lefgren (2004): “Remedial Education and Student Achievement: A Regression Discontinuity Analysis” Review of Economics and Statistics 86 (1): 226-244.
16. [16] Heckman, J. J., Ichimura, H., Smith, J. and Todd, P. (1998): “Characterizing Selection bias using Experimental Data” Econometrica 66(5): 1017-1098.
17. [17] Heckman, J., J. Stixrud and S. Urzua (2006): “The E¤ects of Cognitive and Noncognitive Abilities on Labour Market Outcomes and Social Behavior” Journal of Labor Economics 24(3): 411-480.
18. [18] Hirano, K., G. Imbens and G. Ridder (2003) “E¢cient Estimation of Average Treatment E¤ects Using the Estimated Propensity Score” Econometrica 71(4): 1161-1189.
19. [19] Holmlund, H. and O. Silva (2014): “Targeting Non-Cognitive Skills to Improve Cognitive Outcomes: Evidence from a Remedial Education Intervention” Journal of Human Capital 8(2): 126-160.
20. [20] Hospido, L., E. Villanueva and G. Zamarro (2015): “Finance for All : The Impact of Financial Literacy Training in Compulsory Secondary Education in Spain” Banco de España WP 1502 2015, IZA DP 8902 2015
21. [21] Kanbur, R. (2014): “Globalization and Inequality” Working Papers 180163, Cornell University, Department of Applied Economics and Management.
22. [22] Kremer, M., C. Brannen and R. Glennerster (2013): “The Challenge of Education and Learning in the Developing World” Science 340 : 6130 (April 19): 297-300.
23. [23] Lavy, V. and A. Schlosser (2005): “Targeted Remedial Education for Underperforming Teenagers: Costs and Bene…ts” Journal of Labor Economics 23 (4): 839-874.
24. [24] Myoung-Jae Lee (2005) “Micro-Econometrics for Policy, Program, and Treatment Effects”, Advanced Texts in Econometrics (Ed. Ganger, C. W. J. y Mizon, G. E.), Oxford University Press.
25. [25] OECD (2011) PISA 2009. Technical Report OECD Publishing, OECD, Paris.
26. [26] OECD (2013). Crisis squeezes income and puts pressure on inequality and poverty. OECD Publishing, Paris.
27. [27] Rosenbaum, P. R. and Rubin, D. B. (1983): “The Central Role of the Propensity Score in Observational Studies for Causal E¤ects” Biometrika (70): 41-75.
28. [28] Rosenbaum, P. R. and Rubin, D. B. (1984): “Reducing Bias in Observational Studies Using Subclassi…cation on the Propensity Score” Journal of the American Statistical Association 79: 516-524.
29. [29] Spanish Ministry of Education (2016): Estadísticas de las Enseñanzas no universitarias, Ministerio de Educación Cultura y Deporte. Spain. Available at http://www.mecd.gob.es/servicios-al-ciudadano-mecd/estadisticas/educacion/nouniversitaria/centros/centros-servicios-estadisticas.html (accessed 15 July 2016).
30. [30] U.S: Department of Education, O¢ce of the Under Secretary (2003): When schools stay open late: the national evaluation of the Twenty-First Century Community Learning Centers Program, …rst year …ndings. U.S. Department of Education, O¢ce of the Under Secretary, Washington, DC.

Appendix 1: Variable Description

We describe all of the variables used in our estimations (the original variable names in the PISA database are presented in capital letters).

PAE-Immediacy: Dummy variable that equals 1 for students at schools that participated in the PAE during the 2011/12 academic year (0 for students at schools that never participated in the PAE). Source: INEE.

PAE-Intensity 1-2 years: Dummy variable that equals 1 for students at schools that participated in the PAE during the 2010/11 and/or 2011/12 academic years (0 for students at schools that never participated in the PAE). Source: INEE.

PAE-Intensity 3-4 years: Dummy variable that equals 1 for students at schools that participated in the PAE for 3 out of 4 of the academic years between 2008/09 and 2011/12 (0 for students at schools that never participated in the PAE). Source: INEE.

Reading: The average of the …ve plausible values of literacy outcomes. Source: Students questionnaire PISA 2012.

Reading25: Dummy variable equal to 1 for students with reading score below the …rst quartile of the reading scores distribution for all public schools, i.e., 420.1. Source: Students questionnaire PISA 2012.

Immigrant: Dummy variable equal to 1 for non-native students, i.e., …rst- or secondgeneration immigrants (IMMIG above 1). Source: Students questionnaire PISA 2012.

Repeater once: Dummy variable equal to 1 for students attending grade 9 (ST01Q01 equal to 9). Source: Students questionnaire PISA 2012.

Repeater more once: Dummy variable equal to 1 for students attending grade 8 or lower (ST01Q01 lower or equal to 8). Source: Students questionnaire PISA 2012.

_² Attended pre-primary: Source: Dummy variable equal to 1 for students attending pre-primary schools for more than one year (ST05Q01 above 2). Source: Students questionnaire PISA 2012.

Mother/father educated: Dummy variable equal to 1 for students whose mother/father attended at least tertiary education. Source: Parents questionnaire PISA 2012.

Index educational materials: Index of whether the home possesses a desk and a quiet place to study, a computer and/or educational software, books to help with schoolwork and a dictionary (HEDRES variable ranges from -3.93 to 9999.00 in the international dataset). Source: Parents questionnaire PISA 2012.

Students educ parents: % Students at school with educ. parents: Percentage of students at the school whose mother and father attended at least tertiary education. Source: Parents questionnaire PISA 2012.

ESCS: Dummy variable equal to 1 if the school is above the last third in the index of economic, social and cultural status distribution for all public schools. Source: Parents questionnaire PISA 2012.

School size: Total school enrolment. Source: School questionnaire PISA 2012.

Presion: Perceptions of principals about parents exerting pressure towards the school to set high academic standards and to have their students achieve them (SC24Q01 below 3, which is the answer to what best characterizes parental expectations towards your school. Possible answers range from 1-there is constant pressure- to 3 -pressure largely absent). Source: School questionnaire PISA 2012.

Proportion of Dropout students at School: Proportion of students who left the school without the certi…cate that allows them to enter post-secondary or vocational education, apprenticeships or employment (generated from 2-digit variable SC23Q01). Source: School questionnaire PISA 2012.

Proportion of Dropout students (percentile 75): Dummy variable equal to 1 if the school is above the 75th percentile in the evaluation sample in the proportion of dropout students at school.

Student Teacher Ratio at School: number of students per teacher at the school (generated from 2-digit variable STRATIO). Source: School questionnaire PISA 2012.

Rural School: Dummy variable equal to 1 if the school is located in a village or a small town and to 0 if located in a town, a city or large city (SC03Q01 is the principal answer to school location). Source: School questionnaire PISA 2012.

Principal Enhances Reputation: Dummy variable equal to 1 if the principal says that he/she works to enhance the school’s reputation in the community once or more than once per week (SC34Q01 above 5). Source: School questionnaire PISA 2012.

Students Admittance: Dummy variable equal to 1 if the principal says that the school has responsibility for the student’s admittance (SC33Q09C equal to 1). Source: School questionnaire PISA 2012.

Sta¤ decision: Dummy variable equal to 1 if the principal says that the school has responsibility for sta¤ hiring decisions (SC34Q10 between 4 and 6 both included). Source: School questionnaire PISA 2012.

Review Work: Dummy variable equal to 1 if the principal says that he/she reviews work produced by students when evaluating classroom instruction (SC34Q20 above 5). Source: School questionnaire PISA 2012.

Discuss Problems: Dummy variable equal to 1 if the principal says that he/she takes the initiative to discuss matters when the teacher has a problem in the classroom (SC34Q07 above 5). Source: School questionnaire PISA 2012.

Assess: Dummy variable equal to 1 if the “Use of Assessment” index (ASSESS) is equal to 5 or 6 in the school. This index measures the extent to which assessments of students are used to inform parents of their child’s progress, to make decisions about students retention or promotion, to group students for instructional purposes, to compare the school to district or national performance, etc. Source: School questionnaire PISA 2012.

Table 6: Summary Statistics: Treated and control

Note: Treated students under Treatment PAE-Immediacy. Treated and control in columns (1) and (2) are sample averages. Treated and controls in columns (4) and (5) are averages of the treated and control group when the sample is reweighted by the (inverse of the) probability of receiving PAE-Immediacy treatment predicted by the set of individual, socioeconomic and school variables in this table. (i) column (4) and column (1) should be exactly the same as treated students receive a weight of 1, however they do not coincide due to the existence of missing values in the weight variable (observe the reduced number of observations in columns (4) and (5). We test for mean differences: *** p<0.01, ** p<0.05, * p<0.1. Source: PISA 2012

Table 7: Propensity score estimation

Note: The probability of participating in the program is estimated using a Logit model which, in addition to the covariates shown in the table, includes the following control variables: regional dummies (Andalusia, Aragon, Castile Leon, Catalonia, Extremadura, Galicia, Murcia and Navarre) and dummies to capture missing values in some variables (attended pre-primary, mother educated and school size). *** p<0.01, ** p<0.05, * p<0.1. Robust standard errors in parenthesis. Source: PISA 2012.

Table 10: The impact of the PAE program: subsamples

Note: P<50 (resp. P>50) refers to the subsample of students below (above) the median of the reading distribution for the sample of all public schools. Reading75 indicates the probability of having a reading score above the third quartile of the reading distribution. *** p<0.01, ** p<0.05, * p<0.1. Robust standard standard errors in parenthesis. Standard errors for the IPWE are corrected for clustering at the school level. Source: PISA 2012.

Table 11: The impact of PAE program: Urban vs rural

Note: Urban (resp. rural) schools are those located in a community of more (resp. less) than 15,000 people. *** p<0.01, ** p<0.05, * p<0.1. Robust standard standard errors in parenthesis. Standard errors for the IPWE are corrected for clustering at the school level. Source: PISA 2012.

Table 12: Summary Statistics: Treated and controls. Selection bias

Note: Treated: students attending schools that did not participate in the PAE program before 2009 but participated after 2009 (either during 2009/10, 2010/11 or 2011/12). Treated and control in columns (1) and (2) are sample averages. Weighted controls are averages of the control group when the sample is reweighted by the (inverse of the) probability of participating in PAE only after 2009 predicted by the set of individual, socioeconomic and schoo variables in this table. Source: PISA 2009

Table 13: Determinants of PAE participation only after 2009

Note: The probability of participating in PAE program only after 2009 is estimated using a logit model model which, in addition to the covariates shown in the table, includes the following control variables: regional dummies (Aragon, Balearic Islands, Cantabria, Galicia, La Rioja, Murcia and Basque Country) and dummies to capture missing values in some variables (attended pre-primary, mother and father educated). *** p<0.01, ** p<0.05, * p<0.1, Robust standard errors in parenthesis. Source: PISA 2009

Table 14: Impact of PAE participation (only after 2009)

Note: The dependent variable is the student’s probability of belonging to the first quartile of the reading distribution in the PISA 2009 exams for public schools (Reading25) and the student’s reading score in the PISA 2009 exams. The estimation method in the first row is ordinary least squares. Estimation methods in the rest of rows are similar to the ones used in Tables 8 and 10 above. *** p<0.01, ** p<0.05, * p<0.1, Robust standard errors are clustered at the school level.

Table 15: The impact of the PAE program: schools

Note: The first two rows in Panel A and Panel B report Inverse Probability Weighting Estimator without covariates (IPWEnc) or with covariates (IPWEwc). The covariates included in the estimation are the ones used for the propensity score (Table 7); Rows 3 to 6 in Panel A and Panel B report nearest neighbour propensity score estimators using two neighbours NNPS(2), four NNPS(4), etc. *** p<0.01, ** p<0.05, * p<0.1. Robust standard standard errors in parenthesis. Standard errors for the IPWE are corrected for clustering at the school level. Source: PISA 2012

Table A.1: Summary Statistics: treated and control. Urban vs rural schools

Note: Treated students under Treatment PAE-Immediacy. Treated and control in columns (1) and (2) and (4) and (5) are sample averages. Weighted controls in columns (3) and (6) are averages of the control group when the sample is reweighted by the (inverse of the) probability of receiving PAE-Immediacy treatment predicted by the set of individual, socioeconomic and school variables in this table. Source: PISA 2012

Table A.2: Summary Statistics: Evaluation sample and PISA sample. Schools

Note: Evaluation sample: PISA sample excluding students in private schools and schools which joined other remedial programs. See the text for details. Source: PISA 2012

Table A.3: Summary Statistics: Treated and control. Schools

Note: Treated schools under Treatment PAE- Immediacy. Treated and control in columns (1) and (2) are sample averages. Weighted controls in columns (3) are averages of the control group when the sample is reweighted by the (inverse of the) probability of receiving PAE-Immediacy treatment predicted by the se of school variables in this table. Source: PISA 2012

Table A.4: Propensity score estimation. Schools

Note: The probability of participating in the program is estimated using a Logit model which, in addition to the covariates shown in the table, includes the following control variables: regional dummies (Andalusia, Aragon, Asturias, Balearic Islands, Cantabria, Castile Leon, Catalonia, Extremadura, Galicia, La Rioja and Navarre) and dummies to capture missing values in some variables (school size). *** p<0.01, ** p<0.05, * p<0.1, Standard errors in parenthesis. Source: PISA 2012