Editor Responsável: Fábio Waltenberg
This article assesses the impact of the Programa de Educação Integral (PEI) implemented in the state of São Paulo (Brazil) on test scores and school characteristics. Using difference-in-differences and leads and lags methods, we found positive and significant effects of the program on performance in Mathematics (0.469 standard deviations) and Portuguese (0.462 standard deviations) for ninth grade students. The impact is greater if the school receives the program for a longer time. Also, the program reduced the disparities of scores within schools. We also identified that participant schools undergo changes in their infrastructure and students’ socioeconomic profile.
O artigo avalia o impacto do Programa de Educação Integral (PEI) implementado no Estado de São Paulo (Brasil) sobre o desempenho educacional (SAEB) e características das escolas participantes. Usando diferenças em diferenças e lead and lags, encontramos efeitos positivos e significativos sobre o desempenho em matemática (0.469 desvio-padrão) e português (0.462 desvio-padrão) para os estudantes do 9º ano do ensino fundamental. O impacto é maior se a escola recebe o programa há mais tempo. O programa também reduziu a desigualdade de notas dentro das escolas. Também identificamos que as escolas participantes apresentaram mudanças em sua infraestrutura e perfil socioeconômico dos alunos.
Brazil has undergone a major expansion of education in recent decades, but the quality of the educational system still remains a problem. To improve student learning, several efforts are being made, including lengthening the school day. Data from 2018 shows that Brazil has a school day shorter than the average of the OECD countries and shorter than other Latin American countries and the United States.
Aligned with the national guideline, the São Paulo state government established two full-time education programs: the Escola de Ensino Integral (hereafter ETI) and the Programa de Ensino Integral (hereafter PEI). ETI was created in 2005, before PEI, and contemplates the final years of elementary school with the extension of the school day (
It is not straightforward to assume that longer school hours imply gains in academic performance. For example, the federal program Mais Educação, created in 2007 with the intention of expanding full-time education seems to have been unsuccessful in increasing student performance (
The aim of this paper is to evaluate the impacts of PEI on student performance, and to analyze whether there are changes in the composition of students and in the infrastructure of the schools that receive the program. These changes may be a mechanism by which the average academic performance of treated schools improves, but are taken into account when estimating the impact of the program. We use differences-in-differences and leads and lags strategies to estimate the impact of PEI on ninth graders students’ academic performance. As a robustness analysis of the results, we use the propensity score matching methodology in conjunction with differences-in-differences. We find that PEI has improved the schools’ average test score for mathematics and Portuguese. This improvement occurs in the first year that the school receives the program, and is amplified in the following years. The program also reduces the dispersion of scores among students in the school. Moreover, the composition analysis shows that the program also impacted positively the number of students in each class, but negatively the number of classes. Regarding the socioeconomic composition of students, PEI either attracts students with higher socioeconomic profiles, repels students with a lower profile, or both.
This paper contributes to the literature in three aspects. First, it is the evaluation of a program that, until now, does not have published quantitative academic studies to assess its impacts. Second, PEI differs from other full-time schooling programs implemented in Brazil at the time, and there are positive results on school performance. These lead to a reflection on the essential characteristics of a successful full-time education program. Third, it provides evidence that full-time school programs are attracting students with higher socioeconomic profiles.
In addition to this introduction, this paper presents the rationale behind the extended school time in Section 2, followed by a brief literature review in Section 3, presenting empirical results from national and international full-time school programs. We describe the PEI program in section 4, and provide descriptive statistics in Section 5. Section 6 presents the methodology that is used in the impact estimations, which are presented in Section 7. We divided this last section into three parts: the impact on academic performance (Section 7.1), robustness analysis (Section 7.2) and composition analysis (Section 7.3). Finally, Section 8 discusses the results and Section 9 concludes.
The educational production function relates the inputs to the maximum possible learning and is based on the production function in the firm theory. Among the various inputs, we have those related to families, peers, and the school, such as teacher quality and available infrastructure (library, room size). Among the school’s inputs, we have the time the student stays at school (i.e., the number of hours per school-day, the number of days in the week, or the number of school days in the year). The theoretical foundations that place time as one of the central elements for learning are in the model proposed by
The numerator of this ratio depends on the time allocated for learning (i.e., instruction time). Also, it depends on their level of perseverance, expressly the time engaged in learning. The denominator of the ratio, that is, the time needed for students to learn depends on their aptitude, the quality of instruction they receive and their ability to understand the instruction.
Although
However, the additional instruction time may have an effect on student’s performance if combined with other changes in learning.
The program has also changed the teacher’s work contract, reinforcing the teacher’s bond with the school, with an impact also on the teacher’s salary. And greater accountability was also inaugurated, with peer and student evaluation of teachers, with the possibility of sanctions. All these variables enter into the educational production function, with possible impacts on learning, in addition to classroom time and the learning environment. There are papers in the literature that analyze the impact of these other variables on learning. For example, teacher turnover (
Systematic review from early research revealed that much evidence about increasing school-days fails to address causal inference (
Since experimental designs are rare, many of them rely on quasi-experimental strategies, including differences-in-differences. For example, in Japan,
Evidence from Chile also reinforces the positive impact of full-time schools on learning.
Despite the evidence from the work cited above, student covariates may be a confounder factor in determining the impact of full-time schools, since student characteristics are an input of the educational production function. The Coleman Report was the seminal study that pointed out the importance of student characteristics in determining student outcomes (
Several international studies show the impact of increasing the learning hours in school, however only a few of them explore the mechanisms behind these effects.
In 2007, the Brazilian national government created the program Mais Educação, which was implemented full-time in several schools. The program consists of lengthening the time students stay in school, providing different activities in this extra time from the standard curriculum. However, several studies conclude that the program has not resulted in a positive effect on the academic performance of participants (
Programs that not only expand the school-day, but also include pedagogical and school-management innovations may be more successful. For example,
Taking advantage of the heterogeneity in program implementation,
In addition,
The impact of full-time school on schools’ composition is also analyzed by
There are two different full-time school programs in São Paulo, one is PEI, the object of analysis of this study, and the other one is ETI (Escola de Tempo Integral).
We didn’t find any impact evaluation regarding specifically the full-time school program in São Paulo (PEI).
The state of São Paulo implemented the Programa de Ensino Integral (PEI) in 2012, using as reference the Ginásio Pernambucano experience. This model, implemented first in one public school in the state of Pernambuco in 2004, served as the standard for the full-time school state program in Pernambuco, and it was the inspiration for the Ginásio Experimental Carioca, in Rio de Janeiro (
The student protagonism takes shape, for example, in the Life Project, a mandatory subject in elementary and high school. In this subject, young people trace their journey beyond school and their actions to achieve their goals and dreams in life. Teachers guide these actions, but it is the students who put them into practice. This project is materialized in a document that follows the student during his or her school career and is constantly revisited. Other school activities also make the young person an active subject of learning, such as student clubs and class leaders.
In addition to the Life Project, the diversified curriculum is composed of elective courses designed by the school’s own teachers, and chosen by the students. These subjects promote interdisciplinary work, and should be a space for experimentation and for diversification. ETI (the other full-time school education program in São Paulo) also features a diversified curriculum. However, it differs from PEI in that these diversified activities are not in consonance with the regular curriculum, being taught exclusively in the afternoon shift. In addition, the diversified part of the ETI is decided by the state secretary, and it is not up to the school to change according to its needs, as in the case of the PEI (
Besides the differentiated pedagogical methodology, PEI presents significant changes in the teaching career. The teachers follow the exclusive dedication regime, working 40 hours a week, and receiving a bonus of 75% of their base salary for this full dedication. In addition, teachers and managers undergo an internal evaluation by students and other teachers, and their permanence in the school is conditioned based on this evaluation.
The priority for enrollment in these schools is for students who were already enrolled prior to the implementation. Schools join the PEI on a voluntary basis through the school board, and the secretary of education is responsible for evaluating whether the school will receive the program. PEI began in 2012 with 12 high schools, expanded in 2013 to elementary schools, and by 2015, 157 elementary schools were participating in the program.
Although the program’s pedagogical methodology is not exclusively focused on students’ academic performance, the question is whether the PEI has an impact on student performance in mathematics and Portuguese. This evaluation is important, since it allows comparison with other schools not participating in the program. This paper aims to answer this question, and analyze whether there are changes in the schools’ composition and the schools infrastructure.
We considered in our analysis São Paulo state schools in the years 2009, 2011, 2013, and 2015. We measure the students’ performance using data from basic education national assessment in Brazil (SAEB), which measures learning in mathematics and Portuguese. All data used is aggregated at the school level. We used the average scores of the schools for the ninth grade (last grade of elementary school) as the main outcome variable. Besides this, we also used other national data available to characterize the schools and their students (School Census and SAEB). Information about the participation of schools in the PEI was made available by the São Paulo Secretary of Education.
Source: School Census, SAEB, São Paulo Secretary of Education.
Year:
2009
2011
2013
2015
PEI
0
0
22
153
Regular full-time
266
196
180
258
State schools
3678
3638
3622
3531
The descriptive statistics are presented in
Note: Asterisks indicate Student’s t test for the difference between the means of the treatment group and the control group (alternative hypothesis: the difference between the means is different from zero). ***significance at 1% level; **significance at 5% level, *significance at 10% level. Variables beginning with “No.” are counts, variables beginning with “Prop.” are the proportion of students in the school who possess a given characteristic, and variables beginning with “Has” are the proportion of schools who possess a given characteristic. The variables “Prop. mothers...” refer to the proportion of the mothers or the women responsible for the student, which informs the answers about her characteristics in the SAEB. The variable “Regular full-time” was constructed from the number of hours reported in the School Census for each class. This is a
Treatment
Control
Mean
Standard Deviation
Mean
Standard Deviation
School census
No. total students***
364.289
143.679
506.481
256.893
No. teachers
121.793
40.550
126.789
56.657
No. staff***
61.393
20.230
70.015
27.618
No. classes***
11.400
3.995
14.712
6.611
Prop. of men**
0.518
0.034
0.512
0.030
Prop. white***
0.511
0.156
0.557
0.196
Prop. teachers with graduation
0.241
0.241
0.255
0.238
Prop. who use public transport ***
0.098
0.175
0.149
0.223
Has sport court***
0.985
0.121
0.950
0.218
Has computer lab.
0.963
0.190
0.964
0.187
Has science lab.***
0.459
0.500
0.314
0.464
Has library
0.059
0.237
0.058
0.233
Regular full-time***
0.481
0.502
0.057
0.231
SAEB
Mathematics test score**
244.804
13.883
242.502
14.818
Portuguese test score**
242.162
13.817
239.808
14.867
Prop. mothers with only high school***
0.179
0.074
0.159
0.074
Prop. mothers with higher education*
0.047
0.038
0.041
0.036
Prop. illiterate mothers***
0.022
0.021
0.029
0.025
Prop. at right age*
0.663
0.117
0.643
0.119
Prop. who work ***
0.097
0.047
0.122
0.056
Prop. had late entrance
0.013
0.015
0.013
0.014
Prop. who had only public school
0.653
0.118
0.656
0.112
Prop. had already failed **
0.142
0.060
0.153
0.065
Prop. had already dropout**
0.026
0.020
0.029
0.021
Prop. with Fridge
0.725
0.111
0.714
0.113
Prop. with TV
0.708
0.107
0.693
0.110
Prop. with Computer***
0.480
0.136
0.437
0.145
Prop. with domestic worker*
0.065
0.035
0.059
0.033
Mean No. of bathrooms
1.380
0.142
1.372
0.159
The proportion of schools in 2009 that already had full-time education is also significantly higher in the treatment group, which may indicate the program’s preference for schools that already had an extended school day, although this is not a rule.
In relation to test scores, graphs (a) and (b) in
Although these graphs provide support for the common trend hypothesis, it was expected that the detachment would start in 2013, the first year of PEI’s implementation in elementary school. But as reported in
There is evidence that schools follow a common trend through the first year of receiving the program. However, a concern arises about the validity of this hypothesis for Portuguese proficiency, given that there is a reduction in the average score of schools prior to the entry into the program (2014 and 2015). This issue will be formally addressed by the leads and lags model (section 7.1).
The first objective of this paper is to estimate the impact of PEI on students’ performance. To achieve this goal, we used the difference-in-differences identification strategy (
Let Y 0it be the mean test score of school i in year t, if school i does not participate in the PEI in year t. Let Y 1it be the mean test score of school i in year t if it participates in the program in period t. Let
where D i is a dummy that indicates whether the school participates in the PEI in any year.
Controlling for observed variables is important because changes in these variables over time can be confounding. For example, participation in the PEI may have caused changes in the profile of students attending the school and its infrastructure
The basis of the differences-in-differences strategy are the assumption that unobservable factors are fixed in time and the common trend assumption, which in this case means assuming that schools have a common time fixed effect. That is:
where λ t is a common time fixed effect across state schools and δ is the average impact of the PEI on Y. Furthermore, it is also assumed that this impact is homogeneous across groups of schools with different entrance years in PEI.
where ∈
The δ coefficient in equation 4 is interpreted as the average impact of the PEI on the grade of participating schools over the entire period that the program was in place. To better understand the program, we can estimate this impact diluted across years of participation. We use the leads and lags’ strategy proposed by
where r is an integer, D it−r is a dummy indicating that treatment at school i occurred in year t − r. Similarly, D it+r indicates treatment at school i in year t + r. The terms δ −r and δ r are coefficients to be estimated. The rest of the notation follows what was previously defined.
Following
PEI start
2013
2014
2015
2009
2011
2013
2015
We present the estimations of regressions 4 and 5 in Section 7.1, considering as dependent variables the average school proficiency in mathematics and in Portuguese. We estimate each of the regressions with and without the covariates X it . As a robustness analysis of the results, in Section 7.2 we perform a matching before the difference-in-differences estimation, restricting the control and treatment group to schools with similar observable characteristics in 2009. In Section 7.3, we perform the composition analysis by estimating the differences-in-differences regression with school characteristics, student characteristics and the coefficient of variation of test scores as dependent variables.
Note: *p<0.1;**p<0.05;***p<0.01 Note: Standard deviations in parentheses. School-level variables. ***significance at the 1% level, **significance at the 5% level, *significance at the 10% level. All regressions include school fixed effects and time fixed effects.
mathematics scores
Portuguese scores
(1)
(2)
(3)
(4)
PEI
9.136 ***
7.069 ***
9.760 ***
6.979 ***
(0.774)
(0.838)
(0.850)
(0.903)
Regular full-time
−0.031
−0.450
(0.743)
(0.801)
Covariates
X
X
Observations
14.469
14.469
14.469
14.469
R 2
0.013
0.178
0.012
0.207
The estimates indicate that the impact of the PEI is significant at a similar level in both subjects. The inclusion of covariates is important to explain part of the variability in scores, controlling also for heterogeneity in the evolution of observable characteristics across schools. In terms of sample standard deviation
Note: *p<0.1;**p<0.05;***p<0.01 Note: Standard deviations in parentheses. School-level variables. ***significance at the 1% level, **significance at the 5% level, *significance at the 10% level. All regressions include school fixed effects and time fixed effects.
mathematics scores
Portuguese scores
(1)
(2)
(3)
(4)
Lead 4
- 1.413
0.154
-2.606
- 0.705
(1.704)
(1.561)
(1.870)
(1.682)
Lead 3
0.579
1.553
- 0.464
0.821
(1.314)
(1.202)
(1.442)
(1.296)
Lead 2
- 2.069
- 1.106
-3.878 ***
- 2.707
(1.701)
(1.558)
(1.867)
(1.680)
Lead 1
- 0.237
1.066
- 2.011
- 0.424
(1.316)
(1.205)
(1.444)
(1.299)
Lag 0
5.090 ***
4.326 ***
4.734 **
3.685 **
(1.722)
(1.654)
(1.890)
(1.783)
Lag 1
9.743 ***
8.651 ***
9.354 ***
7.702 ***
(1.311)
(1.280)
(1.439)
(1.380)
Lag 2
14.600 ***
10.786 ***
13.184 ***
8.483 ***
(2.458)
(2.295)
(2.697)
(2.474)
Regular full-time
0.171
-0.271
(0.747)
(0.805)
Covariates
X
X
Observations
14.469
14.469
14.469
14.469
R 2
0.015
0.179
0.014
0.208
We see that the lead 2 of the model for Portuguese proficiency without covariates is significant, although small (column 3
In the previous analysis, we used all state schools that did not receive PEIs as the control group. In this section we will restrict the control group to schools that have similar characteristics to the treatment, using propensity score matching. We calculate the probability of treatment p(x) based on characteristics of the school and the students that are part of the school, prior to treatment, in 2009. For the identification of the treatment effect, we need to ensure (
1.
2.
To choose the covariates that determine p(x), we used the hit or miss technique, accompanied with statistical significance analysis
Note: *p<0.1;**p<0.05;***p<0.01 Note: Standard deviations in parentheses. School-level variables. ***significance at the 1% level, **significance at the 5% level, *significance at the 10% level.
Treatment
No. total students
-0.005 ***
(0.001)
Prop. mothers with higher education
1.653 *
(1.000)
Prop. had already failed
-1.895 ***
(0.640)
Has sport court
0.629 **
(0.275)
No. classes
0.151 ***
(0.039)
No. teachers with graduation per student
0.805 ***
(0.543)
Constant
- 1.884 ***
(0.303)
Observations
3.679
Log Likelihood
-537,968
Akaike Inf. Crit.
1.089,94
The higher the total number of students, the lower the probability of being treated. However, the relationship is inverse with respect to the number of classes. This indicates that the institutions that are contemplated by the PEI have fewer students per class. In addition, there are suggestions that the schools in the program have better infrastructure and teacher qualifications: having a sports court increases the probability of treatment, and also having a higher proportion of teachers with a graduation degree. Furthermore, the higher the proportion of students in the school who have already failed at some grade, the lower the chance of being treated.
The higher the total number of students, the lower the probability of being treated. However, the relationship is inverse with respect to the number of classes. This indicates that the institutions that are contemplated by the PEI have fewer students per class. In addition, there are suggestions that the schools in the program have better infrastructure and teacher qualifications: having a sports court increases the probability of treatment, and also having a higher proportion of teachers with a graduation degree. Furthermore, the higher the proportion of students in the school who have already failed at some grade, the lower the chance of being treated.
The model is able to correctly predict 76% of treated schools (specificity), and 61% of control schools (sensitivity). Part of the difficulty in correctly predicting more schools is due to the imbalance between the number of treated and untreated schools: while 135 schools participated in the PEI, 3543 did not. The complete Hit or Miss statistics are available in
Given p(x) calculated using the probit, we performed a matching with the two nearest neighbors with a maximum distance (caliper) of 0.02 on the propensity score. The matching was done with replacement, and to ensure the common support hypothesis, 1 observation was removed from the treatment group. In
Furthermore, to verify the quality of the matching and also to corroborate the conditional independence hypothesis, a balance analysis of the covariates before and after treatment was performed. The analysis was done in terms of the standardized mean difference.
For additional robustness, two other pairings were also performed: one using the Radius method with caliper of 0.02 and with replacement; and another using Kernel with a window of 0.02. No observations were discarded in both methods to ensure common support. The common support plots and the covariate balance for these methods are in
The results of the three methods are reported in
Note: *p<0.1;**p<0.05;***p<0.01 Note: Standard deviations in parentheses. School-level variables. *** significance at the 1 % level, ** significance at the 5 % level, * significance at the 10 % level. All regressions include school fixed effects and time fixed effects.
Dependent variable:
Math scores
Caliper
Kernel
Radius
PEI
9.049 ***
7.830 ***
9.274 ***
7.658 ***
9.275 ***
7.697 ***
(1.069)
(1.327)
(0.241)
(0.352)
(0.247)
(0.351)
Covariates
X
X
X
Observations
1,440
1,440
14,055
14,055
14,055
14,055
R 2
0.068
0.241
0.013
0.160
0.013
0.160
PEI
9.745 ***
8.406 ***
9.725 ***
7.654 ***
9.727 ***
7.668 ***
(1.130)
(1.374)
(0.260)
(0.367)
(0.260)
(0.367)
Covariates
X
X
X
Obserations
1,440
1,440
14,055
14,055
14,055
14,055
R 2
0.071
0.274
0.012
0.193
0.012
0.193
The covariates used are the same as in the differences-in-differences method. When we do the matching, we control for covariates in the baseline, but if there is a change in characteristics between the control and treatment group during the analyzed period, the effect of the program on student proficiency may be “contaminated” by the change in these characteristics. Thus, we estimate controlling also for covariates. In the Radius and Kernel, although no observations were discarded, the number of observations differs from the differences-in-differences because there are observations that did not enter the estimation (had weight equal to zero), and in addition, 17 PEI schools were created after 2009, and therefore did not enter the estimation.
The results are similar to those found in subsection 7.1. If we take the Kernel estimations with covariates as a basis and compare it to the previous difference-in-differences estimations, the Kernel ones are 0.589 (3.9% of a standard deviation) and 0.675 (4.5% of a standard deviation) higher for mathematics and Portuguese, respectively. These differences correspond to less than 10% of the estimated impact. Also, both methods’ 90% confidence interval for the estimates overlaps for mathematics and Portuguese.
In The previous subsection, we showed that the full-day school program had an impact on test scores, but does it have an impact on the schools’ composition or its infrastructure? For instance, has the program attracted students with a higher socioeconomic level? Have the treatment schools hired more teachers per class? In order to address these questions, we estimate, using difference-in-differences, the following model:
where the variable
The dependent variables are divided into school infrastructure, students’ socioeconomic characteristics, other students’ characteristics, and coefficient of variation (CV) in grades (standard deviation of school’s test scores divided by the mean of school’s test scores). For the first three categories of variables, in addition to the results of the differences-in-differences regression, we calculate the sample standard deviation of variable
Some of the
Note: Standard deviations in parentheses. School-level variables. ***significance at the 1% level, **significance at the 5% level, *significance at the 10% level.
School infrastructure
No. classes
-0.366*
6.245
-0.059
0.176***
-0.064
0.191***
-0.07
0
(0.204)
(0.035)
(0.037)
No. total students
25.506***
236.912
0.108
-
-
-
-
0.001
(7.508)
No. teachers per class
-0.847***
1.276
-0.664
-
-
-
-
0.009
(0.087)
Has computer lab.
-0.005
0.17
-0.029
-0.213
0
-0.722
0
0
(0.013)
(0.531)
(0.573)
Has science lab.
0.37***
0.469
0.789
0.068
0
0.303
0
0.027
(0.022)
(0.319)
(0.343)
Has library
-0.023
0.256
-0.09
-0.581*
0.013
-0.259
0
0
(0.023)
(0.304)
(0.328)
No. staff per student
-0.022***
0.046
-0.478
2.459
0
0.544
0
0.009
(0.002)
(3.144)
(3.389)
Prop. teachers with graduation
0.018*
0.185
0.097
0.092
0
-0.963
0
0
(0.01)
(1.718)
(1.852)
The number of classes per student is also significantly impacted: while there is a negative impact in the number of classes (−0.059 standard deviations), a positive impact is observed in the number of students (0.108 standard deviations). The absolute number of classes and students dropped, on average, in all schools. Thus the results indicate that schools receiving the PEI had a lesser drop in the number of students, and greater in the number of classes. The lower number of classes is possibly due to the reduction of classes in the afternoon period, as a result of the full school day. These impacts are accompanied by a negative impact in the number of teachers per class (−0.664 standard deviations). This reduction can be explained by the exclusive dedication work-regime imposed by the program. The teachers must work exclusively in one school, probably with a larger workload than before the program.
Note: Standard deviations in parentheses. School-level variables. *** significance at the 1 % level, ** significance at the 5 % level, * significance at the 10 % level.
Students’ socioeconomic characteristics
Prop. illiterate mothers
0.004** (0.002)
0.024
0.167
-26.764*** (4.1)
-0.107
-26.943*** (4.419)
-0.108
0
Prop. mothers with only high school
0.026*** (0.005)
0.077
0.338
14.002*** (1.606)
0.364
17.636*** (1.731)
0.459
0.002
Prop. mothers with higher education
0.011*** (0.004)
0.08
0.137
19.166*** (2.142)
0.211
22.664*** (2.309)
0.249
0.001
Prop. who work
-0.013*** (0.004)
0.055
-0.236
-16.52*** (1.918)
0.215
-30.811*** (2.067)
0.401
0.001
Prop. with domestic worker
0.007** (0.003)
0.039
0.180
-17.054*** (2.259)
-0.119
-25.666*** (2.435)
-0.18
0
Prop. with TV
0.051***
0.102
0.5
0.833
0
8.733**
0.445
0.004
(0.008)
(3.778)
(4.072)
Prop. with Computer
0.042***
0.153
0.275
18.897***
0.794
17.148***
0.72
0.003
(0.007)
(1.54)
(1.66)
Mean No. of bathrooms
0.016**
0.157
0.102
3.534***
0.057
2.541***
0.041
0
(0.008)
(0.879)
(0.947)
Prop. who use public transport
0.026***
0.231
0.113
-1.5
0
-2.265**
-0.059
0.001
(0.007)
(1.004)
(1.082)
The results point out a significant and positive impact of the PEI on the proportion of students with mothers who have completed higher education, on the proportion of mothers who have completed up to high school, and on the proportion of illiterate mothers. In terms of standard deviations, the impact on the proportion of mothers who completed up to high school is considerably larger than the others. The results also point out that mothers with higher education and mothers with up to high school positively impact grades, unlike illiterate mothers. Considering the estimates, attracting students with more educated mothers is associated with an increase in grades, while attracting students with illiterate mothers is associated with a decrease.
The impact of the program on the proportion of working students is significant and negative. This is expected since one of the features of the PEI is the extension of the number of hours a student spends in school, making it more difficult for students to work. In addition, the estimates indicate a positive impact of the program on asset ownership variables, with the largest impact in terms of sample standard deviations being on the proportion who own a TV (0.5 standard deviations).
These results support the hypothesis that the changes implemented by the program attracted students with a different profile to the participating schools. More specifically, the results presented in
Note: Standard deviations in parentheses. School-level variables. ***significance at the 1% level, **significance at the 5% level, *significance at the 10% level.
δ
σ
δ
β
δ
β
δ
Prop. of white
-0.005
0.169
-0.03
0.749
0
1.555
0
0
(0.006)
(1.082)
(1.166)
Prop. of men
-0.016***
0.031
-0.516
-5.573**
0.089
-20.258***
0.324
0.004
(0.002)
(2.793)
(3.011)
Prop. at right age
0.071***
0.111
0.64
-10.427***
-0.74
-11.696***
-0.83
0.008
(0.008)
(2.645)
(2.851)
Prop. who had only public school
0.049***
0.104
0.471
14.493***
0.71
18.928***
0.927
0.004
(0.007)
(2.352)
(2.535)
Prop. had already failed
-0.015***
0.066
-0.227
-35.196***
0.528
-38.827***
0.582
0.001
(0.005)
(1.813)
(1.954)
Prop. had already dropout
-0.003*
0.024
-0.125
-27.027***
0.081
-23.798***
0.071
0
(0.002)
(3.758)
(4.051)
Prop. had late entrance
0
0.014
0
-48.57***
0
-53.593***
0
0
(0.001)
(5.44)
(5.863)
Also negative and significant is the estimated impact of the PEI on the proportion of students who have already failed and the proportion of dropouts. Consequently, the impact is positive and significant for the proportion of right-aged students. These results indicate that either the PEI attracted better students, or else it improved these characteristics in students during the period in which it was implemented. It is also observed that the proportion of students who have only studied in public school increased. That is, one hypothesis is that the students attracted by the PEI were the students from other public schools and, according to the previous results, students with higher socioeconomic profile and better performance from these other public schools. Another hypothesis is that the students who stayed in the participants’ schools are the ones who have this profile, while others might have been repelled, as mentioned previously.
Note: *p<0.1;**p<0.05;***p<0.01 Note: Standard deviations in parentheses. School-level variables. ***significance at the 1% level, **significance at the 5% level, *significance at the 10% level. All regressions include school fixed effects and time fixed effects
Math CV
Portuguese CV
(1)
(2)
(3)
(4)
PEI
- 0.001
- 0.003 *
-0.011 **
-0.008 **
(0.002)
(0.002)
(0.002)
(0.002)
Covariates
X
X
Observations
14,469
14,469
14,469
14,469
R 2
0.0001
0.027
0.003
0.066
The estimates indicate that the PEI decreases the dispersion of the Portuguese and math scores. The impact of the PEI on the CV in terms of sample standard deviations in the models controlling for covariates is 0.167 for math and 0.364 for Portuguese. The only specification that showed a non-significant result was for the CV of mathematics without the use of covariates. This indicates that some of the covariates included in the model varied in schools that received the PEI in the sense of increasing the dispersion of math scores.
The results found corroborate the hypothesis that the PEI improves the average performance of schools on test scores. We found a positive impact of the program on ninth grade math (0.469 standard deviations) and Portuguese (0.462 standard deviations) scores in treatment schools. This result equals approximately one year of learning for the two subjects, when we compare the differences in scores between the final years of elementary school and high school.
The estimated impact for PEI is higher when compared to the impacts found in the evaluation of other programs across Latin America. The average impact estimated by
Our results are also higher than those identified by
In the compositional analysis, the results indicate that schools receiving the PEI had a lesser drop in the number of students and had a greater drop in the number of classes than control schools. These impacts are accompanied by a negative impact in the number of teachers per class and staff per student. The program also seems to attract a higher proportion of stu-dents with a higher socioeconomic profile, measured by the increase of the proportion of students with computers, TVs, and with highly educated mothers. However, this change in composition is not the driver of the es-timated effect, given that the impact evaluation method used controls for these observable characteristics. The drivers of the estimated effect may come from the unique characteristics of the PEI. For example, the regime of exclusive dedication for teachers can lead to an improvement in the quality of classes, since the teacher manages their time in only one school, not spending time commuting, and can improve their productivity. Besides this, the increase in salaries can be a stimulus to improve the quality of the classes. Another incentive is the internal evaluation, since teachers are evaluated by their own colleagues and by the students.
Still regarding teachers, although we controlled for training, we did not control for experience. A minimum of three years of experience is required to be a teacher at a school which receives PEI. Thus, part of the effect also comes from this characteristic. Furthermore, when estimating the effect of the program, we did not control for teacher turnover. In the program, teachers are assigned to the same school for at least three years, and there is an absence of teachers hired under temporary contracts. Thus, in future research it is worth investigating whether teacher tenure in the PEI school is longer, and whether this tenure has an impact on proficiency.
The diversified curriculum of the program, as well as the pedagogical proposal of placing the student as the protagonist, may also explain part of this estimated effect. This hypothesis is substantiated by the result of
Because of these characteristics, the effect of the PEI is believed to differ from the effect of other full-time school programs. In the estimated results we find no significant impact of the regular full-time school (school day of longer than 7 hours) on scores. This corroborates the literature, which argues that increasing the school day alone is not an effective policy for improving grades (
Although the results presented here are robust to different methodological specifications, some limitations are worth noticing. The first limitation is that we can’t explore the drivers of the result. Our results can be driven by several different factors (e.g., enhancement of teacher quality, curriculum changes, or pedagogical structure) that could not be disentangled. Second, non-observable factors that vary over time that affect students’ performance can bias our results. For example, if full-time schools attract more motivated students, and if this is not captured by our students’ socioeconomic covariates included in the model, our estimates can be biased upwards. The third limitation is that we used public data that doesn’t allow us to assign the treatment to the student. If this were possible, we could include students’ fixed effects and control for these non-observable factors, such as motivation. Also, we could explore in more detail the changes in the composition, perceiving if PEI attracted higher socioeconomic students and expelled disadvantaged students. Furthermore, it would be possible to estimate heterogeneous effects, perceiving if the enhancement in the test scores is driven by the students that were already top-performance before the program’s implementation. If the panel-data at the student level is available, further research can be dedicated to analyzing PEI heterogeneous effects and the gross changes in composition.
It is worth noting that the PEI difference from the other programs comes at a higher cost. While the average annual cost per student in a part-time state school is U$1393, and in a school that has the ETI is U$1474, the average annual cost of the PEI is U$1869, 34% higher than the part-time state schools average (
However, it is also important to highlight different benefits that the PEI can generate in the long term. Micro evidence shows that school performance has an impact on future wages, even controlled for years of schooling (
Finally, this paper contributes to the policy discussion in two aspects. First, it verifies the success of the policy, in terms of test scores, which can be replicated in other states or countries. Second, the finding that PEI attracted students of higher socioeconomic status, opens the discussion to the challenge of implementing policies that include, in its conception, individuals of low socioeconomic status.
Note: *p<0.1;**p<0.05;***p<0.01 Note: Standard deviations in parentheses. School-level variables. ***significance at the 1% level, **significance at the 5% level, *significance at the 10% level. All regressions include school fixed effects and time fixed effects.
Dependent variable:
mathematics scores
Portuguese scores
(Dif-Dif)
(Caliper)
(Dif-Dif)
(Caliper)
PEI
7.069***
7.830***
6.979***
8.406***
(0.838)
(1.327)
(0.903)
(1.374)
Prop. illiterate mothers
−26.764***
−7.975
−26.943***
−16.917
(4.100)
(13.195)
(4.419)
(13.654)
Prop. mothers with higher education
19.166***
17.190***
22.664***
20.670***
(2.142)
(6.654)
(2.309)
(6.886)
Prop. who work
−16.520***
−17.817***
−30.811***
−40.396***
(1.918)
(5.920)
(2.067)
(6.126)
Prop. with domestic worker
−17.054***
−4.070
−25.666***
−8.239
(2.259)
(7.195)
(2.435)
(7.446)
Prop. with Fridge
−0.070
6.335
−2.170
10.315
(4.153)
(12.604)
(4.476)
(13.043)
Prop. with TV
0.833
-6.973
8.733**
4.663
(3.778)
(11.872)
(4.072)
(12.285)
Prop. with Computer
18.897***
19.304***
17.148***
13.111***
(1.540)
(4.702)
(1.660)
(4.866)
Mean No. of bathrooms
3.534***
6.421**
2.541***
3.391
(0.879)
(2.749)
(0.947)
(2.845)
Prop. who use public transport
−1.500
−1.567
−2.265**
−4.907
(1.004)
(3.268)
(1.082)
(3.382)
Regular full-time
−0.031
−1.228
−0.450
−2.122
(0.743)
(1.484)
(0.801)
(1.536)
Prop. white
0.749
6.574
1.555
2.638
(1.082)
(4.063)
(1.166)
(4.204)
Prop. of men
−5.573**
−3.977
−20.258***
−8.953
(2.793)
(9.117)
(3.011)
(9.435)
Prop. at right age
−10.427***
−20.223**
−11.696***
−23.119***
(2.645)
(7.953)
(2.851)
(8.230)
Prop. who had only public school
14.493***
21.578***
18.928***
20.654***
(2.352)
(6.865)
(2.535)
7.104)
Prop. had already failed
−35.196***
−46.752***
−38.827***
−49.978***
(1.813)
(5.569)
(1.954)
(5.763)
Prop. had already dropout
−27.027***
−17.380
−23.798***
−22.952*
(3.758)
(11.700)
(4.051)
(12.107)
Prop. had late entrance
−48.570***
−44.965**
−53.593***
−47.372***
(5.440)
(17.710)
(5.863)
18.326)
Has computer lab.
−0.213
3.602
−0.722
1.141
(0.531)
(2.201)
(0.573
(2.277)
Has science lab.
0.068
−0.253
0.303
−0.497
(0.319)
(0.964)
(0.343)
(0.997)
Has library
−0.581*
1.564
−0.259
1.584
(0.304)
(1.036)
(0.328)
(1.072)
No. staff per student
2.459
−1.800
0.544
−5.286
(3.144)
(9.272)
(3.389)
(9.595)
Has sport court
0.497
−4.006
0.845
−3.322
(0.706)
(3.396)
(0.760)
(3.514)
No. classes
0.176***
0.255*
0.191***
0.225
(0.035)
(0.154)
(0.037)
(0.159)
Prop. teachers with graduation
0.092
8.836*
−0.963
2.501
(1.718)
(4.746)
(1.852)
(4.911)
No. teachers with graduation per student
−4.077
−31.461**
−0.734
−10.830
(5.861)
(14.002)
(6.317)
(14.489)
Observations
14,469
1,440
14,469
1,440
R 2
0.178
0.241
0.207
0.274
Before Matching
After Matching
Mean Treatment
Mean Controls
Mean difference
Mean Treatment
Mean Controls
Diff. Of means
% reduction
Distance
0.06
0.04
0.02
0.06
0.06
0.00
99.97
No. total students
364.30
506.48
-142.18
364.30
355.10
9.20
93.53
Prop. mothers with higher education
0.05
0.04
0.01
0.05
0.05
0.00
87.71
Prop. had already failed
0.14
0.15
-0.01
0.14
0.15
0.01
40.56
Has sport court
0.99
0.95
0.04
0.99
0.98
0.01
79.10
No. classes
11.40
14.71
-3.31
11.40
11.24
0.16
95.19
No. teachers with graduation
0.08
0.07
0.01
0.08
0.07
0.01
41.47
2172
33
1371
102
Sensitivity
Specificity
Precision
0.613
0.760
0.985
No. total students: number of students from 6th to 9th grade No. teachers: number of teachers from 6th to 9th grade No. staff: number of staff in school (includes teachers) No. classes: number of classes in school Prop. of men: proportion (from 0 to 1) of students from 6th to 9th grade who are male Prop. teachers with graduation: proportion (from 0 to 1) of teachers from 6th to 9th grade who have a specialization, or master’s or doctoral degree Prop. who use public transport: proportion (from 0 to 1) of students from 6th to 9th grade who use public transport to go to school Has sport court: dummy variable (0 or 1) indicating if the school has sport court Has computer lab.: dummy variable (0 or 1) indicating if the school has a computer laboratory Has science lab.: dummy variable (0 or 1) indicating if the school has a science laboratory Has library: dummy variable (0 or 1) indicating if the school has a library Regular full-time: dummy variable (0 or 1) indicating if the school has at least one final year elementary school class with a school day longer than 7 hours.
Prop. mothers with only high school: proportion (from 0 to 1) of ninth-graders students that report their mother (or the caregiver woman) has no school, or lower secondary school or complete high school Prop. mothers with higher education: proportion (from 0 to 1) of ninth-graders students that report their mother (or the caregiver woman) has completed higher education Prop. illiterate mothers: proportion (from 0 to 1) of ninth-graders students that report their mother (or the caregiver woman) is illiterate Prop. at right age: proportion (from 0 to 1) of students 15 years old or younger in the year of reference (the right age for 9th grade is 14). Prop. who work: proportion (from 0 to 1) of ninth-graders students that work outside the household Prop. had late entrance: proportion (from 0 to 1) of ninth-graders students that entered after the 1st year of elementary school Prop. who had only public school: proportion (from 0 to 1) of ninth-graders students who had studied only in public schools Prop. had already failed: proportion (from 0 to 1) of ninth-graders students who had already failed at least one time Prop. had already dropout: proportion (from 0 to 1) of ninth-graders students who had dropout school Prop. with Fridge: proportion (from 0 to 1) of ninth-graders students who have a fridge in the household Prop. with TV: proportion (from 0 to 1) of ninth-graders students who have a TV in the household Prop. with Computer: proportion (from 0 to 1) of ninth-graders students who have a computer in the household Prop. with domestic workers: proportion (from 0 to 1) of ninth-graders students who have domestic workers in the household Mean No. of bathrooms: average number of bathrooms in ninth-graders students’ households
According to PISA data (2018), the learning time per week in Brazil was 25.7 hours. The OECD average was 27.5; Chile, Peru, Mexico and the United States were 31.1, 30.4, 28.4 and 30.4, respectively.
A standard school in Brazil is part-time, where the length of a school-day ranges from 4 to 5 hours (
The upper elementary school corresponds with the sixth to ninth grade.
The average number of hours per week in upper elementary school in São Paulo state schools was 27.5 (School Census, 2017). ETI and PEI schools have on average, respectively, 35.4 hours and 43.3 hours per week (
All counts presented in
As already pointed out,
Since
Although recent literature shows that the standard two-way fixed effects linear regression specification is not robust to heterogeneous treatment effects (
Known as Granger’s causality test (
The sample standard deviation, calculated using the grade of all schools in all years of analysis, for mathematics and Portuguese are 15.09 and 15.11, respectively. In terms of SAEB scale standard deviation (45), the impacts would be 0.157 and 0.155 in mathematics and Portuguese, respectively.
When compared to the impact of lag 0, lag 1, and lag 2, the impact calculated for lead 1 is 1.2, 2.4 and 3.4 times smaller, respectively.
Method described by
This covariate was chosen due to the large discrepancy between the groups prior to treatment, described in
Further estimations were done with other probability of treatment models and there were no significant differences.
Given that
We calculate the sample standard deviation considering the schools that were not receiving PEIs, and considering only the odd years from 2009 through 2015.
In 2011, 9.5% of boys and 5% of girls aged 10 to 15 were working (
The covariates used are the same ones used in the differences-in-differences in Section 7.1.
The average differences in scores between high school and the final years of elementary school are 22.4 and 26.5 points for mathematics and Portuguese, respectively, which is equivalent to 7.45 and 8.83 points on average per year. The average difference was calculated using São Paulo state network and the SAEB editions of 2011, 2013, and 2015.
See
(
While the PEI covers an average of 338 students per school, the ETI covers an average of 212 students (
The values were converted using the exchange rate from December 30, 2016.
I21, I26, I28.