Estudios sobre la Economía Española - 2020/07

Gender Gap and Multiple Choice Exams in Public Selection Processes

J.I. Conde-Ruiz (Fedea and Universidad Complutense de Madrid)

J.J. Ganuza

(Universitat Pompeu Fabra and Barcelona GSE)

M. García

(Fedea and Universidad Complutense de Madrid)

fedea

Las opiniones recogidas en este documento son las de sus autores y no coinciden necesariamente con las de FEDEA.

J.I. Conde-Ruiz (Fedea and Universidad Complutense de Madrid)

J.J. Ganuza (Universitat Pompeu Fabra and Barcelona GSE)

M. García (Fedea and Universidad Complutense de Madrid)

Resumen No Técnico

Los exámenes tipo test (pruebas de elección múltiple) se utilizan comúnmente en el sector público en sus procedimientos de contratación y selección, así como en la entrada en algunas profesiones (abogados, médicos, profesores, etc.). Además, se consideran como una evaluación objetiva de los méritos que es clave para el sector público. El diseño de pruebas de opción múltiple es una tarea compleja. Una característica importante de este diseño es decidir la puntuación de las respuestas erróneas y omitidas. Normalmente, las respuestas erróneas se penalizan y las preguntas omitidas no. El argumento para penalizar las respuestas erróneas es evitar la suerte en la evaluación del conocimiento del participante. Sin embargo, penalizar las respuestas incorrectas también puede tener un impacto negativo en la evaluación de los examinandos con aversión al riesgo o con un menor grado de confianza. Existe una amplia evidencia empírica de que, en este tipo de pruebas, las mujeres son, en promedio, más adversas al riesgo y con menor grado de confianza que los hombres, y por lo tanto suelen omitir más preguntas cuando estas están penalizadas. Si las mujeres son más adversas al riesgo que los hombres y por lo tanto dejan sin contestar mas preguntas que los hombres, esto supone que, ceteris paribus, los hombres ocuparan las primeras posiciones en los resultados de este tipo de exámenes.

Si este comportamiento se observara en las pruebas de selección del sector publico, la introducción de la penalización en las respuestas erróneas reduciría el rendimiento medio de las mujeres y su contratación o promoción, simplemente porque arriesgan menos que los hombres. Esta es una advertencia importante, ya que este sesgo, de existir, podría aumentar la brecha entre los géneros en el sector público y puede ser un obstáculo importante en las políticas de contratación pública para atraer talentos.

Para ello analizamos el examen nacional MIR (“Médico Interno Residente”) del año 2019. Cada año el Ministerio de Sanidad español abre puestos en programas de formación de postgrado en más de 50 especialidades. Casi todos los puestos de trabajo para un licenciado en medicina en España requieren la realización de un postgrado en el hospital, y en consecuencia, la gran mayoría de los graduados realizan este examen MIR. Los candidatos a las plazas ofertadas por el Ministerio deben realizar este examen MIR nacional, que consiste en un test de elección múltiple con penalización por respuestas erróneas. Los graduados son clasificados en un ranking según un promedio ponderado de su puntuación en el examen tipo test MIR (un 90%) y sus notas en la carrera de medicina (un 10%). A continuación, los graduados eligen secuencialmente entre todas las posiciones de postgrado disponibles. El primer candidato en el ranking elige su posición de residencia preferida. Luego el segundo candidato en el ranking escoge entre las posiciones restantes. El proceso continúa hasta que todos los puestos están asignados. Por lo tanto, el resultado de la prueba MIR determina en gran medida la carrera profesional de los graduados médicos, y por este motivo los candidatos médicos invierten mucho tiempo, muchos de ellos asisten a escuelas especializadas, para prepararse para este examen MIR.

Nuestra contribución a esta literatura consiste en estudiar el rendimiento relativo por género en exámenes tupo test con penalización, en un entorno real en el que los examinandos tienen mucho en juego, como ocurre en la mayoría de los procesos de selección de funcionarios públicos. De acuerdo con esto, queremos inferir si las diferencias de comportamiento por género respecto a las preguntas omitidas que se encuentran en la literatura académica se mantienen cuando los examinandos están bien preparados para el test, pues se juegan gran parte de su futuro, y se preparan específicamente para este tipo de pruebas.

El resultado principal es que, en esta prueba de máxima importancia para los examinados donde se juegan todo su futuro en dicho examen, en contraposición con la literatura previa, encontramos una diferencia significativa en el comportamiento entre hombres y mujeres en cuanto a las preguntas omitidas muy inferior a lo encontrado en la literatura previa. En particular, controlando por otros factores, encontramos que las mujeres dejan en blanco tan solo un promedio de 0,82 preguntas más que los hombres de las 225 que incluye el examen MIR.

Nuestros resultados parecen sugerir, que cuando las mujeres se preparan intensamente y entrenan la prueba, aunque omiten más preguntas que los hombres, la brecha de género es mucho más pequeña que la encontrada en la literatura hasta la fecha. Por lo tanto, no hemos encontrado evidencia para cuestionar el uso de la prueba de elección múltiple con penalización en el proceso de selección pública de funcionarios.

La principal limitación de nuestro análisis es que no tenemos el contra factual de lo que habría pasado si la prueba se hiciera sin penalización por respuesta errónea. Sin embargo, nuestros resultados pueden sugerir que es muy poco probable que el rendimiento de las mujeres mejore significativamente si este tipo de prueba se realizara sin penalización. Esto se debe a que la diferencia de rendimiento entre los sexos es más significativa entre los participantes que obtienen mejores resultados y, que son, además, los que dejan sin contestar muy pocas preguntas. Por lo tanto, las mujeres que aumentarían el número de preguntas contestadas probablemente no serían las que obtienen mejores resultados en la prueba MIR.

J.I. Conde-Ruiz^a, J.J. Ganuza^b and M. García^a

February, 2020

Abstract

Multiple choice tests are commonly used by the public sector in their recruitment and selection procedures as well as in the regulation of entry for some professions (lawyers, physicians, etc.). Empirical and experimental literature has found evidence that females skip more questions on these tests undermining their performance. This bias could increase the gender gap in the public sector, and it can be an important caveat of the public recruitment policies for attracting talent. Using data of the Spanish “MIR (Médico Interno Residente)” national exam of 2019, we analyze if gender di↵erences in behavior arise in high-stakes tests, in which the outcome of the test has long term impact on the test takers careers. We find that when a female prepares intensively and trains for the test, although she skips more questions than men, the e↵ect is significantly smaller than in the previous literature. Moreover, we still find small di↵erences in the exam performance between men and female, this gender gap in performance is greater for the best candidates.

Keywords: Multiplechoice Test, Gender Gap, Competition, Tournaments.

JEL classification: D81, H30, H83, I20, J16

Corresponding author: nacho.conderuiz@gmail.com

^a Fedea and Universidad Complutense de Madrid; ^bUniversitat Pompeu Fabra and Barcelona GSE.

We acknowledge Pedro Rey, Nagore Iriberri, Walter García-Fontes, the editors and the two anonymous referees for useful comments and suggestions. Juan-José Ganuza gratefully acknowledges the support of the Barcelona GSE Research, the government of Catalonia, and the Spanish Ministry of Education and Science Through Project ECO2017-89240-P. Forthcoming at Hacienda Pblica Espaola /Review of Public Economics.

1 Introduction

Multiple choice tests are commonly used by the public sector in their recruitment and selection procedures as well as in the regulation of entry for some professions (lawyers, physicians, teachers, etc.).¹ These tests o↵er many advantages in terms of time and costs (especially when there is a large number of exam takers). Further, they are seen as an objective evaluation on merits which is key for the public sector. The design of multiple choice tests is a complex task. An important feature of this design is deciding the scoring of wrong and omitted answers. Typically, wrong answers are penalized, and omitted questions are not. The argument for penalizing wrong answers is to prevent guessing. However, penalizing wrong answers may have also a negative impact on the evaluation of risk averse and less confident test takers. As there is evidence that women are, on average, more risk averse and less confident than men,² they may skip more questions when wrong answers are penalized reducing their performance. This is an important warning, since this bias could increase the gender gap in the public sector and it can be an important caveat of the public recruitment policies for attracting talent.

There is a growing literature that investigates the relative performance of females and males on multiple choice tests. Ben-Shakhar and Sinai (1991) analyze PET tests in Israel, showing that females skip more questions. Similarly Ramos and Lambating (1996), Pekkari nen (2015), and Akyol, Key and Krishna (2016) provide observational evidence (mainly based on university entrance exams) about a clear gender gap for multiple choice tests. These papers di↵er in the empirical methodological approaches as well as in the magnitude of their findings; however, together they demonstrate that females skip more questions than males undermining their performance. Baldiga (2014) confirms this finding in an experi mental setting controlling for students’ knowledge. In a field experiment, Iriberri and Rey (2019b) goes further analyzing the risk aversion di↵erences between females and males, not only shows that using a di↵erential scoring rule for omitted questions and wrong answers has a significant negative impact on the gender gap in performance, also that females skip more questions even when wrong answers are not penalized and answering all questions is a dominant strategy.³

¹Multiple choice tests are common in Spain for civil servant selection procedures, they are also used for admission procedures in universities and other educational organizations (for example, Scholastic Aptitude Test (SAT) and Graduate Record Examination (GRE)), and for accreditation in many professions, such as the Bar exams for lawyers.

²See Iriberri and Rey-Biel (2019) for a more detailed discussion of the evidence that indicates that women are on average more risk averse and less confident than men

Our contribution to this literature is to study the relative performance by gender in a real setting in which test takers have very much at stake, as in most of the selection processes for public servants carried out by the public sector. According to this, we want to infer if the di↵erences in behavior by gender regarding omitted questions found in the literature hold when test-takers are well prepared for the test and invest in training.

We will analyze the Spanish “MIR (Médico Interno Residente)” national exam of 2019. Every year the Spanish Ministry of Health opens postgraduate training program positions in more than 50 specialties. Almost all of the jobs for a medical graduate in Spain require postgraduate work in hospital, and consequently, the vast majority of graduates take this exam. The matching process, between medical school graduates and residency training positions is regulated at the national level. The allocation mechanism is a serial dictatorship. The eligible candidates must take this national exam: a multiple choice test with penaliza tion for wrong answers. Graduates are then ranked according to a weighted average of their test score (90%) and their grades in the medical school (10%). Then, graduates sequentially choose among of all available residency training positions. The first candidate in the rank chooses his preferred residency training position. Then the second candidate in the rank chooses from the remaining positions. The process continues until all the positions are allocated. The best hospitals and the high-value specialities (as plastic surgery, dermatology or cardiology) are selected by the top ranked candidates. Therefore, the result of the test determines to a large extent the professional career of medical graduates, and explains that medical candidates invest a lot of time and many of them attend specialized schools to prepare for this exam.

³Iriberri and Rey (2019b) consider di↵erent scoring rules. In particular, they analyze a scoring rule that rewards omitted questions rather than penalizing wrong answers. Espinosa and Gardeazabal (2013) show that when test takers are risk averse this scoring rule lead to less omitted questions than penalizing wrong answer, although both approaches are equivalent under risk neutrality.

On this high-stakes test, we do not find a significant di↵erence in behavior between men and women regarding omitted questions. When females prepare intensively and train the test, although they skip more questions than men, the gender gap is small. We also find small di↵erences in performance on the exam between females and males, this gap is higher for the best candidates. Given that the MIR exam is a tournament with several awards, this finding may align with literature that shows and attempts to explain the underperformance of women in competitive environments.⁴ On this high-stakes for the candidates, the gender gap is significantly smaller than those found in the previous studies.

This paper is structured as follows. In the next section we present the data and descriptive statistics. Section 3 presents the main results, and Section 4 synthesizes our findings, situating them in prior results and o↵ering implications.

2 Data and Descriptive Statistics

The process to become a doctor in Spain is as follows⁵. After completing a six-year university degree, graduates need also a postgraduate specialization residency (MIR) in Hospitals of the National Health System, as a necessary step in order to be able to practice Medicine, in either public or private institutions. This process of postgraduate specialized health formation (i.e., the positions o↵ered to become a Resident in one of more than 50 medical specialties) in Spain is known as the MIR exam since MIR is the acronym designed for ”Médico Interno Residente” (i.e. ”Internal Resident Doctor”). For our analysis, we use data from the 2019 edition of the MIR exam. On this year, the Spanish Health System o↵ered 6,797 positions for medical graduates.

The selection procedure for access to specialty medical training is based on the consti tutional principles of equality, merit and capacity for public employment. The selection process is based on a test of knowledge (using a multiple choice exam), which accounts for 90% in the final score, and the candidates’ academic performance in the medical school, which serves as the remaining 10%. The exam is a five-hour test consisting of 225 questions with each having four options to select and only one ”correct” answer. The multiple-choice score is obtained by the sum of each valid answer, that receives a value of three points, and one point is subtracted for each of the incorrect answers. Omitted questions are not assessed. The individual total score of each applicant on the test is calculated from the sum of the score obtained on multiple-choice and the score given for academic merits. The significant weight of the multiple-choice exam on the final result of the MIR highlights the importance of the exam, where the candidates are, in five hours, deciding practically all their professional trajectory.

⁴See, for example, Gneezy, Niederle, and Rustichini (2003) and Iriberri and Rey (2019).

⁵See García-Estañ (2018) for further details.

In our data set, described in Table 1, we have the following information for each candidate: i) academic record, ii) exam performance, and the iii) number of right, wrong, and omitted questions on the test. We classify students by gender according to their first name. For this purpose, we rely on three di↵erent databases as in Beneito, Boscá, Ferri and García (2018): the first-names database published by the U.S. Social Security Administration, created using data from Social Security card applications, the database constructed by Tang et al. (2011), who use Facebook to collect data on first names and self-reported gender, and finally the names database developed by Bagues and Campa (2018). Any candidate who (a) falls within the [0.05 0.95] probability interval of being male/female or (b) cannot be found in any of the databases are excluded.⁶ In 2019, 14,187 medical graduates took the MIR exam.⁷ In our analysis we have decided to remove two groups of candidates because they have a number of places assigned to them (a specific quota) and, therefore, they are not competing directly with the rest of candidates. Firstly, the disabled group. The regulations in force require that 7% of the specialized health training places o↵ered be reserved for people with disabilities. In 2019 there were 476 places reserved for the disabled. Secondly, doctors without a residence permit. This year the Ministry of Health has set a quota of 272 places for those admitted from outside the EU, which represents 4% of the total supply. However, in our database we include those foreigners admitted to take the exam on an equal terms with the local Spanish candidates. That is, those foreign doctors who present one of the following situations: community regime, permanent residence, or temporary residence. In total, 2,076 doctor immigrants took the MIR exam in 2019 under the same conditions as the Spanish students.

⁶After using the three databases, the number of non-classified students is 408 (2.6%).

⁷There were 15,519 registered students for MIR exam, but only 14,187 took it.

Table 1. Descriptive Statistics for the Complete Sample Notes: The p-value are for the F-test of equality of variable means across gender.

In our final sample we have 11,695 graduates who have taken the MIR exam in 2019, of which 64.35% are women and 35.65% are men. Table 1, shows the average performance on the exam and the academic record by gender. Both genders have a very similar average academic record (1.88 for women versus 1.89 for men), while on average men perform slightly better, a 1.37% , than women on the exam. In the top 10%, the exam performance gender gaps are more substantial. The academic record is constructed with the average grade of all the courses taken in the undergraduate degree⁸.

Regarding the number of question omitted, participants skip a very small number of questions. Women on average omit 10.25 questions (out of 225) while men leave the exam on average with no answer for 9.62 questions. The average gender gaps are also very small for the right and wrong answers. Women, on average, obtained 129.97 correct questions and 84.79 incorrect questions. While, men, on average, obtained 131.17 correct answers and 84.21 incorrect answers. If we examine those scoring in the top 10%, we can see how the main di↵erences between men and women increase. Specifically, men, on average, answered 152.77 questions correctly, while women answered 149.22. Gender di↵erences for incorrect answers (67.04 versus 65.83) or in omitted questions (8.74 versus 6.40) are also larger.

⁸It is important to keep in mind that the grades of each subject have a discrete value: i) C (i.e. ”Aprobado” is 1 point; ii) B, (i.e. ”Notable”) are 2 points; iii) A (i.e. ”Sobresaliente”) are 3 points and iv) A+ (i.e. ”Matrícula de Honor”) are 4 points.

Density distribution functions of performance and academic record by gender for the complete sample are shown in Figure 1. Men stand out in the tails of the distribution. That is, there are more men among those who have very low and very high performance. The same happens with the distribution by result of academic records, although perhaps less sharpened in the tails⁹.

Figure 1. Density distributions for Exam (a) MIR Performance and (b) Academic Record by Sex. (a) MIR Performance

(b) Academic Record Notes: The vertical line represents the 90th percentile in the distribution.

We will also analyze the distribution of omitted questions by gender. As we have highlighted in the introduction, there is a growing academic literature that shows how multiple choice tests generate gender gaps in performance. Omitted questions, on a multiple choice exam with penalty, have been identified as the key mechanism behind this fact. Women have di↵erent attitudes than men towards risk; they are more risk averse, and they omit more questions.

⁹In the Appendix we show the same distributions but for the top 10% MIR performance candidates.

Figure 2 shows the distribution of omitted questions by gender. We found two interesting facts. First, both men and women leave few questions omitted and a high percentage answer all questions. Second, there are more men who answer all the questions than women. Specifically, 30.3% of women and 36.4% of men answer all questions. These di↵erences are also observed when we analyze the distributions of the 10% that best performs the MIR exam (see Figure 2). To interpret this result, it is important to take into account that the penalization system of the MIR exam does not penalize guessing for risk-neutral exam takers. The expected outcome of guessing is equivalent to skip the question, since with probability 3/4 - the answer is wrong)- and one point is discounted; and with probability 1/4 -the answer is right- and three point are added. In addition, for many questions is very easy to identify a clearly wrong answer and then guessing would be a dominant strategy for a risk-neutral test taker. This explains why schools that help the candidates to prepare for the exam strongly advise against skipping questions.

It is also interesting to analyze the percentage of women and men in each decile of both the distribution of academic records and the distribution of the MIR result. Figure 3 shows how women are underrepresented in the first and last decile in the distribution of academic records. And, with respect to the distribution of MIR results, women are underrepresented in the first two and last two deciles. It is noteworthy that among the 10% with the best MIR score there is practically parity between men and women, when 63% of women and 37% of men took the MIR exam.

Finally, the relationship between the omitted questions regarding the academic record and the result of the MIR is summarized in Figure 4. We see how there is a direct relationship between the omitted questions and the result of the MIR: the better is the result, the lower is the number of omitted questions. However, this relationship is not linear when we relate the omitted questions and the academic record. We find, for example, that the decile with higher academic records, on average, leaves more questions omitted than the previous decile, with worse academic records (mainly in the case of women).

Figure 2. Density distribution of Omitted Questions. (a) Complete sample vs (b) top 10% MIR performance students. (a) Complete sample

(b) Top 10% MIR performance Notes: The vertical line represents the mean of the distribution.

Figure 3. Proportion of men and women by deciles. (a) MIR Exam Performance vs (b) Academic Record. (a) MIR Exam Performance

(b) Academic Record Notes: The horizontal line represents the mean of the distribution.

Figure 4. Average Number of omitted by Sex. (a) MIR Exam Performance deciles vs (b) Academic Record deciles. (a) MIR Exam Performance

(b) Academic Record

3 Main Results

First, we examine to what extent men perform better on the MIR exam when compared to women. We standardised values with mean zero and standard deviation (SD) of . The main estimates are presented in Table 2. Columns (1)-(3) show regressions where the dependent variable is the result of the MIR test and the independent variable is a dummy for gender (Female). Column (1) does not include any control or explanatory variable beyond the gender. Column (2) includes the academic record as control and Column (3) also adds a new dummy variable for nationality (National Origin). In the three specifications of Table 1 we can see how female participants underperform compared to male participants. In the column (3) model, when we control for nationality, the Female coecient is significant, ¹⁰In the Appendix we replicate the main results with non standardised test scores.

with a gender gap of 5.7% of one standard deviation (SD) of the MIR exam score. It is also important to point out that native participants perform better in the MIR exam than foreign participants¹¹. This gender gap, being very statistically significant, is much smaller than other gaps estimated in the literature for similar models. In particular, Iriberri and Rey-Biel (2019) estimates the gender gap in the results of a mathematics multiple-choice test organized by the Region of Madrid. Controlling for academic record, they find a gender gap six times larger than we found here.

Table 2. Gender Di↵erential in Performance

* p<0.10 ** p<0.05 *** p<0.01. Robust Standard Errors.

We then explore how female participants perform as compared to male participants, among participants who perform better on the MIR exam (i.e. the top 10%). Table 3 complements our analysis by examining the performance of the MIR exam throughout the distribution of results. Gender gaps are estimated using quantile regression techniques, which allows us to estimate the gender gap at various points (percentiles) in the results distribution to compare the participants’ gender gaps according to where they are located in the distribution¹². Table 3 shows how di↵erentials in performance on the MIR exam between men and women are higher at the top of the distribution than at the middle or bottom¹³. One possible explanation is that at the top results level, there is greater competitive pressure to achieve the best available seats. And, as shown by the seminal work of Gneezy, Niederle and Rustichini (2003), women under-perform relative to men in competitive environments¹⁴. Iriberri and Rey-Biel (2019) also found that, when the competitive pressure increases, women do worse on multiple choice exams than men.

¹¹The non native (foreign) participant has on average 18.12 omitted answers versus 8.70 for natives; and 106,46 right questions versus 134.31 for natives participants. In terms of gender gaps, the male non natives have 2.36 more right questions than female non natives (107.90 versus 105.54), while performance gender gap for natives is 1.69 (135.41 correct answers versus 133.72). Similarly, non natives males have omitted 2.63 questions less than female not natives (16.52 versus 19.15), while the omitted questions gender gap fo natives is - 0.52 (8.36 versus 8.88)

¹²Quantile regressions estimate the impact of changes in control variables on percentiles specific to the

Table 3. Gender Di↵erential in Performance by quantiles

* p<0.10 ** p<0.05 *** p<0.01.

Table 4 presents the results of similar estimations but with di↵erent dependent variables: i) the number of omitted answers (column (1)); ii) the number of correct answers (column (2)); and iii) the number of wrong answers (column (3)). According to this results, female underperformance could be explained by an increase of the number of omitted answers, a decrease in the number of right answers, and an increase in wrong answers. As in previous results, gender gaps are all significant, but small in practical size. In particular, controlling for other factors, women leave on average 0.82¹⁵ more questions unanswered than men, out of the 225 included in the MIR exam (i.e. 0.36% of the total questions).

Table 5 shows the same estimates of Table 4 but for the 10% that obtained the best results on the MIR examination. We see nearly the same results as the whole sample, but when we explain the omitted questions, the gender gap remains significant, although smaller. Notice that if we consider the participants who are in the 10% that best did the MIR exam, and we look at the result of the estimation in column (1) in Table 5, the Female control variable is three times smaller to explain the number of omitted questions than what was obtained in the estimation with the complete sample (column (1) in Table 5). That is, we have seen how the better the participants are (i.e. those that obtain a better result on the MIR exam) the more evident female underperformance become, as compared to male counterparts yet the role of gender in explaining omitted questions is lower. Therefore, this result rises doubts about what are the driving forces behind the performance of female participants, and about their attribution to the design of the test.

dependent variable, just as the estimation of ordinary least squares measures the e↵ect of changes in control variables on the mean of the dependent variable. Therefore, they allow the relationship between the dependent variable and the control variables to di↵er throughout the distribution of results.

¹³In the Appendix we have the graphical representation of the quantile regression process.

¹⁴There is extensive academic literature showing how women perform worse than men when competitive pressures increase. Among others we can highlight Jurajda and M¨unich (2011), Ors, Palomino and Peyrache (2013) or Buser, Niederle and Oosterbeek (2014).

¹⁵See table A.3 in the Appendix for non standardize gender di↵erential.

Table 4. Gender Di↵erential: Omitted, Right and Wrong questions

* p<0.10 ** p<0.05 *** p<0.01. Robust Standard Errors.

Moreover, it is important to point out that the behavior of the top 10 per cent of test takers can be driven by the goals that they pursue. Some of the most competitive medical specializations in terms of admissions scores tend to be very male dominated (e.g. surgery). Then, if females, for preferences, focus on less demanding specialization they would have less incentive to struggle for the top scores and are likely to take less risks on the exam. We further discuss the role that heterogeneity in goals may have on gender performance gaps in conclusions, discussing this interesting avenue for future research.

As is stated previously, there is a growing academic literature that shows that women participants omit more questions when there is a penalty for wrong answers. In particular, recently Iriberri and Rey-Biel (2019b) analyze these hypotheses -Brave Boys and Play-it-Safe Girls-, and they find that female participants omit more questions when there is a reward for omitted questions. They also showed that this gender di↵erence, which is stronger among high ability participants, hurts females for final scores and rankings. Contrary to Iriberry and Rey-Biel (2019b) and most of the previous literature¹⁶, we have found that, although that there are statistical significant di↵erences in number of omitted questions between male and female, the di↵erence is so small that the omitted questions are unlikely to be behind the underperformance of females in this high-stake exam, in which the outcome of the test has long term impact on the graduate careers.

Table 5. Gender Di↵erential Top 10% MIR Exam Performance: Omitted, Right and Wrong questions

* p<0.10 ** p<0.05 *** p<0.01. Robust Standard Errors.

Funk and Perrone (2016) and Akyol, Key and Krishna (2016) also found that the e↵ect of omitted questions in multiple choice exam has very little or no e↵ect over the performance in the exam. In particular, Funk and Perrone (2016) argue that a potential explanation of their result is that the exam they analyze is not high-stake exam. Our analysis goes against this hypotheses by showing the same results in a setting with higher stake than the one they consider.

¹⁶For example, Co↵man and Klinowski (2019) found that in the national college entry exam in Chile women skip significantly more questions than men on average in a text with penalization. They also report that when penalty was remove from the test for the new cohort of participants, skipped questions almost disappear and the gender gap was reduced by approximately 70 percent.

These previous academic papers challenged the use of multiple-choice tests as an ecient mechanism for sta↵ selection in public administrations. Multiple-choice exams are used in practically all competitive examinations to achieve a civil servant position at any level of the public administrations in Spain. In this paper, using evidence from an exam where doctors pin all to test their professional aspirations or careers, and unlike these authors, we find that the MIR exam’s multiple-choice design does not explain that female participants have a worse result than men. Or at least, we found that the omitted questions can not explain the performance gender gap of the MIR exam.

4 Conclusions

This paper has shown that there are not significant di↵erences in the behavior between male and females when they take multiple choice high-stakes tests. Previous literature has provided evidence that women skip more questions because of risk aversion and lack of confidence. On the contrary, we have shown that when women prepare and train well for the test because their professional future is at stake, the number of unanswered questions with respect to men is very small and has little impact over the final outcome of the MIR exam. Therefore, we have not found enough evidence to question the use of the multiple choice test with penalty in the public selection process of ocials.

One limitation of our analysis is that we don’t have the contrafactual of what would have happened if the test were done without penalty. However, our results may suggest that it is very unlikely that the performance of the females improve significantly without penalization in this tournament test setting. This is because the performance gender gap is more significant among the best test takers that they skip very few questions. Therefore, the females that would increase the number of questions answered are likely not to be the high performance ones.

Our result, we have analyzed the Spanish “MIR (Médico Interno Residente)” national exam of 2017. This is a very unique data base for the features of the exam and the extremely high stakes for test takers. In the future, we plan to continue working with this interesting data to overcame some limitations of the present paper. For example, in this test with very high stake, risk aversion should play a very important role on shaping the behavior of participants. Therefore, it would be interesting to introduce some measure of risk aversion to investigate how much of the observed gender gap can be explained by di↵erences on risk aversion between males and females as other papers in the literature have done. A quite related point is to analyze how goals determine the risk attitude of test takers. As we have explained in the main text, the MIR is a tournament with heterogeneous awards (not all test takers rank the awards in the same way). Then, candidates that pursue a very demanding position may be are willing to take more risks (answering most of the questions) than others that pursue a less demanding position, that may tend to have a more conservative behaviour. This is because, the first ones only get the position if they get a very high score in the exam and the second ones only don’t get the position if they get a very low score. The MIR test may be an interesting setting for understanding how the behavior of test takers change depending on how dicult and competitive is the goal they pursue. In addition, it would be important to investigate gender di↵erences in goals they pursue and using this insight to better understand the gender gaps reported in this paper.

References

1. Akyol, S.P., Key, J. and Krishna, K. (2016) ”Hit or Miss? Test Taking Behavior in Multiple Choice ExamffiNBER Working Paper Nr. 22401.
2. Anderson, J. (1989) ”Sex-related Di↵erences on Objective Tests among Undergraduates” Educational Studies in Mathematics, 20: 165-177.
3. Bagues, M., Campa, P. (2018). ”Can Gender Quotas in Candidate Lists Empower Women? Evidence from a Regression Discontinuity Design”. CEPR Discussion Paper No. 12149
4. Baldiga, K., (2014) ”Gender Di↵ererences in Willingness to Guess” Management Science 60(2) 434-448.
5. Ben-Shakhar, G. and Sinai, Y. (1991). Gender di↵erences in multiple-choice tests: The role of di↵erential guessing tendencies. Journal of Educational Measurement, 28(1), 23-35.
6. Beneito, P., Boscá, J.E., Ferri, J. and García, M., 2018. ”Women across Subfields in Economics: Relative Performance and Beliefs” Working Papers 2018-06, FEDEA.
7. Buser, T., M. Niederle and H. Oosterbeek (2014) ”Gender, Competitiveness and Career Choices”. The Quarterly Journal of Economics, 129(3): 1409-1447.
8. Co↵man, K.B. and D. Klinowski (2019) ” The Impact of Penalties for Wrong Answers on the Gender Gap in Test Scores”, Harvard Business School Working Paper 19-017.
9. Espinosa, M. P., and J. Gardeazabal (2013) ”Do Students Behave Rationally in Multiple Choice Tests? Evidence from a Field Experiment.” Journal of Economics and Management, 9(2): 107-135.
10. Funk, P. and Perrone, H. (2016) ”Gender Di↵erences in Academic Performance: The Role of Negative Marking in Multiple-Choice Exams” Working Paper.
11. García-Estañ (2018) ”Studying Medicine and being a doctor in Spain” AMEE MedEd-Publish, Version 1 (7 December 2018), www.mededpublish.org
12. Gneezy, U., M. Niederle and A. Rustichini (2003) ”Performance in Competitive Environments: Gender Di↵erences” The Quarterly Journal of Economics, 118 (3):1049- 1074.
13. Iriberri, N. and Rey-Biel, Pedro (2019). ”Competitive pressure widens the gender gap in performance: Evidence from a two-stage competition in mathematics”. The Economic Journal, 129 (620), pp. 1863-1893.
14. Iriberri, N. and Rey-Biel, Pedro (2019b). ”Brave Boys and Play-it-Safe Girls: Gender Di↵erences in Willingness to Guess in a Large Scale Natural Field Experiment”, mimeo.
15. Jurajda, S., and D. M¨unich (2011) ”Gender Gap in Performance under Competitive Pressure: Admissions to Czech Universities” American Economic Review Papers and Proceedings, 101(3): 514-18.
16. Ors, E., F. Palomino and E. Peyrache (2013) ”Performance Gender Gap: Does Compe- tition Matter?” Journal of Labor Economics, 31(3): 443-499.
17. Pekkarinen, T. (2015) ”Gender di↵erences in behaviour under competitive pressure : Evidence on omission patterns in university entrance examinations” Journal of Economic Behavior and Organization Vol. 115, 07.2015, p. 94-110.
18. Ramos, I. and Lambating, J. (1996). ”Gender Di↵erence in Risk-Taking Behavior and their Relationship to SAT-Mathematics Performance.” School Science and Mathematics, 96(4): 202-207.
19. Swineford, F. (1941) ”Analysis of a Personality Trait” Journal of Educational Psychology, 45: 81-90.
20. Tang, C, Ross K., Saxena, N. and Chen, R. (2011). ”What’s in a name: a study of names, gender inference, and gender behavior in facebook”. Database Systems for Advanced Applications, 344-356.
21. Tannenbaum, D. (2012. ”Do Gender Di↵erences in Risk Aversion Explain the Gender Gap in SAT Scores? Uncovering Risk Attitudes and the Test Score Gap.” Mimeo, University of Chicago.

Appendix

Figure A.1. Density distributions for Exam Performance (left) and Academic Record Performance (right) by Sex. Top 10% Exam Performance Students Notes: The vertical line represents the mean of the distribution.

Figure A.2. Quantile Regression Process representation.

Academic_Perf

Table A.1. Non Standardize Gender Di↵erential in Performance

* p<0.10 ** p<0.05 *** p<0.01.

Table A.2. Non Standardize Gender Di↵erential in Performance by quantiles

* p<0.10 ** p<0.05 *** p<0.01.

Table A.3. Non Standardize Gender Di↵erential: Omitted, Right and Wrong questions

* p<0.10 ** p<0.05 *** p<0.01.

Table A.4. Non Standardize Gender Di↵erential Top 10% MIR Exam Performance: Omitted, Right and Wrong questions

* p<0.10 ** p<0.05 *** p<0.01.