arxiv: v1 [cs.cl] 8 Feb 2021

(1)

How True is GPT-2?

An Empirical Analysis of Intersectional Occupational Biases

Hannah Kirk* 1 2 _{Yennie Jun}* 1 2 _{Haider Iqbal}* 1 3 _{Elias Benussi}* 1 3 _{Filippo Volpin}* 1 4 _{Fr´ed´eric A. Dreyer}* 1 5

Aleksandar Shtedritski1 6 Yuki M. Asano1 6

Abstract

The capabilities of natural language models trained on large-scale data have increased im-mensely over the past few years. Downstream applications are at risk of inheriting biases con-tained in these models, with potential negative consequences especially for marginalized groups. In this paper, we analyze the occupational biases of a popular generative language model, GPT-2, intersecting gender with five protected categories: religion, sexuality, ethnicity, political affiliation, and name origin. Using a novel data collection pipeline we collect 396k sentence completions of GPT-2 and find: (i) The machine-predicted jobs are less diverse and more stereotypical for women than for men, especially for intersections; (ii) Fitting 262 logistic models shows intersec-tional interactions to be highly relevant for occu-pational associations; (iii) For a given job, GPT-2 reflects the societal skew of gender and ethnicity in the US, and in some cases, pulls the distribu-tion towards gender parity, raising the normative question of what language models should learn. Code is available athttps://github.com/

oxai/intersectional_gpt2.

1. Introduction

The advent of deep learning and massive growth in training data have led to natural language models surpassing humans on numerous benchmarks (Wang et al.,2018;2019;He et al., 2020;Adiwardana et al.,2020). However, asBender et al. (2021) states, these models can exacerbate existing biases in data and perpetuate stereotypical associations to the harm of marginalized communities. Simultaneously, pre-trained models have become readily accessible via APIs, allowing non-experts to apply these tools in their own applications.

*

Equal contribution 1Oxford Artificial Intelligence Society

2

Oxford Internet Institute, 3Dept. of Computer Science, 4Dept.

of Economics, 5Dept. of Physics, 6Dept. of Engineering Science,

University of Oxford, Oxford, United Kingdom. Correspondence to: Hannah Kirk <[email protected]>.

Preprint.

0.0 0.2 0.4 0.6 0.8 1.0

Share of Women (GPT-2 pred.) 0.0 0.2 0.4 0.6 0.8 1.0

Share of Women (US 2019 data)

babysitter bus driver carpenter chef cook laborer maid mechanic model nurse photographer plumber receptionist secretary security guard social worker teacher truck driver Woman [0.0238] +Asian [0.0004] +Black [0.0013] +Hispanic [0.0049] +White [0.0182]

Figure 1. GPT-2 Monte-Carlo prediction vs ground truth US population share. GPT-2’s predictions with regards to intersec-tional characteristics are highly stereotypical – yet they are closely aligned to the US population data. We show the predicted values for gender intersected with ethnicity along with the [Mean-Squared Errors] and annotate example jobs for the gender-only predictions.

These developments in generative language models substan-tiate a need to understand the potential for biases towards protected classes, such as gender and ethinicity. Within the specific context of fairness in AI-assisted hiring, we focus on analyzing one popular, publicly available language model (GPT-2). By generating 396k samples, we empirically an-alyze which occupations GPT-2 preferentially associates with intersections of gender and protected classes. The paper provides the following contributions:

• a detailed data collection protocol for studying inter-sectional biases of generative language models, • the analysis of the intersectional biases of (gender ×

[ethnicity, religion, sexuality, political affiliation, name origin]) present in GPT-2,

• a comparison of GPT-2’s predictions with real-world occupation frequencies, as shown in Fig.1.

(2)

Asian Black Hispanic White Ethnicity Buddhist Christian Hindu Jewish Muslim Religion Gay/Lesbian Straight Sexuality Conservative Liberal Political Africa Americas Asia Europe Oceania Continent X Y Man Woman Base GPT-2 NER The works as a ...X Y Z works as a ... Z {NAME} Man Woman Janitor Writer Teacher ... ... Man Woman Janitor Writer Teacher ... ... Man Woman Janitor Writer Teacher ... ... Man Woman Janitor Writer Teacher ... ... Man Woman Janitor Writer Teacher ... ... Frequency for orX Y Z

Figure 2. Data Collection Process. We collect 396,000 responses from GPT-2, and retrieve “titles” via Stanford CoreNLP’s Named Entity Recognition (NER) to analyze the predicted occupational distribution for various intersectional categories.

2. Related Work

Bias in classical NLP models. Extensive research has shown that unrestricted training of natural language mod-els inherits human biases and, in some cases, amplifies them. (Bolukbasi et al.,2016;Caliskan et al.,2017;Zhao et al.,2018;Gonen & Goldberg,2019). Negative generaliza-tions, stereotypes, or misrepresentations of particular social groups can be learned by generative language models ( Blod-gett et al.,2020). These learned associations can persist in tasks such as machine translation (Stanovsky et al.,2019), dialogue systems (Liu et al.,2020), and hate speech classi-fiers (Kennedy et al.,2020). There have been many attempts to identify, quantify, and de-bias context-independent word embeddings such as word2vec and GloVe (Bolukbasi et al., 2016;Zhao et al.,2019;Diaz et al.,2018). While these earlier works investigate low-capacity models, we analyze a high-capacity transformer-based generative language model, GPT-2 (Vaswani et al.,2017;Radford et al.,2019). Bias in generative language models. Transformer-based language models are increasingly replacing approaches us-ing static word embeddus-ings and current state-of-the-art lan-guage models are trained on immense amounts of text col-lected from the Internet (Fedus et al.,2021;Radford et al., 2019;Brown et al.,2020). The vastly improved model per-formance and capacity has motivated many downstream ap-plications, such as dialogue generation (Dinan et al.,2020), automatically generated video captions (Tatman,2017), and job application assessment (Bhatia et al.,2019; Li et al., 2020). In terms of models, GPT-2, a transformer-based lan-guage model trained on 40GB of text from over 8 million documents from the Internet, has been the subject of various studies (Wick et al.,2020;Budzianowski & Vulic,2019; Ethayarajh,2019).

Researchers have attempted to quantify and mitigate

bi-ases in generative language models:Zhao et al.(2019) find systematic gender biases in ELMo’s contextualized word vectors andBhardwaj et al.(2020) measure the ways in which gender bias in BERT affects emotion and sentiment intensity.Kurita et al.(2019) quantify gender bias in BERT by measuring gendered occupations and adjectives, while Sheng et al.(2019) utilize template-based methods to quan-tify gender bias in texts generated by GPT-2, with sentiment scores as a proxy for bias. Building on these methodologies, we focus on intersectional biases of GPT-2 with regards to the domain of occupations.

Intersectional biases. As Crenshaw(1989) explains, in-tersectional biases are a necessary consideration because a single axis of analysis treating gender and race as mutually exclusive categories distorts the reality of marginalized com-munities (such as Black women). More recently,Foulds & Pan(2020) provides definitions of fairness in machine learn-ing systems informed by the framework of intersectionality. The intersections between gender and racial biases have been studied in sentiment analysis (Kiritchenko & Moham-mad,2018) and generative language models such as BERT and GPT-2 (Tan & Celis,2019). As well as race and gender, we extend our analysis to intersections with other legally protected categories that have historically been subject to discrimination: religion, sexuality, and political affiliation.

3. Methods

3.1. Data collection

In order to analyze the potential impact of biases in applica-tions of generative language models, we focus on publicly available models. Among these, we focus on GPT-21_{as it}

1

We applied for research access to GPT-3 (in Oct. 2020) but have received no response from OpenAI.

(3)

How True is GPT-2? An Empirical Analysis of Intersectional Occupational Biases

Table 1. Summary table of data collection showing the number of calls per category and per variant (Var) in addition to thresholds (θ) used for analysis. The cumulative sum of all calls is 396,000.

Category Var Calls Total θ∗ Cum.

Calls Share Base 2 7,000 14,000 35 81% Ethnicity 8 7,000 56,000 140 82% Religion 10 7,000 70,000 175 84% Sexuality 4 7,000 28,000 70 83% Political 4 7,000 28,000 70 82% Continent 200 1,000 200,000 500 76%

Notes: ∗Threshold (θ) for plots to exclude 0.25%

tails of infrequently mentioned jobs

is one of the most widely recognized generative language models. To proxy for the most popular model version, we use the 124M hyperparameter model as it was the most-downloaded version of GPT-2 available on HuggingFace (see Appendix A, Tab. 1) ). We also confirm that our results hold true for another generative model, XLNet (Yang et al., 2019) (see Appendix B).

Our data collection pipeline is shown in Fig.2. We prompt GPT-2 with prefix templates, as defined bySheng et al. (2019). The template is “The [X][Y ] works as a”, where X is one of the following protected classes: ethnicity, reli-gion, sexuality, and political affiliation, and Y is ‘man’ or ‘woman’. For a baseline to intersectional effects, we leave X blank (i.e. “The man/woman works as a”). From these 28 unique templates (Tab.1), we generate 7,000 sentences using GPT-2 (Radford et al.,2019) through the Hugging-Face API with a top-k parameter of 3 and model size ‘small’. Generated sentences are limited to a maximum length of 10 words to capture immediate occupation associations. Name-based templates. An additional prefix template is created of the form “[Z] works as a”, where Z is a name sampled from the most popular male and female first names per country, obtained from Wikipedia.2 We aggregate the names into five geographic groups: Africa, Americas, Asia, Europe, Oceania. We sample 20 names for each geographic group and gender pair, yielding 200 unique templates, from which we generate 1,000 sentences each. By prompting GPT-2 with a template devoid of inherently gendered or racialized terms, such as ‘man/woman’ or ‘Asian/Black’, we can better examine the latent associations when GPT-2 estimates the ethnicity and gender from first names. Occupation entity recognition. For each generated sen-tence, we use the Stanford CoreNLP Named Entity Recog-nizer (NER) (Manning et al.,2014) to extract “job titles.”3

2_{https://en.wikipedia.org/wiki/List_of_}

most_popular_given_names

3_{https://stanfordnlp.github.io/CoreNLP/}

ner.html

NER was unable to detect titles for some sentences which were removed from the dataset, losing 10.6% of gender-occupation sentences and 19.6% of name-gender-occupation sen-tences.4 _{We then create a one-hot encoded frequency}

ma-trix for returned job tokens, combining duplicate jobs (e.g. nurse/nurse practitioner). However, we do not merge job tokens with inherent hierarchies (e.g. assistant profes-sor/professor) or implicit gender associations (e.g. sales-man/salesperson, waitress/waiter). Sentences returning mul-tiple titles (e.g. “The woman works as a waitress and a maid”) were treated as two separate entries in the frequency matrix given that individuals can have more than one job.

3.2. Empirical Analysis

The distribution of returned jobs is highly-skewed with long tails: a few jobs comprise a significant share and many jobs are mentioned infrequently. Therefore, we apply a lower-bound threshold to focus our analysis, removing tokens men-tioned in fewer than 0.25% of total calls which preserves approximately 80% of the sample (Tab.1). For jobs above the threshold, we run a logistic regression on the one-hot matrix and output frequencies to predict p([job] = 1|X, Y ) for the input “The [X][Y ] works as a [job]”. While GPT-2 is a ‘black-box’ model, this predictive modelling attempts to estimate how intersectional categories change GPT-2’s prior on the probability of job associations. By using in-teraction terms, we can study whether intersectionality has additional influence beyond main effects (e.g. the isolated effect of gender and ethnicity). The logistic regression equa-tion includes ‘man’ from the baseline case as the reference group, with dummy variables added for woman, for each intersectional category C, and for interaction terms:

log odds(p(jobi|c)) = β0+ β1Womani+ (1) C X c=1 γicCategoryic + C X c=1 δicCategoryic∗ Womani+ i,

where log odds(p) = log(p/(1 − p)) is the log-odds ratio.

4. Results

We analyze the effect of gender on returned occupational distributions in Sec 4.1and on particular occupations in Sec4.2. We extend these analyses to intersectional associa-tions in Sec4.3and present empirical results derived from logistic regressions in Sec4.4. Finally, we compare and quantify the predicted distributions against real-world US occupational data in Sec4.5.

4

For the names-occupation template, we removed 2000 sen-tences with the job title ‘Princess’ for the African name ‘Princess’.

(4)

100 ₁₀1 ₁₀2 ₁₀3 Log(Rank) 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Share of Total

16 jobs account for 50% of men 8 jobs account for 50% of women

Women (Average) Men (Average)

Figure 3. GPT-2 occupational stereotyping. GPT-2 stereotypes the occupational distribution of women more than that of men.

4.1. Gender differences in distributions

Fig.3ranks the frequency of jobs against the cumulative share. While 16 jobs account for 50% of the outputs for men, only 8 jobs account for the same share for women. Similarly, at the 90% level, men are associated with more jobs than women (66 vs 43, respectively). This suggests that GPT-2 predicts a wider variety of jobs for men and a narrower set of jobs for women. The Gini coefficients5in Tab.2confirm this more unequal distribution for women.

4.2. Gender differences in occupations

In addition to distributional differences, the set of returned jobs also differ by men and women. In Fig.4, we show the proportion of genders in all jobs mentioned more than 35 times for baseline man and woman. We make two obser-vations: first, there is a greater number of jobs dominated by men as compared to women, reflecting the greater di-versity of occupations for men. Second, the occupations seem stereotypical: men are associated with manual jobs such as laborer, truck driver, and mechanic, and with profes-sional jobs such as software engineer and private investiga-tor. Women are associated with domestic and care-giving roles such as babysitter, maid, social worker, and house-wife. Furthermore, over 90% of the returns for ‘prostitute’ were women, and over 90% of returns for ‘software engi-neer’ were men. We only find three jobs for which GPT-2’s outputs suggest a gender-neutral prior over occupations: reporter, lawyer, and sales representative.

5_{G = (}Pn

i=1(2i − n − 1)xi)/(n

Pn

i=1xi), where x is the

observed value, n is the total values observed, and i is the rank is ascending order.

Table 2. Gini coefficients of rank-frequency distributions.

Gender Intersection Gini Coeff Relative Coeff

Base M = 100% Man Base 0.933 100 Man Religion 0.929 99.57 Man Sexuality 0.935 100.21 Man Ethnicity 0.939 100.64 Man Political 0.942 100.96 Woman Base 0.951 101.93 Woman Political 0.951 101.93 Woman Ethnicity 0.956 102.47 Woman Religion 0.956 102.47 Woman Sexuality 0.958 102.68 4.3. Intersectional analysis

The previous analysis indicates considerable differences be-tween genders. We ask the following question: Do these stereotypical male and female jobs change when inter-sectionality is considered? The Gini coefficients (Tab.2) for gender-intersection pairs indicate a greater clustering of women into fewer jobs across all intersections, especially for sexuality and religion. To analyze differences in job associa-tions for each intersection, we display a scatter plot with the equi-proportion line given by (1/|c|, 0) to (0, 1/|c|), where |c| is the number of choices for intersection c. We normalize the axis such that 1/|c| = 1x so that jobs lie on this line if adding intersections has no effect on the gender ratio. We further include a bar plot showing the extremes of the dis-tribution with the top ten jobs with the largest man-woman range.

Ethnicity. For gender and ethnicity intersections (Fig.5), we find a similar pattern of occupations associated with men (plumber, guard, contractor, and police officer) and others with women (secretary, prostitute, model, babysitter). While all ethnicities of women are associated with prostitute, only Black men are. Overall, few occupations are solely associated with men or women of a certain ethnicity, and are mostly distributed over several ethnicities.

Religion. For gender and religion intersections (Fig. 6), Hindu men and women only have associations with non-religious professions (e.g. bouncers and massage therapists). For Christian, Buddhist, and Jewish religions, there is a tendency of GPT-2 towards generating occupations with large man-woman disparities, especially for professional religious occupations: nuns are dominated by Buddhist women, rabbis are dominated by Jewish men, and monks, pastors, and priests are dominated by Buddhist and Christian men.

Sexuality. For gender and sexuality intersections (Fig.7), we find professions such as massage therapist, counselor, and graphic designer to be almost unique to lesbian women, while professions such as detective, plumber, guard, and coach are dominated by straight men. Male-dominated

(5)

laborer cashier monk plumber salesman

truck driver

software engineer

developer

private investigator

mechanic courier contractor

driver

journalist supervisor bus driver

photographer barber priest technician clerk security guard computer programmer waiter carpenter construction worker taxi driver

police officer bouncer janitor doctor translator manager consultant bartender servant chef writer

reporter lawyer

sales representative

teacher cook editor barista

housekeeper secretary receptionist assistant cleaner waitress

nurse

housewife counselor prostitute

social worker maid model caretaker massage therapist babysitter 100% Men Gender Parity 100% Women Male-dominated jobs Female-dominated jobs

Figure 4. Fundamentally skewed output distributions. We show the gender proportions when querying for the base case, i.e. X = {}, Y = {Man, Woman} and present all jobs with greater than 35 = n ∗ 0.25% mentions, making up 81% of returned sentence prompts.

0x 1x 2x 3x

Over-representation Factor (Women)

0x 1x 2x 3x

Over-representation Factor (Men)

babysitter babysitter babysitter guard guard model plumber technician 0.0 0 1 2 3 4 5 6 7 8 9 0.2 0.4 0.6 0.8 1.0 Woman

plumber guard _contractor courier police officer technician secretary prostitute

model babysitter 0.0 0.2 0.4 0.6 0.8 1.0 Man Asian Black Hispanic White

Figure 5. Man-Woman Occupational Split by Ethnicity

0x 1x 2x 3x 4x 5x

butcher counselor housewife massage therapist monk nun pastor shepherd 0 1 2 3 4 5 6 7 8 9 0.0 0.2 0.4 0.6 0.8 1.0 Woman

monk pastor priest missionary bouncer

rabbi guide nun

counselor massage therapist 0.0 0.2 0.4 0.6 0.8 1.0 Man Buddhist Christian Hindu Jewish Muslim

Figure 6. Man-Woman Occupational Split by Religion

professions are almost exclusively straight, whereas female-dominated professions are almost exclusively lesbian. Political affiliation. For gender and political affiliation intersections (Fig. 8), the occupations are similar to the baseline man and woman case presented in Fig.4. Although

occupations are split along the gender axis, some have equal representation across political affiliation. The exception is that liberal men are strongly associated with critic and banker, and conservative men with driver and host. Name origin. For gender and continent name origin inter-sections (Fig.9), jobs are more tightly distributed around the equi-proportion line. This suggests that name origin has less of an effect on the token returned by GPT-2 than when adding an explicit categorical intersection (e.g. ethnicity or religion). Gender continues to be the more significant determinant on the occupations generated by GPT-2, with men being associated with jobs such as mechanic and leader, and women being associated with jobs such as nurse and receptionist.

4.4. Logistic Regression

The previous visual analyses demonstrate the considerable differences in the jobs associated with women and men, and further show that these gender splits remain when inter-sections are added. Next, we address the question: quan-titatively, how important are gendered intersections in determining the job returned by GPT-2? Tab.3presents summary results from 262 logistic regressions, which pre-dict the likelihood of a job being associated with a given sen-tence prompt. We focus on two metrics indicating how often the addition of regressors adds explainability of the outcome: i) The proportions of regressions where the woman dummy and the interactions were significant (p < 0.05), and ii) The change in Pseudo-R2 on the addition of the woman dummy and the interactions.6Statistical results, including the coefficients, for all regressions are in Appendix D. The aggregated results in Tab.3show that the woman dummy is frequently significant, most commonly so in ethnicity

6

We use the McFadden R2 which is calculated by

compar-ing the log-likelihood of a model with no predictors L0,

ver-sus the log-likelihood of the estimated model LM: RM cF2 =

(6)

0x 1x 2x

coach counselor detective graphic designer guard massage therapist plumber therapist 0 1 2 3 4 5 6 7 8 9 0.0 0.2 0.4 0.6 0.8 1.0 Woman

detective plumber guard

police officer courier coach graphic designer counselor therapist massage therapist 0.0 0.2 0.4 0.6 0.8 1.0 Man Lesbian/Gay Straight

Figure 7. Man-Woman Occupational Split by Sexuality

0x 1x 2x

banker bouncer counselor driver host maid model prostitute 0 1 2 3 4 5 6 7 8 9 0.0 0.2 0.4 0.6 0.8 1.0 Woman

truck driver bouncer lobbyist salesman banker driver host critic

prostitute maid 0.0 0.2 0.4 0.6 0.8 1.0 Man Conservative Liberal

Figure 8. Man-Woman Occupational Split by Political

0x 1x 2x

graphic designer nurse receptionist salesman technician therapist waiter waitress 0 1 2 3 4 5 6 7 8 9 0.0 0.2 0.4 0.6 0.8 1.0 Woman

salesman mechanic waiter

graphic designer

leader cook nurse waitress therapist receptionist 0.0 0.2 0.4 0.6 0.8 1.0 Man Africa Americas Asia Europe Oceania

Figure 9. Man-Woman Occupational Split by Name Origin

regressions (71%) and least commonly in political regres-sions (59%). Adding a woman dummy increases the model R2_{on average by +3.3% (percentage points), signifying}

that gender explains additional variation in job prediction. Interactions are significant in approximately one third of regressions, but the additional increase to R2_{is on}

aver-Table 3. Aggregated logistic regression results. We fit a total of 262 logistic regressions and report the number of times the inde-pendent variables contributed significantly to the logistic model,

as well as their average contribution to the Pseudo-R2.

#Jobs Variable Pct. Signif ∆ R2 Ethnicity 55 woman 0.71 3.22 woman:asian 0.29 0.40 woman:black 0.36 woman:hispanic 0.38 woman:white 0.16 Religion 64 woman 0.61 3.31 woman:buddhist 0.19 0.39 woman:christian 0.27 woman:hindu 0.27 woman:jewish 0.33 woman:muslim 0.25 Sexuality 72 woman 0.61 3.36 woman:lesbian 0.35 0.45 woman:straight 0.26 Political 71 woman 0.59 3.47 woman:conservative 0.24 0.46 woman:liberal 0.30

age smaller (+0.4%). There is some variation in the sig-nificance of interactions; for example, {women:hispanic} and {woman:black} are more frequently significant than {woman:white}, and {woman:lesbian} more significant than {woman:straight}. These results suggest that some intersections are more salient in changing the returned job from a given sentence prompt, and may anchor GPT-2 on a stereotypical occupation set. In general, across a wide range of jobs, gender and intersectionality are significant determinants of the token returned by GPT-2.

4.5. Comparison to Labor Market Ground Truth A comparison of GPT-2’s predictions to the true labor mar-ket distribution requires recent data disaggregated by gender and intersection for a granular set of occupations. The 2019 US Labor Force Statistics from the Current Popula-tion Survey (US Labor Bureau of Statistics,2019) reports the gender and ethnicity shares of workers in 567 occupa-tional categories. We recognize a number of limitations of this data, which we address in the discussion. We first select the 50 most frequently mentioned jobs by GPT-2. Then from these, we match GPT-2’s job tokens to real US occupation titles, finding correspondences for 41/50 titles (see Appendix C). We compute GPT-2’s predicted proportional representation for each gender-ethnicity pair, assuming the percentage of women is equal across ethnic-ities: The ‘predicted’ labor force has equal representation across groups because we generate the same number of sentence prompts per pair (n = 7,000). This is not the case in reality, so the predicted proportions are scaled by the true distribution of gender and ethnicity reported in the US Labor Statistics and summarised in Appendix C. The

(7)

scaling factor is γ(c) = G(c)E(c)_D(c)_ˆ , where G(c), E(c) are the gender- and ethnicity-shares of the US data, respectively and ˆD(c) = 12.5% is our artificial “population”-share:

adj. Pred(i, c) = γ(c) × Pred(i, c), (2) where Pred(i) is the share of job i for characteristics c. For jobs reported in the US data, we calculate the difference between the predicted proportions and the true proportions. For a given job, how well does GPT-2 predict the gender-ethnicity split? There are three possible cases: GPT-2 overestimates the true representation of women in female-dominated jobs (exacerbates societal skew), GPT-2 matches the true proportional representation (directly inherits skew), or GPT-2 underestimates the true proportional represen-tation (corrects for skew). In Fig. 1, we find that most predicted values lie close to the ground-truth given by the identity line, indicating a high accuracy in prediction (mean-squared errors of < 2%). In particular, for the gender-ethnicity intersections, the low mean-squared errors indicate a considerable degree of similarity between GPT-2’s pre-dicted distribution and the ground truth distribution, espe-cially for Asian and Black workers. Furthermore, it appears that GPT-2 pulls the distribution further from the extremes; that is, it under-predicts the extent of occupational segre-gation. This is demonstrated by the fact that GPT-2 pre-dicts a higher proportion of women than the ground truth in male-dominated jobs with less than 25% women-share (on average +8.7%) and predicts lower proportions of women in jobs with more than 75% women-share (on average -6.5%). The exceptions to this pattern are courier, bus driver and photographer, for which GPT-2 under-predicts the propor-tion of women, and social worker and model, for which GPT-2 over-predicts the proportion of women.

For a given gender-ethnicity pair, how well does GPT-2 predict the top jobs? This question aims to answer the ex-tent of stereotyping of GPT-2 predictions. Tab.4shows the top five predicted and ground truth jobs for each intersection. GPT-2 predicts a high proportion of baseline women to be waitresses (14%) but only Hispanic women have waitress in the top five occupations, according to the US Labor data. While GPT-2 predicts 18% of Hispanic women to be wait-resses, in reality only 3% of Hispanic women in America work as waitresses. Some of this strong association may be because waitress is an inherently gendered job. GPT-2 also over-predicts the number of nurses, predicting 11% of women to be nurses when in reality only about 4% of American women are nurses. Security guard is consistently overpredicted for men of all ethnicities. Yet security guard only appears as a top job for Black men and at a lower frequency (2%) than the predicted frequency (8%). GPT-2 over-predicts the proportion of janitors for all ethnicities, especially for White and Asian men, for whom janitor does not appear as a top job.

The share of the most popular occupation for each gender is significantly higher for women (waitress at 14%) than for men (security guard at 8%). The cumulative share of the top five occupations is 41% for women, which is more than dou-ble the ground truth observation (17%). While GPT-2 also over-predicts the cumulative share of top five occupations for men, the discrepancy to the US data is smaller (24% vs 10%). While GPT-2’s tendency to aggregate women into a small set of stereotypical jobs was identified in prior analysis (Fig.3and Tab.2), the comparison to US data corroborates this result.

5. Discussion

Demographic distribution per occupation. Overall, we find strong differences in the occupational tokens returned by GPT-2 for gendered sentence prompts. At first glance, it may seem biased that GPT-2 predicts so many women to be maids or secretaries and so few to be plumbers or truck drivers, but in fact, the model predicts less occupational segregation by gender as compared to the US ground truth distribution. It appears that GPT-2 is pulling the skews of the distribution found in reality towards gender parity. For ethnicity, GPT-2 accurately predicts the distribution of occupations in real-world data with low mean-squared er-rors, especially for Asian and Black workers. In addition to gender and ethnicity, adding a religious intersection consid-erably changes the returned jobs, especially for men. For example, GPT-2 predicts 4% of Buddhist men to be monks. There are an estimated 3.75 million Buddhists in the US and approximately 1,000 Buddhist centers and monasteries (Pew Research,2020; Institute for Genealogical Studies, 2020). A back of the envelope calculation shows each of these centers would need to employ more than 70 monks each to reach the 4% threshold. Therefore, it is likely that GPT-2 infers too strong of an association between practising a religion and working in a religious profession. Intersec-tions with continent-based names show that the returned occupations are more similar to those of baseline man and woman. This finding indicates that prompting GPT-2 with explicit intersections like ‘Buddhist man’ or ‘Black woman’ changes the probabilities of returned tokens to a greater ex-tent than a name prompt where GPT-2 must independently ascertain the gender and background of the individual. Occupation distribution per demographic. Despite re-flecting the gender-ethnicity proportions per real-world oc-cupation, GPT-2 notably displays a bias towards predicting greater occupational clustering for both genders, especially for women. The Gini coefficients confirm that the female distribution is more unequal than that of men. For the pre-dictions generated by GPT-2, a larger and more diverse set of occupations are associated with men than with women. Gender-ethnicity predictions do not deviate much from the

(8)

Table 4. Top five jobs per intersectional category with associated proportions of cumulative sum

GPT US

Jobs (Prop) Sum Jobs (Prop) Sum

WOMAN

base waitress (0.14), nurse (0.11), maid (0.06), receptionist (0.05),

teacher (0.05)

0.41 teacher (0.04), nurse (0.04), secretary/assistant (0.03), cashier

(0.03), manager (0.03)

0.17

Asian waitress (0.14), maid (0.11), nurse (0.08), teacher (0.05),

recep-tionist (0.04)

0.42 nurse (0.05), personal appearance worker (0.04), cashier (0.03),

accountant/auditor (0.03), manager (0.03)

0.18

Black waitress (0.18), nurse (0.10), maid (0.07), prostitute (0.05),

teacher (0.04)

0.44 nursing/home health aid (0.07), cashier (0.04), nurse (0.04),

personal care aide (0.03), teacher (0.03)

0.21

Hispanic waitress (0.16), nurse (0.14), receptionist (0.07), maid (0.07),

teacher (0.04)

0.48 maid/housekeeper/cleaner (0.05), cashier (0.04), waiter/waitress

(0.03), secretary/assistant (0.03), nursing/home aide (0.03)

0.18

White waitress (0.17), nurse (0.11), maid (0.07), teacher (0.05),

recep-tionist (0.04)

0.44 teacher (0.04), nurse (0.04), secretary/assistant (0.04), manager

(0.03), cashier (0.03)

0.18

MAN

base security guard (0.08), manager (0.05), waiter (0.04), janitor

(0.04), mechanic (0.03)

0.24 manager (0.04), truck driver (0.04), construction laborer (0.02),

retail sales supervisor (0.02), laborer/ material mover (0.02) 0.14

Asian waiter (0.09), security guard (0.07), manager (0.04), janitor

(0.04), chef (0.03)

0.27 software developer (0.11), manager (0.04), physician/surgeon

(0.02), teacher (0.02), engineer (0.02)

0.21

Black security guard (0.08), waiter (0.07), bartender (0.05), janitor

(0.05), mechanic (0.04)

0.29 truck driver (0.06), laborer/material mover (0.04), janitor (0.03),

manager (0.03), security guard (0.02)

0.18

Hispanic security guard (0.09), janitor (0.07), waiter (0.07), bartender

(0.05), manager (0.05)

0.33 construction laborer (0.06), truck driver (0.04), grounds

mainte-nance worker (0.03), carpenter (0.03), janitor (0.03)

0.19

White waiter (0.06), security guard (0.06), janitor (0.05), mechanic

(0.04), bartender (0.04)

0.25 manager (0.04), truck driver (0.04), construction laborer (0.03),

retail sales supervisor (0.02), laborer/material mover (0.02)

0.15

predictions for baseline man and woman. This signifies that GPT-2 predicts the occupations for women with less variety than for men, regardless of what ethnicity.

This is a different kind of bias than that normally discussed in the algorithmic fairness literature. In reality, large propor-tions of women do work as secretaries, receptionists, and maids, and large proportions of men do work as mechanics, plumbers, and carpenters. Therefore, GPT-2’s bias is not in the jobs associated with women, but in the rate at which it associates women with such a small set of jobs, a pattern exacerbated from the ground truth occupation data. Limitations. This paper is subject to a number of limita-tions. First, our chosen comparison to labor market data renders the ground truth baseline inherently US-centric. Sec-ond, without consistent, granular data on occupational splits by religion, sexuality and political affiliation, we cannot comment on how accurately GPT-2 reflects the ground truth for these intersections. Third, for jobs in the informal sec-tor, such as ‘prostitute’, we cannot compare to real-world incidences. Additionally, if terms such as ‘prostitute’ are commonly used as slurs, GPT-2 may display a bias towards over-estimating their proportion. Finally, by focusing only on two genders, the results do not adequately reflect occu-pational biases which may be associated with non-binary gender identities. Future research is recommended to inves-tigate ground truth comparisons across a broader range of countries against the set of gender-intersections examined in this paper and to comment on a broader spectrum of gender identities. Doing so would be valuable in establishing poten-tial areas of bias which risk being inherited by downstream applications of generative language models such as GPT-2.

6. Conclusion

What should be the goal of generative language models? It is certainly appropriate that they should not exacerbate existing societal biases with regards to occupational segre-gation. It is less clear whether they should reflect or seek to correct skewed societal distributions. Compared to US data, we identify a bias towards returning a small number of stereotypical jobs too many times, especially for women. However, for a given job, we find that GPT-2 reflects soci-etal skew and, in some cases, errs on the side of correcting for it. One proposed reason for this observed pattern is over-representation in the training data towards ‘exceptional cases’. If society expects women to be teachers and nurses, it is possible that there are more training examples scraped from social media platforms or newspaper articles of when men occupy these stereotypes, or vice-versa with plumbers and software developers. It remains to be seen whether a larger training set improves or impairs occupational bi-ases for intersections. Using the methodology developed in this paper, it is possible and would merit future research to determine the effect of model and training set size by comparing GPT-3 relative to its younger sibling analyzed in this work, GPT-2. This work presents the first comprehen-sive analysis of protected class intersections with gender in generative language models, and we hope that it will spark new research further investigating biases on topics relevant to downstream applications and intersectionality in AI more broadly.

(9)

Acknowledgements

This work has been supported by the Oxford AI student society, the EPSRC Centre for Doctoral Training in Au-tonomous Intelligent Machines & Systems [EP/L015897/1] (A.S., Y.M.A.), and the Economic and Social Research Council grant [ES/P000649/1] (H.K). We also thank R. Maria del Rio-Chanona, Gesa Biermann for their useful comments.

References

Adiwardana, D., Luong, M.-T., So, D., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., and Le, Q. V. Towards a human-like open-domain chatbot. ArXiv, abs/2001.09977, 2020.

Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? . In Conference on Fairness, Accountability, and Transparency (FAccT ’21). ACM, New York, NY, USA, 2021.

Bhardwaj, R., Majumder, N., and Poria, S. Investigating gender bias in bert. ArXiv, abs/2009.05021, 2020. Bhatia, V., Rawat, P., Kumar, A., and Shah, R. End-to-end

resume parsing and finding candidates for a job descrip-tion using bert. ArXiv, abs/1910.03089, 2019.

Blodgett, S. L., Barocas, S., Daum’e, H., and Wallach, H. Language (technology) is power: A critical survey of ”bias” in nlp. In ACL, 2020.

Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., and Kalai, A. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NeurIPS, 2016.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020.

Budzianowski, P. and Vulic, I. Hello, it’s GPT-2 - how can I help you? towards the use of pretrained lan-guage models for task-oriented dialogue systems. CoRR, abs/1907.05774, 2019.

Caliskan, A., Bryson, J., and Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science, 356:183 – 186, 2017.

Crenshaw, K. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. 1989. Diaz, M., Johnson, I., Lazar, A., Piper, A., and Gergle, D.

Addressing age-related bias in sentiment analysis. Pro-ceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018.

Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., and Weston, J. Queens are powerful too: Mitigating gen-der bias in dialogue generation. ArXiv, abs/1911.03842, 2020.

Ethayarajh, K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings. CoRR, abs/1909.00512, 2019. Fedus, W., Zoph, B., and Shazeer, N. Switch

transform-ers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.

Foulds, J. and Pan, S. An intersectional definition of fair-ness. 2020 IEEE 36th International Conference on Data Engineering (ICDE), pp. 1918–1921, 2020.

Gonen, H. and Goldberg, Y. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word em-beddings but do not remove them. ArXiv, abs/1903.03862, 2019.

He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654, 2020.

Institute for Genealogical Studies. US:

Reli-gious Records-Part 2, 2020. URL https:

//www.genealogicalstudies.com/eng/

courses.asp?courseID=209.

Kennedy, C. J., Bacon, G., Sahn, A., and von Vacano, C. Constructing interval variables via faceted rasch mea-surement and multitask deep learning: a hate speech application. ArXiv, abs/2009.10277, 2020.

Kiritchenko, S. and Mohammad, S. M. Examining gender and race bias in two hundred sentiment analysis systems. In *SEM@NAACL-HLT, 2018.

Kurita, K., Vyas, N., Pareek, A., Black, A., and Tsvetkov, Y. Measuring bias in contextualized word representations. ArXiv, abs/1906.07337, 2019.

Li, C., Fisher, E. M., Thomas, R., Pittard, S., Hertzberg, V., and Choi, J. D. Competence-level prediction and resume and job description matching using context-aware transformer models. ArXiv, abs/2011.02998, 2020.

(10)

Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., and Tang, J. Does gender matter? towards fairness in dialogue systems. In COLING, 2020.

Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S., and McClosky, D. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations), pp. 55–60. The Association for Com-puter Linguistics, 2014. ISBN 978-1-941643-00-6.

Pew Research. Religious Landscape Study,

2020. URL https://www.pewforum.

org/religious-landscape-study/

religious-tradition/buddhist/.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.

Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. The woman worked as a babysitter: On biases in language generation. ArXiv, abs/1909.01326, 2019.

Stanovsky, G., Smith, N. A., and Zettlemoyer, L. Eval-uating gender bias in machine translation. ArXiv, abs/1906.00591, 2019.

Tan, Y. and Celis, L. Assessing social and intersectional bi-ases in contextualized word representations. In NeurIPS, 2019.

Tatman, R. Gender and dialect bias in youtube’s automatic captions. In EthNLP@EACL, 2017.

US Labor Bureau of Statistics. Employed peons by de-tailed occupation, sex, race, and Hispanic or Latino eth-nicity, 2019. URLhttps://www.bls.gov/cps/

cpsaat11.htm.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurIPS, 2017.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and anal-ysis platform for natural language understanding. In BlackboxNLP@EMNLP, 2018.

Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Super-glue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019.

Wick, M. L., Silverstein, K., Tristan, J., Pocock, A. C., and Johnson, M. Detecting and exorcising statistical demons from language models with anti-models of negative data. ArXiv, abs/2010.11855, 2020.

Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretrain-ing for language understandpretrain-ing. In NeurIPS, 2019. Zhao, J., Zhou, Y., Li, Z., Wang, W., and Chang, K.-W.

Learning gender-neutral word embeddings. In EMNLP, 2018.

Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., and Chang, K.-W. Gender bias in contextualized word embeddings. ArXiv, abs/1904.03310, 2019.

(11)

Supplementary Material:

How True is GPT-2?

An Empirical Analysis of Intersectional Occupational Biases

Note on language used in this paper

In our paper, we focus on the occupational associations with binary gender identities i.e. “man” and “woman”. While we do sometimes refer to jobs dominated by women as ‘female-dominated jobs’, we do not make an explicit comparison to sex, i.e. prompting GPT-2 with the ‘female worker is a...’. We feel strongly about the importance in studying non-binary gender and in ensuring the field of machine learning and AI does not diminish the visibility of non-binary gender identities. In future work, we hope to extend our analysis with the same data collection pipeline. For example, womxn is a umbrella term used in the intersectional feminist community to be inclusive of transgender woman and non-binary individuals. The sentences returned when prompting GPT-2 with ‘womxn’ are primarily of two types: (i) stereotypical job associations e.g. ‘drag queen’, ‘feminist’, ‘crossdresser’ or ‘nurse’, and (ii) not recognizing ‘womxn’ as a person noun e.g. ‘The womxn works as a kind of a noodle shop’, ‘The womxn works as a battery’, ‘The womxn works as a mauve-wool hat’ or ‘The womxn works as a kind of virtual sex toy’. These preliminary findings suggest it is critical for future work to study occupational biases with non-binary gender identities in generative language models.

A. GPT Model Downloads

We select the most downloaded version of GPT-2 available on HuggingFace as a proxy for popularity in use-cases by experts and non-experts alike. In Tab.5we show the original GPT-2 models released by OpenAI (Radford et al.,2019) and available on HuggingFace. The small version has an order of magnitude more downloads as compared to the medium and XL versions. Further, larger models of GPT-2 have been shown to have an increased capability to memorize training information, introducing privacy concerns (Carlini,2020). Finally, while the environment cost of inference is cheap,Bender et al.(2021) discuss how the environmental impact of training scales with model size, and the associated consequences likely disproportionately affect marginalized populations.

Table 5. GPT-2 model available on Huggingface by number by total downloads (accessed 3rd February 2021)

Model # Hyperparameters # Public Downloads

GPT-2 Small 124M 623k

GPT-2 Medium 355M 77k

GPT-2 Large 774M 14k

GPT2 XL 1.5B 70k

B. Comparison with XLNet

XLNet sample generation. In addition to the suite of models released by Open-AI, XLNet is a generalized autoregressive pre-training method which outperforms BERT across a number of benchmark tasks (Yang et al., 2019). To assess the generalizability of our findings, we generate 7,000 sentences for the gender-occupation template (X = {}, Y = {Man, Woman}), and analyze the returned occupational tokens from XLNet. Out of the total 14,000 returned sentences, 4,442 had no title recognized by the Stanford NLP Named Entity Recognizer. This sample loss of 31% is higher than GPT-2 (Tab.6). A plausible reason for this higher sample loss is in the way XLNet generates text which includes extra inverted

(12)

commas.

Table 6. Sample loss from sentences with no detected job title

Model Template Missing Titles Sample Loss

GPT-2 Gender-occupation 20,689 10.6%

GPT-2 Names-occupation 39,203 19.6%

XLNET Gender-occupation 4,442 31.7%

Distributional Analysis. Fig.10shows the rank of jobs against the cumulative share. While 11 jobs account for 50% of the outputs for men, only 5 jobs account for the same share for women. Similarly, considering 90% of the output, women are associated with fewer jobs than men (31 vs 46, respectively). This disparity is similar to the one that we found in GPT-2, suggesting that XLNet also predicts a wider variety of jobs for men and a narrower set of jobs for women.

Table 7. XLNet: Top five jobs for base man and base woman

XLNet Jobs (Proportions) Sum

Woman maid (0.27), waitress (0.14), prostitute (0.05), servant (0.04), nurse (0.04) 0.54 Man carpenter (0.11), mechanic (0.07), maid (0.05), waiter (0.05), taxi driver (0.04) 0.32

Top occupations. Tab.7shows the top five jobs for men and women as predicted by XLNet. Similar to our observations for gender differences predicted by GPT-2, we see a higher cumulative share in the top jobs for women as compared to men. The top job for woman (maid at 27%) represents a substantially larger proportion than the top job for man (carpenter at 11%). Interestingly, men are predicted to be maids 5% of the time, which was a pattern that we did not see with GPT-2. Fig.11shows the proportion of genders in all jobs mentioned more than 35 times for baseline man and woman. This is the same threshold as the one we used to calculate the analogous gender parity graph for GPT-2 jobs. Men and woman are associated with stereotypical jobs, but slightly different ones than those predicted by GPT-2. In this case, we see that men are associated with a variety of jobs, such as courier, barber, teller, magician, and builder. Women are, yet again, associated with domestic and care-giving jobs, such as nanny, housewife, and nurse. Women are also highly associated with jobs such as gardener, bartender, secretary, and prostitute.

100 101 102 Log(Rank) 0.00 0.05 0.10 0.15 0.20 0.25 Share of Total

base_W base_M

Figure 10. XLNet: Occupational distribution for men and women (baseline case). As with GPT-2, the job titles predicted by XLNet are less diverse and more stereotypical for women than for men.

(13)

courier barber teller

contractor magician builder printer plumber

tailor

carpenter driver trader painter weaver hunter cutter taxi driver salesman butcher mechanic

baker

merchant doctor dresser factory worker messenger

waiter manager laborer shopkeeper cashier teacher shoemaker owner clerk therapist vendor janitor servant

cook

housekeeper dressmaker

maid

assistant cleaner prostitute nurse secretary waitress housewife bartender gardener nanny

100% Men

Gender Parity

100% Women

Male-dominated jobs

Female-dominated jobs

Figure 11. XLNet: gender proportions when querying for the base case, i.e. X = {}, Y = {Man, Woman} and show all jobs with greater than 35 = n ∗ 0.25% mentions, making up 65% of returned valid responses.

C. Processing

C.1. Named Entity Recognition

We used Stanford CoreNLP Named Entity Recognition (NER) to extract job titles from the sentences generated by GPT-2. Using this approach resulted in the sample loss of 10.6% for gender-occupation sentences and 19.6% for name-occupation sentences (see Tab.6). The sample loss was due to Stanford CoreNLP NER not recognizing some job titles e.g. “Karima works as a consultant-development worker”, “The man works as a volunteer”, or “The man works as a maintenance man at a local...”.

C.2. Adjustment Factors

When comparing to the US data, some adjustments are made to ensure fair comparison. Firstly, there are no breakdowns by gender and ethnicity in the US Labor Bureau data so we assume the proportion of women are equal across ethnicities. Secondly, for each gender-ethnicity pair, we generate the same number of sentence prompts per pair (n = 7,000). This implies the ‘predicted’ labor force has equal representation across groups which is not the case in reality. Accordingly, the predicted proportions are scaled by the true distribution of gender and ethnicity reported in the US Labor Statistics. The scaling factor is: γ(c) = G(c)E(c)_D(c)_ˆ , where G(c), E(c) are the gender- and ethnicity-shares of the US data, respectively and

ˆ

D(c) = 12.5% is our artificial “population”-share:

adj. Pred(i, c) = γ(c) × Pred(i, c), (3)

where Pred(i) is the share of job i for characteristics c. Tab.8shows the true proportions and the steps made in the adjustment process.

(14)

Table 8. Adjustment calculations.

US Eth. US Gender G-E. Distr. GPT Distr. Correction

(E) (G) (D = G ∗ E) ( ˆD) (γ) Man NA 0.530 0.530 0.500 1.060 Woman NA 0.470 0.470 0.500 0.940 Asian Man 0.065 0.530 0.034 0.125 0.276 Asian Woman 0.065 0.470 0.031 0.125 0.244 Black Man 0.123 0.530 0.065 0.125 0.522 Black Woman 0.123 0.470 0.058 0.125 0.462 Hispanic Man 0.176 0.530 0.093 0.125 0.746 Hispanic Woman 0.176 0.470 0.083 0.125 0.662 White Man 0.777 0.530 0.412 0.125 3.294 White Woman 0.777 0.470 0.365 0.125 2.922

C.3. Matching GPT-2 and US Jobs

The US data has four nested levels of disaggregation e.g. Management, professional, and related occupations → Professional and related occupations → Computer and mathematical occupations → Computer Programmer. For GPT-2’s 50 most frequently mentioned jobs, we match the GPT-2 job title to one in the US data at the lowest nested level, apart from ‘salesperson’ and ‘manager’ which are too general to match to the lowest disaggregation. For these, we match to ‘sales and related occupations’, and ‘management occupations’, respectively. In total, we find correspondences for 41/50 jobs. Jobs were not matched for three reasons: (i) there were too many varied mentions of a job e.g. ‘clerk’ was associated with 25 different jobs spanning finance, law and hospitality sectors, (ii) there was no match for a job e.g. ‘prostitute’ and ‘translator’, (iii) the jobs were inherently gendered e.g. ‘waitress’ and ‘salesman’. There are two further considerations in matching. First, when a GPT-2 job is less general than the US categories. For example, while GPT-2 gave separate predictions for taxi drivers and chauffeurs, the US data only reports ‘taxi drivers and chauffeurs’. Similarly, while GPT-2 gives separate predictions for maids, housekeepers and cleaners, the US category amalgamates these into ‘maids and housekeeping cleaners’. For these cases, we average across GPT-2’s predictions for the relevant jobs, i.e. combining the predictions for maid, housekeeper and cleaner. Second, when GPT-2’s predictions are more general than the US categories. For example, when GPT-2 returns the token of ‘teacher’ but the US data reports ‘postsecondary teachers, ‘preschool and kindergarten teachers’, etc. For these cases, we sum across the US sub-categories. Tab.9gives details on these matches.

(15)

Table 9. Job matches between GPT-2 predicted jobs and US data.

GPT US DATA

babysitter Childcare workers

secretary / assistant Secretaries and administrative assistants

receptionist Receptionists and information clerks

cleaner / housekeeper /

maid Maids and housekeeping cleaners

nurse Registered nurses

social worker Social workers

teacher Postsecondary teachers, Preschool and kindergarten teachers, Elementary and middle school teachers,

Special education teachers

model Models, demonstrators, and product promoters

writer Writers and authors

barista Counter attendants, cafeteria, food concession, and coffee shop

bartender Bartenders

photographer Photographers

bus driver Bus drivers

reporter / journalist News analysts, reporters and correspondents

cook Cooks

doctor Physicians and surgeons

manager Management occupations

janitor Janitors and building cleaners

lawyer Lawyers

barber Barbers

chef Chefs and head cooks

guard / security guard

/ bouncer Security guards and gaming surveillance officers

courier Couriers and messengers

computer programmer Computer programmers

police officer Police and sheriff’s patrol officers

taxi driver / chauffeur /

driver Taxi drivers and chauffeurs

truck driver Driver/sales workers and truck drivers

construction worker /

laborer Construction laborers

carpenter Carpenters

plumber Pipelayers, plumbers, pipefitters, and steamfitters

mechanic Automotive service technicians and mechanics

salesperson Sales and related occupations

EXCLUDED JOBS

clerk Too many sub-categories

technician Too many sub-categories

consultant No entry

contractor No entry

prostitute No entry

translator No entry

salesman Gendered title

waitress Gendered title

(16)

D. Regression Analysis

D.1. Percentage of Significant Coefficients

Tab.10shows the percentage of significant coefficients for each intersection. To produce these results, we run regressions for all jobs mentioned more times than the same threshold values used in the paper. Each regression includes all main effects and interaction terms. We then compute the percentage of significant coefficients for each term across all regressions with baseline man as the reference group. We repeat these steps for each intersection: ethnicity, religion, sexuality and political affiliation. We did not run regression for continent name origin because there was no suitable baseline category given every first name has geographic and gender associations.

Considering religion, the Buddhist term has the higher percentage significance across all regressions (78%), while the Hindu term has the lowest (55%). This supports the findings in the paper that some religions are stronger determinants of jobs than others. Of the interaction terms, woman:buddhist is the least significant (19%). This finding suggests that male jobs are more highly determined by Buddhist membership, but female jobs are less strongly associated with this affiliation. Considering ethnicity, the Hispanic term is most commonly significant (64%), while the Asian term is less commonly significant (42%). The interactions for Hispanic and Black women are more frequently significant than those for White and Asian women. This finding suggests some ethnicity-gender pairs more saliently affect GPT-2’s priors on job associations. Considering sexuality, both sexuality categories (gay/straight) are significant in approximately 50% of regressions. A woman’s intersectional association with being lesbian is more commonly significant than an association with being straight. Considering political affiliation, the liberal term is more commonly significant than the conservative term, and the same pattern apply to gender-political interaction terms.

Finally, we can compare the average significance of categories, gender and their intersections across religion, ethnicity, sexuality and political regressions. Religion main effects are on average significant in 66% of regressions, ethnicity main effects in 53% of regressions, sexuality main effects in 48% of regressions and political main effects in 60% of regressions. This suggests for men, there is higher across-religion variation in predicted jobs than say for across-sexuality variation. The woman dummy is significant in 61% of religion regressions, in 71% of ethnicity regressions, in 61% of sexuality regressions and in 59% of political regressions. This finding demonstrates the woman and man variation is more influential in distinguishing between job affiliations for ethnicity and least influential for political affiliation. Across all regressions, the woman dummy is highly significant suggesting gender is an important determinant of job predictions. Finally, the interaction terms are significant in 26% of religion regressions, in 30% of ethnicity regressions, in 31% of sexuality regressions and in 27% of political regressions. This suggests for women, sexuality and ethnicity are stronger determinants of job associations. Interaction terms are significant in approximately one-third of regressions, while the woman dummy is significant in approximately two-thirds of regressions. This finding suggests, while intersectionality is an relevant determinant of predicted job, gender more strongly influences GPT-2’s priors over occupational associations.

Table 10. Percentage of significant coefficients in logistic regressions by intersection

RELIGION ETHNICITY SEXUALITY POLITICAL

Intercept 0.94 Intercept 0.95 Intercept 0.90 Intercept 0.92

buddhist 0.78 asian 0.42 gay 0.51 conservative 0.55

christian 0.69 black 0.55 straight 0.44 liberal 0.66

hindu 0.55 hispanic 0.64 woman 0.61 woman 0.59

jewish 0.66 white 0.49 woman:lesbian 0.35 woman:conservative 0.24

muslim 0.64 woman 0.71 woman:straight 0.26 woman:liberal 0.30

woman 0.61 woman:asian 0.29 woman:buddhist 0.19 woman:black 0.36 woman:christian 0.27 woman:hispanic 0.38 woman:hindu 0.27 woman:white 0.16 woman:jewish 0.33 woman:muslim 0.25

(17)

D.2. All Regression Results

Fig.12presents the significant p-values in all regressions for main effects and interaction terms. Significant p-values (p < 0.05) are shaded in black, while non-significant terms are left as white. Considering for example ethnicity, there are two axes of variation. First, some jobs have significant p-values across all terms such as supervisor and teacher, indicating these jobs are highly segmented by gender and by ethnicity, but also by their interaction. Jobs with no significant p-values represents cases where the model did not converge which occurred when there was insufficient variation across different demographics. In Fig.13, we present the direction and magnitude of significant coefficients. Any negative coefficients, i.e. those that make the job prediction less likely, are shaded in red. Any positive coefficients, i.e. those that make the job association more likely, are shaded in blue. Any insignificant coefficients (p > 0.05) are left as white. A darker color indicates a larger strength of coefficient. We present all the results so an interested reader can select a certain job and find the associated coefficients for gender and intersections, alongside their interaction terms.

Finally, Fig.14presents the change in Pseudo-R2for all job regressions across ethnicity when the woman dummy is added and when the interaction terms are added. To produce these results, we first run a regression with all the main effects of categorical membership e.g. (‘Asian’, ‘Black’, ‘Hispanic’, ‘White’) but without the woman dummy. Given baseline ‘man’ is the reference group, all gender variation resides in the intercept. Next, we re-add the woman dummy, and observe how the model fit improves. Finally, we run a regression with all main effects and all interaction terms and see what additional variation is explained. The general pattern observed is that the woman dummy has a greater effect on the model fit than the interactions. This finding suggests that while interaction terms for intersectional associations are significant in approximately one-third of job regressions, they explain a lower proportion of variation than gender. Once again, there is considerable variation by job and by intersection, so for detailed insights we invite readers to examine particular occupation-demographic patterns.

(18)

guard monk

supervisor

massage_therapist

technician prostitute secretary laborer caretaker translator plumber police_officer truck_driver

maid

housewife model courier _{salesman contractor sales_clerk} barber cleaner social_worker mechanic driver reporter photographer nurse waitress construction_worker receptionist

servant editor waiter _chauffeur clerk

journalist consultant writer taxi_driver assistant bus_driver

security_guard

bartender carpenter janitor

sales_representative doctor teacher housekeeper manager barista chef lawyer cook Intercept asian black hispanic white woman woman:asian woman:black woman:hispanic woman:white ETHNICITY

pastor monk rabbi

technician

guide priest shepherd missionary

massage_therapist

housewife farmer painter baker laborer tailor contractor

photographer

reporter

shopkeeper gardener nurse courier bouncer

construction_worker

salesman servant plumber truck_driver mechanic

maid

secretary translator cleaner

security_guard

writer

receptionist

sales_representative

porter

paralegal editor _{bus_driver consultant taxi_driver} waitress driver barber doctor librarian assistant waiter bartender teacher

clerk

police_officer journalist chauffeur carpenter barista janitor manager chef lawyer cook housekeeper Intercept buddhist christian hindu jewish muslim woman woman:buddhist woman:christian woman:hindu woman:jewish woman:muslim RELIGION

detective therapist guard coach

graphic_designer

developer stand_in

software_engineer

counselor bodyguard babysitter

massage_therapist monk plumber _contractor private_investigator caretaker construction_worker courier model technician priest computer_programmer truck_driver cleaner

housewife supervisor mechanic bus_driver salesman maid driver

police_officer

bouncer laborer nurse

security_guard

waitress librarian chauffeur taxi_driver clerk assistant _{sales_clerk journalist} waiter barber servant secretary

receptionist

house_painter

carpenter

photographer

barista reporter doctor janitor director manager translator teacher producer editor bartender

writer consultant artist chef sales_representative housekeeper cook _lawyer Intercept gay straight woman woman:gay woman:straight SEXUALITY

plumber courier banker

host

supervisor

driver critic lobbyist maid

technician mechanic barber massage_therapist construction_worker

housewife counselor taxi_driver truck_driver

personal_trainer

activist producer journalist prostitute

financial_adviser

director

contractor

nurse

salesman bouncer translator model

police_officer

private_detective

editor

private_investigator

security_guard

secretary developer executive

social_worker waitress artist sales_manager analyst carpenter receptionist software_engineer

assistant professor lawyer president specialist student writer servant reporter barista cook clerk waiter

computer_programmer

photographer consultant manager janitor teacher doctor

housekeeper chef sales_representative bartender Intercept conservative liberal woman woman:conservative woman:liberal POLITICAL

(19)

guard monk

supervisor

massage_therapist

technician prostitute secretary laborer caretaker translator plumber police_officer truck_driver

maid

housewife model courier _{salesman contractor sales_clerk} barber cleaner

social_worker

mechanic driver reporter

photographer

nurse

waitress

construction_worker

receptionist

servant editor waiter

chauffeur

clerk

journalist consultant writer taxi_driver assistant bus_driver

security_guard

bartender carpenter janitor

sales_representative doctor teacher housekeeper manager barista chef lawyer cook Intercept asian black hispanic white woman woman:asian woman:black woman:hispanic woman:white ETHNICITY

pastor monk rabbi

technician

guide priest shepherd missionary

massage_therapist

housewife farmer painter baker laborer tailor contractor

photographer

reporter

shopkeeper gardener nurse courier bouncer

construction_worker

salesman servant plumber truck_driver mechanic

maid

secretary translator cleaner

security_guard

writer

receptionist

sales_representative

porter

paralegal editor _{bus_driver consultant taxi_driver} waitress driver barber doctor librarian assistant waiter bartender teacher

clerk

police_officer journalist chauffeur carpenter barista janitor manager chef lawyer cook housekeeper Intercept buddhist christian hindu jewish muslim woman woman:buddhist woman:christian woman:hindu woman:jewish woman:muslim RELIGION

detective therapist guard coach

graphic_designer

developer stand_in

software_engineer

counselor bodyguard babysitter

massage_therapist monk plumber contractor private_investigator caretaker construction_worker courier model technician priest computer_programmer truck_driver cleaner

housewife supervisor mechanic bus_driver salesman maid driver

police_officer

bouncer laborer nurse

security_guard

waitress librarian chauffeur taxi_driver clerk assistant

sales_clerk journalist waiter barber servant secretary _receptionist

house_painter

carpenter

photographer barista reporter doctor janitor director manager translator teacher producer

editor bartender writer consultant artist chef sales_representative housekeeper cook lawyer Intercept gay straight woman woman:gay woman:straight SEXUALITY

plumber courier banker

host supervisor driver critic lobbyist maid technician mechanic barber massage_therapist construction_worker housewife counselor taxi_driver truck_driver

personal_trainer

activist producer journalist prostitute

financial_adviser

director

contractor

nurse

salesman bouncer translator model

police_officer

private_detective

editor

private_investigator

security_guard

secretary developer executive

social_worker waitress artist sales_manager analyst carpenter _receptionist software_engineer

assistant professor lawyer president specialist student writer servant reporter barista cook clerk waiter

computer_programmer

photographer consultant manager janitor teacher doctor

housekeeper chef sales_representative bartender Intercept conservative liberal woman woman:conservative woman:liberal POLITICAL 8 6 4 2 0 2 4 8 6 4 2 0 2 4 8 6 4 2 0 2 4 8 6 4 2 0 2 4