Principle of Non-Discrimination in AI Governance | Part I

Part I | Understanding how artificial intelligence (AI) systems can inadvertently or systematically perpetuate discrimination? 

This part of the article explores the need for OECD AI Principle 1.2, focusing on non-discrimination throughout the AI lifecycle. By examining five ways through which bias can infiltrate AI systems—ranging from discriminatory class labels to problematic proxies—the discussion highlights the critical need for robust anti-bias measures in AI governance. Understanding these mechanisms is vital to ensuring equitable and fair AI development amidst a rapidly evolving technological frontier.

Mihir Nigam*

INTRODUCTION

There is an apparent multi-dimensional divergence in how the countries across the globe are approaching the governance of artificial intelligence. The major risk in such situation is that the countries, in absence of a global policy on the matter, would create a fragmented regulatory landscape, reminiscent of the current data transfer regulations. To counter such risk at the earliest, in a major development, the European Union has legislated upon the EU Artificial Intelligence Act (“the Act”). Upon the approval of the Council of European Union on 21 May 2024, the act enters into force across all the EU states on 1 August 2024. The Act is guided with a risk-based approach, and it seeks to regulate the most controversial technology of our times, its uses, and its developers.

The AI Act, among its many objectives, was brought to address the urgency for international alignment on AI governance. The urgency stems from concerns raised by whistleblowers within the industry who are wary of the rapid pace at which development in AI is taking place and the major societal implications that could arise if AI systems are developed and deployed carelessly in a fragmented regulatory landscape.[1] In this context, the AI Principles, prepared by the OECD, have been carefully crafted to minimize risks and provide a foundation for countries to align and formulate their policies. Out of the many principles provided by the OECD, the topic for exploration of this article is OECD AI Principle 1.2 (“Principle 1.2”), the principle that emphasizes non-discrimination throughout the AI system’s life cycle.[2] 

To appreciate this principle, it is first important to understand that how easily discriminatory behavior can creep into the AI system. Barocas and Selbst, in their article have highlighted five ways in which the AI-decision making can lead, unintentionally, to discrimination.[3] In the following paragraphs, I have tried to establish the need for non-discrimination principle in AI governance by discussing the five issues highlighted by Barocas and Selbst, using lucid language and relevant examples.

(1)     Discrimination through “target variables” and “class labels”

During the training of the AI systems, the target outcome desired out of the model is called a “target variable,” and the when the possible values of the target values are divided into mutually exclusive categories they are called “class labels.”[4] They are relevant for our purpose to understand that, sometimes, defining the target labels requires creation of a separate class labels. For example, consider a situation where a company requires an AI system for sorting out the job applications, now in order to select a good employee (target variable); we need to create separate class labels by defining who is really a good employee – the one who is never late at work (class label #1); or the one who performed better by making better sales (class label #2).[5]

Now consider this proposition­—the poor people who often live outside the city and have to travel 40-50km to reach the office and face traffic jam or public transport issue—are the ones who are usually late to work than other employees. Further, if it could be established that immigrants are on average, poorer, and live outside the city. In such case, the choice of going ahead with class label #1 will lead to discrimination with respect to the immigrants with disadvantaged background.

Another example can be, a scenario where an AI system is designed to classify loan applications for a financial institution. Here, the target variable could be “the likelihood of a loan application being approved.” The class labels might be categories such as “Stable Employment,” “Bank Transaction History,” “Place of Residence/Region.” Now, if the first-class label is based on criteria of stable employment, this might disadvantage applicants from rural areas or people with seasonal employment. As, in India, many small farmers or seasonal workers live in less urbanized regions and may have less stable employment due to the seasonal nature of work or fluctuating local markets. Furthermore, the mode of payment in such employments is usually “cash,” therefore, there the transaction that they have will not reflect under their bank transaction history.

Therefore, if these applicants are evaluated on such criteria, the AI might unfairly label them under “low-chance of approval” category even though they might have a solid financial standing.

(2)     Prejudiced Labelling in Training the Model

Data Labelling is the process of manually assigning class labels to data, either based on pre-existing categorizations or through subjective judgments when no labels are available. This process can introduce biases if the labels are based on flawed or prejudiced criteria.[6] For example, if historical decisions or labels used to train an algorithm reflect past prejudices, the algorithm may perpetuate those biases. And there always remains a possibility that AI system is trained on a biased data set. Usually, the AI models are trained on a huge pile of data, and during the training of such model if the data has not been labelled correctly and the measures are not taken to rule out the bias from the data set, this is an accepted fact that the AI system is only going to reproduce such bias. The discriminatory effects of this can be observed in two ways – i.) When the AI system is training on a data that is inherently biased; ii.) When the AI system learns from the biased sample.[7]

For instance, reports surfaced that women-founded start-ups receive significantly less venture capital than those founded by men. In the year 2019, just 2% of venture capital was directed towards start-ups founded by women.[8] Now, if this data is fed to an AI system designed for helping investors to make important investment related decisions, the system can very well suggest an investor to invest in the start-ups founded by a man, because a logical conclusion (that it is the best course of action) can be drawn from the fact that majority of the venture capitalist are diverting their capital towards startups led by a man.  

Such algorithmic bias is not a new problem as it has been an issue since the early development of AI systems. In late 1970s, Dr. Geoffrey Franglen’s admissions algorithm, at St. George’s Hospital Medical School, was under scrutiny. The algorithm was aimed to streamline and standardize the admissions process, however, despite its goal to reduce human bias, it inadvertently reinforced existing prejudices. By using biased historical data, the algorithm unfairly treated non-Caucasian names and female applicants.[9] This early case demonstrated that algorithms, while designed to emulate human decision-making, can perpetuate, and institutionalize biases rather than eliminate them.

" the AI models are trained on a huge pile of data, and during the training of such model if the data has not been labelled correctly and the measures are not taken to rule out the bias from the data set, this is an accepted fact that the AI system is only going to reproduce such bias.

(3)     Discrimination in Data Collection

Data Collection is the phase of the project in which data is ingested from multiple sources. Challenges stem from incomplete or inaccurate data or data that do not represent groups well. For example, if the records for underrepresented groups are less complete or less accurate because of structural biases, this skew might under or overrepresent those groups in your dataset.[10] This could be further skewed when there are socioeconomic factors that alter participation, technology access, or geographic representation. This could therefore systematically bias the result against protected classes whenever decisions were made based on flawed data, due to the failure to represent the data in an unbiased and proportionate form.

Bias in data collection methods can have significant societal consequences. Especially, when the critical governmental decisions are informed by AI systems that are inherently biased due to flaws in their data collection processes, these biases can be perpetuated and exacerbated.

For example, the case of Street Bump application, which utilized GPS technology to report road conditions to municipal authorities, relied on volunteer users to gather data.[11] The application’s website stated: “Volunteers use the Street Bump mobile app to collect road condition data while they drive. This data provides governments with real-time information to address issues and plan long-term infrastructure investments.” However, if there is a lower prevalence of smartphone users among economically disadvantaged populations compared to wealthier individuals, there is a risk that road conditions in poorer areas may be underreported.[12] Consequently, this underrepresentation could result in fewer repairs and less attention to infrastructure problems in these underserved communities.

" Bias in data collection methods can have significant societal consequences. Especially, when the critical governmental decisions are informed by AI systems that are inherently biased due to flaws in their data collection processes, these biases can be perpetuated and exacerbated.

(4)     Discrimination by way of features in an AI system

Incorporating specific features into an AI system’s decision-making process can inadvertently introduce bias against certain groups. It is introduced in essence to ease out the operation process in algorithm, as it becomes impossible for accessing each input completely, due to the system constrains or the cost of the operations. Therefore, certain features or attributes or characteristics are chosen for the purpose of prediction.

For more clarity on this, consider the survey which reported that many employers in India tend to favour candidates who have graduated from prestigious and expensive universities in London.[13] In this backdrop, it becomes important to note that individuals from certain racial or socioeconomic backgrounds may be underrepresented in these elite institutions due to various systemic barriers. Now, a scenario where an AI system is designed to screen job applications and the features selected from sorting applications is the “educational background” of applicants. If the AI is programmed to prioritize candidates from high-status foreign universities, it may disproportionately disadvantage applicants from marginalized racial groups, who are less likely to have attended such institutions. This selection criterion could lead to a biased hiring process, where qualified individuals are overlooked simply due to their educational background rather than their actual capabilities or experiences.

The other example here can be the case of incorporating certain features into health insurance underwriting algorithms. Such algorithm can unintentionally introduce bias against specific groups. For instance, insurance companies prioritize features such as body mass index (“BMI”) and history of pre-existing conditions in their risk assessments.[14] However, individuals from lower socioeconomic backgrounds or marginalized communities may face unique health challenges and barriers that affect these factors differently.

Imagine a scenario where an AI system is designed to evaluate health insurance applications and includes features such as “BMI” and “frequency of doctor visits.” If the AI system is programmed to weigh these features heavily, it may disproportionately disadvantage individuals from communities with limited access to healthcare or those who experience higher rates of chronic conditions due to systemic inequities. For example, people from lower-income neighbourhoods might have low BMIs and less frequent medical check-ups due to financial constraints and lack of healthcare access. This criterion could lead to a biased underwriting process, where qualified individuals are unfairly denied coverage or charged higher premiums based on factors beyond their control, rather than their actual health risks or needs.

(5)     Problem with the proxies

When training an AI system, it is crucial to be aware that some data points in the training set might inadvertently correlate with protected characteristics, even if they are not explicitly included. As noted by Barocas and Selbst, “criteria that are genuinely relevant in making rational and well-informed decisions also happen to serve as reliable proxies for class membership.”[15]

Imagine a real estate company uses an AI system to determine the likelihood of prospective tenants paying their rent on time. The system is trained on historical data that includes factors like income, occupation, and rental history, but does not explicitly include information about religion or caste. The AI system identifies that tenant from certain neighbourhoods, such as affluent areas in Bengaluru or Delhi, tend to have fewer issues with timely rent payments. Consequently, the AI system uses neighbourhood data as a predictor for assessing the likelihood of timely payments.

At first glance, using neighbourhood data might seem neutral.  However, if certain neighbourhoods have strong correlations with socio-economic backgrounds, or even specific castes or religions, then the AI system’s reliance on this neighbourhood information might indirectly disadvantage individuals from less affluent areas who might belong to marginalized communities. These correlations are not directly related to protected characteristics but could serve as proxies. For instance, if lower-income neighbourhoods are more prevalent among certain marginalized groups, the AI’s predictions could result in unfair treatment of applicants from those areas, regardless of their actual financial reliability.

Another example involves the job market. Suppose a company uses an AI-driven recruitment tool to screen job applications for managerial positions. The AI is trained on historical data that includes information about educational institutions attended and previous job titles. If certain prestigious institutions or companies are more commonly associated with certain socio-economic backgrounds or regions, the AI might favour candidates with backgrounds from these prestigious institutions, inadvertently sidelining candidates from less prestigious but equally capable backgrounds.

" The OECD Principle 1.2 is necessitated to guide the creation of AI systems that are not only transparent and accountable but also designed to prevent the perpetuation of bias, thereby helping to break the vicious cycle and foster a more equitable technological landscape.

The Vicious Cycle of Biased Output

Addressing the proxy problem is challenging. Barocas and Selbst highlight that “computer scientists have been unsure how to deal with redundant encodings in datasets.”[16] Simply excluding variables that could be proxies for protected characteristics might remove criteria that are otherwise relevant for making informed decisions. As a result, one potential solution could be to intentionally reduce the overall accuracy of the AI system to ensure that decisions do not systematically disadvantage members of protected classes.[17] This approach aims to balance fairness with accuracy, acknowledging that avoiding discrimination may sometimes come at the cost of less precise predictions.

Alongside these issues, a concerning phenomenon that can arise is the creation of a vicious cycle of bias, particularly with generative AI models. This cycle occurs when an AI model, trained on biased datasets, generates outputs that perpetuate the same biases. These biased outputs then become part of future training datasets, thereby reinforcing, and magnifying the original biases in subsequent iterations of the model.

For example, imagine a generative AI model trained on historical news articles that exhibit biased reporting patterns. If the model generates news content based on these biased patterns, the generated content might reflect and perpetuate the same biases. When this biased content is incorporated into new training datasets, it risks embedding these biases deeper into future models. This cycle not only maintains but can exacerbate existing biases over time, creating a compounding effect that can be difficult to rectify.

This issue is compounded by the “black box” nature of many AI systems, where the decision-making processes are opaque and not easily interpretable. When biases are perpetuated through this feedback loop, it becomes increasingly challenging to identify, understand, and address them. As a result, it might be too late to effectively mitigate the problem once the biases are deeply embedded in the AI system’s behaviour. Therefore, by promoting the design and implementation of AI systems that actively avoid and address bias through, the non-discrimination principle aims to ensure that AI technologies are developed and deployed in ways that uphold fairness and equality.

Therefore, The OECD Principle 1.2 is necessitated to guide the creation of AI systems that are not only transparent and accountable but also designed to prevent the perpetuation of bias, thereby helping to break the vicious cycle and foster a more equitable technological landscape.

 _____________________________________________________________________

*Mihir Nigam is a 5th-year Intellectual Property Law (Hons.) student at National Law University, Jodhpur. He serves as the Team Lead for AI & Data Protection at the Centre for Research in Governance, Institutions, and Public Policy at the same university.

Pdf version below

Part I - Principle of Non-Discrimination in AI Governance.pdf

Download PDF • 316KB

(The next part will cover how different nations are adopting the Principle 1.2 and crafting their laws, regulations, and policies to mitigate biases and ensure ethical AI deployment.)

References:

[1] Pause giant AI experiments: An Open Letter, Future of Life Org, signed by more than 33k signatures. The same can be accessed from here: https://futureoflife.org/open-letter/pause-giant-ai-experiments/ 

[2] Principle 1.2 on Human-centred values and fairness, The OECD AI Principles adopted in May 2019. The same can be accessed from here: https://oecd.ai/en/ai-principles.

[3] Barocas S. and Selbst AD., ‘Big Data’s disparate impact’ (2016) 104 Calif Law Rev 671. The same can be accessed from here: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2477899 

[4] Unit 2, AI Tools by PVP Siddharatha. For understanding on the basics of Machine Learning, Target Variables and Class Labels, refer to: https://www.pvpsiddhartha.ac.in/dep_it/lecture%20notes/AI%20TOOLS/AITools_Unit-2.pdf 

[5] Barocas S. and Selbst AD. (2016)

[6] Braun T., Pekaric I., Apruzzese G., Hong J., Park J.(2024), Understanding the Process of Data Labeling in Cybersecurity Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, online publication date: 8-Apr-2024. The same can be accessed from here: https://dl.acm.org/doi/10.1145/3605098.3636046.

[7] Barocas S. and Selbst AD. (2016)

[8] Global Entrepreneurship Monitor. (2020). Global report 2020/2021. London: Global Entrepreneurship Research Association.

[9] Murdoch JB., The problem with algorithms: magnifying misbehaviour, published on The Guardian on August 14, 2013. The same can be accessed from here: https://www.theguardian.com/news/datablog/2013/aug/14/problem-with-algorithms-magnifying-misbehaviour 

[10] Favaretto, M., De Clercq, E. & Elger, B.S. Big Data and discrimination: perils, promises and solutions. A systematic review. J Big Data 6, 12 (2019). The same can be accessed from here: https://doi.org/10.1186/s40537-019-0177-4. 

[11] Crawford K., The Hidden Biases in Big Data, Harvard Business Review (Analytics and Data Science) published on April 01, 2013. The same can be accessed from here: https://hbr.org/2013/04/the-hidden-biases-in-big-data .

[12] Barocas S. and Selbst AD. (2016)

[13] Campus roundup: Returning London graduates help Indian firms go global, published by the Financial Express on June 24, 2013. The same can be accessed from here: https://www.financialexpress.com/archive/campus-roundup-returning-london-graduates-help-indian-firms-go-global-study/1132802/ 

[14] For reference, consider the factors that affect health insurance policy, by Aditya Birla Capital. https://www.adityabirlacapital.com/abc-of-money/factors-that-affect-your-health-insurance-premium 

[15] Barocas S. and Selbst AD. (2016)

[16] Barocas S. and Selbst AD. (2016)

[17] Prof. Borgesius FZ., Study on Discrimination, Artificial Intelligence, and Algorithmic Decision Making, published by Directorate General of Democracy, Council of Europe. The same can be accessed from here: https://www.coe.int/en/web/artificial-intelligence/-/news-of-the-european-commission-against-racism-and-intolerance-ecri-