Publications
“Algorithms are opinion embedded in mathematics,” wrote American mathematician Cathy O’Neil, author of “Weapons of Math Destruction.” Every algorithm, to some extent, carries with it the values and opinions of those who built it. It is this characteristic that defines it as a technique and as a tool capable of operating wonders, at the same time that allows it to learn from humans, in a process of reproducing certain behaviors, among them, sexism, that is, gender discrimination.
Such reproduction is so likely that last year, Amazon was accused of having created a recruitment algorithm which “learned” that male candidates were preferable over women. On this, the North American giant preferred not to respond to the accusations. In the same vein, a study published in the journal Science in 2017 reveals that systems based on machine learning replicate the same gender and race stereotypes that humans struggle to control. But where do these stereotypes come from?
In the case of Amazon’s algorithm, the problem materialized because the system was trained with resumes submitted for the company’s analysis during the previous ten years, and most of them came from men. In other words, the algorithm itself was not biased, but the database was.
Already in 1996, an article published by Batya Friedman and Helen Nissembaum titled “Bias in computer systems” drew attention to the bias that computer systems can exhibit. They established three categories of biases that the system may contain: pre-existing, technical and emergent. According to the authors, “pre-existing bias originates from social institutions, practices and attitudes. Technical bias arises from technical constraints or considerations. Emergent bias arises in the context of use.“
Furthermore, Friedman and Nissembaum conceptualized a biased system as one that ‘systematically or unfairly discriminates against certain individuals or groups of individuals in favor of others.‘ They go on to state that a system discriminates unjustly if it denies an opportunity or a good, or assigns an undesirable outcome to an individual or group of individuals in unreasonable or inappropriate terms.
When this bias is reflected against women, as in the case of Amazon, it serves to reinforce a behavior already socially established, namely, the inequality between men and women. The fact that women are a minority in higher education and in the technology industry is a verifiable piece of data. It is also common knowledge that the salaries paid to women are, as a rule, lower than those paid to men for the same job functions. Therefore, if a machine, adopted in a certain sector of work and use, reproduces these values, it creates a favorable scenario for the perpetuation of this status, with no chance of modifying the social scenario and, consequently, the lives of women and their participation in areas where they are commonly underrepresented.
Based on the reasoning presented above, another can be inferred: the lack of diversity in technology companies imposes the lack of this same diversity in the products they release to the market. Sara Wachter-Boettcher, author of “Technically wrong: sexist apps, biased algorithms, and other threats of toxic tech,” starts from the premise that the technology industry, predominantly composed of white men, creates products that exclude the needs of women and black people. She believes that the focus on diversity in technology should not only be from the perspective of the workers but also from the perspective of the consumers. The discussion on bias and algorithms favors the game of Leonardo Boff, when he asserts that every point of view is the view from a point: the absence of the view from the point imposes the monochromy of points of view, that is, the defeat of diversity, which, in turn, is a prerequisite for innovation.
Wachter-Boettcher also argues that creators of products that use artificial intelligence need to carefully examine the data they use, paying attention to the biases that emerge from this data: it is important to check the system’s failure rates, because low error rates are important, but this cannot be acceptable when the failure always occurs in relation to the same group of people, since this has exactly the effect of reinforcing the exclusionary patterns and those socially established.
The technical research in the field of artificial intelligence is very advanced, and its scope of application is not exhaustible (even though, in the state of the art, there is talk of limits reached regarding processing power and the like). Currently, AI tools are used: by doctors to make diagnoses; by law firms to advise their clients on the best strategies and probabilities of success for certain actions; and by financial institutions to base their decisions on who will receive loans or credits and who will be hired or not. However, there is something that researchers have only recently started to worry about: the social analysis of these tools. It is necessary to observe and analyze the impact of these decisions made by algorithms on social institutions, how this is impacting, for example, the labor market or the economy, with the offer of credit being decided by machines.
Similarly, the law must also be concerned with this automation of decisions. In the event of a wrong, biased, or harmful decision being made by a machine, who is responsible? How was the decision made? What are the determining factors that led to that result? And when the damages are concentrated in specific groups in society? Many issues are still in dispute among the various social sectors, and their answers require grounding in social studies of the application of these technologies.
But how can the problem of biases in machine learning algorithms be solved? Researchers argue that a major driver of bias in these systems is the training data used for the algorithm. Using databases that are more balanced in social representation helps to diversify the pattern worked on by the algorithm. In this sense, measures must be taken to ensure that the databases are diverse and do not under-represent specific groups. Some researchers are already working on this.
Moreover, it is important that training datasets for algorithms include information on how data has been collected and categorized. According to Zou e Schiebinger,
“[a] complementary approach is to use machine learning itself to identify and quantify biases in algorithms and data. We call this conducting an artificial intelligence audit, in which the auditor is an algorithm that systematically tests that the original machine learning model can identify biases both in the model and in the training data.”
In the end, it is important to emphasize that there are methods that can be used to at least mitigate the occurrence of biases in machine learning algorithms, they just need to be used more frequently. In addition, it is necessary for the development teams of such products to be more diverse, so that this concern appears more emphatically in the technology industry. To achieve this, it is necessary to encourage the participation of women, blacks, indigenous people, members of the LGBTQ+ community, that is, minority groups in general, in the field of technology, so that they can feel increasingly represented in the products released to the market. In more conceptual terms: there is an imperative to combine the will of technique and economic will, prevalent in the development of technologies, with the ethical and scientific debate, consistent with interests that are affected by the technologies, but are ignored in their development.
In the same manner, academic research should focus on the analysis of these tools in an interdisciplinary way. Inserting disciplines or activities (inherent skills construction) other than computer science into research and development teams for tools, and consequently inserting the thinking coming from the humanities, from the creation of algorithms to their implementation, will generate greater reflection on the social, legal, and political effects of their adoption.
Finally, using the example of Amazon’s algorithm mentioned earlier: it is necessary to reflect on the socially established gender patterns, so that they are not perpetuated through technology. I dare to consider that, in an ideal world, technology would be capable of reversing these socially imposed roles and implementing feminism through technique, bringing about the equality for which we have fought so hard.
