University of Southern California Dissertations and Theses

Toward counteralgorithms: the contestation of interpretability in machine learning

(USC Thesis Other)

Toward counteralgorithms: the contestation of interpretability in machine learning

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Copyright 2022 Catherine Griffiths
TOWARD COUNTERALGORITHMS:
THE CONTESTATION OF INTERPRETABILITY IN MACHINE LEARNING
by
Catherine Griffiths
A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(CINEMATIC ARTS (MEDIA ARTS AND PRACTICE))
May 2022
i
Table of Contents
List of Figures .………………………………………………………………………………………… iii
Abstract .……………………………………………………………………………………………….. v
Introduction ……………………………………………………………………………………………. 1
Chapter 1 - On complexity and interpretability ……………………………………………………. 7
Section 1 ……………………………………………………………………………………………….. 7
Complexity ……………………………………………………………………………………. 7
What is the interpretability problem? ………………………………………………………. 7
What is deep learning in the context of artificial intelligence?………………………….. 8
Why is the interpretability problem significant?………………………………………….. 13
Deep learning is a paradigm shift in computing………………………………………… 17
Explainable AI is science’s answer to the interpretability problem……………………. 18
Expanding the interpretability problem outside of science……………………………. 21
Section 2…………………………….…………………………….…………………………………. 24
A critical theory of networks…………………………….…………………………….…… 24
Moving from the problem of interpretability to the contestation of interpretability….. 24
Networks are universal, plural, heterogenous, and intersect…………………….……. 25
Networks as emergent constituent power…………………….…..…..…..…..…..…….. 27
Networks are not neutral…………………….…..…..…..…..…..……..…………………. 28
The relationship of control to deep learning models…………………….……………… 30
The problem with generative knowledge…………………….…………………………… 31
The hostility of complexity…………………….……………………………………………. 34
Chapter 2 - Code is a political language; code is a human language………………………… 40
Section 1………………………………………………………………………………………………. 40
Code is a political language……………………………………………………………….. 40
Political legitimacy in an algocracy……………………………………………………….. 42
The implications of emergent code in Open AI’s GPT3 model………………………… 43
In an algocracy code is law……………………………………………………………….. 44
The first materially affective discourse…………………………………………………… 45
The robustness principle: political philosophy in code…………………………………. 46
Ted Nelson’s two-way computational architecture………………………………………. 47
The problem of bias augmentation……………………………………………………….. 51
Bias augmentation is a form of structural injustice………………………………………. 57
Section 2………………………………………………………………………………………………. 59
Bringing the human into code……………………………………………………………… 59
Human code is interpretable code………………………………………………………… 60
Interpreting code’s meaning is conflicted………………………………………………… 61
Accessibility beyond the black box……………………………………………………….. 65
ii
Chapter 3 - Convolutional Domains: a critical software application…………………………… 70
Section 1……………………………………………………………………………………………… 70
A practice-theory approach to critical software development………………………… 70
Convolutional neural networks…………………………………………………………….. 70
The lineage of convolutional neural network visualizations……………………………. 73
A framework for reflexive software……………………………………………………….. 75
Section 2……………………………………………………………………………………………… 79
An overview of the Convolutional Domain critical software……………………………. 79
Against code agnosticism…………………………………………………………………. 95
Toward contextual code…………………………………………………………………… 101
The politics of labor, surveillance, and the robotization of the worker’s body………. 105
A case study of a CNN used for labor productivity evaluation……………………….. 116
The contestation of interpretability……………………………………………………….. 120
The unmodelled…………………………………………………………………………….. 124
The science of wellbeing……………………………………………………………………129
Using subjective wellbeing research as unmodelled parameters……………………. 132
The dissolution of accountability………………………………………………………….. 136
Moral weights……………………………………………………………………………….. 137
Section 3……………………………………………………………………………………………… 141
Three tactics of critical software design:…………………………………………………. 141
Visualizing algorithms………………………………………………………………………. 141
Slow computation…………………………………………………………………………… 148
Visualizing process…………………………………………………………………………. 150
Conclusion……………………………………………………………………………………………. 156
Bibliography………………………………………………………………………………………….. 158
iii
List of Figures
Figure 1. Agnosticism. The the initial arrangement of the interface upon opening the software.
The images are positioned overlapping, and the user must move them around the screen to
sort and see them all.
………………………………………………………………………………………………..………… 80
Figure 2. Agnosticism. When selected, each image is an input to a convolutional neural
network.
………………………………………………………………………………………………..………… 81
Figure 3. The application contains six different labor contexts in the form of video inputs. This
labor context is agriculture.
………………………………………………………………………………………………..………… 81
Figure 4. This labor context is textiles.
………………………………………………………………………………………………..………… 82
Figure 5. This labor context is warehousing.
………………………………………………………………………………………………..………… 82
Figure 6. This labor context is construction.
………………………………………………………………………………………………..………… 83
Figure 7. This labor context is an office.
………………………………………………………………………………………………..………… 83
Figure 8. This labor context is driving.
………………………………………………………………………………………………..………… 84
Figure 9. Contestation of Interpretability showing the context of agriculture. Information in the
bottom-left cell details specific attributes of the agriculture industry, including the type of
surveillance already implemented or being proposed and its associated ethical issues. The
top-right cell shows the labels and prediction accuracy of the algorithm. The bottom-right cell
shows an excerpt of the algorithm's nodes or filters that the input video passes through before
being assigned a label.
………………………………………………………………………………………………..………… 86
Figure 10. Contestation of Interpretability showing the context of textiles. A user can hover over
each network node to magnify it and see its position in the network structure.
………………………………………………………………………………………………..………… 87
Figure 11. Contestation of Interpretability showing the context of warehousing.
………………………………………………………………………………………………..………… 87
Figure 12. Contestation of Interpretability showing the context of construction.
………………………………………………………………………………………………..………… 88
iv
Figure 13. Contestation of Interpretability showing the context of an office.
………………………………………………………………………………………………..………… 88
Figure 14. Contestation of Interpretability showing the context of driving.
………………………………………………………………………………………………..………… 89
Figure 15. Dissolution of Accountability displays the whole network structure. A user can hover
over each node to magnify it.
………………………………………………………………………………………………..………… 90
Figure 16. Unmodelled showing the proposed counterdata parameters applicable to the
context of agriculture.
………………………………………………………………………………………………..………… 91
Figure 17. Unmodelled showing the counterdata parameters applicable to the context of
textiles.
………………………………………………………………………………………………..………… 92
Figure 18. Unmodelled showing the counterdata parameters applicable to the context of
warehousing.
………………………………………………………………………………………………..………… 92
Figure 19. Unmodelled showing the counterdata parameters applicable to the context of
construction.
………………………………………………………………………………………………..………… 93
Figure 20. Unmodelled showing the counterdata parameters applicable to the context of an
office.
………………………………………………………………………………………………..………… 93
Figure 21. Unmodelled showing the counterdata parameters applicable to the context of
driving.
………………………………………………………………………………………………..………… 94
Figure 22. A film still from Frank and Lillian Gilbreth’s motion studies analyzing bricklayers at
work. Internet Archive.
………………………………………………………………………………………………..………… 107
Figure 23. A film still from Frank and Lillian Gilbreth’s motion studies analyzing an office work.
Internet Archive.
………………………………………………………………………………………………..………… 108
Figure 24. A video still from the software demo of ‘Construction Activity Recognition with Two-
Stream Convnets’, by Luo et al., 2018.
………………………………………………………………………………………………..………… 116
Figure 25. An image from the research paper by Luo et al., 2018, detailing how the image data
is analyzed based on color and motion-tracking in both x and y axes.
………………………………………………………………………………………………..………… 118
v
Abstract
This dissertation aims to reframe an understanding of deep learning algorithms, as they are
applied in socially sensitive domains of society, to one that does not separate technical
explanation from critical discernment, but recognizes them as inextricably interconnected, both
at the level of source code and across intersectional networks of application. I argue that we
need to move away from a solely technical framing of the interpretability problem in machine
learning and move towards a framing of the contestation of interpretability. This is my term to
define a way to grapple with the intersectional ethical, philosophical, and socio-political issues
that deep learning technologies raise. To do this, I have created a practice-theory approach for
developing critical software, software that reflexively presents the means of its own production
for visual and interactive engagement. We need to build technical-critical tools that situate
contestation across intersecting networks and stakes. The problem of bias augmentation is a
second term that I have conceived to situate the ethics of deep learning, precisely the problem
of bias, directly in source code via the algorithm’s weights. I argue that we should critically
reconsider weights as another space of ethical contestation and tool to structure the values
encoded into an algorithm. Through the problem of bias augmentation, the structure of a deep
learning model connects to structural injustice in society. Finally, I propose a strategy for the
foundation of future research in critical algorithm studies and visual design called
‘counteralgorithms’ to contest normative assumptions in algorithms and propose new
counteralgorithmic imaginaries.
Chapter 1 explores the nature of complexity and the interpretability problem at the heart of
deep learning algorithms and how this poses philosophical and ethical problems when
causality becomes inaccessible in generative decision-making systems. Recent advances in
vi
deep learning have enabled broad and rapid deployment across multiple socially sensitive,
high-stakes societal domains, which have not previously been subjected to artificial
intelligence. These domains include the justice system, policing, healthcare, education,
welfare, and employment. When decision-making in such domains is automated by algorithms
that cannot be verified and offer no means of redress, we cannot be sure that such high-stakes
decision-making is correct and free from gender, race, and class biases. Additionally, there are
deeper concerns about the effects of deep learning algorithms on traditions of governance,
democracy, and the robotization of workers. Ceding agency to uninterpretable autonomous
decision-making systems that are distributed across society raises philosophical issues.
Explainable AI (XAI) is a subfield of machine learning created to study the interpretability
problem and extrapolate the internal processing of training deep learning models. However,
the notion of explainability is a particularly reductive framing of the problem by computer
science. I argue that the interpretability problem should be expanded into a distributed
sociotechnical problem. The arts and humanities have a rich legacy of doing interpretative
work. They are equally equipped to investigate the interpretability problem, expanding its
framing to explore the meaning generated within a deep learning network and when that
network intersects other societal networks. The arts and humanities recognize that meaning is
subjective, contextual, culturally defined, and uncertain. I argue that the interpretability
problem in deep learning needs to be reframed as an interdisciplinary problem to incorporate
this notion of contestation, which is philosophical, ethical, and socio-political. The
interpretability problem should be redefined as the contestation of interpretability.
vii
In the second part of chapter 1, I utilize ideas from the humanities to explore a critical theory of
deep networks and to think through the behaviors that complex networks exhibit that also
provide for intersectional socio-political meaning. This includes considering how networks can
reorganize intelligence, control, and power in relation to deep learning. The notion of an
emergent and self-organizing intelligence that is implemented, trusted, and permitted agency
raises epistemological questions about the production of inscrutable and elusive knowledge. In
this way, the interpretability problem critically expresses an unhumanity. From the perspective
of a critical complex network theory, in which interpretability is recognized as a space of
contestation, as I propose, issues of masked hostility and consent arise. The concept of the
swarm has been used to give representation to this faceless form of otherness, which is both
elusive, controlled, and unaccountable. From this perspective, deep networks risk becoming
an expression of a decentralized, deregulated, privatized, neoliberal political imaginary. The
allure of their computational innovation blinds the reality of how they externalize negative
consequences, deny systemic relations, and mask accountability.
Chapter 2 considers ways to rethink code as an inherently political medium, and I propose
programming new computational architectures with this knowledge. I advocate for the concept
of algocracy as a critical way of thinking about shifts in code and power, governance, and
democracy. Algocracy is both an affordance of algorithms directly located in lines of code and
it is a consequence of their application across distributed sociotechnical systems. In the deep
learning era, it is important to keep track of this new form of authority, which structures
behaviors, redefines rights, and negates legitimacy. If we want to address human rights issues,
notions of freedom, and governance, we need to look at how code is written and recognize that
our values are also structured and facilitated there. In this way, code can be thought of as
viii
materially affective critical discourse.
I offer several examples of how social and ethical values can be built into computational
architectures. From notions of trust and accountability to models of economics and democracy,
code can be an important part of how we imagine, design, and structure society. Without such
code consciousness, opposing values and desires can be encoded. Bias augmentation is my
term for thinking through the relationship between deep learning and issues of systemic
injustice. It is a tactic to expand the focus from the idea that algorithmic bias lies solely in the
training data and investigate how the design of an algorithm’s decision-making model, its
internal structure, and iterative process can increase bias from an initial latent seed. Weights in
deep learning could be thought of as a mechanism sensitive to structural inequality. Weights
are mechanisms to emphasize or suppress information in relation to other information. In deep
learning, weights can be designed to sculpt specific outcomes. I argue that we should
consider weights an intersectional domain, both as a computational architecture and a
potential place of inequity in algorithms. Today, the term algorithmic could be synonymous with
the term systemic. Algorithms are sociotechnical infrastructures. In this way, bias augmentation
in deep learning algorithms can play a role in executing a form of systemic injustice.
The second half of chapter 2 focuses on how code should also be thought of as an inherently
human language by bringing the legacy of humanist interpretative thinking to bear on source
code. From the field of critical code studies, we can recognize that code is an omnipresent
cultural text that holds too much influence to be left to machine parsing and computer science,
which would otherwise claim its ideological neutrality. We need to develop different forms of
accessibility to be able to interpret source code’s meaning beyond functionality and hold
ix
space for the contestation of that meaning.
Chapter 3 presents a critical design research project, Convolutional Domains, as an
experiment in critical software development. The project works with a convolutional neural
network (CNN), trained to classify objects in imagery. It is a type of deep learning algorithm
behind many computer vision applications, including facial recognition. In developing this
project, I have referenced a range of other CNN visualization projects that exemplify XAI
techniques to the traditional framing of the interpretability problem. My project advances on
these techniques to propose ways to visualize and critique CNNs from the perspective of my
reframing of the contestation of interpretability. In this hybrid practice-theory approach, I have
developed a framework that I call reflexive software to visually and critically unfold intersecting
sociotechnical systems of computation. I take an abductive approach to knowledge production
by designing software applications that foreground a reflexive consideration of their means of
production.
In the second part of chapter 3, I lay out five core critical design strategies, explore their
theoretical position, and describe their interactive and compositional design attributes in the
software Convolutional Domains. The five core critical design strategies are:
◦ Against code agnosticism
◦ Toward contextual code
◦ The contestation of interpretability
◦ The unmodelled
◦ The dissolution of accountability
x
‘Against code agnosticism’ considers the problem of code and data developed in one context
and ported and reused in a different context. This creates assumptions about false notions of
neutrality, universalism, framelessness, and placelessness in code and data, which allows
them to be developed a-contextually across different societal domains without considering the
ethical conflicts that arise from this agnosticism. The widespread use of the large ImageNet
data set is an example of this. Agnosticism in code and data negates the ideologically
constructed project of data sets and deep learning.
‘Toward contextual code’ presents ways to reinsert lost context, provenance, and place into a
critical reading of a CNN. Situated knowledge is a concept used to contest objectivity, insert
positionality, and counter the drive towards optimization and convergence in favor of
foregrounding a different set of stakes and stakeholders. This critical design strategy aims to
expand on the notion of difference in data and code rather than erase difference for the sake of
legibility and consensus. I frame the entire project around the political context of labor,
specifically workplace surveillance and the robotization of the worker’s body. An emerging
application of CNNs, alongside other machine learning techniques, is taking place in
workplaces to monitor, manage, and evaluate workers for increased productivity and
optimization. This is an extension of a long legacy of industrial labor exploitation through the
C19th and C20th; however, deep learning offers a type of micro surveillance and control that
was not previously available. I use as a case study, applied research into a new method for
using CNNs on an industrial site to monitor and assess the productivity of construction
workers.
xi
‘The contestation of interpretability’ visualizes how to reframe the interpretability problem in
CNNs. Explainability in CNNs is usually reduced to highlighting features in neurons. This
reduces complex socio-political issues to computational logic and method. In my application, I
juxtapose counterdata into the visualization in the form of contextual information and proposals
for alternative data features to deny convergence and create a sense of contention and
ambiguity within the logic of the algorithm.
‘The unmodelled’ presents a speculative counterdata set that moves toward building a
counteralgorithm. The unmodelled is a tactical representation of data parameters that the
algorithm has not modeled and conceptually challenges the assumptions, values, and
ideology at play. The unmodelled data parameters serve as a critical counterdata strategy that
has been conceived from research into subjective wellbeing and wellbeing in the workplace.
The science of subjective wellbeing attempts to account for a phenomenon that resists
measurement: individual and societal feelings around life satisfaction, happiness, and a sense
of meaning in life. I selected metrics from publicly available data sets and wellbeing research
methods that speak to the context of labor, workplaces, and the anticipated rise of algorithmic
management tools. My counterdata parameters juxtapose alternative values and imagined
futures. They point to what is not in the data more than what is. They visualize the gap, the
disruption between meaning and function. They propose a critical algorithm strategy toward
counteralgorithms.
‘The dissolution of accountability’ presents the entire network of neurons and weights. It
visualizes the problem of when the decision-making process is dissolved into a complex
xii
generative system, provenance is lost, causality is forfeited, and consequently, there is a
dissolution of accountability for decision-making agency. This foundational ethical and political
conflict is at the heart of deep learning abstraction deployed into sensitive high-stakes
domains. It presents the dissolution of meaning between the input image and the prediction
label. Far from making an argument about the possibility of legibility, seeing the whole network
and navigating through the increasingly abstract filters is a critical design strategy that
presents the failure of explainability. This scene does not offer a meaningful explanation but a
lack of contestation. I present this view of the algorithm to reject the notion of explainability and
reframe the need to grapple with contestation when we try to explain deep learning. Weights
are also reframed in the context of research into subjective wellbeing to politicize them in
juxtaposition with the concept of moral weights.
The third and final part of chapter 3 offers an additional three critical design tactics: visualizing
algorithms, slow computation, and visualizing process. As a result, they more generally afford
access to working with a range of different algorithms to extrapolate their functions to make
them available for critical discussion.
1
Introduction
This dissertation research has been conducted in the context of the emerging discourse on the
ethics of machine learning technology and the real-world problems that arise when machine
learning algorithms are deployed in socially sensitive high-stakes domains of society,
especially domains that have not previously been subjected to artificial intelligence. Such real-
world problems include algorithms that embed gender and racial bias that transfers into
discriminatory practices. For example, this occurs when machine learning-driven hiring
processes exclude a larger proportion of women or when machine learning-driven policing
practices predict people of color as crime suspects at a higher rate than white people.
Algorithmic bias is a new frontier of social justice that is being reckoned with and resisted.
Women, including women of color, have led much of the most prominent work to reframe the
white male legacy of algorithmic technologies and demand ethical reconciliation and greater
accountability for the harms that algorithms enact.
The problem of machine learning bias partially stems from historical data practices that have
promoted false notions of neutrality and objectivity in data collection which has led to bias
being unintentionally embedded into data sets. This recognition that data is a political space
was explored by Lisa Gitelman in “Raw Data” Is an Oxymoron, in which she argues that data
cannot ever be considered to exist in a raw state, like a natural resource, but are always
undergoing a process of interpretation. (2013) Machine learning algorithms and the data sets
they are trained on reflect historical priorities, preferences, and prejudices; they also reflect the
values and desires of the people and institutions who fund and build them.
2
In her book, Cathy O’Neil coined the term Weapons of Math Destruction to describe the harm
that machine learning models can do when applied in sensitive situations, such as firing a
person from their job while remaining black-boxed and unavailable to scrutiny or appeal.
(2016) A new type of machine learning technique, called deep learning, generatively produces
its own complex model that is inscrutable to both the computer scientists who developed it and
the lay people who use it. Moreover, when such a model is used to make decisions that
meaningfully affect people’s lives, the decision-making process is also inscrutable. This is
known as the interpretability problem, and it is at the heart of the ethical and philosophical
ruptures caused by deep learning algorithms.
Despite many machine learning technologies being built from uninterpretable models and
protected by intellectual property rights and thus unavailable to analysis, researchers have
found ways to analyze and reveal injustice. Joy Buolamwini and Timnit Gebru’s Gender Shades
research project (2018) evaluates the accuracy of facial recognition systems from IBM,
Microsoft, and Face++. Despite the appearance of overall high accuracy, the authors identified
bias in the different error rates across different groups of people based on gender and skin
tone. For example, they found that while the overall accuracy of Face++ was 90.0%, the
accuracy for female subjects was only 78.7% versus 99.3% for male subjects. When
measuring the error rate across gender and skin tone intersectionally, the accuracy for darker
skinned female subjects dropped to 65.5%.
In Algorithms of Oppression, Safiya Umoja Nobel takes issue with the search engine as a
political object and means of oppression. She studied the process of searching for specific
terms, the role of the predictive autocomplete feature, and the organization of image results
3
and found that search engines reproduce harmful stereotypes, biased representations, and
discrimination, especially against black women and girls. (2018) Her research questions the
concept of popularity as an algorithmic model for search and the false assumption of
legitimacy that this model confers on results. Nobel questions what this means for minority
communities living in a majority culture, the unacknowledged profitability of sexist and racist
stereotypes, and the lack of accountability for the harms caused.
Ruha Benjamin’s work at the intersection of race and science and technology studies redefines
algorithmic technologies as structuring a carceral continuum, (2019a) in which algorithms
captivate bodies and imaginations within the carceral state. Benjamin argues that race is a
form of technology, a construct used to structure society, “race itself operates as a tool of vision
and division with often deadly results.” (p.19, Benjamin, 2019b) In her concept of the New Jim
Code, Benjamin connects the legacy of racist policies in the United States to contemporary
algorithmic technologies that reproduce racial inequities in the form of racially biased
surveillance, policing, and imprisonment of African Americans. This reality is simultaneously
negated by the claim of many algorithmic technologies to be racially neutral or even to be
remediations of human racial prejudice. One of the ways this occurs is through whiteness
being encoded as a norm in both society and technology.
From the perspective of algorithms and socio-economics, Virginia Eubanks writes about the
relationship between class, poverty, and technology as an evolution of the C19
th
poorhouse
into a new C21st “digital poorhouse.” (2017) She demonstrates how public welfare services
increasingly use sophisticated predictive algorithms, risk models, and automated eligibility
systems to surveil, manage, and criminalize the poor. Eubanks points out how decision-making
4
tools are tested in low rights environments in which already disempowered people have few
expectations for transparency, accuracy, or accountability of the systems they are subjected to.
While many algorithmic technologies promote a cultural narrative of optimization and
innovation; they underlie a cruel social narrative around who is deserving and who is
undeserving when it comes to poverty.
In Surveillance Capitalism, Shoshana Zuboff argues that the shift toward mass data collection
and machine learning prediction products instantiates a new political economy. (2019) No
longer premised on labor and manufacturing, surveillance capitalism is imbued with a logic of
aggressive extraction of value from increasingly intimate and nuanced details of our lives,
some of which are unknown to ourselves. Moreover, many of today’s algorithmic technologies
were developed quickly outside of regulation, in which the slow process of policy-making has
not been able to keep abreast of ethical considerations. Zuboff considers how capitalism in the
C20
th
has threatened the natural world through environmental degradation, natural resource
depletion, global warming, and species extinction. In comparison, C21
st
surveillance capitalism
threatens the human world by turning human embodied experience into raw material for
behavioral modification products sold on behavioral futures markets. Zuboff argues that this
could lead to people themselves becoming automated so that our futures can be predicted for
other people’s gains. She argues that the highly asymmetrical practice of surveillance, data
collection, privacy transgressions, and behavioral modification calls fundamental assumptions
about humanity and civilization into question.
From a technical vantage point within computer science, Explainable AI is a new subfield of
machine learning that works to address the interpretability problem, as it is framed by
5
computer science, by producing technical explanations of complex algorithms. The concept of
explainability, often supported by visualizations, is presented by the computer science
community as a solution to many issues relating to transparency, agency, and ethics.
Additionally, the mathematician, Hannah Fry, has called for people working in fields related to
new algorithmic technologies to take a Hippocratic oath. (Sample, 2019) Traditionally taken by
medical doctors, a Hippocratic oath is a way to establish an ethical foundation and promise to
the work one does. Now that algorithmic technologies have become so powerful, the public
needs to be protected, and Fry argues that mathematicians and engineers must also learn
from the beginning of their training that technology shapes society’s future in powerful ways
and has the potential to cause great harm.
Two independent research organizations, AI Now and Data & Society, work from a social
science and digital humanities context to investigate the ethics at stake when machine learning
algorithms are applied in new societal domains. Focussing on issues of governance,
democracy, labor rights, and the justice system, their work influences and intersects with
government advising and policy making and is an essential step toward better government
oversight. For example, several countries are developing policy around a person’s right to an
explanation when they are subjected to an algorithmic decision-making system.
It is out of this research context and specific contributions to the emerging discourse at the
intersection of algorithms, ethics, and power that my research begins. My research is
instantiated by developing a critical reconciliation with the interpretability problem and
considering the machine learning subfield of Explainable AI from a cultural perspective.
Specifically, I bring a design research approach that combines experimental thinking grounded
6
in critical theory with abductive and practice-based strategies from the arts and design.
Through this approach, I extend the study of interpretability to recognize the problem as an
inherently contested space. From this recognition, I propose ways to use critical-design
strategies to imagine and design a new algorithmic approach to deep learning.
How can we bring the repository of experimental thinking from the critical arts and design to
bear on the future development of computational technologies, specifically deep learning
algorithms? How can we use critical design practice to create access to technically
obfuscated domains in order to open them to critical engagement by broader publics? Finally,
how can we recognize a tangible and material relationship between writing code and affecting
social and political change in society so that we can reckon with the threat of algocracy?
I propose counteralgorithms as the foundation of my critical design research strategy that
combines critical algorithm studies and visual design to develop critical software applications
that foreground the contestation of normative assumptions in algorithms and propose new
counteralgorithmic imaginaries.
7
Chapter 1 - On Complexity and Interpretability
Section 1
Complexity
I would like to begin by defining complexity as a framework to understand the behavior of
systems where multiple diverse entities interact with each other over a network. The behavior of
complex systems can then be described as structures or patterns that emerge from rule-based
interactions between interdependent entities. (p.21, Page, 2010) Such characteristics make
the behavior of complex systems difficult to predict. The study of complexity also observes that
entities within a system are often capable of adapting their behavior over time, leading to what
is termed complex adaptive systems (CAS). Emergence is a higher order level of information,
such as a pattern, that arises out of non-linear relationships via a process of self-organization.
Melanie Mitchell’s study of complexity describes complex systems through their common
characteristics: the display of complex collective behavior, signal and information processing
from both internal and external sources, and an ability to adapt through learning or evolutionary
processes. (p.12, 2009) This framework allows us to establish relationships between systems
as diverse as insect colonies, brains, immune systems, the global economy, and the internet.
What is the interpretability problem?
The interpretability problem is a new concept specific to machine learning that has arisen with
the development of deep learning algorithms. The interpretability problem is defined as the
technical challenge of understanding how an algorithm arrives at its numerical predictions.
8
“Interpretability is the degree to which a human can understand the cause of a decision,”
(Miller, 2017) which has become largely unintelligible with the development of deep learning
algorithms. A programmer writes a deep learning algorithm that trains on a data set and
generates a model to make future predictions; however, the programmer does not understand
why the algorithm arrived at a particular outcome. While the algorithm might be highly
accurate, its predictions can no longer be reverse-engineered back through the algorithm to
trace its decision-making process. The interpretability problem is a problem of complexity, and
the way patterns emerge as information passes in one direction through a system, however,
those patterns cannot be retraced back to their original information. In a deep learning
algorithm, the relationship between cause and effect, between the initial data set and the final
prediction, is very difficult to observe and make sense of, which leads to various ethical,
philosophical, and socio-political problems.
What is deep learning in the context of artificial intelligence?
Deep learning is a term to describe a subset of algorithmic methods that fall underneath the
umbrella of machine learning, a subfield of artificial intelligence. The three terms, artificial
intelligence, machine learning, and deep learning, are frequently used interchangeably but
their differences are important to this research. Artificial intelligence as a field was founded in
the 1950s as the study of intelligent agents: computational programs which can perceive
information from their environment and take actions towards achieving a specific goal. Artificial
intelligence is more popularly understood as machines, specifically software, that attempt to
simulate the intelligence of humans, animals, or plants. Since the 1950s, the field has passed
through several waves of development to investigate areas including reasoning, perception,
learning, planning, mobility, and language processing. The field of artificial intelligence retains
9
a long-held aim to develop artificial general intelligence (AGI), a simulation of intelligence that
is indistinguishable from human intelligence. While such an aim is entangled in philosophical
questions and potentially poses a threat to humanity, it appears to be still a long way off
realization. This research addresses the developments in applied artificial intelligence over the
last ten years, which raise specific ethical problems: concerning traditions of governance and
the possibility of artificial intelligence systems replacing policy and political debate; concerning
automated labor practices and the threat of mass unemployment; and concerning biased
decision-making via the learning of race, gender, and class-based prejudices.
The field of artificial intelligence comprises several approaches, and up until the 1980s,
symbolic AI was the dominant approach. Symbolic AI uses a series of explicit rules written by a
human programmer for how to perform a task, which is executed linearly, and often referred to
as simple if-else statements. This approach is based on binary logic, requires exact input data,
and insists on precision. This approach is also known as hard computing. From the mid-1980s
onward, there was a shift from hard computing to soft computing, also known as computational
intelligence. Computational intelligence looks at problems that cannot be solved with clear
logic and certainty but can accommodate real-world, complex, and messy problems. This
more contemporary approach is less concerned with the achievement of AGI and more
concerned with smaller, specific problems and their application. Computational intelligence
was initially inspired to develop problem-solving methods loosely based on natural systems
such as evolutionary algorithms, fuzzy systems, swarm intelligence, probabilistic reasoning,
and neural networks. Such algorithms support stochastic, incomplete, uncertain, and adaptive
environments, overcoming the precise binary requirements and knowledge-based systems of
symbolic AI. (Russell and Norvig, 2021)
10
From within this shift, in the 1990s, machine learning was further delineated as a formal branch
of artificial intelligence to study more specifically how intelligent agents can autonomously
learn new knowledge or behaviors from noisy data sets. Unlike symbolic AI, the approach is
tolerant of approximation rather than insisting on accuracy. It is pertinent that machine learning
developed as an approach concurrently with the development of the internet and the large
amounts of digital information that was being generated and stored for the first time. Machine
learning is the creation of a mathematical model derived from sample data, also known as
training data. (Mitchell, 2019) The machine learning model is generated during the training
process and could be described as the process of one algorithm autonomously writing its own
algorithm, which becomes the model. The model is the map of all the patterns, pathways, and
rules that the algorithm generated from the training data, a map that would be impossible for a
human programmer to write by hand. This is often described as the algorithm learning from
experience; in this sense, the data set is a representation of a small part of the world, a micro-
environment, and the algorithm learns from experiential data processing rather than from top-
down instruction.
There are three primary types of machine learning: supervised learning, unsupervised learning,
and reinforcement learning. Supervised learning is perhaps, the most widely used and requires
structured data, also known as labeled data, which is data that consists of paired features and
labels, such as a series of images of animals, each labelled with the correct name of each
animal. During the algorithm’s training process, the specific relationships and dependencies
between the data features and their corresponding label will be learned and modeled.
Supervised learning is modeled on a notion of ground truth defined by the assigned labels.
This process is dynamic: the algorithm’s learning takes place through iterative modification and
11
optimization to minimize errors by self-correcting in relation to its objective, based on what is
often referred to as a loss function. Once the model has been trained, it becomes a discrete
computational artifact that can be embedded into software technologies and used to make
predictions on new data that the model has not previously seen. This method is used to solve
classification problems based on discrete values, such as identifying an animal in an image,
and regression problems, based on continuous values, such as predicting trends in stock
prices.
The second type is unsupervised learning, which works with unstructured data: data that
contains a series of features minus their corresponding label. Unsupervised learning is not
based on a notion of ground truth as there are no labels in the data. Instead, the training
process generates a model to extract relationships based on broader commonalities and
differences within the data, such as patterns, features, groups, and general rules. There are no
predefined prediction categories; therefore the process is more complex, self-organized, less
controllable, and more disposed to the problem of dimensionality. This method is often used to
solve clustering problems, density estimation, association rules, and dimensionality reduction.
Unsupervised learning may be used when there is no labeled data available, potentially due to
the high cost of human time spent labeling data; it can also be used to de-noise a data set,
such as satellite imagery or in recommendation systems. Human judgment plays a more
important role because the algorithm doesn’t provide clear and specific answers, and there are
fewer agreed-on principles to quantify its success; therefore there is more scope for subjective
and heuristic perspectives when working with this method.
12
Reinforcement learning is the third type of machine learning, which gathers data from real-time
interactions with its environment via sensors and cameras. It generates a model of behaviors
by calculating the risk and reward against errors. The model learns to perform better via this
feedback signal, this reinforcement. This approach is most often used in robotics, self-driving
cars, and games.
An artificial neural network is an algorithm that can be used in all three types of machine
learning. The concept of an artificial neural network was initially theorized in the 1940s,
originally loosely inspired by biological neural networks in the human brain; it offered a shift in
thinking in cognitive science and philosophy from a symbolic model of knowledge to a
connectionist model. From the connectionist perspective, an artificial neural network model
demonstrates how a neuron can process and communicate information to other parts of a
network. However, the system was not developed further until the 1970s, when the
backpropagation technique was incorporated into the artificial neural network, which offered
the potential for multiple layers in the network. Biological neurons and synapses are modeled
as nodes and edges in an artificial neural network. Each node in a network contains a single
piece of data, and each edge contains a weight that dynamically adjusts during the learning
process to signify the importance of each node. During the training of an artificial neural
network, the generation of a model is the iterative process of fine-tuning all the weights until an
optimal set have been defined.
It was not until 2006 that multilayered neural networks were rediscovered and successfully
implemented and developed into what we now refer to as deep learning. For the first time, an
algorithm was able to recognize high-level information, such as specific animals, in images
13
from unstructured data using unsupervised learning. The cultural development of deep
learning cannot be separated from the technological context that has seen large increases in
computing power and newly acquired access to big data sets. Deep learning networks are
artificial neural networks with multiple layers that progressively build higher-level information
from the input data. In image processing, including image generation and classification, each
layer in the network serves to identify features in an image, abstracting and compositing
information as it passes through the network. If the images contain faces, for example, one
layer might try to extract the general shape of the head from the background, another layer
might try to extract the feature of a nose, another the mouth, and so on, until the network
generatively arrives at a high-level representation of a face. Convolutional neural networks and
general adversarial networks are two more recent classes of deep learning algorithms that
have been successfully implemented for image processing.
Why is the interpretability problem significant?
A deep learning model is a generative complex system that is largely unreadable. In recent
years, we have seen deep learning algorithms implemented into socially contentious domains,
or high-stakes and decision-critical applications, to make decisions that significantly affect
individual lives. Until recently, such a development was inconceivable to the public and
triggered various ethical and socio-political issues. The dilemmas arising from the use of deep
learning algorithms include the likelihood of them generating mistakes and augmenting biases
hidden in data. An investigation by ProPublica showed how such systems encode racial bias
when used in criminal sentencing. (Angwin, 2016) Cathy O’Neil’s work points out how
algorithms used in the workplace, including job appraisal and recruitment, far from being
neutral mathematical arbiters of job performance, can fuel inequality and poverty, due to them
14
being opaque, untested, and beyond redress. (O’Neil, 2016)
The sensitivity around the use of algorithms to make life determining decisions is exacerbated
by “AI’s Unspoken Problem”, described by Will Knight in a 2016 article in the MIT Technology
Review. He describes the issue as being that an algorithm can not tell us why it made the
decision it did; it can only present its predicted answer to a given question. The algorithms
currently in use do not have the quality of common sense or an awareness of context
incorporated into their models. There is an ethical need to be able to ask a deep learning
algorithm why it arrived at a particular prediction, and to receive an explanation, before further
using them as collaborative tools. (Knight, 2016) Until this is the case, there will be a legitimate
degree of unease, suspicion, and a sense of powerlessness, especially when there are few
opportunities to appeal the decisions of deep learning algorithms.
Deep learning algorithms not only need to be interpretable to their own programmers and other
experts, but also to other stakeholders: judges, police officers, social workers, employers, and
any other non-technical people; who are traditionally expected to make decisions that are
accurate, fair, and transparent. The interpretability problem is a problem of tracing causality,
and it engenders issues of trust, fairness, and accountability. In deep learning models, the
relationship between cause and effect is diffused; the relationship between the initial data set
and the final prediction is almost impossibly obfuscated. Additionally, during the autonomous
generation of the deep learning model, it is imperative to check that only causal relationships
are being modeled and not correlative ones. How can we know whether to accept or reject a
decision if we do not know the decision-making process? Researchers at the University of
Washington created a deep learning classifier to distinguish between images of wolves and
15
images of huskies. The resulting classifier appeared to have a 90% accuracy; however, on
further investigation, it became evident that the model had learned to distinguish snow in the
background of images versus images that did not have any snow in the background. The
algorithm only appeared to be accurate because most images of wolves have snow in the
background and most images of huskies do not. (Ribeiro et al., 2016) The deep learning model
had learned a correlative pattern.
One can see that when such systems are working on more socially sensitive data sets, the
consequences of such mistakes can be grievous. In many situations, it is not sufficient to only
know the prediction; even if it is accurate, there must also be an explanation of how the
prediction came to be in order to build trust and not negate accountability. The black box
problem is traditionally understood to be when algorithms are proprietary, and their code is
locked behind intellectual property (IP) rights, creating a lack of transparency to their inner
workings. In the case of deep learning algorithms, the black box problem extends beyond IP
law to the nature of complex self-generating models that inhibit interpretability and access
differently. The black box problem in deep learning has philosophical and ethical dimensions:
the nature of generating new knowledge systems that resist analysis, interpretation, and
accountability.
Machine learning models are understood to offer a trade-off between accuracy and
explainability; as the accuracy of the algorithm increases, our ability to understand it
decreases. Much ethical criticism of machine learning systems has focussed on the original
data sets as harboring and repeating the same sexist, racist, and classist discrimination found
in society. However, the generative knowledge building inside the model is also important to
16
attempt to study, despite its constant elusion. In artificial intelligence, the cultural shift from
hard coding to the computational intelligence approach has allowed the study of messy,
ambiguous data, such as images and text documents. However, its very inventiveness in being
able to structure meaningful patterns from such ambiguity is deeply bound up with a loss of
interpretability. The notion that an algorithm writes its own algorithm from which humans are
precluded from understanding also presents an uncomfortable power dynamic between
humans and the computational agents that we build and then to which agency is ceded.
Agency is ceded when uninterpretable autonomous decision-making processes are distributed
through societal institutions without recourse.
The interpretability problem within deep learning models also problematizes the issue of
fairness. Some algorithms have been found to encode race, gender, or class biases. (Hao,
2019) (Dastin, 2018) While in many situations, the problem has been at least partly located in
data sets that were not diverse or inclusive enough in the first place, there are still many
unknowns about the generation of the model and the problem of bias augmentation, in which
small biases in a data set can be augmented during the algorithmic process. Additionally, the
interpretability problem engenders the issue of accountability. When causality is dissolved and
decision-making is automated, the human is removed from the process, yet the algorithm is not
accountable for the consequences of its decisions in the way society expects decision-makers
to be accountable. Both scientists and frontline workers, such as social workers, are disturbed
at being removed from decision-making and not being able to explain and justify why a
particular decision has been made. Deep learning technologies have quickly entered the
market due to corporate interest and innovation narratives without serious consideration or
governance. Deep learning systems are only marginally controlled by ethics laws, and they
17
also eclipse traditional governance that takes place through public debate and policy-making
via their obfuscated and fast distribution through new societal domains. The interpretability
problem, via the dissolution of causality, also facilitates the negation of governance and
accountability.
Deep learning is a paradigm shift in computing
Deep learning epitomizes the shift in computational culture that is currently taking place. From
the shift from hard to soft computing in the 1980s, from precise rules to a tolerance for
uncertainty in data, results, and representation, machine learning has now paved the way for a
form of emergent computation to dominate. A generative model is a form of self-organized
code that writes itself. From generative models, we get generative decision-making due to the
application of such models to produce autonomous decision-making. Such generative
decision-making could be formalized as a shift toward a form of generative power that
bypasses the traditional institutions of authority and processes of power that traditionally shape
society. In computer science, this social and political shift emanates from a shift in the scientific
method, from the pursuit of artificial intelligence as a theoretical field, based on a deductive
model of mathematical reasoning and symbolic truths, to machine learning as an empirical
science. In contemporary machine learning, progress is made through heuristic methods of
trial and error, in which reductive notions of accuracy and efficiency measure success. What
does this mean for how we arrive at knowledge and determine the governance of power that
arises from generative prediction models?
Adjacently, the field of AI has also changed as a technical endeavor. From its early days as a
primarily academic discipline, it has emerged as an energetically industrious and applied field,
18
driving neoliberal late-capitalism. Academics/entrepreneurs ‘tinker’ and ‘disrupt’ social
domains driven by corporate interests pursuing innovation imaginaries. The field has also
become highly integrated with a fast-paced digital technology industry, where the
development of tools and systems based on machine learning has been democratized. One no
longer requires a formal education or training in AI. One can be a self-taught programmer, who
develops an AI application and puts it into the technology marketplace. This single application
can have transformative social and ethical consequences across society.
Explainable AI is science’s answer to the interpretability problem
In order to address the interpretability problem, the problem of trying to understand how and
why deep learning models produce the results they do, a new sub-field of machine learning
was initiated, known as Explainable AI or XAI, for the scientific exploration of interpretability.
Explainable AI seeks to develop methods to understand and explain the internal mechanics of
deep learning algorithms in human terms. This notion of explainability is addressed mainly to
other scientists in the field and sometimes, in acknowledgment of the societal trust issue, also
to a broader public. Explainable AI offers a shift from focusing on the input and output data to
focusing on the model and the process of training the model.
Examples of XAI research projects include DARPA, an agency of the US Defense Department,
which funds XAI research that seeks to develop effective explanations based on psychology to
build trust in end users. (Gunning et al., 2019) IBM funds XAI research, including a user-
friendly browser-based demo, the AI Explainability 360 Toolkit. Microsoft funds the FATE
(Fairness, Accountability, Transparency, and Ethics in AI) group. A recent paper produced by
the team gives insight into their approach to the concept of interpretability leading to
19
explainability:
“In the context of local explanations for predictions made by ML models, the
phenomenon to be explained is why a model M predicted output y for input x.
This why-question can be operationalized in different ways. The facts used to
explain this phenomenon may include information about the input features, the
model parameters, the data used to train the model, or the manner in which
the model was trained.” (Alvarez-Melis et al, 2021)
The team defines explainability as a translation process of one-to-one correspondences
between input and output. Furthermore, they propose several design principles for
explainability: explanations should be contrastive, i.e., explain why the model predicted y
instead of alternative y; explanations should be exhaustive, i.e., justify why every alternative y
was not predicted; explanations should be modular and compositional, breaking up
predictions into simple components; explanations should rely on easily-understandable
quantities so that each component is understandable; explanations should be parsimonious,
i.e., only the most relevant facts should be provided as components. (Alvarez-Melis et al, 2021)
This approach employs cognitive science ideas that prioritize legibility and simplicity as
understood within a computer science framing of the interpretability problem.
In an overview of the tactics deployed in XAI titled Explaining Explanations, Gilpin et al. layout
several examples of how to reduce the complexity of the operations with a deep learning
network: proxy models have been used that behave similarly to the original model but are
easier to explain, the LIME method is an example of this, as is the use of decision trees to
20
decompose deep learning networks into more simplified systems; various algorithms for
automated rule extraction have been developed, and saliency maps are used to reveal areas
of the network with higher activations and higher sensitivity in which any changes would most
affect the output, this approach is used in convolutional neural networks for image recognition
to locate the focus of visual activity and extract features. (Gilpin, 2018)
Generally, it is understood that there is always a trade-off between accuracy and explainability
and a discrepancy between interpretability and completeness in deep learning models. The
more accurate a deep learning model’s results are, the less opaque its process is, and vice
versa. Deep learning models are never 100% accurate. We can be sure they produce
mistakes, and thus their predictions are just that, predictions that should not be taken as truths.
A deep learning model can estimate the accuracy percentage against test data; however, it
cannot state which predictions are the mistakes. In this way, XAI seeks to create “models that
are able to summarize the reasons for neural network behavior, gain the trust of users, or
produce insights about the causes of their decisions.” (Gilpin et al., 2018) XAI develops
methods to provide insight into deep learning decision-making processes by enabling parts of
the internal system to be more transparent and turn these moments into more broadly
communicable explanations.
Similarly, when it comes to interpreting deep learning decision-making, the search for a
meaningful explanation conflicts with the completeness and accuracy of the interpretation,
which might only be legible to a very few technical experts. Within computer science, the ideal
would be for XAI to produce complete, accessible explanations for 100% accurate deep
learning models, however, this is not possible. Outside of computer science, this is also
21
potentially not ethically desirable.
Expanding the interpretability problem outside of science
The framing of interpretability in the field of computer science is a problem. It is defined within
the siloed culture of science that reduces problems into smaller discrete parts to manage them
and ultimately solve them. I argue that the interpretability problem should not be reduced but
expanded into a sociotechnical problem that cannot be ethically addressed only by the field of
computer science. It could be argued that with the recent developments in machine learning,
computer science has created a paradigm shift. Today computer scientists, including self-
trained developers, can build technologies that can be quickly applied to domains of society
that were previously off-limits, including justice and welfare, for example.
In computer science, the interpretability problem is defined as a translation process of one-to-
one correspondences in a highly complex system, which merely needs to be improved to
become more accurate, legible, and complete to solve the interpretability problem. However, I
argue that we should first recognize this as a particular framing and reduction of a problem
and its solution by the computer and cognitive science culture. If the frame were to change or
expand, a different approach would be required. The problem with XAI and the solution to the
interpretability problem is that it does not recognize the distributed nature of the problem,
which really takes place across social, political, and ethical domains. XAI, even improved
methods than those that currently exist, are unwilling to grapple with the broader ethical
problems caused by deep learning algorithms used and misused in socially sensitive and
high-stakes societal domains. With certain domains, a fair question is whether such algorithms
should ever be designed in the first place, regardless of the legibility of the explanation of the
22
algorithm.
In the XAI literature, computer scientists at times invoke notions of humanism in the
development of XAI, but it is a form of humanism that is entirely framed by science, one that
seeks accuracy, simple legibility, and completeness. The interpretability problem in deep
learning exceeds the boundaries of science into a broader social terrain. There is a long and
rich legacy of doing interpretative work in the arts and humanities. Interpretation is a core
approach, process, and outcome. Interpretation defines the ontological nature of the arts and
humanities, and I argue that these fields are equally as informed and equipped to explore the
interpretability problem in deep learning, significantly expanding its framing and practice.
Interpretability is a practice of exploring the meaning that is generated between things: words,
images, objects, and systems. The arts and humanities recognize that meaning is subjective,
contextual, culturally defined, and always uncertain, and these fields have a capacity to
manage that ‘problem’ and recognize its value. Furthermore, the arts and humanities recognize
that meaning is inherently contested. I argue that the interpretability problem within deep
learning needs to recognize and incorporate this contestation, which is philosophical, ethical,
and political. Computer science's framing of the interpretability problem needs to recognize the
legacy of knowledge and ideas already built by the arts and humanities around interpretability.
Where the field of computer science argues that explainability solutions are intended, in part,
to support the public to develop trust in deep learning systems, computer science needs to
recognize that trust is already ruptured in their definition of the problem. Even the language of
explainability is narrow and implies an entitlement to define what the public should care about.
Furthermore, it implies that there is only one ultimate or preferable explanation, rather than a
23
space of contested interpretation that needs to be grappled with. When the public is given
access to this contested space of interpretation, and from such a position can grapple with
their own thoughts and feelings about the workings and uses of deep learning algorithms, we
can more accurately speak about building trust.
The goal of XAI needs to incorporate humanistic critique that incites contestation and not only
define linear translations that seek consensus. To do this, experts from more diverse
knowledge backgrounds and public stakeholders need to be a part of the field of XAI to build a
broader cultural recognition that to develop technology is to be interpolated into an ethically
contentious domain. The interpretability problem is a cutting-edge issue at the intersection of
AI technology and ethics. There is a problem in treating the development of AI technologies as
a solely scientific problem with a scientific solution when it is a distributed sociotechnical
problem across society, with a need for interdisciplinary thinking. We need to redefine the
interpretability problem by redefining the term interpretability as a contested space.
24
Section 2
A critical theory of networks
As the method of deep learning relies on a complex network structure to organize and parse
information, we can look to a critical theory of networks for ideas about the broader meaning
this creates. A deep learning model is composed of nodes and connections through which
data passes and functions are executed to finally arrive at new knowledge in the form of a set
of output predictions. Traditionally, the science of networks has offered the topology of graph
theory as the overarching form of representation for the structure of points and their relations,
known as vertices and edges. However, critical theory that engages networks and networked
thinking offers alternative representations of networks and the interpretability problem in how it
manifests across sociotechnical applications.
Moving from the problem of interpretability toward the contestation of interpretability
The interpretability problem, as computer scientists term it, could be rephrased as the problem
of interpretability to open it up as a shared concern across several fields of study. This reminds
us that while the issue of interpretability might be a new one in machine learning, it has a rich
legacy of development in the arts and humanities. From this perspective, its is less a problem
of interpretability that has the potential to be resolved in an absolute sense, and more of an
extensible interrogation and contention of interpretability. From a sociotechnical perspective,
interpretability is not solely an algorithmic problem or a problem of technology, nor is it a new
problem. It is a problem of complex networks, which it is argued here, is an issue of philosophy
and representation, engaging epistemology, literacy, and power.
25
In The Exploit, by Alexander Galloway and Eugene Thacker, the authors present ideas in
network science from a socio-political perspective, implicating networks in an emerging form of
political power. The book was published in 2007 and is contextualized through the
development of the internet, peer-to-peer file sharing, the SARS virus, and terrorist groups as
examples of the new landscape of networked power. Galloway and Thacker’s work still has
many pertinent ideas thirteen years later to think through issues of complexity, power, and
deep learning networks.

Networks are universal, plural, heterogenous, and intersect
There is a quality of universality in networks, from a mathematical perspective, in which “the
geometry of graph theory, the mathematics of dots and lines” constitute a “soul of the network”
(p.112, Galloway, 2007) or a “metaphysics of networks”. (p.18, Galloway, 2007) This essence
of networks enables the correlation of heterogeneous phenomena across network forms:
biological viruses, computer viruses, economies, terrorist networks; they can all be represented
and tolerated through their common geometry. Networks offer an almost immediate
representation, albeit at a simplistic topological level. “To in-form is thus to give shape to matter
(via organization or self-organization) through the instantiation of form - a network
hylomorphism.” (p.112, Galloway, 2007) In this way, representation through form-building is a
mode of, not only communication but of the generation of knowledge.
The heterogeneous domains that networks afford should also be understood as encapsulations
and combinations of several networks. A complex network is ontologically rarely only singular.
Networks are connected to and reliant on other networks. In 2007, Galloway and Thacker wrote
about networks in the context of the 2002 SARS outbreak in China and the terrorist networks
26
that came to prominence following the 2001 World Trade Center attacks in New York. It is
opportune to relate Galloway and Thacker’s ideas to the current juncture, at the time of writing
this dissertation, of global network expression: the COVID-19 pandemic and international
protest movements, including the 2019 Hong Kong protests for civil liberties and the 2020 anti-
racism protests following the police murder of George Floyd in Minneapolis. With the example
of COVID-19, the biological network of the virus cannot be disentangled from the network of
epidemiological infection, or healthcare networks, from communication networks including
national media, government information processes, shifts in personal communications as a
consequence of a lockdown, from institutional networks such as the World Health Organization
and vaccine research, or transit networks including airports and commuter traffic. Networks
inescapably bring other networks together.
The generative complex network that is a deep learning model can be both an object of study
and inextricably intersected with these other networks. During the 2019 Hong Kong protests
against the Chinese government’s amendment of the extradition law that would have infringed
on the civil liberties of the semi-autonomous Hong Kong, predictive policing using facial
recognition technologies came to be a key issue in both the nature of the violence and the
resistance of the protestors. (Jacobson, 2019) Likewise, during the 2020 anti-racism protests in
the US, specifically targeting police brutality, corporate developers of facial recognition
technology, including IBM and Amazon, were pressured to put a moratorium on future
development and sales to police departments due to its implication in racist police brutality.
(Greene, 2020) During the COVID-19 pandemic, contact tracing applications have been
developed to support the lifting of lockdowns and the re-activation of national economies. In
this way, deep learning models are both complex networks in themselves and inextricably
27
interlinked with other complex networks, both for their reliance on large social data sets to train
their models and how they re-interpolate their modeling back into society and other complex
networks through their predictions.
Networks as emergent constituent power
In looking at a critical theory of networks, we gain useful ideas to engage with deep learning
models and the contention of interpretability. In order to contend with networks, Galloway and
Thacker first call to “realiz[e] an ethics and politics of networks, an activation of political
consciousness that is as capable of the critiquing of protocological control as it is capable of
fostering the transformative elements of protocol. What would a network form of practice be
like?” (p.100, Galloway, 2007) Here protocol points to the rules that establish networked
relations and for our purposes, point to the inner dynamics of deep learning algorithms,
suggesting that the task of looking inside deep learning algorithms, at their otherwise hidden
calculations, is a step toward a political consciousness of the algorithm, and a praxis of deep
learning models. As we acknowledge that such mathematical systems are also political
systems, “any theory addressing networks will have to entertain a willingness to theorize at the
technical level.” (p100, Galloway, 2007) Therefore, a praxis of deep learning models means
that “to write theory means to write code,” (p.100, Galloway, 2007) and consequently, to write
code engenders a requirement to engage with critical theory.
The notion that networks need to be understood as a relation between one node and another
node, and also concurrently, as a comprehensive will of relations across the network, and
furthermore, an intersectional emergent agency, is important to how we understand them as
socio-political systems. There is both a system of many individual relations and a larger force
28
of the network as a whole that we need to contend with simultaneously. This brings the issue of
agency to the fore: where is agency and political power in the network, and is agency always
human? Galloway and Thacker ask “Is the multitude always ‘human’? Can the multitude or
netwars not be human and yet still be political’?” (p.153, Galloway, 2007) They attempt to move
the question of agency away from its instrumental use through tools and toward “the nature of
constituent power in the age of networks.” (p.153, Galloway, 2007) Networks and today’s deep
learning systems can also be rethought in terms of how they instrumentalize not just software
applications but an emergent form of constituent power. When we contend with interpretability
in deep learning systems, we contend with this form of emergent constituent power.
Networks are not neutral
There has been a longstanding assumption that technology is a politically neutral domain
ensconced in a pure search for innovation and that its impacts in social and political domains
should be considered separately from its development. That view is now changing as society
acknowledges the impossibility of disentangling technology development from its socio-
political context. Networks contain a similar misinterpretation: neutral structures that merely
host social and political content. However, networks are not neutral either; rather, in
computational, biological, social, or political networks, the network organization always defines
and structures control and power, rather than providing a benign substrate.
“The preponderance of network forms of organization in no way implies an
absence of power relations or of control structures. In fact, it prescribes them.
But the kinds of power relations and control structures are not necessarily
those of top-down hierarchies. They are of another order.” (p.70, Galloway,
29
2007)
As control structures, networks are both technical and political structures simultaneously; the
network forms should not imply an absence of power but rather reveal what network structures
produce, which is:
“the experience of systematicity itself, of an integration of technology, biology,
and politics. Networks structure our experience of the world in a number of
ways, and what is required is a way of understanding how networks are
related to the aggregates of singularities - both human and nonhuman - that
are implicated in the network.” (p.70, Galloway, 2007)
The quality of integrative systematicity as an experience of the world, in which both human and
nonhuman components share an environment, offers a way to understand and live inside
networks without denying their structuring of control. Regarding structure, there is another false
assumption that networks are inherently more horizontal and therefore egalitarian. Galloway
and Thacker refer to this as “the tired mantra of modern political movements, that distributed
networks are liberating, and centralized networks are oppressive. The truism of the Left may
have been accurate in previous decades but must today be reconsidered.” (p.13, Galloway,
2007) In fact, network structures can take on different shapes, including centralized,
decentralized, or distributed, and these various shapes reorganize, not remove, control and
power. Contextualizing this issue in terms of the rhetorics of software, theorist, James J. Brown
Jr, writes about the need to “resist the temptation to theorize networks as open, free,
rhizomatic, or flat. Networks are not free of hierarchies, and they feature top-down assertions of
30
power by way of protocols.” (p.11, Brown, 2015) The nature of a network to have “this
multiplicity of nodes in no way implies an inherently democratic, ecumenical, or egalitarian
order. Quite the opposite. (…) it is this existence-as-such of networks that needs to be thought;
the existence of networks invites us to think in a manner that is appropriate to networks.” (p.13,
Galloway, 2007) Galloway and Thacker invite us to consider a network-appropriate form of
thinking, as one that engages networks philosophically, ideologically, and topologically, to
access how network structures manifest values, attitudes, and strategies and rules of
engagement to express forms of control and power.
The relationship of control to deep learning models
To recap on the previous outline of techniques in AI, the dominant approach used up until the
1990s is known as symbolic AI, and can be described as when a human programmer writes
explicit rules based on exact data using binary logic that is executed linearly. From the 1990s
onwards, there has been a shift toward computational intelligence and machine learning
techniques in which rules are generated or learned experientially from stochastic, incomplete,
and uncertain data by mapping relationships. With symbolic AI, the model is hierarchical,
binary, and explicit; with deep learning, the model is emergent, relational, and messy. In
symbolic AI, control is expressed clearly; however, in deep learning, network control is
dispersed, intangible, masked, yet still very much taking place, just in a reorganized manner. In
deep learning, decision-making does not have a one-to-one relationship from the algorithm to a
social domain but is a computationally generative process that extends and is interpolated
across other social systems. Despite the qualities associated with deep learning - more
horizontal, self-organizing, organic, and related to the real world - this does not mean that deep
learning models are more egalitarian, fair, or natural. On the contrary, they invoke a shift in how
31
power and ethics are encoded and performed. This is the problem of interpretability, the covert
reorganization of control, power, and intelligence, that we must address in developing a critical
theory of deep learning.
The problem with generative knowledge
An epistemological problem arises in taking specific issues with the emergent and self-
organizing quality of deep learning models. In the context of AI, a field originally tasked with
modeling human intelligence in technology, the invention of deep learning is now potentially
creating new forms of knowledge from intelligence. It is not that deep learning models
themselves produce knowledge by themselves, but that they are implemented, trusted, and
permitted agency in a way that shifts notions of knowledge production toward generative
systems that are inscrutable. This raises an epistemological question regarding how society
defines knowledge and arrives at a consensus. The potential nature of such a generative
knowledge, that deep learning algorithms are redefining, is indeterminate and inscrutable,
which is very different from how knowledge is traditionally built and agreed upon in the
sciences and humanities through argument, evidence building and persuasion. Deep learning
networks are potentially shifting the terrain of epistemology. In Protocol, Galloway presents how
control takes place in networks after decentralization, following a long period of culturally
associating control with the centralization of power, while the decentralization of power is
associated with disruption. (Galloway, 2004) This argument maps into deep learning networks
and decentralized networks to posit that generative knowledge, while taking place outside of
the traditional boundaries of epistemology, still expresses control and power via its new ability
to redefine knowledge and simultaneously evade definition.
32
The problem with generative, deep learning-based knowledge is how it eludes palpable
discernment. The underlying structure of this generative knowledge is a complex system or a
multiplicity of complex systems that can be ever-adaptable and hard to pin down, yet reify
whenever required to substantiate and self-validate, "[t]hey are forever incomplete but always
take on a shape." (p.156, Galloway, 2007) How can knowledge be questioned, tested, or
trusted if "this ambient aspect of networks, this environmental aspect" are elusive by nature?
(p.157, Galloway, 2007) The ambient, elemental quality of complexity expresses this
everywhere and nowhere of deep learning intelligence, this "scaling for which both the ‘nothing’
of the network and the ‘universe’ of the network are impossible to depict." (p.156, Galloway,
2007) Ambiance is an appropriate term to describe the capacity of deep learning to
circumvent human control, operate above and below the human, and ultimately decline
interpretation. "Networks are constituted by this tension between unitary aggregation and
anonymous distribution, between the intentionality and agency of individuals and groups on
the one hand, and the uncanny, inhuman intentionality of the network as an ‘abstract’
whole." (p.155, Galloway, 2007) This notion of scaling between the micro and macro of
computations on individual data points and generative patterns is another means of evading
where intelligence, knowledge, and power take place.
There is also a cultural misunderstanding of generative intelligence that serves the slippery
nature of deep learning well, inadvertently bestowing a form of sovereignty. Finally, when a
phenomenon is not understood technically and culturally, there is sometimes a tendency to
imbue it with seductive notions of the supernatural to give it positionality and meaning.
Throughout history, cultural attempts to interpret networked phenomena have taken form as
signs of divine retribution or divine providence, harbingers of the apocalypse, punishment for
33
humanity's hubris, the revenge of nature, the infocalypse, the use of spectral figures haunting
systems and crises.
Deep learning has likewise benefitted from its own mysterious nature, preventing scrutiny and
emboldening further development and fast application. The generative intelligence in deep
learning networks both expresses an otherness and simultaneously denies the other. “The
paradox of networked relations is thus the following: the networked other is always obscured,
but experiencing the essence of the other, even in its obscurity, is assumed to be the ultimate
goal of any networked relation." (p.292, Galloway, 2010) The paradox of the networked other is
how a complex network, as a totality, is inherently obscured as an operational system. Yet, the
phenomenon of emergence in the form of generative intelligence is too captivating to ignore
while still averting the clutches of reason.
From the paradox of networked otherness, generative knowledge also contests detection via a
quality of unhumanity. This is not to suggest that deep learning is anti-human, in the sense of
techno-futurist nihilism, but to raise the question of how human-centered deep learning's
trajectory is. Deep learning certainly contains the human in the sense that it is a technology
constructed by humans, and it is also a system that is constituted by human subjects in the
form of social data points on which its models are trained; however, it transcends the human by
way of permitting its own epistemological agency that humans cannot extrapolate.
The interpretability problem is both the obstacle and the inventiveness of deep learning.
Generative knowledge is an uncomfortable discussion regarding "the unhuman within the
human," (p.154, Galloway, 2007) and a way of escaping "our understanding of networks [as]
34
all-too-human." (p.157, Galloway, 2007) The unhuman is an expression of the interpretability
problem. "The nonhuman quality of networks is precisely what makes them so difficult to grasp.
They are, we suggest a medium of contemporary power." (p.5, Galloway, 2007) Interpretability,
it is argued here, has always been a humanist project, yet has now been appropriated by
computer science to problem-solve the ethical problem of generative knowledge. Generative
knowledge is a new type of knowledge that is at once ethical, political, and technological, yet
computer science's XAI track, extracts and dismisses the scope for exploring the unhuman
with the human, reducing it to the computational binary.
The hostility of complexity
The interpretability problem creates a sense of hostility in complex networks, which James J.
Brown Jr grapples with in his book Ethical Programs on the rhetorics of software. Brown writes
about information networked life and the ways that we are always implicated in its system
without giving our explicit consent. In computational networks, such as the internet, we are
never not implicated in constant engagements, extending and accepting invitations,
responding to, hosting, and decision-making both by humans and nonhuman computational
agents in the network. There is no longer such an experience as being offline, as our devices
do not require our consent to implicate us in this networked life. This is another form of
expression of the interpretability problem, one that engenders a sense of masked hostility.
Brown describes how the internet was designed during an era of more openness and
collaboration in the information technology field, and its underlying structure and rules were
defined under those cultural conditions. Today, the internet has evolved to operate as a much
more hostile environment, in which significant ethical problems have been born of its very
functioning, including its impact on democratic elections, abusive communication practices,
35
and financial hacking.
For Brown, this enforced and unconditional hospitality is a new condition of the era of digital
networks. “This vulnerability means that everything and everyone in a digital network is always
hiding in plain sight,” (p.29, Brown, 2015) and one has no choice but to host a potentially
adverse other. Our individual choices “only happen[] in the face of an ever-present
exposedness to others (…) it is this experience of being held hostage by another that resists
representation, that arrives over and beyond our attempts to make sense of that other.” (p.3,
Brown, 2015) This is a highly asymmetric structure that we can see manifested as highly
targeted advertising and political campaigning, in which we do not have access to our own
data, but others do and can use that access as power, both without our consent and without us
being able to see it taking place. Brown calls for us to “refigure ethics as something beyond
individual choice” (p.3, Brown, 2015) and conceive of new ethical programs for the conditions
of networked life. Brown gives expression to the idea of an enforced, hostile, invisible, and
unconditional hospitality of adverse others. It is another representation of the interpretability
problem as it manifests specifically in deep learning networks and the cultural shift toward that
form of decision-making being further embedded into societal institutions and processes.
Brown’s positioning of the invisible hostility of actions in complex networks is further explored
by Galloway and Thacker, who offer the representation of the swarm to help us grapple with the
facelessness of the interpretability problem in complex networks.
“A swarm attacks from all directions, and intermittently but consistently - it has
no ‘front,’ no battle line, no central point of vulnerability. It is dispersed,
36
distributed, and yet in constant communication. In short, it is a faceless foe, or
a foe stripped of ‘faciality’ as such. So a new problematic emerges (…)
‘facing’ the other, (…) a key element to thinking the ethical relation, what sort
of ethics is possible when the other has no ‘face’ and yet is construed as other
(as friend or foe)? What is the shape of the ethical encounter when one ‘faces’
the swarm?” (p.66, Galloway, 2007)
The phenomenon of the swarm has a sublime quality that is seductive and beguiling. Galloway
and Thacker configure the sublime as a conflict that arises when there is a need to encounter
and address the other in a system that functions to remove the possibility of facing the other.
We cannot know if the network contains hostility, or “an enmity without a face,” (p.65, Galloway,
2007) if we are enthralled by it. The interpretability problem in complex systems contains the
possibility of the same faceless foe and negation of an ethical encounter with the other. Such
an evasion through facelessness can also be interpreted as a strategy of sovereignty:
“A new sovereignty, native to global networks, has recently been established,
resulting in a new alliance between ‘control’ and ‘emergence’. Networks exist
in a new kind of global universalism, but one coextensive with a permanent
state of internal inconsistency and exceptionalism. In this network exception,
what is at stake is a newly defined sense of nodes and edges, dots and lines,
things and events - networked phenomena that are at once biological and
informatic.” (p.22, Galloway, 2007)
37
Control and emergence, while seemingly antagonistic, can provide a sense of inconsistency
and instability that is at one a consistent and stabilizing. The point is that the structure of
complex networks and their data processing, the nodes and edges, is the basis of this power,
not in the content or individual human subjects and stories within the data. Galloway and
Thacker suggest that such a faceless and amorphous structure has its legacy in military
strategy. The question of how high-level intelligence or behavior seemingly emerges from
simple and dumb interactions has been long-enquired. It is about the relationship between
enmity and topology. In complex networks, enmity can be considered a part of topology, in
fact, one type of military diagram is that of the swarm, as an amorphous but coordinated strike.
Traditionally, enmity is understood in terms of a symmetrical relationship, the friend-foe
dynamic. The swarm is a form of control by means of its decision-making structure, but its
power is enacted without a leader; in an instrumental sense, it controls without a controller, its
control is constituted by its own dispersal and multiplicity. This ability to be both deliberate and
formless, controlled and emergent in its masking of decision-making is a nefarious asymmetry
of power and circumvention of accountability.
Manuel Shvartzberg connects the influence of swarms on the political imagination, especially
through design practices. He points out how swarm simulations have influenced many fields,
including computer science, psychology, economics, design, education, and cognitive
science. In design, especially architecture, the image of the swarming sublime connects
aesthetics with an ideological imaginary and has supported a discourse, a set of
computational tools, and the development of real-world projects. In an era of decentralization,
the swarm has reified design’s relationship to sociology and the notion of a self-correcting
collective force.
38
“[A] profusion of images of swarms that quasi-organically welds the notion of
‘magnificence’ to an infrastructure of display, and therefore, to a global
economy of ‘sense’. This relationship between circulation and meaning,
rendered as an organic process, constitutes the most important political
dimension of the imaginary of swarms. The endless repetition of
representations to the effect that society is a swarming aggregation of
individual self-interests is an index of the political power at stake in this
imaginary.” (p.98, Shvartzberg, 2015)
The pervasive power of swarms in the political imaginary is in their designation as being more
natural, and by association, legitimate. Shvartzberg calls for us not to succumb to this
abandonment but to demystify and deflect this techno-cultural infrastructure. He points to the
influence of the swarm imaginary on the philosophy of the MIT Media Lab and its embrace of
the qualities of adaptability, non-linearity, and self-organization in its ethos of participatory and
interdisciplinary design and educational practices. From Craig Reynold’s original flocking
simulation to Seymour Papert’s turtle graphics programming environment, he shows how the
swarm weaves through these experimental educational ideas, connecting them to unregulated
global capital flows. In this sense, the swarm is both technically feasible, aesthetically
seductive, and ideologically desirable.
The swarm enacts a self-actualizing, self-correcting, optimizing, seemingly more natural and
instinctive quality, all the while making invisible its external influences, in the same way that
unregulated free-market capitalism masks government interventions in the form of state
39
subsidies, tax breaks, and bailouts, in its model of success. In the context of academia, MIT
Media Lab promoted an innovative, open, democratic, and anti-disciplinary ethos that quietly
externalized an unscrupulous reliance on a funding model based on military research and
billionaire philanthropists whose wealth was accumulated from sources at odds with its ethos.
The multiple expressions of the swarm include: the invisible guiding hand of the market;
Papert’s self-actualizing, freely desiring pedagogical model for children; and “human
collectives as organic compounds of individual self-interests that would naturally self-organize
and aggregate through capitalist collaboration and competition,” (p.115, Shvartzberg, 2015)
have been appropriated to conveniently externalize many significant interventions outside of its
model of success. The swarm’s notion of collectivity is rarely invoked to support grassroots
political organization, activism, or protest movements but exclusively justified for capitalist self-
organization in the form of competition, deregulation, privatization, and personal advantage by
the 1%. Deep learning networks are a new expression of this legacy of the swarm as a political
imaginary. The aesthetics of emergence, the appeal of decentralization, the assumption that
bottom-up organicism equals legitimacy, and the allure of computational innovation justify
these deep learning networks as ideologically valid. In reality, they also externalize negative
consequences and relations in their systemic dispersal of traceable decisions leading to the
masking of accountability and potential hostility and improper data practices that lead to
implicit racism and sexism. The interpretability problem is a problem of the swarm that is a
problem of the simultaneous enactment and negation of agency.
40
Chapter 2 - Code is a political language; code is a human language
Section 1
Code is a political language
The concept of algocracy was first put forward by sociologist A. Aneesh in his 2006 book
entitled Virtual Migration, to describe a new form of power defined by the rule of algorithms, or
the rule of code, as it is embedded in software. It is a new organizational system that does not
require monitoring through traditional surveillance systems, hierarchies, and bureaucratic
forms of government; but instead governance and surveillance take place through the covert
design of the algorithm and the ways it tacitly shapes behaviors and asserts authority, without
public awareness. The term can be traced back etymologically to ‘kratos,’ the Greek word for
power, which developed into ‘cracy’, and can be contextualized alongside other forms of rule
such as aristocracy, rule by a higher class, or democracy, rule by the people. ‘Algo’ + ‘cracy’
offers us a term for an emerging kind of governance via algorithms.
Aneesh conceived of this idea in the context of changes in global labor organization and
migration, especially between India and the US, due to the increasing use of algorithms and
high-speed networks to manage labor practices. ‘Virtual migration’ is a new disembodied and
dispersed type of migrant labor, one in which workers’ physical bodies remain in their home
countries while their labor is semiotically transformed, extracted, and transferred globally as
data, in the form of jobs in sales, customer service support, research and development,
engineering and design, programming, translation, and accounting. For Aneesh, algocracy is
“concrete, grounded in programming languages and code whose materiality, as readable and
41
undeniable as the words on this page, does not need to be deduced from unverified premises
or generalized from particular events. Virtual migration is a measurable flow of code.” (p.4)
Algocracy is, in part, a measurable affordance of algorithms and also an emergent
consequence of algorithms embedded in distributed sociotechnical systems. This embedding
is not necessarily techno-positive but includes the consequences of the cultural misuse and
misunderstanding of algorithms. There is an aspect of algorithms in which their programmed
rules can be equated to a form of management at a local level; however, the concept of
algocracy as an overarching power structure, while directly shaped by algorithms, takes place
through nuanced intersecting networks of digital technology, economics markets, and global
policy.
Previous forms of organization and control, such as the legacy of bureaucratic organization,
are being superseded by networked algorithms, potentially redefining policy and human rights
processes and definitions. If algorithms can exercise power in the absence of a unifying or
hierarchical organization, how do we locate this power? Aneesh’s answer is: directly in the lines
of code and their ability to structure behaviors, decisions, and rights with significantly reduced
human authority relations. (p.102) Code syntax programs the possible field of action, channels
actions in precise ways, and reduces the possibility of negotiation.
In the context of deep learning algorithms and the process of self-generated new rules and
models, the concept of algocracy takes on greater pertinence due to the rising concerns about
the ethics of automated authority. Algorithms structure behavior and define choices within a
limited possibility space of actions, without people’s awareness or acceptance of those rules.
Instead, algorithms design an environment and cultivate an ambiance of interaction in which
42
there is no alternative but to follow the rules and no awareness of alternative pathways.
Algorithms embed an asymmetry of power beyond traditional forms of organization, in which
one no longer needs to know what the rules are because algorithms have a priori guided
decisions along with programmed logics that ensure adherence to them. This covert
adherence takes form through deep learning classifiers, which are trained on a fixed number of
discrete labels, that predefines a boundary in which an algorithm can make observations and
predictions about a particular phenomenon. These invisible boundaries are defined early on in
the development of an algorithm, they are explicitly written into lines of code, which ensures a
limited set of non-negotiable choices. In this sense, algorithms “do not merely represent the
real; they also produce the real,” (p.115, Aneesh, 2006) they do not just represent rules but
imagine and simulate new ones into existence. This new form of authority that negates
legitimacy is shaped algocratically through power embedded in lines of code. In this way, code
is a political language. Programming is a form of power that structures possible forms of
action, which is an important shift in how power, authority, and governance occur in society.
Political legitimacy in an algocracy
If algorithms push negotiability further out of reach, the question of how to bring algorithms
“back into the realm of negotiability” (p.114) emerges. Aneesh’s conception of algocracy
developed within the context of migration and labor preceded the development of deep
learning algorithms. However, law professor John Danaher has evolved the concept of
algocracy into the contemporary space of our political and legal systems and the process of
decision-making in public policy. He also updates algocracy to recognize the new context of
deep learning, as the prospect of autonomous predictive decision-making and its associated
problem of interpretability, in which algocracy takes on increased potency and risk. In
43
government systems, algocracy enters via the way it structures and constrains how people
interact with data, processes, and their communication. This could be via data mining and,
more concerning, via predictive algorithms that influence decision-making and eventually
become its basis. Danaher explains that the legacy of the notion of legitimacy in political theory
is premised on comprehensibility, deliberation, and participation, and algocracy threatens to
undermine this legitimacy format. Danaher defines the threat of algocracy as twofold: one
arising out of “deference to the epistemic superiority of the algocratic system” (2016), due to
an assumption that algorithms have privileged access to legitimacy-conferring outcomes
beyond that of human decision-making; and due to the problem of opacity. As introduced in
chapter one on the interpretability problem, Danaher also points to the difference between
opacity, as an incomprehensibility to human reasoning, being more problematic than the
hiddenness of algorithms, as in their operating without clear consent or oversight. In deep
learning algorithms, the interpretability problem plays out as a threat to the tradition of
legitimacy in political and public spheres.
The implications of Open AI's GPT3 model
Algocratic systems could further advance into industrial-scale systems and software design
through a recent development in Open AI’s GTP3 API, a deep learning model trained on text to
generate various types of natural language text. It has gained recent attention for developing
the ability to write source code, autonomously design a website and code its functionality.
(Metz, 2020) Previously, we have seen earlier versions of the predictive language generator
write highly accurate, subtly toned, and persuasive human language texts, and now GPT3 is
enabling deep learning algorithms to write HTML source code, which is another level of code
writing its own code. The GPT3 model was not explicitly trained to perform this kind of
44
functionality, but it has inadvertently developed it within the broad scope of data that the model
was trained on. From a techno-positivist perspective, the prospect of autonomous code writing
entire software applications from simple human language instructions might seem like an
exciting advancement that could support bespoke software development by non-
programmers. However, from an ethical perspective, the writing of code moves further away
from human comprehensibility and from the possibility of participating in deliberation and
negotiation that Aneesh and Danaher argue is necessary for political legitimacy and
democracy. If code writes its own code, the notion of code literacy and the role that writing and
reading code takes as an act of political participation is further removed. Additionally, code
could become a skill only available to a class of technical elites, whereas today, while
programming is a technical skill, it can be self-taught and industrialized globally. Maintaining a
connection to code syntax, as a critical practice of writing and reading, is imperative to political
participation, and the GPT3 advancement into autonomous programming threatens this
participation.
In an algocracy code is law
The law professor Lawrence Lessig goes as far as to declare that “Code is law.” (p.6, 1999) He
describes how the nature of code, its form of precise statements embedded in software that
define its functionality, can act as a legal regulator, in the same way, that laws regulate through
statements embedded in constitutions and statutes. Lessig argues that a country’s constitution,
beyond the law, is an architecture that structures and constrains a particular set of core values
with regard to power and freedom. Writing in the context of the early days of the internet,
Lessig points out the naiveté underlying the misunderstanding that freedom is a naturally
occurring phenomenon stemming from the absence of control in society. Freedom has to be
45
structurally built into societal systems to facilitate rights of privacy, free speech, ownership, or
trade. "We can build, or architect, or code cyberspace to protect values that we believe are
fundamental, or we can build, or architect, or code cyberspace to allow those values to
disappear.” (p.6, Lessig,1999) Just as the constitution embeds foundational values into society,
in the context of algocracy, code likewise bears a series of values that protect and encourage
certain kinds of life within it. Lessig warns that computational systems can become tools of
control without the conscious writing of such coded values. In this way, code could replace
law, both as a foundational ethical architecture and as the simultaneous regulator of that ethics.
The first materially affective discourse
Returning to Galloway’s conception of protocol in the age of networks as a materially immanent
form of regulation, he states that “Code is the only language that is executable, meaning that it
is the first discourse that is materially affective.” (p.244, Galloway, 2004) Code imposes its rules
immanently via the very execution of the program rather than an external set of abstract ideals.
With regard to freedom and control in computational systems, Lessig also confirms “the
difference between them is simply a difference in the architectures of control - that is, a
difference in code.” (p.20, Lessig, 1999) Previously, it has been argued that code’s political
consequences take form at the point of it being applied and thus affecting a broader culture.
However, drawing on these two direct statements from Galloway and Lessig, I am arguing for a
more direct recognition that individual lines of code can be political regulations with material
effects in and of themselves. I argue that in an algocracy, code is a political language at the
level of its syntax, and political governance is an affordance of code. Without consciousness of
this deeper political meaning and agency, code can replace the need for external law, policy,
and regulation.
46
The robustness principle: political philosophy in code
The Robustness Principle is a case study of how coded rules embody a particular set of social
values, which can automatically be implemented as laws. The architecture of the internet is
based on a series of written protocols, and TCP/IP is a fundamental protocol that defines how
information is connected, moved, and delivered. TCP/IP structures a distributed peer-to-peer
communication system as a universal language between any two computers, is robust and
flexible and actioned by autonomous agents. In 1980 the computer scientist, Jon Postel,
added the protocol RFC 760, writing that “an implementation should be conservative in its
sending behavior, and liberal in its receiving behavior,” (1980a) and in RFC 761, he describes
the principle of robustness as "be conservative in what you do, be liberal in what you accept
from others.” (1980b) This coded principle of internet communication, also known as Postel’s
Law, after its author, engineers a particular notion of robustness by defining how computers
and programs should treat information from a potentially chaotic input to a more managed
output to achieve maximum interoperability between different systems. Alexander Galloway
highlights how the use of the terms ‘liberal’ and conservative’ to directly write a computational
protocol is unusual, and offers us insight into how “[i]f not a political theory proper, Postel was
certainly articulating a very particular philosophy of organization.” (2016) Furthermore, Eric
Allman points out, the Robustness Principle “was formulated in an Internet of cooperators. The
world has changed a lot since then. Everything, even services that you may think you control,
is suspect.” (2011) During the early days of the development of the internet and its
demonstration of the shift from centralized control to a vast distributed network, a culture of
optimism pervaded which prioritized interoperability. The Robustness Principle was born of this
optimism, but today finds itself architecting communication across a much more hostile
47
computational space of hackers, fake accounts, misinformation, and fraudulent agents, in
which “interoperability and security are at odds with each other.” (2011) Today the Robustness
Principle could be argued to encode a lack of robustness in the context of the shift in hostility
and the need for security. It is an example of how a cultural value can be programmed into a
computational architecture and then continues to underpin and materially execute that value in
the way information is networked long after the society’s values have changed or become more
complicated. Computation is political and value-laden at the level of its code and can structure
a political philosophy via the regulation of the flow of information.
Ted Nelson’s two-way computational architecture
Another case study offers insight into how a critical analysis of a foundational computational
architecture can speculate on the potential restructuring of a social and economic paradigm.
Ted Nelson was a computer scientist working in the early days of the development of the
internet. He invented the concept of hypertext, words that could be computationally linked to
other information, thus creating a more complex series of paths through and around a text
document to derive other relationships, expressions, and meanings without changing the
original version. It was a model in which both agents in the system, the creator and the
audience, or the seller and the buyer, were independent, with a more balanced relationship,
more equal rights, leading to more openness and less privatization by a few agents with more
power. Nelson’s model for internet communication is known as the two-way model, in which
data does not need to be copied, only linked, and therefore context and origin are retained.
Often meaning is dependent on context, and veracity is dependent on origin. However,
derivation and mash-up are still possible. Nelson’s structure was related to architecting a
balance between rights and responsibilities and offered a model intending to afford
48
collaboration and culture rather than the figure of the individual user.
However, Ted Nelson’s model is not the one on which the internet is based today. The
computer scientist Tim Berners-Lee simultaneously developed HTML, the foundational protocol
upon which websites are built and, pertinently, structures only one-way links. This was the
computational architecture that succeeded as the computational architecture of the internet.
The technology theorist, Jaron Lanier, has written about the societal consequences of this
decision to pursue a one-way linked network rather than a two-way linked network in his 2013
book Who Owns the Future?. One-way links reduce the amount of information passing through
a network, reducing the friction on each node and the accountability it bears, making it a more
manageable network to implement. One-way links gave rise to the Google search engine as a
service that could constantly scrape the web for contextual information lost in the one-way
network. The rise of social networks could also be considered services developed to capture
the two-way connections that were unavailable for users to see and trace themselves.
Furthermore, Lanier directly relates the damaging political consequences from such services
that evolved out of the one-way network:
“The current online commerce models create a new kind of class division
between full economic participants and partial economic participants.
That means that there isn’t enough shared economic interest to support
long-term democracy. If we can get to the point of symmetrical
commercial rights, then a large space of potential transaction models
becomes thinkable.” (p.247)
49
He argues that the particular type of data gathering platforms that evolved from the underlying
one-way linked information model has propagated an extreme asymmetry of information
access and power relations, (p.2) giving rise to a winner-takes-all economic pattern across
platform technologies and ultimately contributing to the rise in income inequality. Additionally,
one can speculate that the contemporary phenomenon of platform-based misinformation
campaigns is only possible because one-way links do not keep track of information origin and
accountability.
Lanier returns to the values inherent in Nelson’s two-way linked informational model to
speculate on the effects that such a computational principle might have on society. He points
out that each node knows every other node in the network that is linked to it. “That would mean
you’d know all the websites that point to yours. It would mean you’d know all the financiers who
had leveraged your mortgage. It would mean you’d know all the videos that used your music.”
(p.227). While a two-way system is more challenging to manage and keep updated, it retains
greater visibility of which information is important; it preserves context. Lanier suggests that a
two-way model is an example of a more humanistic computing, because it affords the retention
of information provenance and favors economic and political symmetry between user and
platform owner.
Provenance is important because knowing the origin of information is valuable; it protects from
error and fraud, offers accountability, and could be considered a fundamental civil right.
Economic and political symmetry is an opportunity to foster greater democracy in a platform-
based information economy. When people participate in online platforms by sharing content
50
and communicating with other users, this is a form of labor that generates valuable data, and
people should be paid for their labor and their contribution to creating value. “Information is
people in disguise, and people ought to be paid for the value they contribute that can be sent
or stored in a digital network.” (p.245) In a two-way linked network, individual users could trace
their data points, in the form of content uploaded and interactions made, rather than only the
platform owners having access to this data, and then earn money from their labor through
micropayments. This would prevent the extraction and centralization of wealth from a vast user
base into the hands of a few shareholders, increasing the wealth gap. It would also reverse the
practice of high-value platforms employing very few people by externalizing labor onto their
user base, while shifting the value they create off their balance sheet. Lanier argues that
instead of this, two-way links could enlarge the overall economy.
Lanier makes it explicitly clear when he warns against the supposed technological innovation
of computational models in which “the basic symmetries of a social contract are not expressed
in the foundational architectures of our networks.” (p.247) Lanier refers to the “Nelsonian
solution” as one that “doesn’t destroy the middle classes in the long term.” (p.225) Where a
one-way linked computational model fosters a dramatic information asymmetry and winner-
takes-all ethos, a two-way linked model could foster information distribution more akin to a bell
curve. This would afford broader societal support, more balanced economic opportunity, and a
more healthy democracy, free from hidden manipulation from powerful and privatized platforms
that we currently find ourselves stuck in. Rethinking Ted Nelson’s concept of a two-way linked
structure as the foundational protocol of the internet offers us insight into how a basic
computational model can have vast implications for society, economics, and culture.
51
The problem of bias augmentation
Bias augmentation is a term that I conceived to describe an ethical problem in which an
algorithm augments an existing small or latent bias in a data set during its execution. In the
developing discourse around the ethics of algorithms, much work focuses on the input data or
the training data in the case of neural networks, and the discovery of unfair biases that are
contained within those data sets. However, as we come to understand that large data sets that
are collected from human behavioral patterns and historical compositions of representation are
likely to reflect the same problematic biases that exist in society, we can make the first step
toward understanding how algorithms can make unethical decisions and further propagate
inequity and discriminatory. Such biases include poor representational diversity based on
preconceived notions of neutrality and the replication of stereotypes based on historical
predispositions. Similarly to Cathy O’Neil’s concept of a weapon of math destruction, (2016) my
concept of bias augmentation expands the focus from the idea that algorithmic bias lies solely
in the input or training data, to also investigate the ways that the design of an algorithm’s
decision-making model, its internal structure and iterative process, increases bias from an
initial seed.
Bias augmentation is a term that is useful to the discourse of critical algorithm studies today.
However, the legacy of its meaning emerges outside of machine learning from broader societal
examples of how systems of inequity can propagate more inequity through feedback loops that
reinforce implicit biases, for example, in gender discrimination in the workplace. A historical
example of algorithmic bias can be traced back to the 1970s when St George’s Medical School
in London developed an algorithm to screen student applications. The algorithm was found to
52
negatively weight some values in the data, such as non-European names, and was also proven
to automatically remove points from female candidates, leading to gender and race
discrimination. (Lowry and Macpherson, 1988) The original data set contained latent bias due
to the existing gender and race discrimination in historical medical school admissions data;
however, this was inherited and augmented in the design of the algorithm’s decision-making
structure, compounding the bias. The problem was only proven and addressed seven years
after its implementation following complaints made by staff who noticed the reduction in the
gender and racial diversity of the student body. It is a historical example of how an algorithm's
structural design can generate bias, distinctly from the data set it operates on.
We can look to the United Kingdom’s A-Level exam controversy in 2020 for a contemporary
example of bias augmentation. Due to the Covid-19 pandemic, A-Level exams (the final high
school exams students take and upon which university admissions are determined) were
canceled. Instead, the government authorized the design of an algorithm to determine
students’ final grades. The algorithm was meant to modify the predicted grades that teachers
gave to students earlier in the year and take into account various rankings and historical data.
The algorithm was a standardization model and did not use machine learning. It was designed
to moderate predicted grades, primarily by grading down, to account for grade inflation and
maintain historical grade averages across the country. When the results were published, it was
found that the moderated grades had a disparate effect across students of different
socioeconomic statuses. Students who attended state schools were more likely to be
downgraded versus those who attended private schools, who were more likely to be upgraded,
thus propagating structural inequity. Analysis of the model was found to use cohort size data
and positively weight these values, thus automatically privileging students who went to private
53
schools, which tend to be smaller. The algorithm also considered the school’s historical
performance in each subject, thereby disadvantaging stronger students from more poorly
performing schools. (Hern, 2020)
Following a public outcry, student protests, legal action, and the decision by some universities
to disregard the government’s modified grades and award university places based on
predicted grades, the government was forced to withdraw their algorithmic model and default
back to teacher-predicted grades. Despite the known ethical errors that algorithmic decision-
making can create, the algorithm was rushed through development, applied without broader
consultancy or transparency, and did not provide a clear mechanism for redress. However, it
was a situation that brought the concept of discriminatory algorithmic bias into people’s lives in
a very immediate and material way. Several factors influenced the media attention garnered by
the situation and influenced the government turnaround. These included: the fact that the
algorithm’s unethical impact was felt very suddenly by a large group of the population, the
entire cohort of graduating high school students across the country and their families; and also
that this was a group of people who felt empowered to be vocal and were organized enough to
stage the well-reported ‘Fuck The Algorithm’ protest through the city of London. (Amoore,
2020) From an ethical perspective, the algorithm appeared to be designed to prioritize
statistical parity, an abstract notion of the average fairness of results, over the real disparity of
its socioeconomic bias and unfair impact on individuals. There was no way to achieve both.
This case is another example of how algorithmic models, including those outside of machine
learning, exhibit bias augmentation at the level of their structure and data processing, not only
in the input data.
54
The idea of ‘weighting’ is a critical issue for further understanding bias augmentation and the
ethics of algorithms. I argue that weighting is a technique of bias augmentation and a means of
how algorithms can be mechanisms of structural inequity. In this way, weighting is a political
issue and a discrete and identifiable piece of code syntax, a piece of mathematics, that can
foster inequity in algorithms. From a critical algorithm perspective, ‘weight’ is both a term of
programming syntax and a culturally loaded concept and should be available for greater
interdisciplinary interpretation with the intention of ethical insight and accountability. In the field
of statistics, weighting is a mathematical process to adjust specific data values, both positively
and negatively, to change that data’s proportional contribution to the final result. Weighting is a
way to emphasize or suppress aspects of a phenomenon compared to other aspects in a
group. It is a way to increase or decrease the importance of a piece of information. In statistics,
weighting is used as a correction technique to adjust survey data, correct known biases,
compensate for known errors, or fit data more representatively to a target population to improve
the accuracy of the statistical predictions. However, the many different weighting methods
should also be opened to ethical consideration as artifacts of code in a materially affective
political language.
One way of thinking about a deep learning algorithm is as a generative model of weights. The
training of a deep learning model is a process of primarily selecting, testing, and refining a
series of weights that try to achieve as accurate a match with the target as possible through a
loss function. A deep learning model is the final refinement of a structure of weights applied to
input data to sculpt it toward forming a new outcome. A significant recent development in deep
learning research published by Google reveals that weighting has a significant impact on the
55
accuracy of machine learning models. (D’Amour et al., 2020) A team of 40 researchers sought
to understand the problem of why deep learning algorithms can achieve a high level of
accuracy in a lab setting but perform poorly when applied in a real-world setting. They
discovered that the cause is underspecification, a widespread statistical problem that appears
when an observed phenomenon has many possible causes beyond what the model can
account for. They also stated “that underspecification is ubiquitous in modern applications of
ML (…) with downstream effects on robustness, fairness, and causal grounding.” (D’Amour et
al., 2020) This substantiates the idea that unfair automated decision-making is not solely
caused by biased data, but widespread across the structural design of machine learning
algorithms.
Previously, this lab-to-real-world performance drop was solely explained by the ‘data shift’, a
term to describe the difference in data used in a lab setting, which is likely to be cleaned and
parsed to suit the objective, versus real-world data that is more likely to have noise from having
been captured in less than ideal conditions. The data shift is the difference between the lab
data that an algorithm is trained and tested on and the real-world data it faces in its applied
setting. Within the purview of underspecification, the authors point to several factors in the
design of an algorithm that they found to have only negligible impact on results during lab
testing that turned out to have a significant impact on the results when released into the real
world.
“The protocol is designed to detect underspecification by showing that a
predictor’s performance on stress tests—empirical evaluations that probe the
model’s inductive biases on practically relevant dimensions—is sensitive to
56
arbitrary, iid-performance-preserving choices, such as the choice of random
seed.” (D’Amour et al., 2020)
Random seeds are the values selected to configure the starting weights at the beginning of the
machine learning training process. This process of weight initialization often sets a range of
very small random numbers between -1 and 1 for convenience. These values are then
gradually minimized against a loss function, a process known as gradient descent. The iid
(independent and identically distributed) is an evaluation procedure for the model's predictive
performance. The authors’ experiments prove that something as seemingly insignificant and
overlooked as weight initialization can impact the fairness of a model’s decision-making in real-
world applications and that this aspect has been neglected until now. Pertinent to my argument
is that bias augmentation, separately to biased data, also contributes to unethical predictive
decision-making and that a deep learning model’s weights are an example of this structural
bias. The authors specifically point to the ‘choice of random seed’ as the initial decision to set
the weights’ starting positions as one of the seemingly insignificant factors in algorithm design
that can have adverse and substantial effects in the real world. Weight initialization is a crucial
part of the structural and processing-based aspect of a deep learning neural network and is
tied to its ethical performance.
Ethically, the problem with such design decisions is that they are only discovered after the
algorithm has failed in the real world, potentially having already caused damage to people’s
lives. The paper’s authors do not offer a solution but present a more detailed root cause that
was previously not understood, acknowledging that the problem leads to unreliable and
untrustworthy models for society to contend with. However, the authors do propose that more
rigorous stress testing needs to be performed on individual real-world applications.
57
Bias augmentation is a form of structural injustice
In contemporary public discourse, there is an increasing acknowledgment that injustice and
inequality operate in structural or systemic ways and not only as individual circumstances. The
2008 financial crash gave rise to the Occupy Wall St protest movement that established the
concept of ‘the 1%’ firmly in the public’s imagination. The idea of ‘the 1%’ expresses the trend
toward vastly increasing wealth disparity across the world as a form of systemic injustice. Since
2013, Black Lives Matter, the protest movement that arose in response to the continued
violence and deaths that black people in the US were experiencing at the hands of police, also
brought the concept of racism as a systemic issue more acutely to the forefront of public
consciousness. Black Lives Matter has culminated in the 2020 national race protests sparked
by the police murder of George Floyd, leading to a public shift in race discourse to center on
anti-racism practices. The term white supremacy is also undergoing a re-reckoning. A more
mainstream discussion has begun around the possibility of carceral abolition through the
protest slogan Defund The Police, which aims to redistribute police funding toward investment
in black communities. These are successful contemporary expressions of how racism is
constructed and, therefore, can also be deconstructed in a systemic way. When it comes to
technology’s role in systemic injustice, when we acknowledge that technology was designed to
perpetuate injustice, we can start to conceive how to design technology to deconstruct
injustice. (McIlwain, 2020)
I argue that today, we should consider the term algorithmic to be synonymous with the term
systemic. In this way, algorithms can execute forms of systemic bias, or “algorithmic violence,”
(2017) as Mimi Onuoha conceives it. Algorithms are socio-political infrastructures, legal
58
architectures, and systems of ethics through which structural wealth inequality and structural
racism take place by the very design of their internal structures and processing. I argue that
code is not a neutral mathematical language but a political language that is a materially
executable and mathematically designed political discourse. Through our assumptions of
neutrality, it germinates its own political legitimacy. Code is political at the level of its syntax,
and not only because it uses biased data that merely reflects society's biases. Code is a
structural system where all other structural societal issues can be contained, expressed,
negotiated, and reimagined. Bias augmentation is an example of this structuralism, and the
discovery of weight initialization in the training of deep learning algorithms as being a cause of
unfair decision-making is an example of algorithms generating bias from their structure. I argue
that deep learning weights can be thought of as part of a politically deployed language.
59
Section 2
Bringing the human into code
How do we bring the legacy of humanist interpretative thinking to bear on source code? How
do we summon forth critical theory’s reckoning with ideology, its challenge to structures of
power, and desire for human liberation from oppression, through source code? We are led to
believe that the nature of code, its relentless logic, evades and rejects such inquiries and
possibilities. It remains to be understood whether the distinct approaches of critical theory and
computer science might be incommensurable. Gary Hall argues that instead of seeking
reconciliation, we seek instead to:
“produc[e] a consciously developed theory of their incompatibility. Such a
theory might even be capable of showing how they can both be practiced at
the same time, as two incommensurable positions in an irresolvable yet
productive tension, so that the questions, issues, and approaches specific to
each are capable of generating new findings, insights, and realizations in the
other—to the point where both of their identities are brought into question.”
(2013)
Here, incommensurability becomes a task for recognizing, foregrounding, and holding space.
It requires working in a manner in which interaction is understood to be difficult, and synthesis
is recognized to be far from straightforward. In Feminist in a Software Lab, Tara McPherson
goes further to question whether the core quantitative nature of computational systems is
60
structured and partitioned to be incompatible to deny humanist criticality. “And are our critical
methodologies [also] heirs of this epistemology?” (p.50, 2018) We can also skeptically
question whether computation’s tyrannical logic of partition and separation bears an agency
that denies the accretive and contextual questioning of critical theory, and whether that tyranny
also seeps into and impedes our humanist demands.
Human code is interpretable code
Critical Code Studies (CCS) is an emerging field that brings a humanist interpretative method
to bear on the reading of source code. CCS takes its starting point from the position that code
is a neglected but omnipresent cultural text, which embodies too much importance to be left
solely to machine parsing. CCS offers an approach to engage with code that fosters critical
and interpretative thinking. When we interpret source code, we seek to move beyond its
straightforward functionality to recognize the expanded and more nuanced process by which it
achieves meaning. How code accrues meaning requires us to study its interoperating systems
that include the machine, humans, use of data, societal applications, and also their misuse,
connotation, and reappropriation. Mark Marino, the author of Critical Code Studies and the
director of the Humanities and Critical Code Studies Lab at the University of Southern
California, asks us to situate source code, less in the literary sense and more in the cultural
studies tradition of inquiry, likening the study of code to Fredric Jameson’s textual reading of
the Bonaventure Hotel. (1992)
CCS is premised on a practice of close reading of source code that considers semiotics at the
level of programming syntax and grammar, in a similar way to how one would closely read a
poem. This is different from other code reading methods that might perform statistical
61
language analytics or scrutinize the system's data inputs and resulting cultural artifacts. When
code is placed into culture and analyzed for the meaning it bears beyond immediate
functionality, there is an inevitable shift toward intersectionality and how social relations act as
forces that shape and distort meaning as it is processed through computational systems. This
is very different from the longstanding tradition of thinking of source code as a neutral space of
formal logic, stripped of human significance, whose only reader is the executing machine.
“Interpretation in the humanities sense is neither deciphering nor translating,
but instead uses those techniques (and other hermeneutics) to get at meaning
beyond the isomorphic systems. Interpretation then is not merely decoding
but the production of another kind of knowledge: insight. Those insights may
appear quite arbitrary when someone is approaching the code with a
systematic point of view, when one is attending the logic of the systems rather
than potentially challenging it.” (p.4, Marino, 2020)
CCS seeks to directly challenge, subvert, expand, and deny the incommensurability of formal
logic with human interpretation. From its legacy in the digital humanities, Marino ties subject
formation from reading and writing in the age of letters to reading and writing of code in the
age of digital technology, a “technological apriori that frames our digitized thoughts.” (p.19,
Marino, 2020)
Interpreting code’s meaning is conflicted
Reading and interpreting code, however, is difficult, this should not be understated, and we
have not been trained in how to do it. This problem stems from various causes: the legacy of
62
assuming code to be neutral, the demarcation of scientific thinking from the kind of thinking
done in the arts and humanities, the lack of people who have the combined skills and
determination to both read code technically and read it interpretively, the hiddenness of much
code behind layers of technical obfuscation and intellectual property law, all mean that this
work is especially challenging. Additionally, reading code is not the same as reading other
texts, as code contains assemblages of time-based processes, state changes, and layers of
encapsulation; one has to read code with a computational mindset and also remain in the
space of a continually deferred comprehension.
Code has long been bracketed, technically, culturally, and epistemologically. The endeavor to
now cross this divide and intersect its boundaries is wrought with struggle and rejection,
including methodological disagreement from even its own advocates. Research into the ethics
of algorithms, especially for those whose epistemological legacy lies more toward the social
sciences than the humanities, at times disagrees with a direct reading of code. Mike Ananny’s
work promotes a broader understanding of the concept of an algorithm from its source code,
what he describes as “networked information algorithms as assemblages of institutionally
situated code, practices, and norms with the power to create, sustain, and signify relationships
among people and data through minimally observable, semi-autonomous action”. (2016)
Ananny seeks to address algorithms and their effects across a spectrum of sociotechnical
relationships, in which responsibility is shared across the network, rather than focus on the
object of computer code specifically. He argues against the fixation on source code as holding
the key to unraveling ethical problems and problematizes the connection between source code
and the notion of transparency as an adequate goal. Together with Kate Crawford, they ask
whether looking inside the black box means that an algorithm becomes transparent and
63
therefore more accountable when transparency assumes that what is inside is discernible and
that the audience can comprehend it. (2018) It is an important point, which reflects other
disagreements over how and why to read code and what such scrutiny means, including
Wendy Chun’s warning not to fetishize code (2011) by treating it as a sole source of causality
and retreating into its scientific and essentializing nature to solve what is an expansive and
embedded sociotechnical problem.
However, I argue for a preoccupation with code, at least as part of a multi-moded framework,
to strive for greater computational literacy and direct engagement with algorithms, especially
during a time when algorithms are seen as opaque while simultaneously being given greater
authority and autonomy. Knowledge can be gained by examining the composition of an
algorithm, and to neglect this is to overlook the way algorithms frame problems and contain
important statements. For example, a study by Sandvig et al. on whether an algorithm itself can
be racist describes how an algorithm can be written in multiple ways to achieve the same
result. It can encode normatively positive behavior, define variable ranges within which to
identify skin color, and categorize race, when that would be an unacceptable question,
perhaps even an illegal question, in the context in which the algorithm is being used. (2016) It
follows that if the source code is neglected, this type of unethical behavior cannot be
determined. Reading the code can be a useful strategy to understand algorithms and how
ethics can be encoded at the level of syntax.
We can also look to the field of encryption for further affirmation on why we need to read the
code. In Bruce Schneier’s book Secrets and Lies: Digital Security in a Networked World, he
presents the case for why it is important that encryption algorithms are in the public domain. In
64
cryptography, when an algorithm’s code is open-source, it means that a significant number of
people will have studied the algorithm and identified flaws and weaknesses, leading to
updates, literally meaning that the more eyes that have looked at an encryption algorithm, the
more secure it will be. Whereas, when algorithms are blackboxed and protected by intellectual
property, many people have not scrutinized the code, leaving it open to hidden flaws. (p.117,
2000) We can take this attitude into the field of artificial intelligence to argue for policy to
ensure that in ethically contentious spaces, algorithms need to be available to as much public
scrutiny as possible to ensure ethical robustness.
Interdisciplinary teams are also key to this endeavor. From an investigative journalism
discipline, ProPublica has done a lot to raise public consciousness of how algorithms can
encode racial bias when used in criminal sentencing. In Machine Bias, journalists worked with
mathematicians and data scientists to show how algorithms that predict a person’s likelihood of
reoffending, a measure taken to remove known racial bias, were found to be both inaccurate
and used incorrectly, leading to further bias against African Americans. Likewise, Sandvig et
al., in their study to investigate if an algorithm could be racist, assembled an interdisciplinary
team comprising a computer scientist, an artist, a mathematician, and a social scientist.
I advocate for an interdisciplinary approach that engages directly with source code. We need
to scrutinize code to interpret its socio-political traces while being mindful not to fetishize its
causality; we need to recognize that we legitimize its claim to ideological neutrality by not
reading and interpreting code. For example, the interpretative discourses that address the
ethics of algorithms include voluminous references to algorithms, yet rarely are algorithms
studied as code. This aversion to reading source code relies on the argument that algorithms
65
are disparate sociotechnical systems in which to analyze them would be false essentialism.
However, I disagree with this rigidity and consider algorithms to be discrete, identifiable,
readable entities in many cases, and that it is neglectful to reject engagement with their source
code when it is available.
Accessibility beyond the black box
The issue of access is not only a question of code protected behind intellectual property law
and computational literacy. The question of who and how we trace the complex sociotechnical
threads of an engagement with critical code is also a question about audience and authorship,
and how we define language, and who has a right to challenge it. David Golumbia writes about
the misunderstanding inherent in the conflation of code with human language.
“Computers invite us to view languages on their terms: on the terms by which
computers use formal systems that we have recently decided to call
languages—that is, programming languages. But these closed systems,
subject to univocal, correct, “activating” interpretations, look little like human
language practices, which seems not just to allow but to thrive on ambiguity,
context, and polysemy. Inevitably, a strong intuition of computationalists is that
human language itself must be code-like and that ambiguity and polysemy
are, in some critical sense, imperfections. Note that it is rarely if ever linguists
who propose this view; there is just too much in the everyday world of
language to let this view stand up under even mild critique. But language has
many code-like aspects, and to a greater and lesser extent research
programs have focused on these.” (2009, p.84)
66
The conflation of human language and code privileges code by denying how it constructs its
own linguistic ideology rooted in computer science and distinct from other ideologies. This
logic also works to reject the subversion that humanistic interpretative intersection could bring.
If code is considered ontologically equal to human language, its rigid logic rejects access by
humanistic thinking, allowing only the sort of analyses, interactions, and thoughts that align with
its ideology. Marino argues that CCS is the inverse of this approach.
“Rather than trying to make ambiguous language behave according to the
systematic standards of programming languages, critical code studies seeks
the ambiguous (and unverifiable) connotations of programming languages as
they interact with and travels through other systems of meaning read by
humans and machines.” (p.160, Marino, 2020)
In arguing that code is a human language, I am not arguing that code, as it is currently
constituted, is the same as other human languages, such as English, but that we should
consider code to be available to the full spectrum and legacy of humanistic language critique,
and not bracket, protect, and exceptionalize it from human engagement. However, this broader
notion of access and availability is also precluded by what Marino calls “encoded chauvinism,”
(p.159, 2020) a form of cultural imperialism amongst some programming communities in which
a culture of superiority and hostility arises, in response to alternative voices and critical
positions, that ultimately seeks to exclude and silence and leads to a toxic programming
culture. Therefore, the question of how code can become a human language is also a question
of access as cultural inclusivity. For example, during the 2009 Climategate scandal, emails and
67
data were hacked from the Climate Research Unit of the University of East Anglia in the UK and
falsely presented as evidence of a global warming conspiracy in which scientists colluded to
manipulate climatic data. It is an example of the misreading of code, although still an example
of how code’s meaning can change based on who is reading it. Marino presents this scenario
as a case study in encoded chauvinism through the toxic culture of online discussion forums
that exhibit tribal, chauvinistic, and anger fueled engagements that are more open to snap
judgments and even encourage the misreading of code. The episode marks a point in which
code emerges as a medium through which political discourse can take place.
I refer back to McPherson’s question posed previously as to whether our critical methods have
inherited code’s epistemology, whether the logic of computer science enacts itself onto the
humanities’ attempts to engage code, thus preemptively impeding it by asserting its logic and
limiting the possibility space for interpreting meaning and critical insight. My research argues
that we must not allow this to happen any longer. Instead, we must acknowledge how this takes
place and find ways to reject and bypass this limitation.
To conclude, I present a list of critical code tactics adapted from Monfort et al., Marino, and my
own additions (Griffiths, 2018), an evolving list of techniques and ideas to practically support
ways to read code critically. While a lack of programming literacy is a limitation to this study
method, it should be noted that there are multiple entry points, including for those without a
strong programming background or to support interdisciplinary teams.
68
◦ Reading the natural language comments written within the source code to remind the
programmer how different parts of the program work or to support team development.
◦ Reading programmer-assigned arbitrary variable or function names can give insight
into programming team culture, social context, or the code’s conceptual framework.
◦ Reading the source code’s documentation.
◦ Modification of the code as a textual intervention to compare how a change in syntax
changes how the program executes.
◦ Consideration of the programming language used and its affordances.
◦ Porting a program, or thinking through the implications of how a program might execute
differently if ported to a different programming language or hardware system with
different affordances. Issues of translation, interpretation, and adaptation come to bear.
Several ‘weird’ or esoteric programming languages have been developed to reflexively
probe, provoke, or comment on the nature of programming languages. See for example
Piet.
◦ An analysis of computational structures in the context of a program in a given culture.
◦ An analysis of paradigmatic choices made in the construction of a program.
◦ An interpretation of the methods chosen over others and their connotations.
◦ A consideration of paratextual features.
◦ Recognizing the historical ecosystem of code libraries and authorship by other
programmers, upon which a piece of source code is built and modified.
◦ Recognizing the act of programming beyond singular genesis to an act of reading,
writing, cutting, pasting, patching, and reworking. A form of code discourse as critical
making.
69
◦ An awareness of the program's funding source and the positionally or philosophy of its
authors or funders.
◦ Using source code as a design tool by emphasizing its visual and aesthetic dimensions
to explore alternative patterns as an exploratory, creative, or interventionist exercise.
◦ Following the approach by Friedrich Kittler of interpretation by analogy instead of
interpretation by subjective reflection. Consideration of other processes, structures, and
concepts the code parallels in society and other fields of knowledge.
◦ Placing code into context. Considering the social and political history in which the code
was developed.
◦ Consideration of how the audience for a piece of source code shifts over time, how
code might be read differently by different users, especially its original author, and how
its meaning accrues in circulation rather than being static.
◦ Studying code intersectionally through the lens of other critical theories, including
feminism, gender studies, queer theory, postcolonial studies, and critical race theory.
◦ Speculating on how code relates to particular ideological moves and the possibilities
for its subversion.
◦ A literary or poetic reading of code syntax in parallel with its known functionality to
consider its possible framing, connotations, relations, disconnections, slippages,
assemblages, and materiality.
70
Chapter 3 - Convolutional Domains: a critical software application
Section 1
A practice-theory approach to critical software development
In this third chapter, I consider the arguments developed in the first and second chapters,
including the framing of the interpretability problem and its contestation; critical thinking across
intersectional and generative sociotechnical networks; how source code has the potential to be
a materially affective language to shape law, governance, and values; and how the meaning of
code needs to be interpreted beyond its functionality. Finally, I use these theoretical ideas to
frame and experiment with a method of critical software development. Convolutional Domains
is a practice-based project exploring ways to connect visual and interactive design strategies
with critical thinking from the discourse in ethics and algorithms. It takes the form of a
standalone software that a user can interact with to explore different critical ideas and functions
of a convolutional neural network hypothesized to operate in near-future workplaces.
Convolutional neural networks
A deep learning neural network is a multilayered connectionist computational model generated
from a process of learning patterns in a data set to predict new outcomes. Deep learning
networks are complex adaptive systems that process information collectively and in parallel.
The whole network of neurons builds intelligence emergently by iteratively adapting its own
internal structure over time, based on adjusting weights that control information passing
between nodes. This is what makes deep learning networks so interesting and powerful and
stands in contrast with more traditional procedural algorithms that execute code linearly and
71
are not self-adaptive. (Mitchel, 2019)
A convolutional neural network (CNN) is a category of deep learning algorithm that has been
demonstrated to be very effective in computer vision for uses in object identification, facial
recognition, and activity recognition. Applications of CNNs used in computer vision tasks range
from benign to controversial. These include: facial recognition in surveillance footage to match
people in a police crime database; scanning handwritten documents and analyzing them for
content and meaning; analyzing satellite footage of the environment to monitor climatic
changes and predict natural disasters; object detection in manufacturing processes;
navigation in robotics and autonomous vehicles, including missile guidance in the military;
medical image processing for patient diagnosis; and human genome mapping.
CNNs are frequently used to perform supervised learning with structured image data, images
that have been labeled to define their content and intended to solve image recognition and
classification problems. The contemporary use of CNNs and their influence on modern
computer vision can be traced back to Yann LeCun et al. (1989, 1998) Researchers
incorporated computational convolutional layers into a neural network to extract higher level
features and developed the MNIST data set of labeled images of hand-drawn digits. The
MNIST data set has become a go-to data set in the testing and prototyping of CNNs, and its
classification accuracy serves as a common benchmark in research. After continual
optimization over the years, CNNs trained on the MNIST data set have a test error rate of
0.23%. CNNs derive their name from the convolution operation that functions as an image
processing filter to extract features from images. Convolutional layers are derived by applying
a kernel or matrix to the original input image that focuses on and draws out different parts of an
72
image.
The ImageNet data set is a very large structured data set of images containing more than 14
million hand-annotated images and more than 21,000 categories of images. The creation of
labeled data sets is a significant constraint on the development and adaptation of supervised
learning systems due to the human labor-intensive process of defining labels for each image in
the data set.
In CNNs, the convolutional layer is what defines their particular computational approach. In a
CNN, the network's weights take the form of a convolutional layer, a matrix of weights
generated and passed across the image, shifting pixel values higher or lower to filter the image
to draw out particular features. Specific matrix combinations are known to achieve common
image processing filters such as edge detection, contrast, sharpen, blur, or shift an image from
color to greyscale. Each filter weighs pixels in a particular direction, highlighting different
aspects of an image and gradually detecting different features to build a feature map that
eventually supports the classification process. The depth of a feature map is based on the
number of filters created; a higher number of filters will extract a higher number of features,
enabling the network to perform image recognition better.
During the training process, each layer contributes to condensing the most distinguishing
information from the image and passing that information on to the next layer. During this
process, low-level features such as lines, edges, and general shapes will first be identified to
differentiate the foreground from the background. Features can be identified by looking at
which groups of pixels in an image are activated. In later layers, this information is
73
consecutively built upon to identify higher-level, more complex features. The final layer of the
network will contain the most complex features. In a machine learning context, this process is
known as abstraction. A training process can involve running this process hundreds to millions
of times across a large data set of images to optimize the model. Once training is complete,
the generated model, which includes all of the neurons and filters, can then receive new data
inputs in the form of images that the network was not trained on and has not seen previously.
The model will predict new classifications for objects and activities that it can recognize in
images from these new inputs.
The lineage of convolutional neural network visualizations
The practice-based work developed for the Convolutional Domains project follows a lineage of
other visualization projects from the field of XAI. The list of visualization projects and open-
source computer vision code below offers a comprehensive overview of all the references,
ideas, code, and documentation that I have referred to in the development of this project.
◦ The Building Blocks of Interpretability, 2018, by Chris Olah, Arvind Satyanarayan, Ian
Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev.
https://distill.pub/2018/building-blocks/
◦ Feature Visualization, 2017, by Chris Olah, Alexander Mordvintsev, and Ludwig
Schubert. https://distill.pub/2017/feature-visualization/
◦ Deep Visualization Toolbox, 2015, by Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas
Fuchs, and Hod Lipson. https://yosinski.com/deepvis
◦ An Interactive Node-Link Visualization of Convolutional Neural Networks, 2015, by
Adam W. Harley. https://www.cs.ryerson.ca/~aharley/vis/
74
◦ Visual Interpretability for Deep Learning: a Survey, 2018, by Quanshi Zhang and Song-
Chun Zhu. https://arxiv.org/pdf/1802.00614.pdf
◦ Image Kernels Explained Visually, 2015, by Victor Powell. https://setosa.io/ev/image-
kernels/
◦ GAN Lab, 2018, by Minsuk Kahng, Nikhil Thorat, Polo Chau, Fernanda Viégas, and
Martin Wattenberg. https://poloclub.github.io/ganlab/
◦ TensorBoard Embedding Projector, 2016, by Daniel Smilkov, Nikhil Thorat, Charles
Nicholson, Emily Reif, Fernanda B. Viégas, and Martin Wattenberg. http://
projector.tensorflow.org/
◦ Convnet Viewer, 2016, by Gene Kogan. https://ml4a.github.io/guides/ConvnetViewer/
◦ ofxCv OpenCV wrapper for Open Frameworks, by Kyle McDonald. https://github.com/
kylemcdonald/ofxCv
◦ A Modern Computer Vision Library, by Liu Liu (https://github.com/liuliu/ccv)
The visualizations presented here are all aimed at a traditional notion of explainability in the
machine learning context. My research departs from these examples by using visualization
and interactivity to contest interpretability critically. The Convolutional Domains project was
developed in OpenFrameworks, an open-source programming environment developed by
Zachary Lieberman, Theo Watson, and Arturo Castro, to support creative and interactive
coding projects.
75
A framework for reflexive software
In this dissertation, I develop a framework for praxis, a hybrid form of design practice, and
theoretical research to support the reflexive engagement with algorithms and their ideological
positionality. My framework proposes the development of ‘reflexive software’; a means of
visually and critically unfolding complex intersecting sociotechnical computation systems. I
take an abductive approach to knowledge production via the development of critical and
reflexive software development projects that simultaneously trace relations of power and frame
the social, political, and cultural systems within which knowledge is produced.
Abductive thinking is an idea that originates from the work of the American philosopher Charles
Sanders Peirce that supports the nature of uncertainty, intuition, and doubt in the development
of problem-solving and knowledge production. (Peirce, 1934) In the context of design
research, an abductive approach values the creative, speculative, and even unconscious
connections between ideas and materials that develop when an artist or designer iteratively
produces aesthetic artifacts. There is a value in how new ideas can quickly emerge through
imaginative and experimental transformations, leaps, and juxtapositions. In terms of the
development of new knowledge, an abductive approach tolerates the role that material craft
and subjectivity play in meaning-making. Michael Polanyi further develops this concept with
the idea of tacit knowledge, a type of knowledge that can be implicit in the body and beyond
its articulation through language. (Polanyi, 1966) The way that intellectual knowledge and
know-how can be embodied through our experience in the world is useful to understanding
how to develop research through design practice. Knowledge can be contained in and
transferred through the tools and images we create in culture and inform intuitive and
speculative problem-solving. Nigel Cross sought to formalize these notions into a design
76
method for research and education described as “designerly ways of knowing,” (1982) to
connect the interplay of thinking and doing, or theory and practice, in knowledge development.
More contemporaneously, Crouch and Pearce discuss how design research can be a means to
address wicked problems and issues of complexity in society in areas where there is no
singular solution or agreed perspective. “Both research practice and design practice grow out
of the relationship between agency, action, and the social structure in which they are contested
and validated. The tensions and contradictions within a practice are managed by the
community that defines it.” (Crouch and Pearce, 2012, p.34) Continued attention and
willingness to grapple with irreconcilable complexity without clear resolution are required to
keep problems in the space of constant dialectical confrontation. In Doing Research in Design,
Crouch and Pearce situate design research as an inherently reflexive practice, which is
pertinent to my research method. Reflexivity affords design research a self-reflective and self-
critical position of its own field of practice, institutional modes, received wisdom, and
ideological values, to understand hierarchies, power relations, and contestations. It also affords
the individual design researcher a self-awareness of themselves as positioned in their field and
how their identity, subjectivity, and life experience inform their work. This awareness of
positionality needs to be constantly readdressed based on changing conditions. This notion of
reflexivity in the development of research is not necessarily practiced in the science and
humanities, where researchers do not simultaneously reflect on how their identity and the
construction of their field, historically, institutionally, economically, and socially, define the
research that is produced.
The design researcher is highly aware of working within a society, a culture, and at a particular
political and economic moment in history, and that such historical conditions define their work
77
as much as the specific questions, inquiries, experiments, artistic practices, prototypes, and
theoretical positions they address. This sensitivity to context “locates design thinking in the
context of a dialectical engagement between ideas and the material world, positioning design
in a continuously changing social environment.” (Crouch and Pearce, 2012, p.37)
The praxis framework that I have developed takes the form of an iterative back and forth
process between design research and critical theory to arrive at reflexive software
development. I use creative and critical coding skills as a form of (digital) material production
that encompasses spatial algorithm visualization, slow computation, and the visualization of
process in computation to support an audience’s technical access and comprehension. I
intersect this with training in critical theory to address issues of power and ideology in the
discourse around technology development and the ethical considerations of design decisions.
Critical code studies and critical algorithm studies, as described in Chapter 2, are specific and
nuanced approaches that I utilize to develop a critical approach to design research. I also use
an abductive approach to knowledge production in the way that I use uncertainty, intuition, and
complexity in the development of experimental and speculative proposals for reflexive software
development.
Reflexive software foregrounds the provision of an interactive experience to aid a user’s critical
reflection on software development and the ideological design and application of algorithms. It
is software development that addresses the development of software, a critical design
approach that is built on the awareness of the social and ideological circumstances and
positionality of its own production. Reflexive software is a method that reframes a key research
question in machine learning, ‘how to solve the problem of interpretability’, into ‘how to contest
78
the problem of interpretability’ in order to reframe the value system that drives the research.
Reflexive software seeks to break the fourth wall of computation. It does not necessarily
prioritize a seamless and simple user experience; instead, it attempts to design critical
elements of deconstruction, revelation, discontinuity, and contestation into a user experience in
preparation for a new era of machine learning domination. A typical approach to machine
learning software situates the user in an unconscious, immersive narrative of the innovation of
pure automation and subservience to its logic. Every algorithm is a mechanical turk that
disguises a human subject, institutional design approaches, and an ideological system behind
its development. Reflexive software attempts to offer the user a sense that our algorithmic
infrastructures are contestable and could be designed from within an alternative ideological
position.
In the interests of reflecting on the context, opportunities, and conditions that have enabled this
research to come into being, in tandem with my own specific research questions and interests,
my own positionally as the author of this dissertation is as a white, European, cis-gendered
woman, based in the United States. As a first-generation student and self-trained creative
coder, I was provided the opportunity to develop this work in a funded PhD program within an
experimental practice-based research department located in Los Angeles, California. In
cultural and economic proximity to the digital technology industry of Silicon Valley, the program
has consciously sought to prioritize women and people of color in its student body to counter
the gender and race imbalance in the influence of digital technology development.
79
Section 2
An overview of the Convolutional Domain critical software
Convolutional Domains has been developed in the OpenFrameworks programming
environment. It visualizes a CNN that uses an open-source pre-trained model ‘image-
net-2012.sqlite3’. The model was trained on the open-source ImageNet database. I used the
OpenFrameworks ofxCv wrapper of the CCV library to visualize various functions of the CNN,
including layers, weights, and classifications.
The software contains four scenes that each embody a critical strategy:
◦ Agnosticism
◦ Contestation of Interpretability
◦ Dissolution of Accountability
◦ Unmodelled
In Agnosticism, an interface serves as a home page or index to the software. Six image nodes
appear arranged in an overlapping composition. The nodes can be dragged and sorted
across the screen. Each node shows an except of an image of one of six labor domains:
Construction, Agriculture, Textiles, Driving, Warehousing, and Administration. The interface
expresses a sense of agnosticism in how each labor domain is treated interchangeably. These
nodes are the input videos to the CNN, which can operate across all the labor domains without
distinguishing between them. Each input, each diverse labor domain, is computed by the
algorithm according to the same logic and without concern for difference or context. The
algorithm neutralizes each labor context and processes their pixel values according to the
80
same formula, regardless of the different cultural, social, economic, and ethical concerns at
stake in each domain. This critical tactic visualizes the problem of agnosticism. The algorithm
cannot recognize contextual difference. The user can select one of the domains to enter the
next scene of the software.
Figure 1. Agnosticism. The initial arrangement of the interface upon opening the software. The images are
positioned overlapping, and the user must move them around the screen to sort and see them all.
81
Figure 2. Agnosticism. When selected, each image is an input to a convolutional neural network.
Figure 3. The application contains six different labor contexts in the form of video inputs. This labor context is
agriculture.
82
Figure 4. This labor context is textiles.
Figure 5. This labor context is warehousing.
83
Figure 6. This labor context is construction.
Figure 7. This labor context is an office.
84
Figure 8. This labor context is driving.
In Contestation of Interpretability, the software presents a different scene of multiple
overlapping but static nodes. The full video of the labor domain previously selected now
appears. To the right is a node that visualizes an excerpt of the neurons in the CNN as it
processes the input video image in real-time. We can see different layers of neurons and how
the algorithm consecutively filters and draws out features of the video. The user can hover over
the neurons to expand them and see the precise position of the neuron in the network. At the
top right is a node that visualizes the network’s classification labels being predicted in real-time
and the estimated accuracy. At the bottom left a text node juxtaposes contextual information
that the CNN does not compute about the labor domain.
This scene of the software experiments with the question of how to visualize interpretability as a
contested concept. Traditionally, an XAI solution for CNNs would only show how the neurons
85
filter the image in a way that translates to the classification labels. However, from a critical
perspective, the line from the input video, through the neurons, and into the output
classifications is absurd. The specific choice of input images offers a more humanistic reading,
in which each scene conveys a story, a meaning, that exceeds the logic of the algorithm’s
analysis and modeling. Human meaning is constructed from awareness of broader
interconnecting cultural, social and political contexts. In order to counter the problem of
agnosticism, context is introduced in multiple ways. Each labor domain is diverse in order to
highlight the difference between domains. The text node invites the user to consider nuances
of place, situation, workplace attributes, personal struggles of the worker, ethical issues the
worker faces at work, forms of AI-driven surveillance, and management that the worker might
work experience. Additionally, the input video chosen reframes the input imagery away from
the highly controlled images that CNNs are trained on and captures another quality, another
perspective of the worker’s experience, that refuses to assimilate with the model's demands.
The software is critically designed to foreground contestation and reframe the stakes of
interpretability and explainability.
86
Figure 9. Contestation of Interpretability showing the context of agriculture. Information in the bottom-left cell details
specific attributes of the agriculture industry, including the type of surveillance already implemented or being
proposed and its associated ethical issues. The top-right cell shows the labels and prediction accuracy of the
algorithm. The bottom-right cell shows an excerpt of the algorithm's nodes or filters that the input video passes
through before being assigned a label.
87
Figure 10. Contestation of Interpretability showing the context of textiles. A user can hover over each network node
to magnify it and see its position in the network structure.
Figure 11. Contestation of Interpretability showing the context of warehousing.
88
Figure 12. Contestation of Interpretability showing the context of construction.
Figure 13. Contestation of Interpretability showing the context of an office.
89
Figure 14. Contestation of Interpretability showing the context of driving.
In Dissolution of Accountability, we see the entire deep neural network structure of neurons and
weights. We are not usually granted this perspective because it is not considered relevant to
show so many nodes to a non-technical person, and they do not look particularly different from
each other. Some nodes are brighter, some darker, some reveal more of the original input
image and some only a small part. From input image to classification, this perspective is
supposed to account for the full process of internal decision-making. The user can hover over
each neuron to expand it and see its position in the structure. You can track neurons between
here and the excerpt of neurons visualized in the software scene Contestation of
Interpretability. However, having access hardly supports explainability of the stakes of this
algorithm. It does not account for any meaningful understanding of the responsibility and
power contained within decision-making. Seeing the whole network and navigating through the
increasingly abstract filters, the possibility of legibility fails. My critical software design argues
90
for an aesthetics of contestation to open up the intersecting concepts of interpretability,
decision-making, complexity, social domains, and lived experience to continual contestation
and reckoning. As a critical software design tactic, this scene does not visualize explainability
(as a singularly computational problem) but contestation (as a sociotechnical problem). I
present this perspective as a failure of explainability. I argue that instead it visualizes how
decision-making is lost in the network, and along with that loss, is a dissolution of
accountability.
Figure 15. Dissolution of Accountability displays the whole network structure. A user can hover over each node to
magnify it.
In Unmodelled, alternative data parameters are presented in juxtaposition to the labor domain.
These counterdata parameters have been selected from research into subjective wellbeing in
the context of work. Each set of parameters also speaks specifically to each labor context and
91
the current or near-future concerns of AI surveillance and management systems in that
workplace. The counterdata parameters are unmodelled because the algorithm cannot
incorporate them into its rigid logic. They are speculative and visualize the values and
concerns missing from the algorithm. They point the way toward a practice of
counteralgorithms, algorithms designed to challenge, provoke, and counter normative
assumptions built into algorithms.
Figure 16. Unmodelled showing the proposed counterdata parameters applicable to the context of agriculture.
92
Figure 17. Unmodelled showing the counterdata parameters applicable to the context of textiles.
Figure 18. Unmodelled showing the counterdata parameters applicable to the context of warehousing.
93
Figure 19. Unmodelled showing the counterdata parameters applicable to the context of construction.
Figure 20. Unmodelled showing the counterdata parameters applicable to the context of an office.
94
Figure 21. Unmodelled showing the counterdata parameters applicable to the context of driving.
At any point, the user can return to Agnosticism and select another labor domain and follow its
critical visualizations through the software again. As the beginnings of a counteralgorithm
strategy, the software experiments with what James J. Brown calls “computational rhetoric,”
(p.31, 2015) making arguments in and with software rather than about software. Loukissas
describes it as “interfaces that cause friction” (p.178, 2019) to recontextualize data uses and
abuses. The interface design is a proposal in reflexive software development; the application
simultaneously unfolds computational functionality and criticality, to present the CNN's
application in the context of labor as a failure in the production of knowledge under late
capitalism.
95
Against code agnosticism
Many phenomena in the world, disciplines of knowledge, and societal functions are
computationally irreducible. Many social, political, and cultural issues are inherently messy,
which are sometimes referred to as wicked problems, problems that do not have a solution due
to their contradictory and complex nature. One could describe culture as computationally
irreducible in that it cannot be reduced to a finite set of data points and correlation vectors.
Despite this, deep learning techniques are applied in ways that deny their complexity and
insist on their reducibility. Many aspects of society are incommensurable and yet are being
forced into computational modeling and valuation standards. Computation’s insistence on
reducibility to the computable can only take place by ignoring, externalizing, and being willfully
blind to certain information and relationships. Additionally, continually reinforcing the idea that
all aspects of society can or should be modeled and predicted forces society to adapt itself to
fit in with arbitrary models rather than the other way around. Human rights, diverse cultures,
and ethical standards are sacrificed in the process.
I argue against the practice of code and data agnosticism which paves the way for
computational reducibility. Code agnosticism is when algorithms and data sets are developed
in separation from the social context in which they are intended to be applied, enabling the
algorithms and data sets themselves to be positioned as socially neutral mechanics. This
strategy, which is at the heart of much development in deep learning, attempts to negate a
political or ethical positionality, which I argue to be untenable now that algorithms cannot be
extrapolated from the world that they inhabit and engineer. There is a culture in scientific
research that strives for universal truths, a theory of everything, a notion of mathematical
96
neutrality and purity, and abstraction as an ideal. However, in this new era in which deep
learning algorithms are being applied across societal domains, including in highly socially
sensitive practices that have historically contended with discrimination and equity, abstraction
and universality pose a significant problem to the development of algorithms.
The reliance of deep learning algorithms on very large data sets that require labeling for
supervised learning is one of the ways that the problem of agnosticism plays out. The time and
labor involved in creating a large data set prevents more specific and nuanced data sets from
being developed and creates greater dependence on preexisting data sets developed without
considering various ethical problems. For example, the ImageNet data set of more than 14
million hand-labeled images was originally developed in 2009; it is credited with unlocking the
potential of deep learning. However, it was developed prior to the advancement of the sort of
deep learning applications that we see today and prior to the critical discourse that has raised
ethical concerns with deep learning data sets and algorithms. ImageNet was developed for
socially, culturally, and politically agnostic situations, and assumes an era in which deep
learning technologies could be developed a-contextually, in a so-called neutral context. Yet,
this data set remains foundational to many deep learning applications today.
The problem with how data sets present objectivity is explored in the project ImageNet
Roulette by the artist Trevor Paglen and the AI ethics scholar Kate Crawford. It was a browser-
based tool in which a CNN was trained on the ImageNet data set and in which the public could
upload a photograph of themselves, and the model would classify it. The results were
disturbing, reproducing racist and sexist tropes. Images of people with darker skin tones were
classified as ‘negro, wrongdoer’, images of women wearing bikinis were labeled ‘slattern, slut,
97
slovenly woman, trollop’, other images were labeled with charged terms such as ‘unsuccessful
person’ and ‘drug addict’. The results raised questions about how such labels became part of
the data set in the first instance and who labeled images in this way, especially labels that
cannot be determined solely by how people look. The results of the experimental tool
demonstrate that the very notion of classification will always be prejudiced and represent a
particular worldview because the very conception of categorizing people is an inherently
political and ideological act. Crawford and Paglen explain:
“Datasets aren’t simply raw materials to feed algorithms, but are political
interventions. As such, much of the discussion around “bias” in AI systems
misses the mark: there is no “neutral,” “natural,” or “apolitical” vantage point
that training data can be built upon. There is no easy technical “fix” by shifting
demographics, deleting offensive terms, or seeking equal representation by
skin tone. The whole endeavor of collecting images, categorizing them, and
labeling them is itself a form of politics, filled with questions about who gets to
decide what images mean and what kinds of social and political work those
representations perform.” (2019)
The project was successful because it resulted in the Stanford University data scientists in
charge of the data set withdrawing it temporarily while certain labels were withdrawn. However,
it demonstrates that even with a powerful and influential dataset on which many deep learning
applications are developed, data scientists are not aware of all the labels inside of it or
understand their politicizing nature.
98
Other datasets, such as the MNIST dataset of hand-drawn digits (0-9) and the CIFAR-10
dataset of ten classes of objects (airplane, automobile, bird, cat, deer, dog, frog, horse, ship
truck), are used as common benchmarks in the prototyping, testing, and teaching of deep
learning applications. However, they also support a culture harboring a false claim to a neutral
image or a neutral dataset. Much pedagogy of deep learning is framed around systems that
predict categories between a number 1 or number 2, or a cat or a dog, affirming this as a
technical workflow that can be scaled up and out into any other context without consideration
for the exponential ethical differences that exist just outside of these artificially reductive
categories. A culture in computer science and software development has arisen out of these
artificially reductive and agnostic workflows, a culture in denial of the social complexity of the
societies in which such technologies are implemented.
Furthermore, when CNNs are used for computer vision tasks in socially-sensitive contexts,
every problem and solution is reduced to being defined by how something or someone looks.
This sole focus on visuality is another great reduction in real-world complexity. Feature
extraction, the process by which machine learning abstracts layers of information from data,
produces visual arrangements of pixels that are unrecognizable to humans. This is another way
agnosticism can be problematized as the dissolution of human meaning-making in the
algorithm. As cited previously, D’Amour et al’s research also captures how real-world problems
are created due to the culture of reductive, a-contextual frameworks in laboratory development.
Underspecification is a ubiquitous problem in deep learning and is when a phenomenon has
many possible causes beyond the capacity of an algorithm to model for. It is a reason for
99
algorithms to perform well in laboratory settings and very poorly in real-world settings. (2020)
The difference is in the nuance that exists in context, and that is abstracted through code-
agnosticism.
Computational agnosticism can be traced back in social history, beginning with what I term,
the arithmetization of society. Arithmetic precedes contemporary computation and digitization
processes but can be understood as a precursor to the current automation through machine
learning. In The Software Arts, Warren Sack describes the drive toward arithmetization
beginning with industrial capitalism’s connection of labor and capital through the ability to
measure and estimate work quantitatively. This centered arithmetic within what were previously
distinct disciplines of knowledge. (p.79) What does it mean when various societal domains and
disciplines of knowledge undergo such a universalizing computability? Today, this drive is
being hyper-extended through the application of deep learning’s predictive decision-making
across many different types of knowledge, functionality, and services, even ones that were not
previously subject to such prediction-based practices, such as sentencing in criminal justice.
Sack argues that “implementation-independent algorithms are an impossible idealization.”
(p.81) Deep learning’s desire for such a universalizing approach cannot be achieved without
removing local contextual differences, and while the prediction models might not value these
differences, this does not mean that externalizing difference is ethically right or that reductive
prediction models themselves should be valued. Models designed to function in one context
and learn against one set of trained standards are then applied in a different context in which
an entirely different set of social and cultural issues arise. The original conception of an
algorithm was expressly a finite system, providing a single function in a single context however,
100
deep learning’s version of implementation-independence means that a single dataset and
model can be applied across a vast array of societal domains, exporting a set of goals and
values from one domain to another without consideration for important nuances. The drive to
mold ideas, values, and societal institutions under the logic of computability and prediction
demands a form of agnosticism, in which all ideas, values, and societal institutions are
reduced to an abstraction of their real-world manifestation. This elides the messiness of context
and culture, regardless of what and who is lost in the process of abstraction.
The problem of code agnosticism could also be explored from a critical code studies point of
view. The nature of writing source code is often one in which code is written in a way that is
meant to be as neutral and adaptable as possible. In many cases, source code is meant to be
easily shared, adapted by others, and encapsulated into open source libraries. The very nature
of object-oriented programming languages is in their modularity and extendability. This
produces a system in which code written for a highly particular function can appear less useful
because it appears less easily adaptable to different applications. This leads to variables,
function names, and classes being named in a neutral way, and while their names do not
change how they are executed, when we read code, the problem is not what is there and
named, but what is not there and not named. Naming code attributes refrains from being too
specific, but what if this was not the culture of writing code? On the other hand, code could be
written in a way that values and seeks to build in as much contextual specificity as possible at
the expense of seamless adaptability. Neither approach changes the way code executes, but
one approach values awareness of context over the ease of extendability. Furthermore,
perhaps, the former offers a layer of friction between code being utilized with impunity across
any context. Naming practices in source code create a means of disassociation from its
101
application, which is another way of thinking about code agnosticity.
Toward contextual code
In order to move away from code agnosticism and move toward understanding code in
context, we can look to an emerging discourse that argues for contextualism in code and
algorithmic tools.
Yanni Loukissas argues in All data are local, that there is no data without context and proposes
to move away from the term data set, which evokes the discrete and complete, and instead
grapple with data settings, that are rooted in place, time, and local nuances. Using an example
of the connection between the text of a book and its index, he writes that “too often we attempt
to use a given data set as a complete work, such as a book, rather than an index to something
greater.” (p.3, 2019) We can consider this argument concerning the use of highly portable and
reductive data sets used in deep learning systems to make predictions about complex and
messy issues. Loukissas asks us to reconsider data’s locality and complex attachment to
place: from where did it come, who produced it and when, what instruments were used to
collect it, what kind of conditioned audience was it intended for, and how might these invisible
attributes inform its composition and interpretation.
While his arguments “against digital universalism” (p.9, 2019) speak directly to the culture of
working with data, he also acknowledges that data and algorithms are inextricably intertwined,
and his ideas are equally important to the development of deep learning algorithms. Why do
we strive for the digital to be so place-agnostic, a totalizing system of norms that erases the
myriad of cultures? The myth of placelessness implies that everyone can be treated equally by
102
immutable algorithms. Loukissas concludes that “[o]ne reason universalist aspirations for
digital media have thrived is that they manifest the assumptions of an encompassing and rarely
questioned free market ideology.” (p.10, 2019) We need to insist upon data’s locality and
multiple and specific origins to resist such an ideology. “If left unchallenged, digital
universalism could become a new kind of colonialism in which practitioners at the ‘periphery’
are made to conform to the expectations of a dominant technological culture (…) a self-fulfilling
prophecy by enforcing its own totalizing system of norms.” (p.10, 2019) Loukissas appropriates
ideas developed from science and technology studies, feminist theory, and post-colonial
studies to explore ideas about local, universal, and situated knowledge production.
The concept of situated knowledge comes from Donna Haraway’s proposal to apply feminist
thinking to science. Suppose feminist critiques of masculinity, objectivity, and power were
applied to the production of scientific knowledge. In that case, we might arrive at a form of
situated knowledge, in which positionality, subjectivity, and their inherently contestable natures
are considered to produce greater claims to objective knowledge. (1988) The concept of
situated knowledge is being drawn on by various scholars of computational ethics to propose
a way to counteract the false claims of neutrality and drive toward agnosticism in data and
algorithmic processes. For example, Catherine D’Ignazio and Lauren Klein argue in their book
Data Feminism that context matters for making sense of correlations when working with data.
They also acknowledge situated knowledge as the legacy of this central tenet of feminist
thinking that supports a means of addressing issues of responsibility and ethics in relation to
knowledge production.
103
“Rather than seeing knowledge artifacts, like datasets, as raw input that can be
simply fed into a statistical analysis or data visualization, a feminist approach
insists on connecting data back to the context in which they were produced.
This context allows us, as data scientists, to better understand any functional
limitations of the data and any associated ethical obligations, as well as how
the power and privilege that contributed to their making may be obscuring the
truth.” (p.152, 2020)
However, a discussion of context assumes a certain level of access and availability to that
extensive information. The authors point to the promises and problems of many open data
initiatives, which assume that being open-source and freely available also means that
contextual provenances are traced. In fact, the labor of deciphering data, providing contextual
information, and understanding its significance is intensive and arduous work, which is often
not provisioned for.
Jaron Lanier also takes up the notion of data provenance in his book Who Owns the Future? as
a component of a humanistic computing paradigm. Humanistic computing is another approach
that connects provenance and context in computational systems as a way to achieve better
economic and political symmetry and accountability.
“In humanistic information economics, provenance is treated as a basic right,
similar to the way civil rights and property rights were given a universal
structure in order to make democracy and market capitalism viable.” (p.245,
2013)
104
Lanier argues that retaining provenance counters the way that technology can dehumanize
people, specifically the role and value of people’s labor in the construction of datasets in large
networks, such as social media platforms. Tracing provenance is another approach to building
in context to data and algorithmic systems that ties them to more progressive ethical and social
values. D’Ignazio and Klein further this by arguing that “[r]efusing to acknowledge context is a
power play to avoid power. It is a way to assert authoritativeness and mastery without being
required to address the complexity of what the data actually represent” (p.162, 2020) Data
feminism is an intersectional approach to data science that counters the drive toward
optimization and convergence in favor of addressing the stakes of intersectional power in data.
Sarah Myers West also applies a feminist derived concept of situated knowledge to critique the
discourse of algorithmic fairness. She argues that fairness is often approached too
simplistically when it comes to algorithmic harm when in reality, the notion of what is fair and
unfair is more complicated. Myers West proposes thinking through the ideas of situated
knowledge in the development of algorithmic systems to affirm the entanglements of difference
rather than neutralize them. She argues that:
“[shifting] from individualized notions of fairness to more situated modeling of
algorithmic remediation might create spaces of possibility for new forms of
solidarity and refusal, accounting for the socially embedded nature of our
experiences and seeking to build collective power rather than seeking to
normalize and erase our differences.” (2020)
105
Myers West suggests that algorithmic critique emanating from the humanities could be used
not only to fix and remove harm caused by algorithms, but that algorithms themselves could
also be used as tools of remediation, perhaps even mechanisms of affirmative action. Jill
Walker Rettberg has also built on Haraway’s concept of situated knowledge to rethink context
in algorithmic bias. Her method of situational data analysis analyzes the constructed and
representational nature of data encoded power relationships in social media platforms and
apps to overcome the framelessness and lack of positionality in big data. Rettberg is
concerned with data’s representation and operational performance as it plays out across
normative disciplinary power and environmentally embedded power. (2020)
The politics of labor, surveillance, and the robotization of the worker’s body
The politics of labor is the context into which I am situating Convolutional Domains, my critical
software application, specifically, the surveillance of working bodies in an era of deep learning
industrial late-capitalism. There is an emerging application of convolutional neural networks
and other forms of deep learning, which extend industrial practices of worker monitoring and
evaluation aimed at a precision-optimization of worker productivity. Convolutional Domains
addresses this emerging development of deep learning algorithms as a politics of seeing and
predicting, which creates an ideological shift in the condition of the worker, and the question of
who is seen and what is not seen.
The emerging implementations of machine learning in the workplace have the legacies in the
historical practices of industrial labor exploitation of the late C19th and early C20th. Taylorism
defines the application of the scientific method to workplace management in order to drive
productivity and efficiency. Frederick W. Taylor conceived of the role of the industrial
106
management consultant by developing a systematic approach to the monitoring and
measuring of labor. His approach reduced work tasks into smaller and smaller parts that relied
less on an individual worker’s skill and allowed knowledge to be transferred into systems and
processes. Taylor also developed time studies to analyze micro tasks further. Ultimately this
created a system that could easily interchange and replace workers, enabling employers and
managers to have greater control over the minutiae of worker productivity.
Contemporaries of Taylor were Frank and Lillian Gilbreth, who also contributed to the shift in
industrial management approaches by developing a series of motion studies. The Gilbreths
created a series of short films staged in various workplaces between 1910 to 1924 to
document and analyze factory workers, bricklayers, and even doctors during medical
surgeries. The 35mm black and white film recordings show workers performing various
repetitive micro tasks. (Gilbreth, Frank B., and Lillian M. Gilbreth, 1910-1924) The Gilbreths
staged each scene to include a specially calibrated clock called a microchronometer to record
time in milliseconds. They attached small flashing lights to workers' bodies and hands while
performing tasks to measure the length of time individual body motions required. They also
added vertical and horizontal black and white gridded surfaces into scenes to precisely
measure small arm and hand movements across space. The Gilbreths developed an index
comprising 17 basic worker body movements, or behavioral units, such as ‘search’, ‘select’,
‘transport loaded’, ‘position’, ‘hold’, which can be combined into different work tasks. They
termed these units ‘therbligs’, an amalgam of their own name.
107
Figure 22. A film still from Frank and Lillian Gilbreth’s motion studies analyzing bricklayers at work. Internet Archive.
108
Figure 23. A film still from Frank and Lillian Gilbreth’s motion studies analyzing an office work. Internet Archive.
As Taylor analyzed workers performing micro tasks over time, the Gilbreths studied workers
performing micro tasks as motion. For example, in their film analyzing bricklayers, they
observed how much a worker’s body needed to bend and stoop to pick up bricks and how
many bricks could be picked up in a single grasp. They claimed to increase productivity from
175 bricks per hour to 350 bricks per hour. The Gilbreths initial reasoning driving this research
was to eliminate unnecessary and overly repetitive behaviors for workers and protect workers
from excess fatigue and wasted energy. However, the research was also criticized for leading
to increased mental and physical burdens on workers due to pressure to work more and faster,
ultimately improving industrial efficiency primarily for the benefit of those in power. (Karsten,
109
1996) Today, the legacy of Taylor and the Gilbreths' work remains formalized in industrial and
business efficiency models and labor relations.
In the context of machine learning and the criticism of robotizing workers, Gilbreth’s motion
studies can be understood as an analog precursor. Before algorithmic methods existed, their
work demonstrates how monitoring workers' bodies, the microanalysis of behaviors, and the
transfer of embodied knowledge into management systems, was already an established ethos
in a shift of power away from workers and towards the owners of those systems. In the mid-
C20th, Fordism further transcribed such drivers of control into the more advanced mechanized
factory of assembly lines to increase systematization and standardization, also aimed at
efficiency and productivity. The Gilbreth’s studies continue to feed into shifts in contemporary
labor imaginaries. In the C20th, the worker’s body and its role in the workplace were adapted
to model the process, speed, and scale of mechanical technology, and it follows that in the
C21st, the worker’s body is again being adapted to assimilate with the logics and scales of
predictive algorithmic technologies.
Contemporary attempts to bring machine learning technologies into the workplace evoke more
extreme extensions of Taylorism and Fordism, but with new problems. AI ethics researcher
Kate Crawford points out how “[m]any large corporations are heavily investing in automated
systems in the attempt to extract ever-large volumes of labor from fewer workers. Logics of
efficiency, surveillance, and automation are all converging in the current turn to computational
approaches to managing labor.” (p.55, 2021) This shift has consequences for the previous
progress that has been made with labor rights and the ethos of work. Over the last century or
more, society has developed more progressive values around labor and employment by
110
recognizing labor rights, the need for healthy working conditions, and notions of care and
respect in the workplace. We often hear a narrative that argues that machines have freed
people from the drudgery of rote, physically arduous, and unhealthy work tasks. However,
through the use of machine learning systems in the workplace, we see a regression in these
advances. Employers are now using machine learning to monitor, scrutinize, and assess data
extracted from the micro-behaviors of workers’ bodies, a type of surveillance and control that
was not previously available.
The AI Now Institute points out how the inherent problem of using AI in social domains:
“The integration of AI systems within social and economic domains requires
the transformation of social problems into technical problems, such that they
can be “solved” by AI. This is not a neutral translation: framing a problem so
as to be tractable to an AI system changes and constrains the assumptions
regarding the scope of the problem, and the imaginable solutions.” (AI Now
Institute, 2016a)
Conformance does not only happen in the way that users or workers must subject themselves
to an algorithm's needs to see, track, and process, but social problems themselves must be
redefined for developers to claim that algorithms can solve them, which is a political issue
In her research investigating the use of AI in Amazon warehouses, Kate Crawford presents a
picture of a workplace where the focus has moved away from the worker to prioritize the
machine, in the name of greater efficiency and profit. Workers are exposed to the same
111
scanning and micro monitoring as the packages they move around. Workers must perform to a
‘rate’, an allocated time for completing certain tasks that is directly tied to Amazon’s business
model and therefore cannot be negotiated. Sometimes this rate is defined by robotic
machinery, and workers must perform to the same timekeeping, despite the toll this takes on
human bodies and psyches, which is a toll that robots are immune from. The labor dynamics
have shifted to one in which robots are well-tended while human bodies struggle to perform as
efficiently. Workers must continually adapt to the rhythm and changes of the machines; they
must endure constant micro surveillance and perform a type of work that consists of filling in
the gaps of tasks that machines cannot perform while never being able to apply their own
know-how to their work. On a tour of the factory, Crawford observes that many workers wear
support bandages for strain injuries, and the vending machines are stocked with painkillers.
(p.53, 2021)
We have heard similar stories in which the needs of robots subsume the needs of human
bodies. Amazon workers have complained about being forced to skip bathroom breaks due to
the fear of not being able to keep up with robots. Delivery drivers have complained about
being prevented from wearing sunglasses because they interfere with how computer vision
systems are built to monitor their faces when driving. We have heard about skylights being
removed from factories because natural light reflection interferes with the computer vision
algorithms that robots rely on. Agricultural workers have complained about a form of wage theft
that takes place when robots break down, and they are not paid for the unproductive time it
takes for them to be fixed or when at times, they can work faster than the automated system
but must perform at the robot’s pace instead, lowering their potential wages. Instead of a world
in which robots are designed to make work easier for people, and human-AI collaboration is
112
largely beneficial, “this collaboration is not fairly negotiated. The terms are based on a
significant power asymmetry - is there ever a choice not to collaborate with algorithmic
systems?” (p.58, Crawford, 2021) We hear increasingly more stories of how low-paid and
undervalued human labor is reconfigured to attend to the AI system's health and primacy and
conform to the needs of algorithmic logics. Additionally, the data extracted from workplace
machine learning systems can also be sold on to data brokers as a secondary product to train
other systems or provide industry insight, without workers' permission.
The notion of robotizing humans comes into play through the ways that machine learning is
being implemented in industrial workplaces, in which people are required to conform to the
needs of robots. As a result, human bodies must change their gestures, actions, and behaviors
in a way that is interpretable to machine vision systems, all the while masking the still essential
role of human labor in AI-driven automated systems.
“[I]nstead of asking whether robots will replace humans, I’m interested in how
humans are increasingly treated like robots and what this means for the role of
labor. Many forms of work are shrouded in the term ‘artificial intelligence,’
hiding the fact that people are often performing rote tasks to shore up the
impression that machines can do the work. But large-scale computation is
deeply rooted in and running on the exploitation of human bodies.” (p.56,
Crawford, 2021)
Human labor is not actually replaced but rather dispersed geographically and temporally and
thus devalued. The research organization AI Now released a report that studied the impact of
113
artificial intelligence on labor rights and found that there are still many jobs that are essential to
the effective running of so-called automated infrastructures. Disguising the necessity of human
labor in this field is another way to devalue it. (AI Now Institute, 2016a) Platforms are often
presented to the world as frictionless, automated, and not requiring any oversight; however,
Crawford argues that “[c]ontemporary forms of artificial intelligence are neither artificial or
intelligent.” (p.68, 2021) In fact, underpaid, uncredited, and unseen workers, often employed
through crowdsourcing platforms and in countries where labor is much cheaper, are required
to build, test, and maintain these systems. They are employed to label large datasets
exhaustively, tag images for computer vision systems, check the accuracy of algorithmic
results, and review harmful content that has been posted online. Facebook alone has to hire
thousands of workers to monitor posted and streamed content for violence, exploitation, hate
speech, and misinformation, which can have a detrimental impact on the mental health of
those who have to witness such disturbing content constantly. “Faking AI is an exhausting job,”
(p.64, 2021) states Crawford, amusingly. Additionally, the above list does not touch on the huge
unpaid immaterial labor of platform users who provide content and interaction and whose data
is extracted and monetized.
Lilly Irani’s term “data janitors,” describes more seriously the hidden and undervalued role of
labor at the heart of contemporary technological advancements and gives representation to
those who perform this work. (2015) Mary Gray and Sid Suri refer to this particular form of
silicon valley administered labor practice as “ghost work.” (2019) While essential for AI to
function, it risks creating a new precariat labor force that is disempowered due to the strain
and uncertainty of chronic underemployment and wage theft through zero hour contracts, the
constant threat of replacement and redundancy, insufficient protections, and the structural
114
inability to plan for the future or alternative employment.
Machine learning-based automation in the workplace is not only occurring in industrial settings;
in office-based workplaces surveillance systems are becoming the norm in managerial
practices. Office workers are also subjected to their movements being tracked, the monitoring
of the number and rate of keys hit on a keyboard, the number of emails sent, and meetings
arranged. Algorithms now process job applicants and create shortlists with known sexist and
racist prejudices. In addition, algorithms predict an employee’s likelihood of succeeding and
fulfilling corporate values. Some workplaces even analyze workers' personal social media
interactions for information on appraising their effectiveness in the workplace. Another risk of
machine learning systems taking on greater managerial roles is the lack of redress that workers
have to appeal decisions made by uninterpretable algorithms, whose main role is to provide a
statistical overview of productivity.
“Remote and disembodied management can effectively shift power from
frontline managers (people who can be approached face-to-face and who
may understand nuanced situational context) to executives whose view is
shaped by more impersonal aggregate employee data.” (AI Now Institute,
2016b)
If worker-employer relations are given over to algorithms, there is a risk that employers can use
this technology to avoid accountability, and workers will be left without legal protections. How
can a worker hold an employer accountable for an automated decision of which the decision-
making process cannot be deciphered? AI Now advocates updating the definitions and
115
frameworks for fair labor practices to keep abreast of the changes occurring as a result of
machine learning management systems in the workplace.
Contemporary implementations of machine learning in industrial workplaces present a vision of
production that prioritizes efficiency and profit to the detriment of people's working lives by
putting bodies back into historical conditions of repetitive and physically arduous drudgery.
The contemporary vision for AI in the workplace is never presented as a vision to assist
workers and improve their lives, but is a vision of surveillance that demands human workers
conform their bodies to the logics of algorithms and perform their actions to the needs of
robots. The use of machine learning appears to be shoring up an already unsustainable drive
toward increasing wealth inequality by disregarding any notion of how the economic benefits of
AI technology could be more fairly distributed through society.
116
A case study of a CNN used for labor productivity evaluation
Figure 24. A video still from the software demo of ‘Construction Activity Recognition with Two-Stream Convnets’, by
Luo et al., 2018.
There is an area of emerging research into how advancements in computer vision using CNN
could be applied to the industrial labor context to surveil, monitor, and assess workers'
productivity. In recently published research titled Towards Efficient and Objective Work
Sampling: Recognizing Workers’ Activities in Site Surveillance Videos with Two-Stream
Convolutional Networks, the authors claim to have developed a state-of-the-art application of
computer vision and human activity recognition for the construction industry “to precisely
quantify and benchmark labor productivity, which in turn enables them to evaluate productivity
losses and identify causes.” (Luo et al., 2018) The research extends existing techniques that
use CNNs from previous work-sampling techniques that were limited by only focussing on
equipment or individual workers, to now recognizing a broad array of activities in videos. By
117
combining object tracking, optical flow estimation, and activity recognition and using two
stream CNNs to extract both spatial and temporal information, the research claims to build an
efficient and objective work sampling technique that continuously recognizes a diverse range
of worker activities. The technique is 80.5% accurate, which is considered state-of-the-art.

The system works by setting up surveillance video cameras positioned high up and looking
down over the construction site, such as on a crane. The surveillance video is the input data to
a CNN that continuously processes the image, tracking workers, extracting spatial and
temporal information, classifying their activities, and producing productivity estimates. This
system specifically tracks human-object interactions by monitoring bodily gestures, such as
the movement of individual arms and legs, and bodily actions, such as walking, lifting, and
pushing. The system trackers all workers in view individually and simultaneously. The system is
considered non-invasive because the cameras are not positioned on workers’ bodies, and they
are not close enough to detect faces or perform skeleton-based posture recognition (another
type of body monitoring).
The research aimed to develop a new publicly available training dataset created from the site
surveillance video of a real-life construction project in Hong Kong. The resulting dataset
consists of 1058 three-second video clips that identify a discrete worker activity and are
labeled according to a worker activity taxonomy consisting of 16 categories defined by the
researchers. The categories are: communicating, measuring, moving, preparing-switching,
waiting-resting, formwork-fixing, formwork-machining, formwork-taking-placing, formwork-
transporting, rebar-connecting, rebar-fixing, rebar-machining, rebar-placing, rebar-taking,
rebar-transporting, rebar-welding. The model is trained to identify these activity categories and
118
classify them as one of three modes: productive, semi-productive, and non-productive. With
regard to the category of ‘waiting-resting’, it should be noted that this includes standing still,
standing and drinking water, or standing and wiping perspiration. These categories are
encoded as non-productive. Additionally, manually carrying or shouldering materials such as
rebar (steel rods) around the site is encoded as semi-productive.
Figure 25. An image from the research paper by Luo et al., 2018, detailing how the image data is analyzed based
on color and motion-tracking in both x and y axes.
From a critical perspective, there are various concerns in this case study to highlight. The data
set of videos consists of three-second clips because the authors determined that this was the
amount of time necessary to discretize continuous activities into finite atomic units. This
119
suggests how the algorithm will learn what work activities are from three-second periods and
points toward how workers might have to conform to the learned logics of the algorithm
eventually. Additionally, it is problematic that 80.5% accuracy is considered state of the art
without reflecting on the potential harm that 19.5% of mistakes could cause. If this type of
algorithm is deployed, we should assume it is precedent-setting, and it will expand into other
labor domains. Similar research in this area claims that these techniques are aimed at
increasing safety, for example, by ensuring workers are wearing the correct protective clothing;
however, we should practice skepticism that these claims are covert strategies to test out more
problematic autonomous management practices.
This case study in CNN methods to evaluate productivity, alongside the dataset and activity
labels, strongly recall the legacy and ongoing influence of the Gilbreths' motion studies,
especially their index of therbligs, behavioral units of pre-computational activity units. Again,
this points to the inherent ideological values that drive technology development irrespective of
the nuances of their techniques.
In research by Kellogg et al. titled Algorithms at work: the new contested terrain of control, the
authors conceive that “a model of algorithmic control [is] the new contested terrain of control.”
(2020) Algorithms are reshaping the relations of production between managers and workers,
and that algorithmic control is a new form in labor relations, that extends historical workplace
models of control such as bureaucratic control and technical control. Algorithms are a
“structurally antagonistic character of employer-worker relations. It allows us to understand
algorithmic systems not as neutral tools that facilitate efficiency and improve communication
exchanges, but as contested instruments of control that carry specific ideological
120
preferences.” (2020) The authors point to how “algoactivism tactics allow for individual and
collective resistance of algorithmic control.” (2020) Algoactivism is a complementary concept
to counteralgorithms that can be used to reframe critical algorithm development.
The contestation of interpretability
Previous research into visualizing CNNs for the purposes of explainable AI demonstrations
always uses a neutral input image. As previously cited, CNN visualizations purposely choose
input images showing dogs, cats, pieces of fruit, and numerical digits, images that the
researchers consciously choose to be as culturally and socially neutral as possible. My critical
software uses images that are selected because they cannot be described as neutral; in fact
they are charged images conveying cultural, social, and politically specific contexts. The
images show scenes of different kinds of workers, places of work, labor conditions, and bodies
at work. The images collectively and individually present a context of the politics of labor. My
critical software is also developed in the context of the increasing encroachment of AI
algorithms developed to surveil and monitor productivity in the workplace, for example, as
cited previously in the work of Luo et al.
The curation of input images in my critical software cannot be abstracted from the wider culture
of issues, images, and political discourse that they originate from. What is taking place in the
images, the people, objects, activities, and places that are represented cannot be separated
from the meanings they hold for an audience. The curated images are chosen because they
require human interpretation to read meaning, where meaning is derived from the intersection
of an individual’s own positionality and broader cultural, social, and political context. The deep
learning model does not have access to a notion of human positionality, context, and therefore
121
meaning. As Crawford and Paglen point out in their design-theoretical project ImageNet
Roulette, which critically analyzes the ImageNet data set:
“The circuit between image, label, and referent is flexible and can be
reconstructed in any number of ways to do different kinds of work. What’s
more, those circuits can change over time as the cultural context of an image
shifts, and can mean different things depending on who looks, and where they
are located. Images are open to interpretation and reinterpretation.” (2019)
When a prediction model labels an image, that label influences the way we understand the
image, including when the label abstracts and diminishes meaning. My curation of images
refuses such simplified prediction labels and the consequent abstraction from context. A
rupture is created between images that are charged with cultural, social, and political meaning,
and the discrete and rudimentary prediction labels applied to them. This is a tactic to
foreground interpretability as a contested term. In my critical software application, the ability to
read an image requires a humanistic interpretation that emerges from an awareness of
interconnecting cultural, social, and political contextual issues. Yet, the CNN’s intelligence only
allows it to model within its own simplified logic that creates correlations between aggregations
of pixel values and arbitrary labels. Explainable AI can no longer only frame its task around
such correlations but must engage with the interpretation of meaning. By foregrounding this
rupture, the application forces a politics of seeing, sensing, and predicting into the
development of CNNs for worker monitoring and productivity measuring. It cleaves apart the
logic of computational assimilation that the model demands and advocates that the
122
interpretability problem must be reframed to engage with interpretability as a contested term.
This approach requires algorithms to be coded in context, grapple with the nuances and
contestations that context brings, and also have an awareness of their own coded context
within computation and the assumptions implied with that.
Computer science’s formulation of the interpretability problem is a problem. It reduces complex
socio-political issues to computational logic and method. Computer science’s solution to the
interpretability problem is termed ‘explainable AI’, often abbreviated to XAI, and is described in
chapter 1 of this dissertation. However, explainable AI denies many of the ways in which deep
learning algorithms create and contribute to ethical issues. XAI also insists that such ethical
issues can be revealed and therefore solved through purely mathematical analysis. XAI as an
answer to the interpretability problem is artificially reduced to present a solution. However, this
can only happen by externalizing and denying certain factors. In this dissertation, I argue that
the term interpretability should be reconsidered and understood as an inherently contested
term, one that cannot not, in good faith, be approached solely from the perspective of
computer science, but must engage in the long legacy of interpretation as addressed by the
humanities and arts.
Interpreting deep learning systems in CNNs cannot be reduced to deciphering pixel logics
and the generation of weights. Interpreting deep learning systems cannot be reduced to
translation between two systems when those two systems are incommensurable. Code has
meaning in society in the way its application changes power relations and disrupts ethical
standards, and this requires more than mathematical analysis to interpret and explain. Marino
argues:
123
“interpretation in a humanities context is less about mapping a one-to-one
correspondence and more about identifying connections and resonances
between signs and referents, as well as identifying disconnections and
slippages along with the forces that shape or distort meaning.” (p.42, 2020)
The interpretability problem cannot claim to translate one-to-one correspondences without
considering the connotations of those logics and the slippages that occur at all moments of
translation, including the translation from the developer’s lab to societal domains. Interpretation
is an inherently contested concept, and explainable AI must take up the challenge of
addressing code’s meaning within its framing of the interpretability problem.
Deep learning neural networks are considered to be complex systems: a large number of
intersecting nodes and weights that make it impossible to reverse-engineer predictions.
However, this is just one framing of complexity, framed computationally and thus simplified to fit
within computational logics. In fact, deep learning neural networks’ framing of complexity
should extend into their many intersections with society: data collection, data parsing,
philosophical implications of classification, new agencies that deep learning has in defining
power relations, applications and misapplications of deep learning-driven tools, and the
unintended side-effects on society. Unfortunately, the interpretability problem avoids this
conception of deep learning’s complexity, and I argue that the conception of deep learning as
a complex system is not complex enough; it avoids its true complexity. Instead, what we need
is more akin to Haraway’s concept of tentacular thinking, (2016) an approach that shifts away
from solely binary logics toward nonlinear and multiple trajectories and patternings, where the
tentacular can breach the current definitions and boundaries of traditional disciplines and open
124
us new tentacles of connection and meaning-making.
From this perspective, there cannot be complexity without ambiguity. The field of artificial
intelligence must be willing to grapple with the false reliance on such a narrow framing of
complexity and the consequent uncertainty that real complexity engenders. Therefore, when
framing the interpretability problem and its pursuit of explainability, I argue that interpretability
must always be understood as a contested term. This contestation should be foregrounded
against the assumption that all problems have a computational solution and all computation
problems can be explained computationally. There is a long legacy of interpretative practice in
the arts and humanities, and this is a resource that can be incorporated into how we approach
the interpretability problem.
The unmodelled
Previous research into the interpretability problem has developed explainable visualizations
that seek to plot a pathway between an input image and a set of prediction labels, between the
network's input layer, through its interior node structure, and into the output layer of the
network. This is assumed to be somewhat sufficient to explain how CNNs interpret images and
arrive at their predictions. However, the explainable visualization is constructed to confirm the
algorithm’s own logic, not to question or probe the core assumptions made during its
construction. As a result, explainability, and thus interpretability, are reduced to creating a
linear path between input and output. It is worth noting that neural networks are frequently a
preferred algorithm across many application types because they are considered to have the
highest performance accuracy; however, in previously cited research by Luo et al., state-of-
the-art accuracy is given as 80%. Whether we are considering the problem of inaccuracy and
125
mistakes that an algorithm produces, or we are considering more nuanced concerns around
injustice, the problem with explainable machine learning visualizations is their failure to
recognize how injustices are created systemically and are the result of distributed
sociotechnical processes that neural networks and their developers are implicated in.
Instead of mapping linear and misleading pathways between input and output to posit
explainability, my critical software also visualizes what I term the unmodelled. Other adjacent
terms appear in the emerging discourse of critical data studies, including the terms: antidata,
missing data sets, activist data, counterdata, agonistic data, statactivism, and citizen science.
All these terms were created to take issue with the question: Why is so much data still absent
and needed in a world that seems saturated with data? The art research project The Library of
Missing Datasets by Mimi Onuoha, takes the form of a physical installation, comprising a
traditional office filing cabinet that contains rows of manila file folders that are each labeled with
the title of a dataset. Example labels include: ‘Total number of local and state police
departments using stingray phone trackers (IMSI-catchers)’; ‘People excluded from public
housing because of criminal records’; ‘How much Spotify pays each of its artists per play of
song’; ‘Master database that details if/which Americans are registered to vote in multiple
states’. However, all of the folders are empty, and the labels actually describe data that does
not exist. Such missing datasets signify exclusion, indifference, and hidden biases. The
concept of missing datasets shifts our focus, as Onuoha writes:

“[w]herever large amounts of data are collected, there are often empty spaces
where no data live. The word "missing" is inherently normative. It implies both
a lack and an ought: something does not exist, but it should. That which
126
should be somewhere is not in its expected place; an established system is
disrupted by distinct absence. That which we ignore reveals more than what
we give our attention to. It’s in these things that we find cultural and colloquial
hints of what is deemed important.” (2018)
What is potent about the project is its emphasis on why the data is missing. All of the examples
of missing datasets seem feasible, as though they could or should exist, and their existence
would be helpful in addressing various social issues. The very nature of something not existing
invokes issues of power and access. Onuoha offers four reasons why some datasets do not
exist: a lack of incentive on the part of those who have the resources; the nature of the subject
resists simple quantification; the amount of work involved is perceived to exceed the benefit;
there are advantages to some people for the data to remain uncollected. (2018)
D’Ignazio and Klein also speak to the relationship between missing data and how power
operates. Their research discusses how to apply intersectional feminism to data science in
order to examine how power and oppression are at stake within data. The authors describe the
example of activist data collection so that power can be questioned and challenged and data
science can be transformed. Grassroots data collection efforts can fill in the gaps of missing
data, transforming data collection into a political and transgressive act. This is the case for the
data collection efforts by Maria Salguero, a single woman who has been compiling a
comprehensive dataset on femicide (the gender-based killing of women and girls) in Mexico
since 2016. (Salguero, 2020) Despite various attempts by the Mexican government, the courts,
and a UN symposium on the topic, large numbers of femicides in Mexico have continued.
Salguero recognized that because the state did not collect data on this issue, it reduced their
willingness to take meaningful action. She began to construct a detailed and publicly available
127
dataset that presents a powerful representation of the problem.
D’Ignazio and Klein see Salguero’s work as an example of how missing datasets characterize
the problem of unequal power relations “in which a gendered, racialized order is maintained
through wilful disregard, deferral of responsibility, and organized neglect for data and statistics
about those minoritized bodies who do not hold power.” (p.37, 2020) They also point to how
Salguero’s work, as a form of counterdata collection “demonstrates how data science can be
enlisted on behalf of individuals and communities that need more power on their side.” (p.37,
2020) Without data, it is very difficult to know the scope and scale of a problem and for other
stakeholders, including citizens, journalists, and activists, to access reliable information and
work toward greater justice.
In an interesting arc back toward machine learning, D’Ignazio also works with the Data +
Feminism Lab at MIT to expand the means of femicide data collection using machine learning
algorithms to automate the detection and logging of femicide by analyzing news reports.
(D’Ignazio et al., 2020) The aim is to support the work of data activists by reducing the time
and mental health burden placed on activists who have to read through so many traumatic
news stories. The hope is also to inform policy advocacy by systematizing data collection of
femicide across different contexts. In this way, examining missing data and developing
counterdata strategies are ways that support data and algorithmic accountability.
Loukissas also advocates for a counterdata approach to challenge our normative
understanding of algorithms and question power dynamics in data sets. In machine learning,
data and algorithms are entangled in invisible ways, and we need to be cognizant of their
128
intersections. A comparative approach to data analysis is a means to counter assumptions and
raise questions about data. Loukissas points out that “[d]ata sets of high volume and variety
are often composite collections created across disparate times and places.” (p.169, 2018) By
recognizing the inherent construction and difference within data, we can consider the different
values and norms it embodies. By foregrounding the comparative aspect of data and
maintaining a comparative strategy in our analysis, it helps to raise questions such as:
◦ What is at stake in privileging one data set over the other?
◦ When combining datasets, does this raise or lower levels of uncertainty?
◦ What does a dataset suggest about the identity of its collector (positive or negative)?
◦ What kind of people, places, technologies, and funders support data collection?
◦ What kind of power relations are enacted in the production of a dataset?
◦ Who and what is marginalized by these production practices?
◦ Consider how data might be collected otherwise, by or about people and things that
are left out of current collections.
◦ Are counterdata meant to fill in gaps in existing data, replace those data, or stand in
juxtaposition?
Loukissas writes that “we also need counterdata or even antidata: tactical representations that
challenge the dominant uses of data to secure cultural hegemony, reinforce state power, or
simply increase profit.” (p.91, 2018) Counterdata is a strategy with many affordances, including
the way it can encode different perspectives into an issue, the way it can challenge normativity,
or counteract false or dominant historical representations. Counterdata can contest normative,
traditional, top-down, institutional approaches to data collection, and it can support the
129
reimagining of future alternatives.
In Convolutional Domains, the unmodelled is a critical counterdata strategy that has been
conceived from research into subjective wellbeing and wellbeing in the workplace. As a critical
software project that takes issue with the notions of explainability and interpretability in
machine learning, knowing what an algorithm cannot predict is just as important as knowing
what it does predict. This supports resistance to the idea that predictions made in one context
can be equally applied to another. The concept of the unmodelled in this critical software
project considers both real and speculative data parameters that have not been collected in
the dataset and have not been modeled by the algorithm.
The science of wellbeing
The new science of wellbeing comprises various approaches that attempt to measure both
individual and societal notions of happiness and life satisfaction, including a sense of purpose,
quality of life, and subjective wellbeing. Several countries’ governments now fund research into
the science of wellbeing, as well as academic institutions such as the University of Oxford’s
Wellbeing Research Centre, and international institutions such as the Organisation for
Economic Co-operation and Development (OECD) and the United Nations (UN) Sustainable
Development Solutions Network. For many people, the first encounter with wellbeing research
comes from media reporting on the annual World Happiness Report, organized by the UN
since 2012. Wellbeing research seeks to counteract the dominance of gross domestic product
as the definitive measure of a country’s success and progress and the assumption that
economic growth equates to quality of life. “Being able to measure people’s quality of life is
130
fundamental when assessing the progress of societies. There is now widespread
acknowledgment that measuring subjective well-being is an essential part of measuring quality
of life alongside other social and economic dimensions.” (OECD, 2013, p.270) Wellbeing
research takes place within economic theory and frequently aims to influence governments to
incorporate wellbeing measures into policy decisions.
Historically and philosophically, the concept of wellbeing has long been considered an intrinsic
good, in contrast to wealth which is only valuable as a means to something else. Perhaps the
first conception of wellbeing-driven policy came from Bhutan in the latter part of the C20th,
when the country introduced its paradigm of gross national happiness (GNH), and eventually
incorporated it into the constitution in 2008. A majority Buddhist country, GNH was an early
approach to sustainable development that tried to index and balance the non-economic
aspects of a good quality of life. Additionally, the Easterlin paradox, formulated by Richard
Easterlin in 1974, found that while wealthier people tend to be more satisfied with their lives
than poorer people, increasing average wealth does not increase average life satisfaction.
Considering that much policy is focused on economic growth, addressing this paradox was
central to the emergence of the science of wellbeing and the need to evidentially measure
wellbeing in order for it to be incorporated into policy decision-making. (OECD, 2013, p.146)
Subjective wellbeing metrics are qualitative data approaches that survey individuals and ask
them to evaluate their own thoughts and feelings about their quality of life. The broad
approaches to measuring subjective wellbeing take the form of an evaluative approach in
which individuals reflect on their overall assessment of their life satisfaction; an experiential
approach in which individuals focus on their emotional wellbeing, both positive and negative,
131
in a shorter timeframe; and a eudaimonic approach in which individuals psychologically reflect
on their own sense of meaning and purpose in life. (OECD, 2013, p.10)
There are various approaches to and applications of measuring subjective wellbeing. The
OECD Guidelines on Measuring Subjective Well-being include metrics for work/life balance,
social connections, environmental quality, and civic engagement and governance. The private
company, Gallup, produces the annual Gallup World Poll, which is the farthest-reaching public
opinion poll globally and produces several world indexes, including on wellbeing. Gallup uses
Cantril’s Ladder of Life Scale, a wellbeing assessment in which individuals are asked to
imagine a ladder with steps numbered from zero at the bottom, representing the best possible
life for them, to 10 at the top, representing the worst possible life for them. They are then asked
questions against this scale: On which step of the ladder would you say you personally feel
you stand at this time? The UN’s World Happiness Report is derived from the Gallup World Poll
data. The largest non-commercial academic social survey is the World Values Survey,
produced every five years and scores life satisfaction. Some individual countries also produce
their own data on wellbeing using similar scales as the Cantril Ladder. For example, the UK’s
Office for National Statistics (ONS) produces its own annual population survey that measures
people’s life satisfaction, sense of worth, happiness, and anxiety. Other examples include the
United Arab Emirates creating the role of Minister of State Happiness and New Zealand
instantiating a wellbeing national budget. This suggests that wellbeing research is slowly
finding its way into mainstream policy thinking.
The most recent approach to measuring wellbeing was published in 2020 called the WELLBY
approach, or Wellbeing-Adjusted-Life Years, developed in the context of the Covid-19
132
pandemic to support decision-making around when to lift national lockdowns. (De Neve, 2020)
Where previous approaches focused on health-related quality of life (QUALYs) or disability-
adjusted life years (DALYs), WELLBYs take into account more factors that contribute to a
person’s overall wellbeing than only their health, including subjective measures such as a
person’s own sense of their life satisfaction, and then connect these factors to longevity data. A
WELLBY is a single metric that calculates the average subjective wellbeing a person
experiences each year summed up over a person’s life expectancy.
The 2021 World Happiness Report switched to using WELLBYs as their method for compiling
the world happiness index of counties. (Helliwell et al., 2021) The aim of WELLBYs as a
measure of subjective wellbeing is to give policy-makers a measure against which to track and
orient decision-making in relation to future generations’ wellbeing. The aim should be to
maximize the number of WELLBYs across all people in the present and future generations. It is
a measure against which regulation, spending, and investing should be calculated and
provides a better way to track global progress over time. The WELLBY approach posits a way
to judge a society according to the extent to which it enables people to experience lives that
are both long and full of subjective wellbeing. WELLBYs place a higher value on human life
than previous approaches, and provide a common currency to compare different policy
outcomes such as alleviating poverty, enhancing education, improving mental health, and
averting deaths.
Using subjective wellbeing research as the unmodelled parameters
To generate a list of unmodelled parameters, or missing data, for the critical software project
Convolutional Domains, I extracted, modified, and speculated on an array of metrics from
133
subjective wellbeing research and publicly available data. I selected metrics that relate to the
context of labor, workers, and the workplace, and specifically to metrics that relate to the
anticipated rise in AI and algorithmic management tools. My unmodelled parameters are
modified from various sources: survey questions such as the OECD illustrative examples of
subjective wellbeing measures; (2013, p.249) wellbeing at work metrics such as the ONS’
Measuring National Wellbeing dataset; (2019) proposed metrics from various labor codes such
as the International Labour Organization’s convention no.98 on the right to collective
bargaining; (1949) and parameters that I have identified from my critical research into machine
learning algorithms used in or potentially being applied in a labor context. This includes
research published by Data & Society, titled Poverty Lawgorithms, which looks at the new
frontier of workers’ rights and digital wage theft. (Gilman, 2020) Labour context applications
that I consider include various forms of workplace surveillance, productivity measures,
automated workplace management systems such as microtask assignment and scheduling,
and wage calculators.
It should be noted that there are criticisms of wellbeing research approaches, and these
include: the issue that most wellbeing measures compare performance only at a national level
and not on a smaller scale; discrepancies have been found between evaluations and
experiences of wellbeing in which a country can be ranked low in the World Happiness Index
and high in Gallup’s Positive Experience Index; arguments against wellbeing being calculated
into a single metric; questions about whether survey questions can really capture people’s
notion of wellbeing; and problems with scale linearity, in which all increases and decreases on
a 0-10 scale are equal, when in reality a person’s increase in wellbeing from a 5-6 may be
much more meaningful than a person’s increase from an 8-9.
134
However, in my research, I draw on the metrics, values, and proposed parameters from
wellbeing research, not from a scientific training in statistics, economics, and welfare policy,
but as a critical design researcher. My aim in drawing on subjective wellbeing research is to
form an interdisciplinary experiment in formulating a set of counterdata parameters as a
speculative ethics to juxtapose different value systems. The unmodelled are presented as
alternative or imagined features of a dataset that take the form of outliers, non-equal, and non-
correlated features. The unmodelled points to what is not in the data more than what is. It
visualizes the gap, the disruption between meaning and function.
Subjective wellbeing parameters are graphically interpolated with the structure of a machine
learning model as a tactic of critical interface design. The application does not model these
parameters into the CNN algorithm; they stand separately, traversing the system with the
possibility of a different vision and value of labor and worker’s lived experiences. They are
missing datasets, as Onuoha conceptualized. The unmodelled serve to bring awareness to the
lack of incentive by people who fund and work in the development of AI technology to engage
with such issues. The unmodelled open-up resistance to simplistic quantifiable models with
messier, contextually grounded lived experiences. The unmodelled begin to conceive of the
different kinds and technicalities of labor needed to address more complex, nuanced, and
sensitively ethical issues in AI in the workplace. The unmodelled point to the advantage of this
absence of these data parameters to those in power. The unmodelled reveal whose priorities
are being modeled and whose are not.
135
I do not necessarily propose these counterdata parameters as a corrective solution to a
productivity algorithm. This project does not advocate solving the problems of labor rights and
relations with a better algorithm. The unmodelled parameters are designed to haunt (Kenway et
al., 2010) the concept of labor-management, productivity, optimization, and convergence,
fixated on profit and economic growth, with concepts of wellbeing and an ethics of subjective
life satisfaction. The unmodelled parameters are designed to haunt a computational model with
features of life and labor that appear to be incommensurable. The work seeks to raise the
question: are such features incommensurable? And also: what could the future of work look like
through the lens of artificial intelligence versus the lens of subjective wellbeing? Is a human-
centered artificial intelligence possible that optimizes for subjective wellbeing, or is such a
proposition a distraction from the harm that widespread artificial intelligence in the workplace
will ultimately cause? How can we conceptualize machine learning, and how can we build
technologies beyond accuracy, efficiency, and homophily?
The unmodelled strategy is to foreground the contestation of interpretability through critical
interface design. The unmodelled propose a critical algorithm strategy towards
counteralgorithms. Loukissas advocates that “researchers and practitioners who wish to
challenge the normative assumptions in existing algorithms must create their own
counterdata.” (2019, p.175) Towards the end of the book, Loukissas refers to the example of
Grassroots Mapping, a community-led lab that uses DIY techniques to create alternative data
and maps to support its civic science and environmental initiatives. He refers to the
counterdata projects of Grassroots Mapping as an example of what could be developed into
“counteralgorithms”. (p.178) All data is local is an important work of critical data studies that,
amongst other topics, explores the concept of counterdata in detail but uses the term
136
counteralgorithm very briefly. I would like to take the term counteralgorithm as the foundation of
my design research strategy into critical algorithm studies and visual design to propose
counteralgorithmic critical software that foreground the contestation of normative assumptions
in algorithms and propose new counteralgorithmic imaginaries.
The dissolution of accountability
When a deep learning model, a complex system, is used for automated decision-making within
sensitive high-stakes contexts, its decisions (which are really predictions that we turn into
decisions) cannot be reverse-engineered, and their logic cannot be traced back through the
algorithm to understand how a decision came to be determined, whether there was a mistake
made in the algorithmic process, or where bias or discrimination unfairly influenced a decision.
The results cannot be appealed or redressed. Neither the programmer who developed the
algorithm nor the system user can extract and make legible the internal decision-making
process.
The system of nodes and weights does provide a structure and language of decision-making,
but it is incommensurable with the meaning rendered by the decision outcome. The generative
effect created by the iterative process of weights being applied to all the neurons in the
network, a process of continual abstraction until the features, or high-level pixel groupings that
the network determines, bear no recognition to the human need for meaning-making. We are
excluded from the process of meaning and can only blindly accept the results, the decisions,
that always bear great meaning in our lived experiences.
137
When a decision-making process is dissolved into a complex generative system, provenance
is lost, causality is forfeited, and consequently, there is a dissolution of accountability for the
decision-making agency. This is a foundational ethical and political conflict at the heart of deep
learning abstraction deployed into sensitive high-stakes domains, such as employment,
healthcare, justice, policing, and welfare. For example, in labor relations, how do we question a
decision, and how do we mediate disputes when human managers neither make the decision
nor understand the decision-making logic. This is a form of algocracy, when agency is diffused
and distributed across algorithmic networks without the means to resist or dispute. When we
think about deep learning algorithms and high-stakes decision-making, we need to consider
the stakes of decisions within this system of dissolved accountability. When accountability for
decisions is transferred algocractically, without political debate, governance, or public
discourse, but tacitly through deep learning’s obfuscated design and authority, we arrive at the
demise of accountability. The agency granted to deep learning algorithms simultaneously
claims authority through technical prowess and negates accountability through the dissolution
of causality.
Moral weights
Giving weights a visual language is to foreground them to politicize them. Weights are one of
the most important parts of the model and the most obfuscated in their influence on the
system. These little multicolored grids are the ‘convolutions’ in convolutional neural networks;
they are how this particular technique in image-based machine learning was innovated, and
achieved higher accuracy that allowed them to be deployed in real-world domains.
Convolutions (also known by the terms kernels, matrices, and filters) are passed over each
node to distill information and draw out different aspects of the image in a process also
138
referred to as feature detection. This involves differentiating between foreground and
background, larger shapes from smaller shapes, shapes within shapes, parts of shapes, and
collections of shapes. The design tactic of visualizing and foregrounding weights speaks to
Galloway and Thacker’s argument that “a network is as much a technical system as it is a
political one, any theory addressing networks will have to entertain a willingness to theorize at
the technical level (…) Today to write theory means to write code.” (2007, p.100) I would like to
invert and go further with this argument: today, we should consider that to write code is also to
write theory, which is to intervene and rewrite social and political discourse, redefine ethics,
instantiate laws, and redefine governance. The real-world ethical implications of deep learning
algorithms mean that, at some level, writing code and defining political dynamics are the same
thing.
As an act of interdisciplinary and creative thinking and abductive speculation, I would like to
critically juxtapose the concept of weights in deep learning systems with the concept of moral
weights in social science research. A deep learning model is the outcome of a generative
process of defining and refining the most optimal set of weights. Linguistically, the term ‘weight’
is both a mathematical construct, a piece of coding syntax, and a loaded social construct that
denotes a vector of value. It is an interdisciplinary term that should be available for greater
critical interpretation in the relationship between coding algorithms and coding social values.
Across all fields, to weight something is to give something more or less importance, to
emphasize or suppress its contribution to a larger system. In practice, it is a way to correct a
known bias, to reproportion representation, and to compensate for influence. Weighting
becomes a cross-disciplinary critical issue for thinking through ethics across deep learning
models and social structural inequity. In this way, weights are another example of a materially
139
affective political language, a way to relate the politics of equity to code syntax. The visualized
weights in Convolutional Domains are the final refinement of a structure of weights that sculpt it
toward a specific outcome when applied to the input video. There is a relationship between the
structuring of weights in deep learning and the structuring of systemic injustices.
In subjective wellbeing research, there is a concept of moral weights, which describes how
researchers morally value different outcomes. For example, criticism of Cantril’s Ladder of Life
Scale, as a measure of wellbeing, points out how it values each step on the ladder equally. It
could be argued that the difference between 3 and 4 on the ladder is more significant in terms
of a person’s lived experience of health, safety, and happiness, than the difference between 8
and 9 on the ladder, in which a foundation of health, safety, and happiness is already
established. “[M]oral weights are highly sensitive to some of the underlying philosophical
assumptions, particularly the views of the badness of death and population ethics.” (Happier
Lives Institute, 2021) From the perspective of global development policy, giving equal weight to
each step on Cantril’s Ladder could imply that an increase in wellbeing in developing countries
is equal to an increase in wellbeing in developed countries. In this way, weights are both a
computational and moral issue. Across social research, activism, and on-the-ground
development work, questions of morality are being grappled with when weights are assigned in
models to different outcomes.
The notion of moral weights can be determined by the personal values of the people who write
code and build models, whether that is an individual situated in Silicon Valley who seeks to
‘disrupt’ and innovate old markets with new technology, or a person situated in a large-scale
public institution who works to implement funding and policy directives. I propose to use the
140
term moral weights from global development wellbeing initiatives to think about the effects of
weights in deep learning. This is a conscious way to provoke and contest the values behind
predictive decision-making and point to the individual and institutional human subjects that
determine how things are weighted. In the field of global development, defining weights is
about consciously defining whether to prioritize the prevention of deaths during childbirth or a
decrease in poverty, for example. In deep learning, we should also be forced to grapple with
the ethics of defining weights: do we prioritize decreasing workplace injuries, increasing a
worker’s sense of dignity at work, or preventing digital wage theft? A future direction of this
research involves developing design strategies to foreground how morality is a question of
weights in deep learning.
141
Section 3
Three tactics of critical design software
Visualizing algorithms
The visualization of algorithms and computational systems is a tactic that supports broader
public access to algorithms, leading to critical engagement with the ethical discourse of
algorithms. Reading and writing computer code is a technical skill that not everyone has and
can be a long process to acquire. However, there is an increasing social desire for the public
to be able to understand and engage in the ethical and political issues that are arising due to
the expanded role that algorithms play in society. Unfortunately, computer source code and
algorithms can be abstract, complex, opaque, and obfuscated. Furthermore, machine learning
code entails an additional layer of obfuscation due to its complexity and the problem of
interpretability, a form of black box. Additionally, algorithms can be black-boxed due to
intellectual property rights.
Algorithms are no longer simply tools to make things faster, easier, or more efficient. They are
becoming part of the fabric of our social and political lives, autonomously bypassing
governance and political debate as the traditional forums of decision-making and social
change. They are increasingly implemented in a more powerful, pervasive, counterintuitive,
influential, and opportunistic way. The ability to comprehend computational systems in their
structure, scale, and functionality is increasingly more difficult, and exponentially so when we
also need to comprehend their ethical role and make choices about how we want to live with
them. Therefore, the need for the visualization of algorithms takes is the starting point from this
142
new context of comprehensibility and power dynamics.
There is a legacy to the work of visualizing computation for social ends. The experiments I
have drawn on to develop my approach to visualizing algorithms begin with turtle graphics, a
graphic programming device implemented into the Logo programming language in the
late-1960s. Turtle graphics was a way to support people to learn abstract mathematical
concepts in a more embodied, exploratory way. Turtle graphics sought to foster access,
literacy, and engagement with advanced ideas in mathematics for people without training in
science. In his book Mindstorms: Children, Computers and Powerful Ideas, the social agenda
behind Logo and turtle graphics were discussed by one of its co-developers, Seymour Papert.
(1993) The visual concept of the turtle was extended by Mitchel Resnick into StarLogo, a visual
programming language developed to visually simulate concepts of decentralization, at a time
when decentralization was also a social and political shift being grappled with. (Resnick, 1995)
StarLogo is an important example of the connection between code literacy and critical and
social issues.
In Aesthetic Computing, Paul Fishwick calls for new ways to bring the history of visual art to
bear on computation. He points out that when science looks toward visual art for references of
visual language and expression, it tends to do so in a reductive, formal way, favoring classical
notions of beauty, simplicity, and elegance. If there is an aesthetics of mathematics, “it involves
concepts such as invariance, symmetry, parsimony, proportion, and harmony.” (p.9, Fishwick,
2006) This denies the full array of richness and diversity of thinking in the visual arts. So far,
when computation is visualized, it fails to take into account the true evolution and innovations in
aesthetics from the Enlightenment to the present. Other examples of how algorithms have been
143
visualized, which I have referenced in the development of my tactic visualizing algorithms,
include projects such as:
◦ Algorithm Visualizer by Jinseo Park (https://algorithm-visualizer.org),
◦ Setosa’s Visual Explanations by Victor Powell and Lewis Lehe (https://setosa.io),
◦ VisuAlgo by Dr Steven Halim (https://visualgo.net),
◦ 100 Days of Visualizing Algorithms by Jamie Charry (https://
100daysofalgorithms.tumblr.com),
◦ Visualizing Algorithms by Mike Bostock (https://bost.ocks.org/mike/algorithms), who
later went on to develop the data visualization languages D3.js and Observable,
◦ R2D3 by Stephanie Yee and Tony Chu (http://www.r2d3.us),
◦ Sorting by Res Yuan (https://renyuan.io/sorting).
While these examples are useful to understand, explain, and debug technical functionality, they
do not serve to visualize algorithms to support the understanding, explanation, and debugging
of social issues in algorithms or even engage with algorithms critically.
Throughout different movements in the history of art, representation has taken many different
visual forms, including: experiments in subversion, multi-perspectivism, new ways of looking,
reflexivity, and pluralism; as well as aesthetic devices for contesting power structures and
definitions; and aesthetic formulations of criticality and self-awareness. Such aesthetic
innovations bring new social, cultural, and political insights. Visual representation in science
has traditionally enforced a visual minimalism in the name of abstraction and theoretical proof,
but as Fishwick argues, “representation need not compromise the goal of abstraction.” (p.255)
144
He proposes that metaphor, analogy, embodiment, and envisioning can catalyze the role of the
visual in addressing ideas in science.
Other people working in the space of computation and critical thinking have experimented with
visualization to support critical engagement, which I have referenced in developing my tactic
of visualizing algorithms. The Anatomy of AI by Kate Crawford and Vladan Joler (https://
anatomyof.ai), studied Amazon’s Echo and described itself as an anatomical map of human
labor, data, and planetary resources. AI Explanations, led by Caroline Sinders and co-
developed by an interdisciplinary team (https://www.ai-explanations.com), is a web-based
project. Videos of people from different backgrounds are presented explaining various facets
of artificial intelligence from a critical perspective, highlighting terms and challenging the solely
technical perspective that explainability usually takes. The project How Do You See Me? by
Heather Dewey-Hagborg (https://deweyhagborg.com/projects/how-do-you-see-me),
experiments with visual forms that reveal the inner workings of a facial recognition algorithm. In
the artwork Myriad (Tulips) (http://annaridler.com/myriad-tulips), the artist Anna Ridler presents
a data set of images on a wall, each with handwritten labels. The work physically visualizes the
human labor and time required to build machine learning systems, which are usually narrated
as fully autonomous but often rely on low-paid crowdsourced labor in developing countries.
The above projects have all been references for artworks and interventions that seek to give
visual representation to critical ideas in artificial intelligence. When developing my tactic of
visualizing algorithms, I have also sought to incorporate critical code studies' methodology. I
start with a critical engagement with source code rather than a critical engagement with the
artifacts, systems, and ethical discourse around code. I then proceed to experiment with a
145
visual form for critical ideas that emerge from engaging with source code directly. In this way, I
argue that my tactic of visualizing algorithms is an addition to and an advancement in critical
code studies. I have previously written about my approach to combining the fields of data
visualization and critical code studies to create, what I have termed, computational
visualization. (Griffiths, 2018) In combination with the digital humanities, art and design
practice can use contemporary visual design methods, including interaction design,
generative design, and game development, to support a critical exegesis of algorithms.
Visualization is a tactic that supports access to algorithms that, in turn, supports critical
engagement in the issues raised by algorithms being deployed in socially sensitive domains.
By graphically revealing structure, animating temporality, interacting with scale, and
analogizing context, visualization supports computation to be less abstract and opaque. It
fosters greater access, comprehension, and exploration of advanced ideas. Visualizing data,
rules, operations, pathways, emergence, and outcomes, but specifically to support critical
thinking. That algorithms have particular shapes, styles, atmospheres, contexts, and meanings
beyond reading lines of code is the outcome of visualization. Visualization of algorithms
supports a broader public to engage with the complex issues that algorithms’ in socially
sensitive spaces bring up and ultimately fosters humanity in algorithms.
In Convolutional Domains, visualization, interaction design, and awareness of generative
aesthetics support a critical engagement with the ethical questions that have arisen due to
CNNs being applied, or speculated to one day being applied, to the sensitive context of labor
relations. In Convolutional Domains, the neural network itself is visualized directly from its
source code, meaning that the source code itself also drives the real-time visualization of the
algorithm during execution and is not a separate animation. This gives a user access to the
146
inside of an algorithm and its operations without the need to understand its source code. This
is one level of opening the black box. The inner structure of the neural network can be seen in
its totality, composed of nodes, filters, and layers. The visualization unfolds the algorithm
spatially, mapping the process from the input image that passes through the network of nodes
and filters to the output prediction labels.
Visualization spatially unwinds the algorithm and also addresses this unwinding at the level of
encapsulation. Encapsulation in object-oriented programming is when variables and functions
are bundled together in separate and individual units, known as objects, that are stored in
libraries. These objects are then used in a script via a class name that calls or refers to the
object without the need to import all the code of the object inside the script. Encapsulation is
an approach in coding that means that when we read source code, some parts of the code
actually point to other functionality outside the script. I address this encapsulation issue in my
tactic of visualizing algorithms by visualizing some encapsulated objects, such as the neural
networks’ weights, that are drawn by going a layer deeper than the initial source code to the
OpenCV libraries that contain the source code for these objects. In this way, source code is
visualized through spatially unfolding across multiple layers of encapsulation. Critically, this
tactic spatializes and maps the algorithm’s inner decision-making structure.
Visualization also gives insight into the approach of ‘feature extraction’ that has been one of the
ways that computer science has attempted to explain or interpret a neural network’s decision-
making. However, I contest this approach because feature extraction, while useful, only
explains how a CNN process an image based on how it looks and not what it means. This
leads to the separation of a neural network’s logic from the meaning it holds in society, and it
147
further entrenches the reductive use of the term ‘interpretability’ in the field of explainable AI. In
Convolutional Domains, the application is designed so that a user can hover over different
nodes to expand them and see visual features in more detail. However, this interaction does
not support a greater understanding of interpretability, but an understanding that
interpretability fails to explain meaning.
Visualization also displays the predicted class that the CNN gives to the image and a series of
the algorithm’s alternative guesses and their accuracy. Alternative guesses, or potential
decisions, are not information that a public audience would usually be granted access to, but
this supports the idea that a machine learning algorithm’s decisions are relational, debatable,
mistaken, and at times ridiculous. Visualizing alternative guesses or potential decisions is a
way to visualize uncertainty and instability in the algorithm.
Visualizing the network’s weights is also an aspect of a machine learning algorithm that is rarely
offered to the public for consideration. When weights are in the form of source code, they are
technically difficult to comprehend; however, when given visual representation, weights can be
understood as image filters that adjust images and draw out different aspects or features.
Weights can be seen as one-to-one pixel relations and manipulations on an image. In this way,
visualization also supports weights as part of a critical discussion by showing their influence,
literal weighting, and biasing of an algorithm’s interpretation of an image. However, again, this
visibility is not offered to support a greater understanding of interpretability but to support an
understanding of interpretability that fails to explain social meaning and is limited to explaining
one-to-one pixel manipulations. By giving visual representation to weights to show their literality
and limitation, and by presenting weights in the critical context of the problem of bias
148
augmentation, I argue that weights in machine learning could be considered a political
language, a means of influencing decision-making.
In combination, visualization and interaction design are a way to create an experience for a
viewer to engage with the inner workings of a CNN actively, parse its structure, see its
encapsulated elements, contend with its limitations for explainable AI, and support critical
questioning. Visualizing algorithms is also an advancement in critical code studies. It takes
insights from humanistically reading source code and providing those critical insights through
visual representation for broader public engagement.
Slow computation
Slow computation is a critical design tactic inspired by studying a genre of video games known
as ‘programming games’ and ideas from systems thinking. Programming games incorporate
aspects of computer code into gameplay, either through a game’s mechanic, narrative, or
aesthetic design. The game SpaceChem, released in 2011 and developed by Zachtronics, is a
game that simulates computational processes and time in interesting ways. SpaceChem
utilizes a temporal scale to switch between slow and emergent processing speeds. The player
can view computations at a one-to-one scale, which provides the ability to process
computation at a human scale of comprehension, including visually identifying where bugs in
the system arise and view computational processing at a very fast scale, leading to an
emergent understanding of the process.
149
I experimented with this temporal scale in previous research when designing interactive
interfaces to visualize algorithms. In my design research projects Visualizing Algorithms, Parts I
and II, I visualized a machine learning generated decision tree. A critical component of these
projects was the ability of an audience to simulate the computational processing time of the
algorithm with a human-scale perception of time. This revealed the flow of data through the
algorithm, individual points of decision-making, and the visualization of errors. This is achieved
by slowing down every process to a simulated trajectory between an initial condition and the
final location achieved after classification. In computational time this process happens
simultaneously, and it is easy to overlook how the algorithm can make incorrect decisions at
individual nodes. Time is mapped to a variable, enabling acceleration, deceleration, or even
reversal of the process, like an animation timeline, allowing the audience to explore the inner
workings of the algorithm visually.
The tactic of slow computation is used here to think about unfolding computation and decision-
making. Where the tactic of visualizing algorithms unfolded functionality spatially, slow
computation supports unfolding functionality over time. I argue that slow computation is a
design tactic for a more humanistic approach to computation. The quality of slowness
regarding computation allows an audience to access a different feeling and type of
engagement with algorithms, enabling them to develop different ways of knowing, assessing,
and critiquing algorithms than through their traditional technical abstractions. The development
of new algorithmic technologies has been approached in recent history through an ethos
defined by the Facebook mantra ‘move fast and break things’. However, now that algorithmic
technologies are entering into uncharted domains that raise significant ethical issues, slowness
is a valuable tactic of resistance to uncriticality and the harnessing of criticality. Slowness can
also support the demystification of technical prowess. In machine learning, an innovation
150
narrative has developed that enigmatizes and glorifies the technically obfuscated qualities
behind artificial intelligence, such as automation, prediction, and decision-making. Slowness
can partly unravel this narrative to reveal more awkward and rudimentary correlations and
labeling systems that are far from a cryptic intelligence. Slow computation supports criticality
by enabling reflective time to be spent with an algorithmic process and the ethical questions it
raises.
Visualizing process
Data visualization emerged as a contemporary practice due to the need to parse vast amounts
of newly created data: big data due to data harvesting on social platforms and access to large
open data sets. Data visualization has burgeoned over the last two decades from a need to
communicate the technical insights produced by data analysis of complex issues to a broader
audience. However, the emerging critique of data from the field of critical data studies also
applies to data visualization. The problems of bias in data, false claims of objectivity, a lack of
provenance, and the narrative of neutrality that has played out across data practices, are all
problems that also exist within data visualization. Many influential data visualization
practitioners have assumed data to be a given, a fact. Edward Tufte, a statistician and early
pioneer in data visualization, does not address data as a subjective or ethical space but
prioritizes the quantitative display of information based on notions of legibility and clarity. (Tufte,
1983) Likewise, Stuart Card, in his influential work on using visualization to think through data
issues, focuses on “the use of computer-supported, interactive, visual representations of
abstract data to amplify cognition.” (p.1, Card, 1999) Such notions of cognition and clarity do
not encompass making legible critical issues in data analysis that relate to power and ethics.
151
According to this version of data visualization, data is the input to the system, a raw material.
The output is an abstracted and insightful representation of the data presented as just as much
of a fact as the original data. This version of data visualization does not grapple with the point
of view of its creator, the point of view of the person or institution who created the data, or the
influence that such a visualization might have in promoting a particular point of view.
The practice of data visualization has many advantages and often comes from noble
intentions. It has attempted to address the difficult to grasp, statistical nature of many social
and political issues, providing a non-technical audience with information that would otherwise
be lost inside technical studies and vast data repositories. Data visualization has developed its
own accessible visual language “revealing patterns and relationships not known or not so
easily deduced without the aid of the visual representation of information.” (p.11, Meirelles,
2013) However, the field of critical data studies is fundamentally changing this foundation of
working with data, including its visualization, challenging many of the problematic assumptions
made about data during its visualization.
In building a methodology that visualizes algorithms, I must reckon with the issues that have
befallen the visualization of data. Visualizing algorithms could entail many of the same
problematic assumptions if it is not inherently connected to critical algorithm studies. The tactic
developed here seeks to frame the visualization of algorithms to support their technical
understanding and, more importantly, to support their critical engagement. In Design For
Information, Meirelles refers to data visualizations as “cognitive artifacts” (p.13, 2013) for the
way in which they support our understanding of information. The term artifact also reinforces
an idea of data visualization as an output or consequence of a process that has taken place in
152
the past. Within data visualization, a designer or analyst begins with a static data set but does
not question how the data came to exist or how the algorithm parses and mathematically
restructures the data. Loukissas’ concept of shifting from thinking about data sets to data
settings (2019) is also pertinent to visualization practices and how the meaning contained in
data’s context can also be stripped or revealed during the visualization process.
When we shift our gaze from data towards the processing of data by algorithms, we are forced
to think about process, the time-based process that takes place in algorithms that is different
from the static and contained nature of input data. In my approach, visualizing algorithms is a
way to think about unfolding computation, both over time and across space. Data and
algorithms are entangled, and the shift towards process and unfolding supports us to engage
with provenance and point of view. This type of study can also speculate on what Mimi Ohuoha
calls missing data sets, a consideration of what does not exist as much as what does, a
consideration of the possibility space of an algorithm, and the paths not taken as much as the
paths that are taken.
In an era of big data, analyzing data, and parsing it, is an interpretative process, and that
interpretation is an algorithmic process. Usually, this algorithmic interpretation remains black-
boxed, and it is presumed that the only relevant thing to show is the results of data analysis
and visualization. This is where tactics to visualize algorithms and computational processes
take their departure. Visualizing computational process is part of the design research
framework to develop a critical inquiry into algorithms. It supports a broader audience to gain
access and engage with algorithms, which would otherwise require technical knowledge. A
key component is a focus on process, both temporal and spatial, in which data is parsed,
153
forked, and on which decisions are executed. It is about thinking through and visualizing how
the computational process works in real-time to expose or interpret a cause, pattern, or
resulting artifact.
Computational visualization seeks to understand how the algorithm executes and why it
produces its results, whether it is a conventional sorting algorithm or a machine learning
algorithm with significant social and political implications. In this way, it is possible to access
and visualize the processes that underlie the computational systems that increasingly drive key
functions in our society. Temporally, data tends to be fixed and retrospective; we can think of
data visualization as addressing something that has taken place in the past, in the sense that a
data set is an account of a situation that has now ended, at least in terms of the boundaries of
the data set. On the other hand, computational visualization can be both in-process and a
projection of the future, in the sense that it is identifying a process that is currently in use to
understand how it unfolds. When we visualize an algorithmic process, we can see the decision
model that has been created, which invites the question of whether that model is the most
suitable or what changes could be made to it, fostering further engagement. In data
visualization, there is no data model or decision model, and therefore there is no alternative to
understanding how we arrived at a particular visual solution. Computational visualization
identifies a model, then visualizes it to share and propagate that knowledge and cultivate
greater computational literacy.
Practically, deep learning algorithms present an important challenge to the visualization of
algorithms in that visualization cannot support a linear reverse-engineering of decision-making.
As complex systems that generatively produce their own rules, they resist traditional
154
explainability. Visualizing process is problematized. My term, computational unfolding, was
conceived to recognize the lack of linear relationships and point to how the unfolding can be a
broader approach encompassing a more non-linear multilayered intersecting networked
process. What form can it take if unfolding cannot be a linear, one-to-one relational process?
Theoretically, the issue of generative rules and non-linear relationships has been
conceptualized by the scientist Stephen Wolfram as computational irreducibility. (p.742, 2002)
Computationally irreducible phenomena and problems cannot be deconstructed in a traditional
way that seeks to understand higher-order meaning from lower-order constituent parts. In A
New Kind of Science, his ten-year study on complexity, Wolfram argues that this approach
represents a paradigm shift in science, away from a Newtonian model of mathematics and
symbolic logic and towards a computational model of complexity. (2002) The principle of
computational irreducibility says that the only way to determine the answer to a computationally
irreducible question is to perform, or simulate the computation, in a time-based way. Wolfram’s
concept points us toward a philosophy for why visualizing real-time computational processes is
meaningful. Meaning takes place through the process; the process itself plays a role in
determining the outcome; therefore, building tactics for different ways to unfold the process is
a useful skill in critical computational work. In this way, we visualize the process not just for
technical legibility but to unfold an algorithm critically and philosophically.
Finally, computational unfolding, following Wolfram’s concept of computational irreducibility,
could allow for a form of runtime critique. Galloway and Thacker have argued that open source
is insufficient, “something like an ‘open runtime’ movement might also be required, in which the
dialectic of obscurantism and transparency, a longtime stalwart in aesthetics and philosophy, is
155
interrogated as a central problem, if not the central problem, of software.” (p.292, 2007)
Likewise, if we think about the practice of debugging in code, a runtime environment is also
necessary; code cannot be debugged by reading static source code in separation from its
execution. Debugging is a runtime, process-based practice. In this way, visualizing algorithms
and computational unfolding could be a form of runtime critique, an advancement on the
method of critical code studies, which has tended to focus on source code separately to
execution, and sets up a different approach to working with algorithms as mediums of
philosophical and ethical meaning.
156
Conclusion
The work presented in this dissertation calls for the development of a counteralgorithmic
practice in machine learning to enable the contestation of interpretability and the production of
new counteralgorithmic imaginaries. I have used a combination of approaches, including
critical algorithm studies, critical code studies, and visual design. I have introduced several
new terms into the research: computational unfolding, the problem of bias augmentation, the
contestation of interpretability, and counteralgorithms. I have created a framework grounded in
an abductive hybrid practice-theory approach to research. The framework produces critical
software tools to generate new visual-critical knowledge that challenges normative
assumptions in algorithms and provokes new models of value in algorithms. Visualization and
interaction design are tactics to render technically obfuscated algorithms accessible to
broader publics for philosophical, social, and political critique.
I posit that the interpretability problem should not be framed so reductively by computer
science. Instead, it should be expanded in the way I describe to better reconcile a truer
version of complexity that encompasses deep learning networks and socio-political networks.
The issue of context is a rich and nuanced space of possibility for future algorithm design, and
when context is stripped and agnosticism is prized, meaning is lost. Additionally, this research
shows that the perspectives and practices of artists, activists, and on-the-ground stakeholders
are key contributions to the development of critical and creative counteralgorithmic
imaginaries. Future equitable algorithm design can only take place by centering those voices
and bottom-up practices.
157
Future directions for this research include developing speculative and experimental deep
learning models trained on subjective wellbeing data to more tangibly provoke how alternative
values can be modeled, however imperfectly. The current status of the practice-based
component of this research is a single-screen software application. I intend to physically
expand the software into a spatial and embodied multi-screen-based installation experience to
explore how a broader public engages with the ideas when presented in a different medium. I
also intend to work further with the input image component of the deep learning network in the
form of ethnographic filmmaking. I want to use a cinematic framing to explore how the
perspectives and lived experiences of people subjected to autonomous decision-making
experience such technology.
158
Bibliography
AI Now Institute. 2016a. “AI Now 2016 Report: The Social and Economic Implications of
Artificial Intelligence Technologies in the Near-Term.” New York, NY.
AI Now Institute. 2016b. “AI Now 2016 Primers, Workshop Primer: Labor & AI.” New York, NY.
Allman, Eric. 2011. “The Robustness Principle Reconsidered.” Communications of the ACM 54
(8): 40–45.
Alvarez-Melis, David, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, and Jennifer Vaughan
Wortman. 2021. “From Human Explanation to Model Interpretability: A Framework Based on
Weight of Evidence.” In 9th AAAI Conference on Human Computation and Crowdsourcing
(HCOMP 2021).
Amoore, Louise. 2020. “Why ‘Ditch the Algorithm’ Is the Future of Political Protest.” The
Guardian, August 19, 2020. https://www.theguardian.com/commentisfree/2020/aug/19/ditch-
the-algorithm-generation-students-a-levels-politics.
Ananny, Mike. 2016. “Toward an Ethics of Algorithms: Convening, Observation, Probability, and
Timeliness.” Science, Technology, & Human Values 41 (1). https://doi.org/
10.1177/0162243915606523.
Ananny, Mike, and Kate Crawford. 2018. “Seeing without Knowing: Limitations of the
Transparency Ideal and Its Application to Algorithmic Accountability.” New Media and Society
20 (3). https://doi.org/10.1177/1461444816676645.
Aneesh, A. 2006. Virtual Migration: The Programming of Globalization. Durham, NC: Duke
University Press.
Angwin, Julia, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. “Machine Bias.”
ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-
in-criminal-sentencing.
Benjamin, Ruha. 2019a. Captivating Technology: Race, Carceral Technoscience, and
Liberatory Imagination in Everyday Life. Durham, NC: Duke University Press.
Benjamin, Ruha. 2019b. Race After Technology: Abolitionist Tools for the New Jim Code.
Cambridge, UK: Polity Press.
Borenstein, Greg. 2014. “OpenCV for Processing.” Version 0.5.2. [Computer software] https://
github.com/atduskgreg/opencv-processing.
Brown Jr, James, J. 2015. Ethical Programs: Hospitality and the Rhetorics of Software. Ann
Arbor, Michigan: University of Michigan Press.
Buolamwini, Joy, and Timnit Gebru. 2018. “Gender Shades: Intersectional Accuracy Disparities
in Commercial Gender Classification.” In Proceedings of Machine Learning Research, 81:1–15.
159
Card, Stuart. K, Jock D. Mackinlay, and Ben Schneiderman. 1999. Readings in Information
Visualization: Using Vision to Think. United Kingdom: Elsevier Science.
Chun, Wendy Hui Kyong. 2008. “On ‘Sourcery,’ or Code as Fetish.” Configurations 16 (3):
299-324,427.
Crawford, Kate. 2021. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial
Intelligence. New Haven, CT: Yale University Press.
Crawford, Kate, and Trevor Paglen. 2019. “Excavating AI: The Politics of Images in Machine
Learning Training Sets.” 2019.
Cross, Nigel. 1982. “Designerly Ways of Knowing.” Design Studies 3 (4): 221–27. https://
doi.org/10.1016/0142-694X(82)90040-0.
Crouch, Christopher, and Jane Pearce. 2012. Doing Research in Design. London, UK:
Bloomsbury Academic.
D’Amour, Alexander, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex
Beutel, Christina Chen, et al. 2020. “Underspecification Presents Challenges for Credibility in
Modern Machine Learning.” http://arxiv.org/abs/2011.03395.
Danaher, John. 2016. “The Threat of Algocracy: Reality, Resistance and Accommodation.”
Philosophy & Technology 29 (3): 245–68. https://doi.org/10.1007/s13347-015-0211-1.
Dastin, Jeffrey. 2018. “Amazon Scraps Secret AI Recruiting Tool That Showed Bias against
Women.” Reuters, October 10, 2018. https://www.reuters.com/article/us-amazon-com-jobs-
automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-
idUSKCN1MK08G.
De Neve, Jan-Emmanuel, Andrew E. Clark, Christian Krekel, and Gus O’Donnell. 2020. “Taking
a Wellbeing Years Approach to Policy Choice.” BMJ 371. https://doi.org/10.1136/bmj.m3853.
D’Ignazio, Catherine, and Lauren F. Klein. 2020. Data Feminism. Cambridge, MA: MIT Press.
D’Ignazio, Catherine, Helena Suárez Val, Silvana Fumega, Harini Suresh, Isadora Cruxên,
Wonyoung So, Ángeles Martinez, and Mariel García-Montes. 2020. “Feminicide & Machine
Learning: Detecting Gender-Based Violence to Strengthen Civil Sector Activism.” In
Mechanism Design for Social Good (MD4SG).
Dennett, Daniel. 2003. Freedom Evolves. New York: Viking.
Eubanks, Virginia. 2017. Automating Inequality: How High-Tech Tools Profile, Police, and
Punish the Poor. New York, NY: St. Martin’s Press.
Fishwick, Paul A. 2006. Aesthetic Computing. Cambridge, MA: MIT Press.
Fry, Ben, and Casey Reas. 2001. “Processing.” Version 3.5.3. [Computer software] https://
processing.org/.
160
Galloway, Alexander R. 2004. Protocol: How Control Exists after Decentralization. Cambridge,
MA: MIT Press.
Galloway, Alexander R., and Eugene Thacker. 2007. The Exploit: A Theory of Networks.
Minneapolis, Minnesota: University of Minnesota Press.
Galloway, Alexander. 2010. “Networks.” In Critical Terms for Media Studies, edited by W. J. T.
Mitchell and Mark Hansen, 280–96. Chicagco, IL: University of Chicago Press.
Galloway, Alexander R. 2016. “Versity.” 2016. http://cultureandcommunication.org/galloway/
versity#more-983.
Gilbreth, Frank B., and Lillian M. Gilbreth. (1910-1924) “Original Films of Frank B. Gilbreth.”
Internet Archive. https://archive.org/details/OriginalFilm
Gilman, Michele. 2020. “Poverty Lawgorithms: A Poverty Lawyer’s Guide to Fighting Automated
Decision-Making Harms on Low-Income Communities.” Data & Society
Gilpin, Leilani H., David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.
2018. “Explaining Explanations: An Overview of Interpretability of Machine Learning.” In 2018
IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA).
Gitelman, Lisa. 2013. “Raw Data” Is an Oxymoron. Cambridge, MA: MIT Press.
Golumbia, David. 2009. The Cultural Logic of Computation. Cambridge, MA: Harvard
University Press.
Gray, Mary L., and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building
a New Global Underclass. Boston, MA: Houghton Mifflin Harcourt.
Greene, Jay. 2020. “Microsoft Won’t Sell Police Its Facial-Recognition Technology, Following
Similar Moves by Amazon and IBM.” Washington Post, June 11, 2020.
Griffiths, Catherine. 2018. “Visual Tactics Toward an Ethical Debugging.” Digital Culture &
Society 4 (1): 217–26. https://doi.org/10.14361/dcs-2018-0113.
Gunning, David, and David Aha. 2019. “DARPA’s Explainable Artificial Intelligence (XAI)
Program.” AI Magazine 40 (2): 44–58. https://doi.org/10.1609/aimag.v40i2.2850.
Hall, Gary. 2013. “Toward a Postdigital Humanities: Cultural Analytics and the Computational
Turn to Data-Driven Scholarship.” American Literature 85 (4). https://doi.org/
10.1215/00029831-2367337.
Hao, Karen. 2019. “This Is How AI Bias Really Happens - and Why It’s so Hard to Fix.” MIT
Technology Review, 2019. https://www.technologyreview.com/2019/02/04/137602/this-is-how-
ai-bias-really-happensand-why-its-so-hard-to-fix/.
161
Happier Lives Institute. 2021. “2020 Annual Review.” https://www.happierlivesinstitute.org/
annual_review_2020.html.
Haraway, Donna. 1988. “Situated Knowledges: The Science Question in Feminism and the
Privilege of Partial Perspective.” Feminist Studies 14 (3): 575–99.
Haraway, Donna J. 2016. Staying with the Trouble: Making Kin in the Chthulucene. Durham,
NC: Duke University Press.
Helliwell, John F., Richard Layard, Jeffrey D. Sachs, Jan-Emmanuel De Neve, eds. 2021.
“World Happiness Report 2021.” New York: Sustainable Development Solutions Network.
Hern, Alex. 2020. “Ofqual’s A-Level Algorithm: Why Did It Fail to Make the Grade?” The
Guardian, August 21, 2020. https://www.theguardian.com/education/2020/aug/21/ofqual-
exams-algorithm-why-did-it-fail-make-grade-a-levels.
International Labour Organisation. 1949. Right to Organise and Collective Bargaining
Convention (No. 98). shorturl.at/fvFKM
Irani, Lilly. 2015. “Justice for ‘Data Janitors.’” Public Books, 2015. https://www.publicbooks.org/
justice-for-data-janitors/.
Jacobson, Adam. 2019. “Hong Kong Protesters Use Laser Pointers to Deter Police, Scramble
Facial Recognition.” CBC News, August 11, 2019. https://www.cbc.ca/news/world/hong-kong-
protest-lasers-facial-recognition-technology-1.5240651.
Jameson, Fredric. 1992. Postmodernism, or, The Cultural Logic of Late Capitalism. Durham,
NC: Duke University Press.
Karsten, Luchien. 1996. “Writing and the Advent of Scientific Management: The Case of Time
and Motion Studies.” Scandinavian Journal of Management 12 (1): 41–55.
Kellogg, Katherine C., Melissa A. Valentine, and Angele Christin. 2020. “Algorithms at Work:
The New Contested Terrain of Control.” Academy of Management Annals 14 (1). https://doi.org/
10.5465/annals.2018.0174.
Kenway, Jane, Elizabeth Bullen, Johannah Fahey, and Simon Robb. 2010. Haunting the
Knowledge Economy. London, UK: Routledge.
Knight, Will. 2016. “AI’s Unspoken Problem.” MIT Technology Review 119 (5).
Lanier, Jaron. 2013. Who Owns the Future? New York, NY: Simon & Schuster.
LeCun, Y, B Boser, J. S. Denker, D Henderson, R. E. Howard, W Hubbard, and L. D. Jackel.
1989. “Backpropagation Applied to Handwritten Zip Code Recognition.” Neural Computation 1
(4): 541–51. https://doi.org/10.1162/neco.1989.1.4.541.
162
LeCun, Yann, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. “Gradient-Based
Learning Applied to Document Recognition.” In Proceedings of the IEEE, 2278–2324. https://
doi.org/10.1109/5.726791.
Lessig, Lawrence. 1999. Code and Other Laws of Cyberspace. New York, NY: Basic Books.
Loukissas, Yanni A. 2019. All Data Are Local: Thinking Critically in a Data-Driven World.
Cambridge, MA: MIT Press.
Lowry, Stella, and Gordon MacPherson. 1988. “A Blot on the Profession.” British Medical
Journal 296 (6623): 657–58. https://doi.org/10.1136/bmj.296.6623.657.
Luo, Xiaochun, Heng Li, Dongping Cao, Yantao Yu, Xincong Yang, and Ting Huang. 2018.
“Towards Efficient and Objective Work Sampling: Recognizing Workers’ Activities in Site
Surveillance Videos with Two-Stream Convolutional Networks.” Automation in Construction 94:
360–70. https://doi.org/10.1016/j.autcon.2018.07.011.
Marino, Mark C. 2020. Critical Code Studies. Cambridge, MA: MIT Press.
Marino, Mark C. 2006. “Critical Code Studies.” Electronic Book Review, no. December 4, 2006.
https://electronicbookreview.com/essay/critical-code-studies/.
Marino, Mark C. 2018. “Race and Black Codes.” Critical Code Studies Working Group. 2018.
McIlwain, Charlton. 2020. “Of Course Technology Perpetuates Racism. It Was Designed That
Way.” MIT Technology Review, June 3, 2020. https://www.technologyreview.com/
2020/06/03/1002589/technology-perpetuates-racism-by-design-simulmatics-charlton-
mcilwain/.
McPherson, Tara. 2018. Feminist in a Software Lab: Difference and Design. Cambridge, MA:
Harvard University Press.
Meirelles, Isabel. 2013. Design for Information. Beverly, MA: Rockport Publishers.
Metz, Cade. 2020. “Meet GPT-3. It Has Learned to Code (and Blog and Argue).” New York
Times, November 24, 2020. https://www.nytimes.com/2020/11/24/science/artificial-intelligence-
ai-gpt3.html.
Miller, Tim. “Explanation in artificial intelligence: Insights from the social sciences.” arXiv
Preprint arXiv:1706.07269. (2017)
Mitchell, Melanie. 2009. Complexity: A Guided Tour. Oxford, UK: Oxford University Press.
Mitchell, Melanie. 2019. Artificial Intelligence: A Guide for Thinking Humans. New York, NY:
Picador.
Montfort, Nick, Patsy Baudoin, John Bell, Ian Bogost, Jeremy Douglass, Mark C. Marino,
Michael Mateas, Casey Reas, Sample Mark, and Noah Vawter. 2012. 10 PRINT
CHR$(205.5+RND(1));:GOTO 10. Cambridge, MA: MIT Press.
163
Myers West, Sarah. 2020. “Redistribution and Rekognition: A Feminist Critique of Algorithmic
Fairness.” Catalyst: Feminism, Theory, Technoscience 2 (6): 1–24.
Nobel, Safiya Umoja. 2018. Algorithms of Oppression: How Search Engines Reinforce Racism.
New York, NY: NYU Press.
OEDC. 2013. “OECD Guidelines on Measuring Subjective Well-Being.” https://doi.org/
10.1787/9789264191655-en.
Office for National Statistics. 2019. “Measuring National Wellbeing, October 2019 Release.”
https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/datasets/
headlineestimatesofpersonalwellbeing.
O’Neil, Cathy. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and
Threatens Democracy. New York, NY: Crown.
Onuoha, Mimi. 2017. “Notes on Algorithmic Violence.” GitHub. 2017. https://github.com/
MimiOnuoha/On-Algorithmic-Violence.
Onuoha, Mimi. 2018. “On Missing Data Sets.” GitHub. 2018. https://github.com/MimiOnuoha/
missing-datasets
Page, Scott E. 2010. Diversity and Complexity. Princeton, NJ: Princeton University Press.
Papert, Seymour. 1993. Mindstorms: Children, Computers and Powerful Ideas. New York, NY:
Basic Books.
Peirce, Charles Sanders. 1934. “Lectures on Pragmatism.” In Collected Papers of Charles
Sanders Peirce, Vols 1-6, 105–7.
Polanyi, Michael. 1966. The Tacit Dimension. Chicago, IL: University of Chicago Press.
Postel, Jon. 1980a. “Internet Protocol.” RFC 760. 1980. https://web.archive.org/web/
20200620191650/https://tools.ietf.org/html/rfc760.
Postel, Jon. 1980b. “Transmission Control Protocol.” RFC 761. 1980. https://web.archive.org/
web/20200620191657/https://tools.ietf.org/html/rfc761.
Resnick, Mitchel. 1995. Turtles, Termites and Traffic Jams: Explorations in Massively Parallel
Microworlds. Cambridge, MA: MIT Press.
Rettberg, Jill Walker. 2020. “Situated Data Analysis: A New Method for Analysing Encoded
Power Relationships in Social Media Platforms and Apps.” Humanities & Social Sciences
Communications 7 (5).
Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2016. “‘Why Should I Trust You?’
Explaining the Predictions of Any Classifier.” Proceedings of the ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining, 1135–44. https://doi.org/
10.1145/2939672.2939778.
164
Russel, Stuart J., and Peter Norvig. 2021. Artificial Intelligence: A Modern Approach, 4th
Edition. Hoboken, NJ: Pearson.
Sack, Warren. 2019. The Software Arts. Cambridge, MA: MIT Press.
Salguero, Maria. 2020. Google Maps. “Mapa Nacional de Feminicidios en México reportados
por la prensa.” Accessed November 1, 2021. shorturl.at/agjtE
Sample, Ian. 2019. “Maths and Tech Specialists Need Hippocratic Oath, Says Academic.” The
Guardian, August 16, 2019. https://www.theguardian.com/science/2019/aug/16/
mathematicians-need-doctor-style-hippocratic-oath-says-academic-hannah-fry.
Sandvig, Christian, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2016. “When the
Algorithm Itself Is a Racist: Diagnosing Ethical Harm in the Basic Components of Software.”
International Journal of Communication 10: 4972–90.
Schneier, Bruce. 2000. Secrets and Lies: Digital Security in a Networked World. Indianapolis,
IN: John Wiley & Sons.
Shvartzberg, Manuel. 2015. “Play Turtle, Do It Yourself. Flocks, Swarms, Schools, and the
Architectural-Political Imaginary.” In The Politics of Parametricism: Digital Technologies in
Architecture, edited by Matthew Poole and Manuel Shvartzberg, 94–122. London: Bloomsbury.
Thorp, Jer. 2017. “You Say Data, I Say System.” Hacker Noon, July 13, 2017.
Tufte, Edward. R. 1983. The Visual Display of Quantitative Information. Cheshire, CT: Graphics
Press.
Wolfram, Stephen. 2002. A New Kind of Science. Champaign, IL: Wolfram Media.
Zuboff, Shoshana. 2019. Surveillance Capitalism: The Fight for a Human Future at the Frontier
of Power. New York, NY: Public Affairs.

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Coding.Care: guidebooks for intersectional AI

PDF

Building straggler-resilient and private machine learning systems in the cloud

PDF

Deep learning techniques for supervised pedestrian detection and critically-supervised object detection

PDF

Exploring complexity reduction in deep learning

PDF

Infrastructures of the imagination: building new worlds in media, art, & design

PDF

Video camera technology in the digital age: Industry standards and the culture of videography

PDF

I can't afford the luxury of just being an actress: the politicized work of African American actresses on television

PDF

Viral selves: Cellphones, selfies and the self-fashioning subject in contemporary India

PDF

GM TV: sports television and the managerial turn

PDF

The body and its "thumbnails": the work of the image in mobile-imaging

PDF

Machine learning techniques for perceptual quality enhancement and semantic image segmentation

PDF

Simulation and machine learning at exascale

PDF

Emergent media technologies and the production of new urban spaces

PDF

Revenge of the fanboy: convergence culture and the politics of incorporation

PDF

Object classification based on neural-network-inspired image transforms

PDF

More lives to live?: archiving and repurposing the daytime soap opera

PDF

Shadows of stardom: Latina actresses in the 1930's Hollywood produced Spanish language films

PDF

Designing data-effective machine learning pipeline in application to physics and material science

PDF

Machine-learning approaches for modeling of complex materials and media

PDF

Deep convolutional neural networks for remote sensing investigation of looting of the archeological site of Al-Lisht, Egypt

Asset Metadata

Creator Griffiths, Catherine (author)

Core Title Toward counteralgorithms: the contestation of interpretability in machine learning

School School of Cinematic Arts

Degree Doctor of Philosophy

Degree Program Cinematic Arts (Media Arts and Practice)

Degree Conferral Date 2022-05

Publication Date 05/27/2022

Defense Date 05/09/2022

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag activity recognition,arts research,convolutional neural networks,counteralgorithms,critical algorithm studies,critical code studies,critical software,ethics of algorithms,interpretability problem,machine learning,OAI-PMH Harvest,politics of code,politics of labor,unmodelled,Visualization

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Kratky, Andreas (committee chair), Anderson, Steve (committee member), Marino, Mark (committee member), McPherson, Tara (committee member)

Creator Email catherine-griffiths@hotmail.com,griffitc@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC111339152

Unique identifier UC111339152

Document Type Dissertation

Format application/pdf (imt)

Rights Griffiths, Catherine

Internet Media Type application/pdf

Type texts

Source 20220608-usctheses-batch-945 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu