Labor Optics

Machine learning-driven systems are being speedily developed for deployment into new domains of society that have not previously been subjected to generative decision-making processes. Workplaces are socially-sensitive domains in which monitoring software, people’s bodies, and labor relations are in constant agitation. When surveillance infrastructures of networked cameras and sensors feed into machine learning algorithms, new forms of worker tracking, analysis, and modeling become possible on an imperceptible scale. This can lead to what is termed “autonomous decision-making,” or when evaluation processes and decisions are made by generative algorithms without the direct involvement of human managers.1

Constructions sites have become an important space in which to demo and test the role of machine learning in the workplace. This is evident in the development of “smart construction labs” in universities across the world, from the University of Cambridge to Hong Kong Polytechnic University, NYU Abu Dhabi, and Stanford University,2 as well as various privatized start-ups that have emerged from academia, such as Nextera Robotics and Oculai.3 One particular technology that has recently been the focus of many labs and start-ups is based on human activity recognition (HAR) algorithms, which combine video camera footage of construction sites, captured by surveillance cameras, with computer vision analysis to track workers’ bodies. Similar to the more widely known object recognition, in which image datasets are labeled and used to train a model to recognize objects in images, HAR operates on video footage to recognize the human body in motion, analyze its precise behaviors, and predict the type of activity being performed by the body.4 Many early HAR proof of concepts were developed in a sports science context, in which a small dataset was created of clean video footage showing people doing various sports activities, from which the shape and movements of the body could be clearly defined to extract a simplistic animated skeleton model. To achieve this, several algorithmic processes are combined, including computer vision analysis (contour detection, image segmentation, optical flow) and machine learning methods (pose estimation, skeleton recognition).

The context of sports science to develop a proof of concept for HAR makes sense since it already conducts a large amount of monitoring of athletes’ bodies for training purposes, and the athletes involved consent to this as part of their development. However, HAR algorithms have recently been applied and developed within the context of smart construction, as what is now called worker activity recognition (WAR) algorithms, which track, analyze, and label the precise movements of workers. This can be an analysis of hands operating a machine, or the time it takes a worker to carry equipment from one area to another. It can include identifying which task a worker is performing, or how long a worker’s body is standing still versus moving. Many WAR prototypes are also trained to predict how productive each worker’s body is over time, by evaluating the task in relation to performance metrics. In this way, WAR algorithms begin to perform a disembodied managerial agency over worker’s real bodies, even replacing human managers. Construction sites are one of the primary use cases for prototyping WAR algorithms.5

The Testing Grounds for the Future of Work

The labor policy discourse on the “future of work” has long examined the consequences of technology on employment, but has recently found renewed gravity in trying to grapple with how digitization, smart technologies, artificial intelligence, and robotics will transform the employment landscape, reshaping job roles, income levels, the quality and dignity of work, and the design of workplace environments.6 This future is often presented optimistically, focusing on the innovative transformation of work while decentering the ethical risks and associated social and political disenfranchisement of workers. Within this context, WAR algorithms appear as part of the continual advancement of technology in the workplace towards automation and optimization. They share many similarities with historical methods of scientific management of workers, dating back to the early twentieth century time-motion studies of Frank and Lillian Gilbreth, who used analog film, physical clocks, and gridded surfaces to analyze workers’ bodies, measure productivity, and redesign work processes to maximize productivity and implement strict managerial control.7

The process of vectorizing and digitizing workers’ bodies and their labor into increasingly infinitesimal points means that workers can only be valued for their productivity metrics.8 But WAR penetrates not only into bodies, but also into people’s mental health, dignity, and labor rights. And they do so covertly, behind the closed doors of proprietary, black boxed, complex systems. The development of WAR represents an apparent continuation of the norm of twentieth-century exploitative labor practices, but it actually has the potential go much further. The use of AI in the workplace risks reneging on many hard-fought-for labor rights, like the right to fair and equal pay, protection against unfair termination and the right to know why you have been fired, and the right to know if you are being recorded by a camera or audio device.9 At the same time, it abjectly fails to imagine anything better for the future of workers, such as creating work that is more fulfilling, imbuing work with greater dignity, or addressing the problem of increasing wealth inequality.

An alternative way to investigate the future of work is to study the testing grounds of technology rather than the technology itself. Noortje Marres and David Stark point out that while technology is tested through specific prototypes and testing environments, “it is the very fabric of the social that is being put to the test. To understand how testing and the social relate today, we must investigate how testing operates on social life, through the modification of its settings.”10 WAR can therefore be understood not only as a discrete algorithm or function of a technology that is being tested to integrate it into a system, but a testing ground of labor politics in the so-called smart workplace. Prototypes and applications of WAR algorithms are testing workers, unions, and the public to ascertain how much we will tolerate, what we will accommodate and consent to, how we will adapt, where we will say no, and how loudly. Worker activity recognition software is still in an early stage of development. As prototypes, they are tools that are not yet fully implemented in the world, in real workplaces. Thus, there is still space to question their ontology, critique the code, shape the ethics, and reconsider the futures of work they engender.

Computational Objects of Politics

Many WAR algorithms developed in smart construction labs are in part publicly accessible and made available through peer-review publications and open-source repositories, such as Github. One early and pertinent WAR prototype, developed by a team at Hong Kong Polytechnic University, “allows managers to quantify precisely and benchmark labor productivity.”11 To do this, the researchers defined an activity taxonomy with sixteen classes of activities, including “Measuring,” “Placing formwork,” and “Transporting rebar.” All activity classes are evaluated on a scale from “productive,” “semi-productive,” and “non-productive.” One activity class is named “Resting,” which is specified as “standing still, standing and drinking water, or standing and wiping perspiration,” and is evaluated as non-productive.

In order to develop this prototype, the researchers created a new video dataset to train and test their algorithm from surveillance camera footage of a construction site. They uploaded the dataset to Github, so when combined with the activity taxonomy, it provides insight into the design of the model. That the “Resting” class, which monitors when workers wipe sweat from their brow, is calculated as unproductive time, demonstrates how the algorithm constructs a treatment of the worker’s body in a way that denies any ethics of care. This level of micro-analysis goes further than existing forms of workplace management and notions of optimization. In this way, machine learning models in the workplace can be precisely designed to encode practices that will either advance or regress human ethics, social values, and labor rights. Despite the fact that this politics is materially programmed into the algorithm, it can often go unacknowledged by the developers and funders of such technology. 

Moving away from research and into practice, there are examples of WAR algorithms being taken out of public deliberation and made proprietary, thus blackboxing their inner workings. The company Everguard uses a combination of computer vision, machine learning, and wearable sensors “to provide a holistic approach to keeping workers safe, healthy, and productive.”12 They have trademarked the concept of “Worker-Centric AI,” which is described as a real-time platform that improves safety and productivity. But is safety merely a more agreeable visage for a software that masks the enforcement of productivity as its primary function? What, after all, is meant by health and safety? Does the software work to protect against exhaustion, dehydration, and the mental strain caused by constant micro-surveillance? Innovative tools that appear to foreground health and safety narratives today can be easily repurposed toward other kinds of machine learning–driven predictions tomorrow, including those that could covertly impede workers’ privacy and rights.

As WAR algorithms become packaged into comprehensive worker management software systems, we can speculate on the emergence of a new category of AI-driven platforms designed to autonomously oversee, track, coordinate, and drive productivity across workplaces. Ethical and legal concerns arise around the use of autonomous systems, which have the potential to integrate the tracking of workers with productivity evaluations to be used in job appraisals and lead to employment decisions. Beyond concerns over data privacy, this and related potentials raise questions of: transparency over which data metrics are being used and whether workers know the true implications; how work that falls outside computable metrics is quantified; the mental health impacts of working under constant micro-surveillance and the culture of fear, stress, and low morale that can easily be normalized through such systems; and how the depersonalization of the human body into trackable data points entails a consequent loss of ways to value the humane, dignified, and meaningful aspects of work.13

Through the lens of an AI-driven algocracy, WAR and autonomous management software risk destabilizing worker rights and dignified work while asserting authority over labor relations.14 Such an outcome could happen because the nature of machine learning algorithms, as complex generative systems, means that the traceability of prediction-based decision-making is dissolved. The outcome of their decisions cannot be reverse engineered, which removes the possibility for them to be questioned, reevaluated, contested, or rejected as mistakes. In this regard, there is a risk that the authority of machine learning-driven autonomous decision-making is given precedence over what was previously human decision-making. If WAR is implemented at scale, the right to question and appeal a decision about a worker’s job performance is lost. And when decision-making is obscured, the accountability for those decisions is also dissolved.

Reflexive Software

By defining a series of tactics that allow the decision-making process of algorithms as well as the legitimacy of their claims to be scrutinized and contested, I have developed a methodology to study WAR algorithms through what I call “reflexive software.” As a design methodology, reflexive software involves developing custom software to explore learning algorithms and visualize not only linear technical explanations of how autonomous decision-making systems function, but also their social, ethical, and political encodings.15 Reflexive software reframes a key research question in machine learning: the “interpretability problem,” which refers to the difficulty of precisely tracing the path from input to output through the internal network of a generative algorithm, in order to determine the cause or reasoning behind an algorithmic decision.16 This problem has ethical stakes when the output is a decision that is made in a socially sensitive context, such as the decision to fire a worker. Instead of asking how to solve the problem of interpretability in generative systems, a partly impossible task, reflexive software development can focus on contestation of the problem of interpretability itself through the limitations of explainability solutions, and reframe the value system that drives its development. 

Labor Optics is a reflexive software project that critically reads WAR software and training datasets to develop an alternative series of interfaces and visual simulations to augment concerns and contest machine learning logics. It also reinterprets functionality and meaning from a humanistic labor rights and relations perspective. Labor Optics makes use of five tactics that each explore different visualization techniques to gain insight and literacy in how the algorithm sees, how it processes information, and how it arrives at decisions.