Skip to content

Project

Algorithmic Cultural Vandalism

A robot develops new cultural symbols through drawing with humans

Alessandro Mac-Nelly, Pietro Lugaro

Year2024
Additional informationArs Electronica Campus Award 2024 — Grand Prize · Award Ceremony

Project Overview

ACV (Algorithmic Cultural Vandalism) is an evolving installation that probes how cultural symbols are transformed when filtered through algorithmic systems. A human visitor and a robotic arm, guided by an AI trained on a diverse archive of glyphs, talismans, scripts and signs, take turns sketching pseudo-symbols. This collaborative drawing process highlights the tension between meaningful sign and abstract doodle and embraces the playful misunderstandings that arise when human intuition meets machine morphology.

Rather than decoding or interpreting, ACV reduces every stroke to pixels and binary data. The AI "Actor" proposes shapes and a "Critic" (an image recognizer) evaluates them via reinforcement learning. As participants attempt to reinfuse their marks with meaning, the machine discovers strategies, sometimes tricking the Critic, to maximize its scores. This dynamic reveals how algorithmic valuation can distort and reframe even our most familiar cultural forms.

Drawing on Federico Campagna's concept of Technic in Technic and Magic: The Reconstruction of Reality (2018), in which reality is cut up into measurable units for endless recombination, ACV critiques extractive systems that catalog symbols as data commodities. The installation itself becomes a living diagram of this Cosmogony, questioning at each turn when a mark stops being a doodle and starts acting like a symbol.

Designed for broad engagement, ACV has been tested in workshops with children and adults alike. These sessions spark conversations about semiotics, creativity and the delicate line between abstraction and representation. Every interaction is archived and fuels research into whether consistent morphological or gestalt patterns emerge from what we often dismiss as mere scribbles.

Looking forward, we are expanding the AI's archive with additional global symbols, refining the reinforcement learning algorithm and developing new workshops that blend playful experimentation with semiotic exploration. We are also preparing a visual publication alongside academic papers and planning to evolve ACV into a continuous installation that grows, learns and keeps the debate on cultural datafication alive.

Dataset of strokes

The dataset consists of 349 individual images documenting the drawing steps within 65 shared drawings. Before and after each robotic turn, the integrated camera records the current state of the drawing. The images are subsequently binarised and colour-coded so that each newly added stroke can be considered as an individual drawing event: red marks the human strokes, blue the robotic strokes. Black strokes form the existing context in which the contributions of both actors merge into a shared drawing.

The following sheets reproduce the complete dataset as presented in the appendix of my university paper, Mensch und KI Roboter im zeichnerischen Zusammenspiel: wiederkehrende Zeichen-Phänomene, written for Strukturen und Systeme with Prof. Susanne Hauser at UdK Berlin / TU Berlin. Figure numbers, source pages and the original stroke ranges are given in the gallery captions.

Drawing steps 1–96 · Appendix, Figure 6, page 17
Drawing steps 1–96 · Appendix, Figure 6, page 17
Drawing steps 97–192 · Appendix, Figure 7, page 18
Drawing steps 97–192 · Appendix, Figure 7, page 18
Drawing steps 193–284 · Appendix, Figure 8, page 19
Drawing steps 193–284 · Appendix, Figure 8, page 19
Drawing steps 285–349 · Appendix, Figure 9, page 20
Drawing steps 285–349 · Appendix, Figure 9, page 20

1 / 4

Observations from the study

The study investigates recurring phenomena in the drawing interaction between humans and an AI-controlled robot. Communication takes place exclusively through alternating strokes on paper. The morphological properties of each drawing step are recorded in a multidimensional vector, while the semantic level is examined through an open, observation-based classification. A central aspect is the human interpretation of machine-generated drawing actions: the ACV model has no semantic understanding, yet its interventions are integrated, interpreted or ignored by the participants.

Compared with the robot's strokes, human strokes cover less inked area, extend over a greater spatial range and are less complex. Neither actor shows an overall preference for horizontal, vertical or diagonal orientation. Humans initiate the drawing process in 97 percent of cases. Recurring sequences suggest initial, context-dependent rules: a vertical robotic stroke tends to be followed by a vertical human stroke; a straight robotic stroke is followed by a more complex human response, while a highly complex robotic stroke is followed by a simpler, less extensive response with less inked area. Robotic interventions on the left or at the top of the sheet tend to be answered on the right or below. This pattern corresponds to Western reading direction and raises the question of whether different cultural conventions would produce different responses.

The semantic analysis shows that 27.4 percent of human drawing steps are classified as pictorial, compared with only 2.6 percent of robotic steps. Pictorial motifs occur predominantly in the early phases of a drawing, while compositional additions and scribbles tend to appear later. Once an initial basis of meaning has been established, the drawing process appears to continue this context rather than generate new motifs. Meaning emerges from the relationship between the drawn marks and their interpretation, even where no semantic intention exists on the robot's side.

A recurring example is the Tic-Tac-Toe scenario: a participant draws a playing field and makes the first move, expecting the robot to take the next turn. When its intervention does not follow the familiar rules, it can be perceived as a disruption, leading the participant to abandon the game and question the robot's intelligence. In other encounters, unconventional robotic strokes are perceived as creative and constructively integrated into the drawing process. These transitions between ignoring, integrating and reinterpreting the robot's interventions are central to the interaction.

Both analyses reveal methodological challenges. Morphological properties lie on a continuous spectrum, and individual drawing actions can belong to several semantic categories. Clear boundaries are therefore difficult to establish. The observed sequences suggest a possible visual grammar in the sense of Lorraine Daston's context-dependent “thick rules” in Rules: A Short History of What We Live By (2022): patterns that emerge from situations and remain open to interpretation. Whether a unified grammar of human–AI drawing interaction can be derived remains an open question, calling for further investigation and comparison across cultures.

Analysis figures and tables

The figures on morphological properties, clustering and semantic classification are reproduced below, followed by all three numbered tables from the appendix.

Figure 3: Continuous transitions between morphological properties · Page 10
Figure 3: Continuous transitions between morphological properties · Page 10
Figure 4: Morphological clusters in a two-dimensional UMAP projection · Page 11
Figure 4: Morphological clusters in a two-dimensional UMAP projection · Page 11
Figure 5: Distribution of drawing steps by semantic category · Page 12
Figure 5: Distribution of drawing steps by semantic category · Page 12
Table 1: Morphological vectors · Appendix, page 15
Table 1: Morphological vectors · Appendix, page 15
Table 2: Robotic strokes and subsequent human responses · Appendix, page 15
Table 2: Robotic strokes and subsequent human responses · Appendix, page 15
Table 3: Semantic categories and their assignment · Appendix, page 16
Table 3: Semantic categories and their assignment · Appendix, page 16

1 / 6

References and links

Exhibited at

Impressions

1 / 7

Relationships