Keywords

1 Introduction

This paper first aims to identify a number of keywords towards a clear understanding of the notion of ‘interaction’ in the global context of contemporary art. A categorization of interactive systems follows based on the type of emergent relationships afforded by specific orientations. A reference chart is developed showing features of interactive systems; from instrumental responsive systems to systems displaying unpredictable yet coherent life-like behavior. Finally, two of the author’s audiovisual installations document the connection between artistic vision and effective material instantiation. However, first of all, we address the innovative work of two groundbreaking artists; Edward Ihnatowicz and Gordon Pask – the notion of a ‘learning system’ is implicit to the approach of both.

2 Two Pioneers

Ihnatowicz is best known for designing The Senster (early 1970’s), a large computer driven robotic structure, probably the first one designed with behavioral rather than cosmetic objectives in mind. The machine had a repertoire of responsive behaviors in relation to what it was sensing in the environment i.e. capturing sounds and detecting people’s movement. Ihnatowicz pioneered the notion of grounded cognition insisting the process of perception being informed by mechanical manipulation, the physical processes articulating the body being explicit to the emergence of meaning. In a 1977 paper, pondering the challenge to design complex artificial systems, Ihnatowicz turns to nature suggesting: “it may help to treat the whole problem as an exercise in the simulation of evolution and try to design a system, although simple to begin with, could learn from experience…” (Ihnatowics 1977, p. 6) – however, the artist was ahead of his time, the scientific discipline of evolutionary computing would emerge about a decade later.

Gordon Pask pioneered a hybrid art integrating advanced engineering, early notions of artificial intelligence and human-machine interaction. For example, his 1968 piece A Colloquy of Mobiles consists of suspended electro-mechanical ‘creatures’ mutually communicating sound patterns and flashing lights outlining a socially oriented reactive and adaptive environment (Pask 1971). Human participants may interfere with the mobiles, complex behavior issues as the system includes adaptive sensing, memory, variable delays and various feedback loops – in other words “the machine is designed to entrain the performer and to couple him into the system” (Pask 1971, p. 80). Interestingly, mobiles were characterized as ‘male’ and ‘female’ and designed to be partially incompatible, required to communicate in order to co-operate while focusing on a common goal and finally, the biology-inspired notion of autonomous low-level action; mobiles are decoupled while still maintaining their individual integrity.

Both examples provide fascinating and compelling experiences since they establish an intimate link between artefact and viewer/participant through interfacing mechanical and bodily behavior. One projects a form of ‘life’ in the object – a strong affective experience follows from the appreciation of essentially simple machine activity – people project intelligence and life-like properties in inorganic artifacts. As a matter of fact, both Pask and Ihnatowicz implemented forms of adaptation and learning in their respective systems. For example, The Senster maintained a kind of dynamic sound-map documenting acoustic activity in space informing subsequent audio and spatial sensing.

3 Features of Interactive Systems

As artists, we take continuous inspiration from examples of fascinating (1) morphology and (2) behavior in natural, biological and social environments. What if we could identify the structural components leading to the complexity observed? It is quickly understood that interesting non-linear behavior issues from local interaction of minute building blocks leading to either global functionality or evolution towards a point attractor signaling global dysfunctionality: from blood cells in the immune system, to massive neuronal firing in the brain underpinning the notion of consciousness, to self-organization in ant societies, to a sudden revolution ending centralized decision-making and many other biological, social, and economic systems (Resnick 1997).

When considering the power of reciprocal interactions between distinct living species, symbiotic interaction becomes a vital force, quite often even the key to survival. Just one example: intimate symbiotic interactions develops between the sea anemone and the clown fish – the fish needs to hide and find protection from predators, it also eats algae while the anemone needs protection from competition with algae. Both entities mutually coexist in a functional relationship.

From this perspective, and in the light of implementing some observed aspects of interaction in natural/biological workspaces, let’s frame a number of observations and opinions towards the accomplishment of truly interactive systems.

  1. (1)

    Interaction is about exploring a particular space of options, including physical spaces explored through bodily engagement as well as conceptual spaces (often considered of infinite dimension) explored through speculative approaches. Exploration of unknown options is contrasted with viewing interaction as problem solving.

  2. (2)

    Interaction implies the simultaneous activity of partners engaging in mutual action, coexisting in a common biotope, be it physical, biological, cultural, social or any playground offering common resources, preferences or opportunities.

  3. (3)

    In social workspaces, interaction is often about sharing speculative initiative in mutual understanding. Individual agents interact by contributing to a pool of options in continuous flux, the environment is modulated by the very act of sharing and participation. All participating actors (human or synthetic) are at equal levels of authority.

  4. (4)

    Interaction is concerned with the discovery of initially hidden relationships in high-dimensional spaces. For example, affinities between the components of an interactive audiovisual installation (logic, sensors, mapping…) might be obscure at first and inform the act of anticipation. Then, balancing perception and prediction becomes a dynamic process highly impacting the nature and depth of the aesthetic experience.

  5. (5)

    In biological – including human-centered – workspaces, the concept of motivation plays a crucial role in driving complex interaction processes. Therefore, it makes sense to develop intrinsically motivation driven interactive systems. Such systems are prime examples of a methodology referred to as speculative computing. For example, a program develops sounds or images and tracks the complexity of the user’s response aiming to figure out whether the user agrees or disagrees with the system’s suggestion. An experimental platform for interactive composing integrating artificial motivations and machine learning is documented in Beyls (2017).

  6. (6)

    Interactive systems should accommodate unpredictable input (on purpose or accidental) and offer non-linear behavior. In addition, while facing extreme conditions (in terms of internal logic or external conditions) the system should not break down but support graceful degradation in performance.

  7. (7)

    Interactive systems should not merely offer procedural performance but dynamic behavior. Such systems are driven by positive feedback, change entails additional change, potentially steering the system into complex chaotic behavior – the human (inter)actor again engages in a predictive process balancing cause, effect and fluctuating levels of complexity.

  8. (8)

    Finally, the development of interactive/generative systems offering rich and compelling modes of experience and participation is an act of aesthetic introspection – some idea is implemented, and the system talks back to the programmer exposing unanticipated forms of behavior. The exploration-based method is indeed equally paramount in the development of new work. Therefore, in creating new work let’s start from scratch and without prejudice – exceedingly difficult given the conditioning complexity of the digital medium itself. Then, intuition, rather than logic, is a more unselfish adviser informing aesthetics-driven software development.

Observing the above considerations, we conclude interaction to issue from the discovery and management of speculative relationships in high-dimensional spaces.

A deep and outstanding form of interaction subsists when contact leads to permanent modification of the cognitive or material qualities of interacting entities. The 1991 work entitled “Untitled. Portrait if Ross in LA” by Felix Gonzales-Torres is a prime example (Mulder 2007). It consists of a pile of candies on a gallery floor, it weighs 175 lbs, the actual weight of Torres’ lover before losing weight as a consequence of AIDS. A visitor is invited to eat a candy – an enduring and irrevocable trace is left in his/her metabolism. The work of art becomes a grounded experience; the memory of Ross lives on in the body of a participant. In this case, interaction is a one-to-one exchange, where interaction itself entails transformations in the work of art as well as the participant. Human and artefact develop a bond, a sort of relationship extending in multiple dimensions; spreading from the purely materialistic to the cognitive to the purely subconscious. The notion of relationship is key to the categorization and understanding of different types of human/machine interaction. Interactive art can be understood according to the affordance of relationships based on (1) the formation of intimacy, i.e. a person submits to manipulation by an object, or (2) the object embodies the person – a relationship of instrumental control (Fels 2000).

A communication-theoretical definition of interactivity equally considers the notion of relationship: “Part and parcel of a system is the notion of ‘relationship’. Interactional systems then, shall be two or more communicants in the process of, or at the level of, defining the nature of their relationship (Littlejohn 1989). This definition stresses a form of reciprocal, mutual engagement informing action.

Gonzales-Torres’ work clearly implies a relationship of the former type. However, a conversational interaction between object and human – consider a person writing and debugging a computer program – is understood as contemplation driven while the object equally embodies the person. Aesthetic experiences follow from (intermediate) results obtained, a process of flow (Csikszentmihalyi 2009) issues from variable, dynamic associations between human input and continuous machine feedback.

Now think of someone playing the piano, not just open ended – to get the picture, consider Cecil Taylor in concert. A perception of deep connectivity between instrument and musician becomes readily apparent; Taylor embodies the instrument. An intimate relationship develops, and the aesthetic and emotional experience follows from controlling the piano. So, in contrast to contemplation, we share the excitement by imagining of engaging in physical action echoing the musician at work, especially considering the role of expression to convey a musical message.

What happens if we interact with static objects? Consider Rothko Chapel in Houston, TX. Rothko was convinced that paintings had the potential to provoke profound spiritual experiences (Rothko 2006). He claimed, “I paint big to be intimate”, stressing the relationship between two forces: the emotional impact and corporal engagement of the viewer’s body in relation to the canvas. The power of the aesthetic experience surely emerges as a complex dynamical process in both of these dimensions; the viewer continuously adapts through conceptual as well as physical adjustment. Again, aesthetic functionality follows from action rather than static observation. Engaging with a painting is a one-way interaction, the viewer absorbs signals from the object, however the object itself remains unchanged.

Another type of interaction happens in games; for example, people interacting collectively conditioned and constrained by a system of rules defining the game (Salen and Zimmerman 2004). Ironically, emotion and general excitement follows from submission to the objective options implicit to the game or even the constraints characteristic to the interface. Simple game strategies equally perform in exchanges between people; consider Japanese sumo wrestling, both players contribute momentum and energy until someone steps outside the circle – a complex dynamic process with mutual push-pull interaction terminating in a point attractor. An onlooker develops excitement by tracking performance over time (the expression of a relationship informed by competition) and through the active prediction of an imagined winner.

Finally, we may think of a human-machine relationship as the actual specification of information drift between both parties and their interpretation strategies. A computational architecture supporting unbiased human-machine interaction in the context of non-idiomatic musical improvisation implements relationships and autonomous machine motivations as actual computational components (Beyls 2018). In short, relationship objects are non-linear maps linking various momentary features of human musical input. Motivation objects are binary vectors specifying dynamic associations between relationship outputs and two competing motivations: (1) integration: connect to musical material suggested by the human performer) or (2) expression: perform in isolation from the context. An algorithm tracks melodic distance between consecutive statements by man and machine, thereby evaluating the efficiency of currently dominating motivation – effectively implementing a form of reinforcement learning. However, further technical details are beyond the scope of this paper.

4 Towards a Living Interface

We designed a map targeting the identification of a wide-ranging collection of keywords supporting a comparative study of interactive systems. Our map basically situates responsive systems on the left-hand side and systems offering life-like behavior on the right-hand side. Various features (appearing as labels on the map) are spread out in between, their horizontal position being proportional to their conceptual distance to both extremes (Fig. 1). In essence, while reading from left to right, the notion of ‘interface’ gradually changes meaning: from supporting explicit control to accommodating implicit behavior.

Fig. 1.
figure 1

From responsive to life-like interactive systems

Full size image

Let’s consider systems complexity while browsing the map from left to right. The notion of predictability of system response is crucial in evaluating the functionality of interactive systems. For instance, a musician might expect total instrumental control and a tight relationship between instrumental gesture and sounding result. Responsive systems provide reactive behavior, they react by triggering a response usually selected from a palette of hardwired responsive options. The net result is a type of designed procedural automation, responses are typically both highly constrained and fit a predictable, premeditated behavioral niche.

Conversational systems follow a linguistic paradigm, interactive composing (Bresson and Chadabe 2017) offers many examples; human and machine intermittingly contribute musical arguments to a common, evolving social musical fabric, echoing musical fragments, however without exact repetition. This is conceptually close to the notion of ‘reflective systems’ where one conceives of the computer as a kind of qualitative mirror imitating, transforming and reflecting user actions in an augmented audiovisual language. Instrumental control still persists since perception of a clear causal link between action and reaction remains apparent. In other words: “The interactive artist must strike a balance between the interactor’s sense of control, which enforces identification, and the richness of the responsive system’s behaviour, which keep the system from becoming closed.” (Rockeby 1996).

Playing an acoustic instrument entails interaction with a sounding body. A painting evolves from physical interaction with the mechanical properties of paint, canvas and brush. Once again, an intimate physical relationship between artist and medium totally defines the creative process. Richard Feynman explains the phenomenon to historian Charles Weiner:

Weiner once remarked casually that [a batch of notes and sketches] represented “a record of [Feynman’s] day-to-day work,” and Feynman reacted sharply. “I actually did the work on the paper,” he said. “Well,” Weiner said, “the work was done in your head, but the record of it is still here.” [Feynman’s reply:] “No, it’s not a record, not really, it’s working. You have to work on paper and this is the paper. Okay?” (Clark 2013 p. 258).

Digital instruments offer sophisticated control structures for mapping physical activity to sounding algorithms, the performer embodies the instrument, actions inform sound and sound informs actions, man and machine coalesce into a dynamic superstructure. An excellent example is “The Hands” conceived by the late Dutch composer/performer Waisvisz (1985). Expressive instrumental abilities relate to the notion of affective interaction, the expression of emotion-driven intentions to create complex sounds in a wish to share the excitement with a receptive audience. The process is double: (1) a musician interactively navigates the expressive palette of the instrument (effectively a hybrid of hardware and software) and (2) the audience’s listening process is dynamic: continuously balancing sonic expectations with perceived instrumental gestures.

The notion of expectation is equally central to the creative engagement model (Bilda et al. 2008) suggesting a strong relationship between the practice of interactive art and experience design in the research discipline of Human Computer Interaction (HCI). Engagement relates to human values in interaction such as pleasure, frustration, challenge and anger. A connection is suggested between the kind of interaction (intended vs. unintended actions) and the nature of the system’s reaction i.e. causal, predictable feedback vs. uncertain, unexpected responses. Bilda’s model identifies the psychological layers at play in the perception of interactive art – from adaptation and anticipation to deeper understanding of the dynamics of the work’s internal relationships and receptiveness to external influence.

Then, from a more global perspective, we suggest the level of engagement, and therefore the strength of the aesthetic experience, to relate strongly to the dynamics of the anticipation process. Cognitive impact of a work of art follows from evaluating an immeasurable number of competing anticipations – typically, all potentially either implicitly or explicitly available in a complex interactive work of art. However, more universally, any artwork addresses the viewer’s motivated engagement – it reveals itself through the complexity of anticipation, the flow between meaning and mystery. We conclude art to exist as a “qualitative oscillator”.

5 Case Studies

Much of my work involves humans interacting with virtual creatures organized as self-organizing distributed agencies. One recent audiovisual installation, entitled Crickets, suggests a form of symbiotic interaction where humans and creatures coexist in a common biotope. Creatures appreciate the complexity of human input (seen through a camera and computer vision), the quantity (how much external energy is perceived) and quality (the complexity) of gestural activity. In other words, Crickets acts as a dynamic boundary – a living interface – creating a hybrid of organic and synthetic life. Aesthetic survival of both components is interdependent; artificial creatures anticipate delivery of human energy and humans expect complex and interesting behavior in return – human and machine engage in a non-trivial relationship. From the science of complex dynamical systems, we know positive feedback potentially pushes global behavior into a chaotic attractor leading to unexpected, momentarily peaking complexity (Fig. 2).

Fig. 2.
figure 2

Crickets (Color figure online)

Full size image

Crickets implements a simple particle system. Particles within a range of social proximity coalesce into spatiotemporal superstructures envisioned as dynamic creatures. Particles dissipate energy while moving in 2D space and engage in awake/sleep cycles according to energy level. A nascent connection between any two particles is reflected in a sound. A computer vision component creates an abstraction of the camera image: a low-resolution grid captures changes in pixel brightness to update grid cell values, incrementing or decrementing conditioned by a given parametric inertia. The grid acts as a slowly fading image memory, it is visualized by mapping the brightness value to a HSB color scale. An algorithm continuously computes the centroid of change of camera image – when sufficiently apparent, particles are attracted and move towards the centroid. The particle world features a specific non-binary gender system (5 options); when within the centroid zone, particles reproduce as a function of their energy and conditioned by a reproduction table – a 5 by 5 matrix enumerating reproduction probabilities. As a result, the particle population remains dynamic since positively influenced by human activity and limited by a given maximum capacity. The interplay of countless distributed internal forces and unconditioned external input affords a compelling audiovisual experience. A particular, non-inclusive cognitive link between action and reaction becomes apparent, however, the perception of fractional unidentified activity remains since particle world behavior is dynamic rather than merely procedural.

A second example is Dusk, a real-time web-driven audiovisual installation. Dusk extracts data from changes in external real-world events detected by online web cameras; this signal is normalized and used to condition social interaction in a particle world. Consequently, live data in a physical universe is relayed to condition interaction in a virtual abstract world. The subject of Dusk is the cyclic activity (the rhythm of timed trajectories of boats, the cycle of night and day…) in the harbor of Amsterdam.

A global brightness interval 256-element vector is computed from the signed values (−255 to 255) of the difference of very pixel in two consecutive camera images. Thus, for every potential brightness value (0~255), the sum of all occurrences of that value in both images is collected and subtracted – a single signed value results and is added to the vector. Note the grey-scale visualization of the interval image for both cameras in Fig. 3. Finally, the vector is reorganized as a 16 by 16 two-dimensional array, the array is visualized in red (positive values) and green (negative values). The array is updated as soon as a webcam provides a fresh live image.

Fig. 3.
figure 3

Dusk (Color figure online)

Full size image

Now, the array specifies how any two particles interact in their 2D world; when within a given (parametric) sensitivity range, a particle temporarily links to its neighbor (visualized by a green line segment) and addresses a single cell in the array using both particles’ ID as an index pointer. Formally, particle interaction is specified as follows:

$$ \begin{aligned} & Particle \, system = \left\{ {M,R,A} \right\} \\ & A = S = array_{(12,12)} \\ & M = \left\{ {m_{1} \ldots m_{8} } \right\} \\ & a_{m} \in \left\{ {0 \ldots 11} \right\} \\ & D_{{(m_{1} ,m_{2} )}} < S_{{(am_{1} ,am_{2} )}} \Rightarrow (a_{m} \leftarrow R_{{(am_{1} ,am_{2} )}} ,a_{{m_{2} }} \leftarrow R_{{(am_{2} ,am_{1} )}} ) \\ \end{aligned} $$

Algorithm A is equivalent to S, a sensitivity matrix; it holds numerical values expressing a threshold for interaction between any two types of particles to take place. The system contains 16 particles stated by M with particles moving in a direction defined by their momentary angle. The state space of angles is discrete with a resolution of 30°, therefore relative angles range between 0 and 11 to cover full circle. Note that angle resolution has a strong impact on the scope of temporal complexity of the system as a whole; higher resolution will expand the state space exponentially.

Interaction rules R are described explicitly as regular 12 by 12 element arrays. The angles of interacting particles receive specific interpretation. Both angles are interpreted as to index locations in the array – to retrieve the new values of both respective angles. The array is a simple and compact way to represent and condition how particles potentially interact. Global state space of all possible matrix rules is hugeFootnote 1 and thus considered virtually infinite. The effect of the rule array is qualitative control over collisions. Both particles update their respective angle when interaction becomes effective, i.e. when their physical distance is less than their mutual interaction threshold defined by array S.

6 Discussion and Conclusion

This paper suggests a feature map to study and evaluate various approaches to interactive art, starting from two poles: responsive and life-like systems. Relevant qualities are organized and labeled in between. Some artists expect a totally deterministic mapping scheme between user actions and systems’ response, for example, while playing a computer-based musical instrument. Others prefer to participate, to express influence over an otherwise autonomous self-organizing system. From our experience, when initially confronted with a complex system like Crickets, many museum goers spontaneously engage in a responsive interaction format i.e. creating bodily gestures in a wish to activate the system and hope for immediate substantial audiovisual feedback. While the actual system demonstrates complex evolving audiovisual output, many people do not initially develop a spontaneous cognitive link in a wish to interrelate with unpredictable yet coherent life-like systems’ behavior. However, while gradually adapting and developing an understanding of systems’ generative logic and perception modalities, users shift from an instrumental/reactive format into a more participation-oriented format.

Interactive art systems provide a playground for experimental evaluation of how individuals develop sensitivities and understanding of artistic statements in general. We consider the intensity of the aesthetic experience to relate to the dynamics of the appreciation process, insight/resonance is not understood as a process of static perception. Embodied interaction offers methodologies and pragmatic technology to develop knowledge of the cognitive processes underpinning deep forms of human-machine interaction. Generally speaking, the manifestation of art can be viewed as an emergent property of a grounded complex dynamical system; a global meaning emerges from local interactions between an infinite number of components. Art exists as a floating network spanning in an unknown number of dimensions in time and space.