Form Follows Sound: Workshops on Sonic Interaction Design

In the context of Human Computer Interactions, Sonic Interaction Design is commonly considered as a design practice that exploits the use of sound to facilitate users’ interactions with products and services, mediated by computational technologies. As interaction with sound may be task oriented or experience-based, an understanding of the nature of contextual aspects of bodily sonic experience and action–sound relationships are important factors to consider for designing richer sonic interactions.

Our approach to Sonic Interaction Design looks at the relationship between human movement and everyday listening experience to look at novel ways of imagining interactions with sound and inform the design of gestural-sound mappings using interactive technologies. To do so, we investigate a user-centric approach that first considers the affordances of sounds to evoke gestures in order to imagine embodied interaction with sound, and based on this, generates interaction models for interaction designers wishing to work with sound.

We designed Form Follows Sound, a series of workshops in Sonic Interaction Design in which participants drew upon their everyday sonic experiences to ideate imagined sonic interactions, and then realise interactive prototypes.

We carried out the workshop 4 times in 4 different places, with a total of 43 participants of varying degrees of experience with sound and music:

• New York, a two-day workshop at Parsons The New School for Design;
• Paris, a two-day workshop at IRCAM Centre Pompidou as part of the European summer school, Human-Computer Confluence (HC2);
• London, a one-day workshop at Goldsmiths College;
• Zurich, a one-day workshop at the ZHdK academy of art as an activity within a teaching module in Sonic Interaction Design.

Participants’ background included graphic and interaction design, theatre and dance performance, music technology, engineering, rehabilitation, physics, bioengineering, social science and art.

Workshops Structure

The workshops started with an ideation phase, which does not involve technology. The aim of this phase was to generate ideas for action-sound relationships based on memories of sounds from participants’ everyday lives. This phase consisted of three different activities:

 

1. Sonic Incident. Remembering, describing and presenting personal sonic experience in which users’ movements and situation can be re-evoked. Sounds are first presented using vocal imitation and then discussed using words. This method helps to discuss sound – in its physical and contextual aspects – and uses it as case-study for imagining interactions.
2. Sound and Action cards. Tagging activity to facilitate users to enrich their description of the sonic incident.
3. Imagining Sonic Interactions. Imagining interaction with the sonic incident using body movement, as a form of re-embodiment of sound from the perspective of the user.

 

The ideation phase was structured in this way to aid participants to (1) access to sonic qualities and the context of the experience described in the incident and generate design ideas, and (2) to include aspects of body movements which we could explore with motion-tracking based technology in following phases of design.

The following realisation phase brought the participants to the realisation of working technology prototypes of the imagined sonic interactions. We created breakout groups to facilitate a group dynamic of mutual negotiation, teaching and understanding. We provided a technological toolkit to realise a wide range of gestural sound interactions. The toolkit is an open project available on Github, and includes gesture sensors, and software tools for motion data processing, mappings and sound synthesis. The system allows real time interaction, meaning that sound is sculpted and modified live as movement is performed. Finally each group selected one imagined scenario from the set of sonic incidents described in the ideation phase and developed it in an interactive prototype.

Workshops’ contributions to the research

From a research perspective, this project provided three contributions:

 

1. Sonic Incidents and re-embodiment as an ideation process for designing sonic interactions based on gestures;
2. Non technology-centred methodology for enhancing and realising innovative interactions that exploits our sonic sensitivity;
3. Three models of embodied sonic interactions based on action-sound relationships as a contribution for interaction design: Substituting, Manipulating, and Conducting.

Overall, our workshops methodology helps participants without specialist audio engineering or musical training to work with sound, representing a change in meaning of sound in their practices, by encouraging them to think about, discuss, and manipulate sound through our methodology.

This short video shows the structure of the workshops, then gives details about the Sonic Incident, the Gestural-Sound toolkit and finally presents the three embodied sonic interaction models.

This research has been published in a paper presented at ACM CHI2015, available here: Form Follows Sound: Form Follows Sound: Designing Interactions from Sonic Memories

Alessandro Altavilla, Baptiste Caramiaux and Atau Tanaka.