Form Follows Sound Workshops: Sonic Experience and Embodied Interactions

FFS-Resized-0069

Form Follows Sound: Workshops on Sonic Interaction Design

In the context of Human Computer Interactions, Sonic Interaction Design is commonly considered as a design practice that exploits the use of sound to facilitate users’ interactions with products and services, mediated by computational technologies. As interaction with sound may be task oriented or experience-based, an understanding of the nature of contextual aspects of bodily sonic experience and action–sound relationships are important factors to consider for designing richer sonic interactions.

Our approach to Sonic Interaction Design looks at the relationship between human movement and everyday listening experience to look at novel ways of imagining interactions with sound and inform the design of gestural-sound mappings using interactive technologies. To do so, we investigate a user-centric approach that first considers the affordances of sounds to evoke gestures in order to imagine embodied interaction with sound, and based on this, generates interaction models for interaction designers wishing to work with sound.

We designed Form Follows Sound, a series of workshops in Sonic Interaction Design in which participants drew upon their everyday sonic experiences to ideate imagined sonic interactions, and then realise interactive prototypes.

We carried out the workshop 4 times in 4 different places, with a total of 43 participants of varying degrees of experience with sound and music:

Participants’ background included graphic and interaction design, theatre and dance performance, music technology, engineering, rehabilitation, physics, bioengineering, social science and art.

Workshops Structure

The workshops started with an ideation phase, which does not involve technology. The aim of this phase was to generate ideas for action-sound relationships based on memories of sounds from participants’ everyday lives. This phase consisted of three different activities:

 

  1. Sonic Incident. Remembering, describing and presenting personal sonic experience in which users’ movements and situation can be re-evoked. Sounds are first presented using vocal imitation and then discussed using words. This method helps to discuss sound – in its physical and contextual aspects – and uses it as case-study for imagining interactions.
  2. Sound and Action cards. Tagging activity to facilitate users to enrich their description of the sonic incident.
  3. Imagining Sonic Interactions. Imagining interaction with the sonic incident using body movement, as a form of re-embodiment of sound from the perspective of the user.

 

FFS-Resized-0002

Sonic Incident

IMG_2302

Sound and Action Descriptor Cards

FFS-Resized-0004

Imagining Interaction with Sound

The ideation phase was structured in this way to aid participants to (1) access to sonic qualities and the context of the experience described in the incident and generate design ideas, and (2) to include aspects of body movements which we could explore with motion-tracking based technology in following phases of design.

The following realisation phase brought the participants to the realisation of working technology prototypes of the imagined sonic interactions. We created breakout groups to facilitate a group dynamic of mutual negotiation, teaching and understanding. We provided a technological toolkit to realise a wide range of gestural sound interactions. The toolkit is an open project available on Github, and includes gesture sensors, and software tools for motion data processing, mappings and sound synthesis. The system allows real time interaction, meaning that sound is sculpted and modified live as movement is performed. Finally each group selected one imagined scenario from the set of sonic incidents described in the ideation phase and developed it in an interactive prototype.

FFS-Resized-0037

The Gestural Sound Design Toolkit

FFS-Resized-0017

Prototyping

ESIW-HCC-Ircam-0005

Scenario Development 

ESIW-HCC-Ircam-0014

Project Presentation: The Snow Cracker, an immersive sonic storytelling which shows an example of substituting interaction with sound.

IMG_2354

Project Presentation: The Super-Champagne Man

Workshops’ contributions to the research

From a research perspective, this project provided three contributions:

 

  1. Sonic Incidents and re-embodiment as an ideation process for designing sonic interactions based on gestures;
  2. Non technology-centred methodology for enhancing and realising innovative interactions that exploits our sonic sensitivity;
  3. Three models of embodied sonic interactions based on action-sound relationships as a contribution for interaction design: Substituting, Manipulating, and Conducting.

Overall, our workshops methodology helps participants without specialist audio engineering or musical training to work with sound, representing a change in meaning of sound in their practices, by encouraging them to think about, discuss, and manipulate sound through our methodology.

This short video shows the structure of the workshops, then gives details about the Sonic Incident, the Gestural-Sound toolkit and finally presents the three embodied sonic interaction models.

This research has been published in a paper presented at ACM CHI2015, available here: Form Follows Sound: Form Follows Sound: Designing Interactions from Sonic Memories

Alessandro Altavilla, Baptiste Caramiaux and Atau Tanaka.