The algorithm allows for early recognition of gestures (i.e. recognizing a gesture as soon as it starts) and allows for estimating gesture variation (such as speed, scale, orientation, …) of a live gesture according to the learned (pre-recorded) templates.
The algorithm has been developed in the context of expressive continuous control of digital media at IRCAM Centre Pompidou (by Baptiste Caramiaux, Nicola Montecchio, Frédéric Bevilacqua) and interfaced in various interfaces (MaxMSP, pure data, openFrameworks) at Goldsmiths (by Baptiste Caramiaux, Igor Correia, Matthew Gingold, Alvaro Sarasua) under the MGM project.
The algorithm and its evaluations have been published in ACM Transaction on Interactive Intelligent Systems:
- B. Caramiaux, N. Montecchio, A. Tanaka, and F. Bevilacqua. Adaptive Gesture Recognition with Variation Estimation for Interactive Systems. ACM Transactions on Interactive Intelligent Systems (TiiS). 4 (4). December 2014
The code has been released on Github:
https://github.com/bcaramiaux/ofxGVF.
Here we present the content of the article, an example of application is given here, other examples can be found on Github directly.
Motivations
Gesture is increasingly used in Human Computer Interaction (HCI) involving several forms of activity recognition. This is a need to elaborate interaction paradigms based on body movements (e.g. hand, whole body) or tangible interfaces that could enable natural and fluid interaction.
Computational model
The model is based on a state-space model involving a non-linear dynamical system. The state space comprises the variations that have to be estimated in realtime. The dynamics of the variations are linear and rely on noise parameters. The realtime estimations of gesture variations at a given instant knowing the previous estimation and the current gesture observation is based on particle filtering (importance sampling). The idea is to spread particles, that are potential values for the variations, and weight these particles according to the current observation.
Evaluations
We assessed the algorithm in a gesture recognition task. We performed this evaluation on a database from the state of the art and comparing it with established techniques. Specifically we used the database presented with the $1 dollar recognizer (a non real-time 2d gesture recogniser).
Then we assessed the adaptation feature (adaptation to gesture variation) on synthetic data. Details can be found in the article.
Application
We built an application that uses of sound feedback to respond to salient gesture variation. Performing a given gesture initiates a specific sound, which are then continuously manipulated depending how the gesture is performed. Interaction with the application involves the two fundamental aspects of the interaction model given by the algorithm: selection (discrete command from gesture recognition) and manipulation (continuous command from adaptation to gesture variation). Note that we also investigated this interaction model in a graphic application.
In this application the gesture are free-space gestures (3-dimensional). See below a schema of the application: gesture recognition and adaptation to sound processing:
