Interactive Music for Harmonic Dissonance
An art-science experiment where brains and bodies dance together
Art & Science
When art and science meet and both can realise their own ambitions, goals and ideas, interesting and innovative applications often arise. Wow, fancy words, but doesn't that happen anyway? Where do art and science stop coloring within their own lines at all and can we actually define these two domains? Well… great news, we don't need to do that in Private Kitchen, because science and art, just like politics, don't necessarily need to be discussed at the (cutting) table. What does?
Recipes, ingredients and how the resulting dish tastes!
The above recipe was developed as an experiment working towards a new performance for ICK in 2021. This performance was building on the longstanding relationship (since 2018) between the installation’s initial creators, Dikker + Oostrik, and the ICK. Part of the experiment was to explore the scientific research domain on how our brains and bodies synchronise and achieve a sense of "togetherness".
Harmonic Dissonance 2021/2022 recipe and ingredients: Combine a neuroscientist, a dance researcher plus 2 dancers, 1 creative coder and 2 interaction composers with 4 networked audiovisual installations equipped with infrared lights and cameras.
Let it simmer for 2 weeks in an open pressure cooker, stir occasionally, finally add handpan, violin and an audience at the end and serve it up!
The Harmonic Dissonance team created this next video, together with Private Kitchen, in which they explain their experimental project.
One of the team’s missions was to create a performance with dancers, live music and an interactive audiovisual installation. The installation was created to explore the scientific research domain on how our brains and bodies synchronize and achieve a sense of "togetherness", of belonging, and human social connectedness. (within and while performing in a scientific research domain).
The team named this interactive audiovisual installation the “Harmonic Dissonance Quartet” after the four large vertical screens with speakers and cameras that detect the poses of the people standing in front of them.
By means of computer vision (a certain form of Artificial Intelligence), the pose data of the user (in case of the performance, the dancer) is registered by infrared cameras and converted into usable data for visuals and the interactive music.
This interactive music was implemented and performed in FMOD, a programme that is mainly used in the context of interactive game music & sound and that can be linked to various data streams in a fairly user-friendly way.
As a debut for FMOD being used in rapid prototyping of interactive music in the context of choreography and dance, this game audio middleware proved to be very useful!
The first version of the musical building blocks were composed and implemented in advance, with a predetermined concept where the music would build up from ‘very little and small movements, in a slightly dreamy context and with gentle breath’ to ‘bigger and fuller, more eventful and with a clear (heart)beat’, but only going to a more ‘epiphanic and uplifting’ version when more instruments (or dancers) would be ‘in sync’. An ‘ultimate musical, most uplifting version’ would only be reached when ‘dancing / moving in synchronicity’.
FMOD, thanks to it’s ‘live update’ functionality was a great solution in quickly adapting settings and mapping and also the musical material itself in relation to the actions of the dancers.
Both Ivo and Than recorded and implemented new musical material in the days that they worked with the dancers, scientist, developer and choreographer. A truly iterative and collaborative creative process!
screenshots of FMOD, as used in preparation and during runtime
The Harmonic Dissonance Quartet was musically divided into 4 instruments representing handpan, percussion, violin and piano. Depending on the (compiled) data, the instruments were eventually playing states like 'quietly', ‘softly and subtly’, or in the most exuberant version, ‘fully and loudly’.
This interplay between instruments, musical interplay and linkage to poses and movement led to a beautiful synergy between the dancer 'cranking up' the music, which he/she then would continue to move on to (and thereby continue to drive the music through his movements).
Needless to say to a platform for composers that building up musical energy is not the same as breaking it down, releasing some tension. Therefore in good FMOD practice, the ‘way back’ in energy would lead via another pathway, or transitions, for certain instruments than the way up. Simple as this may sound the eventual solution in FMOD was rather new and would, together with the making of of the interactive music, supply enough information to write an entire article.
Quote Than : “Working with two very talented dancers and an at least even talented choreographer was very inspiring and a great delight.
However, my personal highlight and confirmation that this interactive Audiovisual installation has huge potential was when seeing a very young ballerina (age ±5 years) continuously dancing on the music she was actually generating with her own movements!”
The future work and publications of all separate Harmonic Dissonance team members is very promising, ranging from breakthroughs in neuropsychological science, dance research and interactive application design. As a team however, they expect and plan to elaborate on the findings and new arisen questions as well as the ambition to bring the Harmonic Dissonance to a next level.
A next planned possibility to experience the Harmonic Dissonance Quartet will be in Doornburgh - buitenplaatsdoornburgh.nl/agenda - in the entire month of February 2022 and Harmonic Dissonance will also be exhibited in NXT museum, Amsterdam in the 1st week of March : https://nxtmuseum.com
Than van Nispen tot Pannerden & Ivo van Dijk
new media composers
Than van Nispen tot Pannerden and Ivo van Dijk can best be summarised in this context as interactive media composers and musicians.
However, they are so much more than that. Than for instance is both a biologist (he learned about Deep Ecology and Anthropocene extinctions during his MSc at the Utrecht University and acquired his ‘mad scientist’ mojo there) and a music composer for games and interaction, which he studied for at the HKU, giving him two other Master degrees (MMus, MA). Besides life and working on (solutions for)(interactive) compositions, he nowadays enjoys spending his parttime life as a lecturer and researcher at HKU School of Music and Technology where he currently works in fields as Interactive music for (Art) Games and Installations, Music as Biology, BioArt, inclusion, Artificial Intelligence and Sonic Interaction Design. Than has a longstanding collaboration with Ivo van Dijk.
Ivo started his musical career behind a drumkit in the heavier scene of music: Symphonic Metal, a love which both Than and Ivo share. His background extended in creating interactive music for games, realising theatre productions, rock opera’s, and performing with dancers. After a couple of years of entrepreneurial activities in videogames and escape rooms, Ivo discovered a spiritual and intuitive layer of musical abilities, resulting in performing ceremonial music, breathwork sessions and sound healings: Often dubbed as a “Sorcerer of Sound” by people who undergo his musical antics in these settings.. All of these activities have now been combined under his (and his wife’s) musical/creative outlet: UniversalSoundShifts.
Grateful for their wonderful collaborations with other composers, arrangers and the Metropole Orkest in the realisation of Karmaflow: The Rockopera Videogame, they worked on several musical performances with often a high level of audience participation, influencing the outcome of the music.
