How could we use everyday human movement to orchestrate a musical soundscape? This experiment uses video footage from the iconic Abbey Road zebra crossing in London, to conduct a virtual orchestra to perform scores from The Beatles Abbey Road album (naturally). Here, active commuters are transformed into musical instruments, forming a unique new kind of urban marching band.

Making music from moving images: Video frames are analysed in real-time using computer vision AI TensorFlow JS, which detects human figures that are shown as coloured 'clouds'. Data from human poses is then used to build a set of 'movement' variables that control different aspects of a musical system built using the Web Audio API and the ToneJS library. This prototype runs entirely in your web browser. The video source can be adapted to run from a live stream or directly from a web cam - in fact, the whole live experience could be run on a smart phone.

The musical system comprises over a dozen instruments (built from sampled recordings), that are each given a piece of a music score to perform (from a MIDI file). The number of people in view controls how many instruments are audible - turning each person into a part of the band. Basic effects such as audio (instrument) positioning/panning, and reverb are mapped to spatial and movement characteristics. And the music tempo fluctuates depending on the time of day. More work will be done to explore more 'pleasing' and diverse experiences - however even this basic example demonstrates how a generative system can create unique and endless musical performances.