is an augmented reality musical instrument composed of tonal cubes arranged in a 3D spiral. Musicians use freehand gestures to activate tonal drone synthesis, playing combinations of cubes to form chords or individual cubes to explore tones with poly-synthesised textures.
What could a musical instrument look and feel like when it no longer needs a physical surface? cube. explores how spatial placement, movement, and gesture can become expressive musical controls in head-mounted AR system.
The virtual instrument can be positioned in the surrounding space, allowing musicians to move around it or place themselves within its large-scale form.
I designed and prototyped three AR interaction concepts through iterative self-testing, followed by a study of the final design with 20 musicians. The work explored learnability, creative exploration, and the relationship between spatial interaction and musical expression.
The instrument evolved through autobiographical design: building, playing, and reflecting on each prototype to explore how spatial interaction could support musical expression.
The first design drew on a physical synthesiser interface: a virtual panel of buttons and faders mapped to synthesis parameters on one side of a 3D cube. The left side provided tuning controls for two oscillators, while the right side contained an ADSR envelope and a filter. Musicians could operate the controls with hand gestures and freely reposition the whole cube in space.
This design explored how a traditional synthesiser interface would work in an AR headset. Through self-testing, I found that precise freehand manipulation of the panel was slow through the hand-tracking interface, making real-time sound tuning difficult.
The second design simplified the interface into two connected cubes, each representing an oscillator, with the movement of one modulating the other. Sound controls were mapped to movement around the performer’s body, extending interaction beyond a single panel in front of them.
Using far-pointer hand interactions, musicians could move the cubes horizontally and vertically or change their size to adjust the sound. Vertical movement controlled pitch, horizontal movement adjusted modulation, and size controlled volume. Filter controls were hidden with predefined settings, allowing performers to walk freely while continuing to shape the synthesis process.
In self-testing, consolidating multiple sound parameters into spatial controls made the interface easier to use. The simplification also reduced sonic diversity and musical richness.
The final design consists of individual sound cubes, each representing a musical note, arranged as a cylindrical MIDI keyboard. Musicians activate a cube by intersecting it with a finger. Full-hand tracking allows multiple notes to be played simultaneously, supporting chords and different note combinations.
The cylinder was sized so that most cubes could be reached from one standing position. Musicians could grab its centre to rotate the arrangement and access different notes. Colours distinguish tones, while rim lights identify semitones.
As in the second prototype, synthesis parameters were predefined. The design focused on ease of finger interaction and flexibility in musical composition, allowing performers to explore individual notes and chords within a spatial arrangement.
I built the prototypes in Unity with Microsoft MRTK, using C# and DisunityST for real-time audio synthesis.
The study with 20 musicians explored how mobility and the tangibility of spatial interaction shaped their experience. The research identified opportunities for intuitive, exploratory interaction, alongside a need for customisable sound controls to accommodate different creative preferences.