VisualTalk
Ethnographic Research into Collaboration between Electronic Music Performers and Producers
My Role -
Researcher, Strategist, Designer and Software Developer
Personal Project
Brief
A laptop is designed to be used by a single user. Digital Audio Workstations (DAWs) are complicated software that require a high level of specificity from the individual user.
Music is emotional. English does not have a lay-man's lexicon to discuss music with the technical, emotive specificity required by the DAW. This is not a problem when only one musician is using the software but collaboration requires communication.
How do two musicians collaborate using a DAW on a laptop when they are composing music together?
Solution
VisualTalk is a music visualisation software which helps music producers and performers reference audio when collaborating to compose music on a DAW. It was written in Max4Live and Processing and runs live through Ableton.
It visualises separate tracks in the DAW as circles. Their x-position, diameter and hue reflect changes in the tracks' pitch, loudness and spectral centroid. The visuals open up another pool of words for collaborators to use to reference audio.
Key Skills - Ethnographic Research, Thematic Analysis, Conversation Analysis, Software Development
Research
I designed a two-year ethnographic study to research how musicians organise their interactions around the DAW.
I recorded audio, video and the laptop screen of a performer-producer duo composing their new album over four days. This data was then analysed using thematic and conversation analysis.
The most interesting insights were the methods the musicians used to reference audio in order to discuss it in interactions:
Here's the whole thematic map for the other nerds out there:
Insights
The musicians organised their interactions around the audio playback by including the laptop in conversation as if it was an active contributor itself.
This was only observed during collaborative editing tasks, as shown below:
Organisation of communication space during editing tasks (above) vs recording tasks (below)
The themes showed distinct differences in the way the producer and performer used the tools around them to reference audio playback. The producer used the audio playback as a direct reference in his utterances, resulting in quickly-grounded interactions:
Extract from a conversation analysis where A is the producer and B is the performer
Whereas, the performer had to employ a wide range of communication tools (mimicking and gesture), often resulting in mis-communications:
Extract from a conversation analysis where A is the producer, B is the performer and C is the researcher
The performer repeatedly struggled to reference specific sounds in the audio. Replaying sounds was an effective way of referencing audio and even though the performer had the technical skill to use the DAW, he never opted for this option.
How could I design to help the performer to reference audio during interactions?
Software Development
VisualTalk uses Max4Live and Processing to analyse audio within Ableton and form live visuals representing each track. The audio output from each track is routed through a Max4Live patch which analyses the pitch, loudness and spectral centroid. These parameters were carefully researched and chosen based on their psychoacoustic qualities; changes in all three relate strongly to changes in how we perceive sound. These values are then sent to a Processing sketch via OSC messages. The values are used to control the x-position, diameter and colour of circles. Each circle represents a track. The visualisation creates a common language which can be drawn upon to reference audio. By clicking on a circle the user can solo a track. This interaction breaks the single-user paradigm of a laptop, allowing the performer direct control of tracks to help reference sound.
Data structure of the software
Evaluation
I returned to the original duo and conducted an evaluation of the initial software design. I took videos of their interaction with the software and conducted interviews.
Shot from the evaluation and a close-up of the software screen view
Analysis of the evaluation spurred a design for a new iteration in which:
1. The colours would be coded to reflect the emotional quality of the chords being played
2. A horizontal view showing the circles changing over time would be adopted to show the brief history of the changes to add context
Publication
An extended abstract of the initial field notes evaluation was published at Creativity and Cognition conference in June 2023 where I presented the project poster.
The research is being edited into a full-length submission.
Here is a track I recorded with the duo in-between all the research.