MooZk by SonicViz

 

Forces

“I believe that all the individuals who have created works of value aren't inventors, but catalysts of existing forces. They simply know how to look and to understand.”

Zbigniew kaakowski

Sensorband

 

This deceptively simple quote, which I came across during my initial research, defined my applied research approach to this project. The hard part, which doesn’t take a rocket scientist to figure out, is learning how to look and understand.

Aims, Expected outcomes and Significance

 

The aim of this project was to investigate new computer interface metaphors for the

real-time and simultaneous performance of dynamic imagery and sound.

 

In terms of outcomes, the project was designed to provide the groundwork needed to develop a new generation of commercial interactive audiovisual products including, but not limited to:

 

The focus of the project was therefore on rapid software prototyping to enable the elicitation of the key Human-Computer interaction issues, which will possibly contribute to the success of the aforementioned products in the emerging marketplace.

 

An implementation of the agile development manifesto (http://agilemanifesto.org/) was chosen based around the four phases of Project Inception, Elaboration, Construction, and Transition.  Within the Elaboration and Construction phases an Iterative Feature Driven Development cycle was implemented to manage the development of the actual prototypes.

Project Inception

 

An intensive literature and product review was conducted in the following main areas:

DesignGoal

Figure 1 SonicViz REV design goal: Synaesthetic musical bonding of Sound and Image

Main resources (130+) were ICMC and ACMA papers, MIT Media Lab, IRCAM, SIGGRAPH and SIGCHI, innumerable books, Computer Game industry, and the music and VJ communities with associated tools and products.

 

All these resources were necessary to build up a knowledge base from scratch (being a first time virtual instrument builder) of a very broad problem domain.

 

Java was selected as the development language for its cross platform nature and emerging status as an embedded systems language. The KOAN generative music engine was also chosen at this stage due to its accessibility, commercial positioning and extensive API.

 

Project Elaboration

Three Java prototypes were developed to test concepts in Musical aLife, soundtoys, and sound/music composition through navigation of VR environments. Based on positive feedback received from international testers of an early concept test applet it was decided to proceed with development of a simple soundtoy for the REV festival, as it seemed to have most potential to achieve the project goals within the constraints of being immediately accessible to the general public in a festival context (many thanks go to Linsey Pollak for this insight). 

 

Project Construction

 “MooZk is a visual-music soundtoy that uses a pen tablet to draw sonic textures onto a constantly regenerating digital canvas”.

 

A design decision was made to use a combination of robust proprietary, open source, freeware and developed software for the REV throwaway prototype. These components had been located and evaluated during the inception and elaboration phases. This was for practical reasons to allow a reasonably complex application to be developed by one person in a short timescale, but was also an experiment to ascertain the commercial implications for possible products developed using such a range of components. This is becoming a common business model nowadays and I figured the only way to learn about the real technology management issues (licensing, integration, IP control, support, etc) was to “learn by doing”. This also involved signing NDA’s with some technology providers to gain access to new technologies, in addition to obtaining public performance permission.

Components used directly or indirectly:

 

The other main critical success factor in developing this prototype was “test early, test often”, both for technical and aesthetic performance evaluation. The rapid feedback from a diverse group of external testers enabled me to steer the REV prototype towards a successful evolution.

Project Transition

It was decided early on to “go big” and project the visual interface to create an immersive experience as possible for the festival. QUT and the Powerhouse staff did a fantastic job with location and equipment that largely contributed to the success of this. We had a few audio conflicts, but it’s not really a music festival if you can’t crank the amp up now and again, is it? 

The response was generally positive, and it achieved the core objective of stimulating intelligent responses from a wide range of perceptive people for suggestions of possible applications in the emerging visual-music instrument field. These are now under review to decide if it is feasible to develop any commercial or open-source applications from this early research – which is a whole different ball game.

 

Main points of learning that I see as being fundamental to the successful design and development of visual-music systems are:

·        An understanding of the role of gesture for performer and audience. Knowledge of current and emerging gestural control trends in terms of sensors (input), processors (hardware/software), and feedback devices (output) is essential.

·        Visual Literacy skills, which can be developed through problem solving books such as “Visual Literacy” by Wilde and Wilde (ISBN 0-8230-5620-1). These skills are important for designing possible visual-music compositional frameworks and performance storyboards.

·        Knowledge of Interaction design, Human Factors, HCI (Human-Computer Interaction), Usability engineering, and Communication Design. To my mind these are all different lenses through which to view more or less the same problem domain, but all have slightly different takes which are valid dependant upon the context of application usage.

·        A deep knowledge of Patterns and AntiPatterns at all levels of use. Patterns help you identify procedures, designs, and codes that work. AntiPatterns do the opposite, but are just as important to help speed development time and leverage other people’s knowledge. They also help to be able to quickly analyse, filter, and categorise a broad range of resource material, an important component of commercially focused R&D.

·        Generative music composition knowledge through use of commercial systems such as Koan. Open source packages such as Jmusic are also available for experimentation.

·        A solid understanding of development methodologies and toolsets. When you need to “shift gear” in terms of development, you may need to adapt your development methodology to suit the changed context. Appropriate toolsets are critical to supporting and managing this.

 

I’d like to thank the REV teams and students at both QUT and the BrisbanePowerhouse for their invaluable help. Special thanks to Pete and Tim Cole from SSEYO for their help and the superb KOAN engine.

 

Have fun,

paulcohen@ozemail.com.au BSc, MTM

www.sonicviz.com