The Production of the Synesthetic Experience

The D.I.Y. Guide to Digital Creativity

In the beginning there was John Whitney, (along with a few others), computer graphics pioneer and author of 'Digital Harmony'. He was one of the first proponents of the Computer as a medium that would bring music and 'dynamic visuals' together to form what has been described as a synesthetic experience, whereby, electronic music and electronic colouring-action combine to make an inseparable whole that is much greater than its parts.' John Whitney Sr., 1990

This vision of the medium is unfolding at present right under our noses; the relevant means and methods of production are currently in their most crucial stages of evolution. A new medium which will engage all the senses in a rich 'total' experience is on its way. The experiments of today will establish the form of delivery for tomorrow. Technology is advancing and is only now beginning to catch up with the expectations which have recently so often disappointed.

As with any new medium an understanding of its potential generally is not realised until the pioneers have experimented, made mistakes and taken the initiative to do something that nobody would have thought possible. When analysing the potentials of the computer medium academics are at pains to express the parallels with the early days of film, whereby the medium only came of age when the film makers understood that to engage an audience film relies on a narrative, i.e. that it was not moving landscape painting. What we, the producers, should now be doing is asking ourselves how do we go about creating these new experiences, what are the essential components and how do we combine them.

A more useful analogy to make, is that the production processes of creative computer applications will be very like those employed in the film industry. A film maker needs to employ a wide cross-section of disciplines-. photography, lighting, audio, theatre, narrative, editing, etc... Similarly in the computer industry applications will require people from backgrounds in computing of course (whatever this means), the visual arts, music, literature and so on.

The one commercial area that is beginning to pave the way forward, in respect to the development of the computer as a truly creative medium with potential for artists, is the games industry. The games Industry is making use of an increasingly broad cross-section of skills in its production processes. There is a sense that it is coming of age. This feeling goes hand in hand with the present development in `console' technology (gaming machines): the coming of the 'third generation platforms' is upon us. These machines are the Sony PlayStation, the Sega Saturn and the Ultra 64. The miniaturised machines herald for the computer industry what the WalkMan' did for the personal hi-fi industry, and it is no surprise that Sony is now leading the field. The performances rival if not exceed present high end graphic workstations. Comparing the price of a games machine to a workstation just does not make sense. The next generation platforms will retail at between 5200 and 5400, whereas a workstation could be anything between 52,500 and 525,000. The reason for this is very straightforward NUMBERS. Statistics aside, will these machines live up to our expectations? And more importantly what kind of titles will developers create to ensure that they do?

These new machines will enter the living rooms of millions of people globally, and for the first time will provide computer generated interactive graphics of a quality approaching broadcast. It is this quality issue in interactive graphics which has been a threshold which needs to be transcended for the creation of engaging interactive computer based applications. There has been a widespread resistance beyond the computer gamers to the medium for this reason, among others. The improvement in visuals will distract the emphasis from hand to eye coordination, shooting and jumping and hand the floor to entertainment and the enjoyment of what will be an experience: a combination of narrative, gameplay and sumptuous visuals and audio. Players will neither be passive viewers nor nervous response mechanisms; they will be involved in an experience which is both interactive and visually engaging, which will engender the suspension of disbelief.

Developments in sophisticated graphics software are fuelling this impetus towards the interactive medium. 2D packages such as Adobe PhotoShop are leading the way in still image creation and manipulation. In the 3D world the field is led by Alias and Wavefront, both recently purchased by Silicon Graphics to consolidate its leading position in the computer graphics hardware and software industry. Up until the past couple of years 3D character animation has been a difficult and frustrating task, now with these tools sophisticated characters with attributes such as flexible skin, particle system hair and intelligent collision detection, mean that convincing characters may be created. Intelligent characters exhibiting autonomous behaviour in a virtual environment will also be facilitated by libraries of AI behavioural software which will free developers from the arduous task of repeatedly reinventing the wheel. A company called 'Katrix', advertising in ,Virtual Reality - Special Report', Spring 1995, offer software 'that builds characters'. The Libraries are designed to short circuit programming for 3D computer games with 'C' programming libraries going by the names of 'Limb Coordination', 'Rule-Based Control' and 'Neural Network'. This is not a plug (honest), it is an indication that the industry is wising up to the fact that producers of experiences need sophisticated yet high level tools. On the graphic programming front developers are also realising the importance of optimised 3D graphical APIs, again releasing them from writing code which already exists to concentrate on the form and content of their applications. The analogy could be made to painting in that very few artists will mix their own paints; although some painters would argue that you should. This point will be an issue of contention for the foreseeable future between programmers as there are always those who will argue that you would not be able to add your own unique features. The argument between assembler and C language programmers rages on. This will increasingly become a moot point as standards evolve. There are only so many ways you can draw a polygon to screen and it is unlikely that any developer creating their own polygon filling function from scratch would be able to make it more efficient than a team of people whose job it is to spend all their time optimising low level graphics routines.

Although these software and hardware developments may appear without creative significance, they are the developers' tools of the future. From a commercial perspective game producers and multimedia developers will not be able to compete for the market of cyber consumers if they are not creating the most impressive graphics or the most convincing and autonomous characters. Similarly artists, or if you like the equivalent of the film director for the games industry, will have to engage with issues of quality as their efforts will always be judged by what is state of the art. It should not be ignored that being at the front also involves state of the art prices. Lessons could be learnt from the film industry with regards to this, computer experiences take many skilled man hours to be developed and need to be funded accordingly to produce fulfilling experiences. Film directors manage to fund avant-garde movies, so why shouldn't computer 'experience' directors be able to raise the necessary capital?

The importance of these software and hardware tools cannot be overemphasised. A director of 'experiences' must have an overall picture of what all these tools are capable of and what the limitations and potentials are. They also need an overview of the totality of the production processes and understand how to integrate the graphics, programming and sound. The creative team should not be in the business of creating tools. This has been predominantly the case up until recently. Animators would write their own software extensions for existing packages to give them a unique look and feel over their competition, it has been a case of my Ray Tracer is better than yours. A similar scenario exists in the games industry. This will not last much longer. Animators and designers should be concentrating their efforts on design and content. . High level authoring tools already exist for creating interactive audio-visual experiences for top of the range Silicon Graphics machines. Wavefront has recently developed a high level game authoring package called 'Activation'. Obviously optimisation and 'fudges' are still required to create experiences of acceptable quality for domestic hardware, but it will not be long before the hacker programmer is taken off games design and concentrates on tool development. In the industry Argonaut is already doing exactly this, there are now two divisions to the company: one concentrating on software tools, namely their 3D API called 'Brender' (or Blazing Render) and another in game development.

In the art world where funds are not comparable to the games or film industry, the notion of collaboration between people of different disciplines takes on a new meaning. Visual artists, programmers, composers could all work collaboratively on these new synesthetic experiences. Interactivity is already a strong theme in many contemporary artworks; other notions such as Virtual Reality and the Game could be introduced, if artists adopted a production model based on that of game development; they would be in a better position to realise their visions and create in-depth experiences. There are only so many hours in the day to eat, sleep, drink, programme in C, compose and record music, create sophisticated graphics, animate, raise funds, source software and hardware, maintain equipment, take photographs, produce and composite video, and generally relax.

THE END