Virtual Reality: Give Us a Visual Clue

Mike King, London Guildhall University
kingm@lgu.ac.uk
July 1996

Introduction

3D computer graphics routines can be characterised on a spectrum from physics to faking it. The more physics involved the more realistic, though slow, while the more faking the faster. We also see that the more faking required, the more the sensibilities of the artist are needed. Real-time virtual reality, with its high demands on processing power, involves more faking it than in other 3D computer graphics applications, but the fact is that the artist's eye is needed across the spectrum, and this is only for appearances: the artist's wider cultural antennae are also vital for content.

This paper examines the range of the artistic skills necessary in VR, and at the same time looks at some of the philosophical implications of the technology. 

What is VR?

There are different ways of defining virtual reality. For the purposes of this paper I will consider it simply as an interactive technology that recreates some aspect of human experience. Not all would agree that interactivity is essential to virtual reality systems, but without it we leave out a prime element in human experience: free will. Free will, from a philosophical standpoint, is a difficult thing to pin down, but a virtual reality system that leaves out any possibility for the user to make an input turns it into a version of cinema. However, as this paper focuses on the visual clues that make us accept a computer-generated version of reality as worth engaging with, the interactivity question is not so central. How systems can model aspects of human experience and what 'real' reality is becomes relevant however.

Reality and the Role of the Brain

One of the reasons why the subject of virtual reality is so interesting is that it can challenge our thinking about ordinary reality. In fact, the question of what 'reality' is and to what extent the mind and brain construct it for us is one that has engaged thinkers all through history, and recent research has made the answers no easier to come by. Some groups, often with a religious or mystical background, suggest that the world we encounter through our sense perceptions is less real than their essences (Plato's 'forms'), while others propose a spirit world which is more 'real' to them than the material (Rudolf Steiner for example). The Scottish philosopher John Hume was much taken up with the problem of how we seem to construct an orderly world from disparate and jumbled sense impressions; he found that the more one concentrated on them the more the world seemed to fall apart in fact. In the end he despaired of a solution and suggested (perhaps ironically) that 'the only remedy was inattention'. The work of contemporary philosopher Mary Warnock has explored this theme further, with reference also to Kant. Most empiricists however are quite certain that there is a real object world 'out there'. Empiricists, unlike Plato and his followers, believe that knowledge of the world can only come from careful observation of it as presented through the five senses, and through instruments, such as the telescope or microscope, which extend them. Interestingly not all scientists hold this view, Erwin Schroedinger for example saying that 'the world is given to me only once, not one existing and one perceived'. To firmly hold either one or the other view seems unreasonable to me however; it is more interesting to suspend final judgement for the moment and look at what open-minded research is telling us.

One interesting area of research that poses problems for the question of a 'reality-out-there' is dream studies. Brain researchers have found that the chemical and electrical activities in the human brain are different and distinct in the three main states of consciousness: waking, REM sleep, and dreamless sleep. (REM sleep is named after the Rapid Eye Movements which takes place in dreaming). Research has shown that the visual cortex of the brain (the part involved in vision) is stimulated in dreaming as if impulses were coming from the eyes: the brain is 'constructing' a visual reality along with the tactile and emotional drama of the dream. Even more surprising is the fact that the dreamer responds to the dream environment by generating motor signals that would normally cause walking, running, fighting and so on. A chemical that floods the brain during sleeping normally inhibits the transmission of these motor impulses to the rest of the body, though with one exception: the eyes. Technically the body is said to be in a state of suppressed muscle tone. Sometimes, if the impulses are too strong (as in a nightmare perhaps) the dreamer may move or speak, or even wrestle with phantom opponents. Genuine sleepwalking is an extreme case of this. Interestingly, the same is true of many higher mammals, cats for instance. One can often observe a cat's tail twitch as it sleeps, and science has shown that it must be dreaming because all the chemical and electrical states in the cat's brain correspond closely to those in humans.

A form of dreaming where the participant can control their dream environment is called lucid dreaming, and has been the subject of recent research. It has been discovered that most people make good subjects for lucid dreaming and can be trained to gain degrees of control over their dreams. They have even been able to receive messages from lucid dreamers via eye movements, as these do not share the suppressed muscle tone of the rest of the body. The messages have been used to confirm events in dreams, and these moments can be correlated with measured brain activity and the dreamer's subsequent account of events. A group in the US have been working with artists and lucid dreaming with interesting results on their paintings.

The conclusion of some who are involved in this research is that 'waking is like dreaming with the imagination suppressed, while dreaming is like waking with muscle tone suppressed.' The relevance of this to virtual reality is partly philosophical, and partly practical, as there is no doubt that the impetus of VR research will be to move inexorably from modern total-immersion systems involving computer graphics visual systems to what is loosely called 'brain-stem' virtual reality where the user is 'plugged in' directly to the technology. Such possibilities remain at present in the domain of science fiction, such as Neuromancer, or in the wildly speculative physics of Frank Tipler in The Physics of Immortality.

A Taxonomy of VR

Before contemplating the future possibilities of brain-stem VR, let us draw up a rough and ready taxonomy of virtual reality systems, so that we can better see where computer imaging fits in at present.

•	text-based, such as Muds and Moos over the Internet: keyboard and on-screen text - input via keyboard, feedback via text on screen
•	graphics-based,  such as games: input via mouse or joystick, feedback via graphics on screen, sound cues
•	professional simulators, such as aircraft, ship, tank and road vehicle systems: input via mock-driving controls such as steering wheel, rudder, flight console, feedback via graphics on wide screen, with the addition of sound and possibly kinetics such as cockpit judder, lean etc.
•	low-level immersion, such as homebrew VR kits, arcade VR games consoles: input via a range of devices including the dataglove, bodysuit, spatial positioning sensors, feedback from head-up display, audio cues.
•	high-level or 'total' immersion: same as the previous category but using state of the art devices and supercomputer processing power.
•	brain-stem VR: a direct connection between a computer and our brain via synaptic connections (an idea popularised by William Gibson in Neuromancer).
•	up-loading the mind: leaving the body behind altogether and digitally encoding (uploading) our personalities into the computer (a highly speculative idea proposed by the physicist Frank Tipler).

Constructing a World

Whatever level of virtual reality we deal with the creators of the system need to construct an artificial or imaginary world. With the exception of text-based systems this world is presented via computer-generated imagery of varying sophistication. At its simplest a world is 'modelled', that is a database of elements and their relationships and behaviours are constructed, and then 'rendered', that is made visible through the use of computer imaging. As our normal world is three-dimensional most VR worlds are modelled in 3D. In text-based systems the model is described textually, and the 'rendering' of it takes place in the user's imagination, ranging from a typical dungeon in a castle to the hot-tub in LambdaMoo.

Total immersion: Brains in Vats?

Before looking more closely at computer graphics issues in VR, I want to make another philosophical detour. Daniel Denett tells us in his Consciousness Explained that philosophers in the last century were interested in the 'brain in a vat' problem. The idea here is to explore the philosophical implications of a scenario where a mad (or evil) scientist has scooped out our brain, placed it in a vat, and connected us up to electrical apparatus which receives and returns synaptic impulses so as to fool us that we were living an ordinary life. Denett briefly mentions VR in this context, but we are in a position here to pursue this a little further. The main technical objection to the scenario is in what is known as 'combinatorial explosion'. Because the brain's owner may make over a period of time so many unpredictable decisions (because it must still be allowed to exercise 'free will' for the illusion to work), it would become impossible to calculate all the possible activities of the person, and so be able to provide electrical impulses that covered all the eventualities. For example, one might walk the same route to work every day, which means that the mad scientist only needs to construct part of a city (in computer terms: model and render). But what if our person in the vat decides, on impulse, to turn down a side-street one day? Or, far worse, suddenly decides to go on a holiday to a foreign city and, again on impulse, turn into an obscure alleyway? Our scientist would need to reconstruct the entire planet and all its inhabitants in order to guarantee that the illusion never broke down.

The scenario was dropped by philosophers because of this basic objection to it, but with VR we have to consider it again. Even the most basic of computer games programmers is in the mad scientist's position: how much of a 'world' is one going to construct for the interactive participant? We find even at this crude level of VR an answer to combinatorial explosion: procedural modelling. This is a technique whereby, for example, cities can be constructed using a rule-based system: by abstracting out the main principles whereby cities grow and their elements are constructed and appear to us, we can generate cities (or whatever) 'on the fly'. In addition we need (in visual terms) to be able to render any view of these constructed environments on the fly, but this is a separate problem requiring only that there is adequate processing power. (For the brain in the vat an inadequate external processing system might result in 'picture loss' if turning one's virtual head rapidly.) Recent advances in computer graphics modelling and rendering techniques mean that the philosophers will need to revisit the brain in the vat problem.

In The Physics of Immortality the physicist Frank Tipler makes the assumption that computers in the far future will be powerful enough to serve a vast number of 'brains in vats', in fact every one who ever lived (or might have lived). Tipler does not in fact suggest preserving peoples' brains: his theory proposes that our personalities (or souls or whatever) are uploaded into the computer itself, and that the machine will be powerful enough to simulate life for everyone who every lived – a digital immortality. This is the most extreme form of VR yet proposed, and, although Tipler argues for it in great detail (with considerable support from equations and physics) most scientists are highly sceptical of his ideas. Probably the biggest stumbling block is the question of the computability of mind, that is whether the mind/brain functioning can be simulated, even in principle, by a computer (or technically a Turing machine). This is a very interesting debate which I discuss further in Artificial Consciousness – Artificial Art. The next section will throw up further doubts about Tipler's proposal.

Physics vs. Faking it

Apart from text-based and 2D VR systems we are mainly dealing with 3D environments, all of whose features can be expressed, understood, and coded via the laws of physics. In practice however this is rarely done because of the amount of computing power required, and the whole evolution of computer graphics has in fact been a history of 'faking it.'

Faking starts at the modelling stage, where an internal representation or 3D database is built of the artificial world. Various techniques exist, but the commonest method is to represent objects using polygons. A great deal of skill is involved in modelling objects to an appropriate level of accuracy using the minimum number of polygons: in the games industry for example typical 3D scenes may have a maximum polygon count in the low hundreds to a few thousands. An artists' job working on such a game may involve the modelling of a realistic looking car with just 30 polygons. This is a skill that artists develop from their traditional training in the arts, and involves an intuitive understanding of how visual clues work, and how to abstract the key visual characteristics from complex natural phenomena. What if we were to turn to physics to help us in the modelling stage? If we really wanted to build a virtual reality that imitated beyond any doubt the real world then we probably would have to use physics – right down to the molecular level. The appearance and behaviour of objects depends on this: the exact distribution of mass in a car for example determines the way it corners; the exact distribution of pigments and carriers in the car's paintwork determines its finish (and whether the car looks new and expensive or old and cheap). This, for me, is another of the objections I have to Tipler's virtual universe: a convincing reality requires modelling at the molecular (or even atomic) level, and you would need a processor for every molecule or atom. 'Molecular computing' as it is called does look in fact like a possibility, though a remote one, but even if we could build an information processor at the molecular size, we would land up needing one per molecule in our model: in other words you would need a whole universe to model a universe! 

A more realistic use of physics in modelling has been mentioned earlier: procedural modelling. This requires again that we abstract out some high-level behaviour from a natural phenomenon, for instance rules about the way plants grow, clouds form, and so on. One common technique in this area is called particle systems, where simple physical laws are used to control large numbers of particles (represented usually only by a location and a small amount of other data such as mass, direction, speed, and colour). This is a solution somewhere between (in terms of the processing power needed) molecular representation and polygon modelling.

However, modelling plays only a small part of the overall process of producing 'cheap' but convincing VR. Turning now to the rendering stage of computer visualisation, we find that even more faking it goes on, though the physics in fact is well-understood. There are two main areas for faking: firstly in a range of texturing techniques that are used to make up deficiencies in the modelling, and secondly in the rendering itself. The earliest faking of a 'realistic' scene derived from the polygon representation of objects: simply shade each polygon according to how much light falls on it. This gives a crude faceted realism to a scene that a wire-frame rendering can never have. Progress in rendering involved better faking: if shading levels are averaged across the polygons (Gouraud shading) there is a better illusion of smoothness, if polygon normals are interpolated (Phong shading) specular highlights can be introduced to make things look shiny. These are called shading models, and they work in conjunction with lighting models. The simplest lighting model is a classical bit of faking-it given the grand title of Lambert's lighting law. What was needed was a simple mathematical relationship that took the angle between the light direction falling at a point on the surface of an object and the normal to the plane and calculated a light intensity. When this angle is zero the intensity should be a maximum, and when the angle is ninety degrees it should be zero; what, asked Mr Lambert, is a simple mathematical function that does this? Answer: the cosine. The result works, that is it gives reasonably realistic 3D shaded objects, but the maths used bears no relation to the physics of light involved.

A more sophisticated strategy called ray-tracing allows more of the laws of physics to be used, specifically optics. This is a shading technique that can be used with different lighting models, as with the previous shading techniques, but much more realistic results can be obtained. Ray-tracing follows imaginary rays of light through a scene, but in reverse, starting with the eye-point, moving through a pixel of the final image into the 3D space where it may encounter objects, bounce off them or pass through them in differing proportions, and finally encounter a light source. A development of this called radiosity allows for subtle lighting effects to be computed based on the equilibrium of light energy in confined spaces. Ray-tracing takes much more processing power than the rendering techniques described above, and radiosity calculations that much more again. The kind of imagery obtainable is described as photo-realistic, and, as this name suggests, the artistic sensibilities required to work successfully with it are closer to that of the photographer than painter.

Because ray-tracing and radiosity techniques are so processor-intensive they tend to be used for still imaging and animations with high budgets. VR, with its requirements for real-time rendering (absolute minimum 6 frames a second, but preferably 25 up to 60) rules out both of these techniques for the time being. In fact, even more conventional rendering techniques are often too slow, which is where texturing comes in. In any VR system there will be a polygon 'ceiling', that is a maximum number of polygons that the system can render at acceptable frame rates. To improve on the crudity that this would otherwise impose on the look of the virtual world, digital images known as maps are added to the polygons. The most important are texture maps (images such as wood or brick), transparency maps (providing holes in objects) and reflection maps (making objects look shiny). In each case the application of the map reduces the polygon count, often quite dramatically. In a typical top-end flight simulator one might need trees at the end of a runway for realism, but modelling a realistic-looking tree with thousands of polygons is out of the question. What is the minimum number of polygons that we can use to simulate a tree and get away with it? The surprising answer (in the specific context of the flight simulator) is – one! A single polygon with the appropriate texture map (showing trunk, branches and leaves) and an appropriate transparency map (a silhouette of the tree) makes for an acceptable tree. This works because the angle of approach to the tree when landing will have only minute variations – otherwise it would soon be apparent that it was a cardboard cut-out. The artistic skills here, as in a range of VR applications with limited view-points, is more that of the stage-set designer.

If we can use ray-tracing and radiosity we can obtain very good shiny-looking objects because of the accurate reflections in them. Interestingly, even here, a degree of faking is often needed because of the cost of modelling enough world to make realistic reflections. A spoon, held up to the face in a busy street, reflects large chunks of city especially if moved – a modelling task that can be avoided if the reflections don't have to be accurate. The fact is, and artists know this, that the reflection in the spoon is read by the eye/brain merely as a visual clue: the eye is not focused even on the reflected image but uses it in determining that it is a shiny metal (and whether or not it is clean and suitable for eating with, for example). The geniuses of faking-it quickly spotted that almost any reflected image would do, and many systems merely provide a typical indoor reflected image (usually containing a window) and a typical outdoor reflected image (usually with a sky and ground). You've guessed it: another map is used, the reflection map. (For a good discussion of these techniques and how 'chrome' has come to symbolise modern computer graphics, and how the artist's sensibilities are involved in this see 'The Chrome Age' by Sonya Shannon.)

If we consider now what this spectrum of physics-to-faking-it implies we realise that it is all about trade-offs. Faking it gives us speed; physics needs a lot of processing power. Hence the demands of VR for real-time graphics means, for the foreseeable future that more faking than physics is needed. Does our spectrum imply as processing power increases (it will need to by orders of magnitude) that the artistic input will be less? I don't believe so, but the type of artistic skill will change: real-time ray-tracing and radiosity will require the eye more of the photographer than the painter.

Artists' Visual Clues

We have shown that both the modelling and the rendering stage of VR graphics require the artists' sensibilities, given that until Tipler's world becomes true we cannot rely on physics alone to build our virtual reality systems. Let us explore how pre-digital art has exploited a similar strategy of visual clues, albeit rendered in paint or other artistic media. Artists have always abstracted key visual features of the natural and man-made objects that make up their compositions, though it is only in this century that the degree of abstraction has made a quantum leap beyond its previous parameters. If we stick to representational art however, it is clear that the artist somehow 'models' the objects to be represented in their minds, and then uses visual clues drawn from a stock of such clues that comprise a language of the visual. 

When an artist wishes to represent a scene, whether drawn from real life or imaginary, they have to consider how it is lit: for the picture to be convincing there will generally have to be a global light direction which determines which side of objects will be brighter than others, and how shadows fall. Many decisions regarding light have to be taken (perhaps unconsciously), but they are all readily translatable into 3D computer graphics. While a painter has a greater freedom and flexibility to use light in a personal way than with a 3D computer system, he or she still has to be consistent and follow certain rules otherwise the marks made on the canvas will not be readable. Visual clues can cancel each other out (though of course illusionists such as Escher can make use of this.)

Once global decisions are made about lighting, from which many of the necessary visual clues and their execution follow in paint, or other media, objects within the scene have to be differentiated using different surface qualities. Thus apples are shiny (a dab of white is required for the highlight), a carpet is dull but textured, a person's face rich in subtle colouring, wet things reflect other objects.

Reflection in water and other reflective surfaces has long been a stock trick for artists; it is instructive to look at master cartoonists like Giles or Steve Bell to see how, with a few simple vertical strokes, they transform a dry city pavement into a wet, sodden, slippery surface. Refraction, the bending of light through transparent objects, is another easy trick: just shift around the background a bit, or, if it is a severe case, like a bottle or glass sphere, just use background elements in a murky swirl. (Incidentally the long-running Smirnov vodka ads use no refraction for the scene seen through the bottle — probably for clarity —, and it does not detract from the message.)

Artists also use what is known as depth cueing to give the impression of space: objects in the distance are reduced in clarity because of atmospheric effects such as haze or fog. The simplest visual clue here is a desaturation of colour (see Shannon's article again for a discussion of this). One of the difficult problems in painting is to give the elements of the composition sufficient separation, because, lacking stereo vision and parallax, the viewers' normal strategies for disentangling objects are restricted (this comes back to Hume's perceptive comments about how on earth we extract discrete objects from an environment served up to us as a 'porridge' of visual sense impressions). Some painters use thick black lines around objects, while others use a variety of depth clues, including of course the natural one of scale.

Finally, objects that move rapidly in a scene can be given the illusion of motion by blurring them. All these techniques, including this motion blurring, are translatable into software techniques for VR, and all of them require the artists' eye to work successfully.

3D Studio

Given that 3D graphics systems whether for VR or anything else, have to provide the user with a balance on the physics-to-faking-it spectrum combined with a reasonable estimate of the artistic skill the user will have, how do well-known systems compare? How do their designers pitch the functionality of their systems? They also need to take into account how much physics a user knows.

Materials: just provide a name, e.g. marble, brushed aluminium, and leave it to the system? Or provide the tools to create these? The exact balance is difficult, and is matched with the difficulty throughout the modelling and rendering options that can be provided. At London Guildhall University we use a 3D modelling and animation package called 3D Studio, which shows an interesting range of choices in the modelling and rendering functions offered. The software is designed to run on a fast PC, such as a Pentium, but the performance of the average machine is way below that provided by workstations such as the Silicon Graphics machines. Hence ray-tracing is only provided for accurate shadows, and the rest is rendered by a scan-line algorithm.

Virtual Worlds and the Artist

So far we have discussed mainly the artisanship of the artist in constructing acceptable virtual worlds, acceptable from a visual standpoint that is. The real challenge of VR is to harness the available technology at any stage in its development to provide an experience that really engages. It is obvious from people like Tipler that there is a desire to play God, to 'improve' on real reality. This impulse may be the same as the Utopian one that appears through history from Plato to Aldous Huxley, but it may well be that the more utopian the virtual world the more boring it will be. In other words the ground-rules for life in the real world seem to provide for a pretty engaging drama, and it is more likely that good VR scenarios will not abandon the age-old bugbears of the utopian dreamers, such as evil, profit and loss, and death. The artists' role in VR is much greater than the merely visual, encompassing all the creative arts, many of which are akin to those found in theatre or film production. The visual artist has, however, a key role in the look and feel it of it.

Conclusions

We have seen that virtual reality systems comprise a spectrum of applications, most of which rely heavily on the visual. Because of the processing power necessary to provide sophisticated computer-generated imagery, much of it is achieved through cheaper forms that can be called faking. Computer imaging techniques are themselves drawn from a spectrum that can be described as 'from physics to faking it', and the right choice is critical for a workable VR scenario. The skills of the visual artist are essential in working across this spectrum; at one end the visual clues used by painters and illustrators have to be understood, while at the other end the sensibilities of the photographer are essential.

The philosophical implications of VR are another challenging aspect to the artist, in that the designer of a virtual world has to have a profound grasp of the drama of the real world. I leave you with a philosophical thought: would you really find a virtual world designed by utopian thinkers like Plato or Tipler more interesting than one designed by a dramatist like Shakespeare or Hitchcock?

References