I started on the design of the Virtual World, and on learning how to play
it, within only a month of the performances, so I had to become completely
immersed in the creation process.
I had originally thought the piece would be an elaborate VR "demo",
or explication, with clear visual cues for the music, easy to use interfaces,
and lots of funny, Rube Goldberg-like tricks. But, as I worked on the World,
a mood, or an essence, started to emerge, and furthermore it was true to
my emotional and spiritual experience at the time. This was unexpected and
exciting, even if the content was not cheerful. So, I went with a darker
and more intuitive process instead of falling in line with the more familiar
computer culture of clarity and light humor. There have only been rare occasions
when I felt I was programming in an intuitive way, and this was one.
Don't expect the instruments to be immediately understandable, or imagine
that they are easy to play. Rather, they emerged from a creative process
I cannot fully explain, and I learned to play them. I don't think the two
esthetics I'm distinguishing must be mutually exclusive, of course, but
the intuitive side of the equation can't be reliably willed into action.
A synthesis of clarity and mood will come by grace, when it comes.
The first instrument played is called the Rhythm Gimbal. A gimbal is a common
mechanical construction of a hierarchy of rotating joints. The Rhythm Gimbal
resembles a gyroscope. When it is still it is completely white and completely
silent. When I pick it up and move it, it begins to emit sound. Actually
the sound is created by the rings rubbing against each other- they also
change color at contact. Once set in motion, the Rhythm Gimbal will slow
down, but will take a long time to stop completely. If I give the Rhythm
Gimbal a good spin as I release it, it emits an extra set of noises which
are more tinkly, and which slow down as the instrument winds down. Thus,
unless I carefully release it without any spin, it will continue to make
sounds when I'm not looking at it. The "background" sound heard
while I am playing the other instruments comes from the Rhythm Gimbal.
The primary (non-tinkly) Rhythm Gimbal sound is an interpolation of a choir
and an orchestra and some other stuff. The harmony is generated by the momentum
with which internal parts of the instrument hit each other after it has
been released by the hand. Each ring transmits spin to the ring outside
it, creating a complex motion, like pendulums hung on pendulums. The rings
have beads on them. When the beads collide, they change color, and also
force a change in harmony. You know those old attractions at amusement park
arcades, where you hammer a target on the ground with a giant mallet and
see how high you can send a puck on a big vertical ruler? The internal collisions
of the Rhythm Gimbal are flinging virtual pucks around the circle of fifths,
and up the harmonic series, in much the same way. A note is added to the
harmony when the two types of puck reach it approximately at once. All the
harmony, and all the rhythmic texture, comes out of this process. A slow
twist of the Gimbal will tend to choose notes that are close to each other
harmonically, such as musical fifths. What's interesting about the Gimbal
is the music that it turns out when it is spun with a greater energy. There
is a range of harmonic and textural style that can be explored by practicing
the spinning of the Gimbal, varying from a very open, harmonic, and calm
sound, to a crazed dissonance. My favorite is the in-between zone, which
can sound like a cross between late Scriabin and the Barber Adagio (no kidding).
I was almost appalled for a while at how good this simple gizmo was at generating
harmony. Is this all a composer's brain does? But the Gimbal can't be described
properly as an algorithmic music generator. For example, I don't think an
explicit style of initialization could be used to find the right parameters
to make it sing. There is a necessary element of intuitive performance in
the weird harmonies of this curious instrument.
Every note of The Sound of One Hand is generated by my hand movements, as
they are transmitted through the virtual instruments: There are no pre-determined
sequences or groupings of notes. The musical content is entirely improvised
with the sole exception of the timbral range of the instruments. This does
not mean that I can make any arbitrary music, any more than I could with
any other musical instruments. I can't get a specific chord out of the Rhythm
Gimbal reliably, but I can get a feel out of a chord progression, because
I can influence when chords change and how radical the change will be. This
does not feel like less control to me, but rather like a different kind
of control. The test of an instrument is not what it can do, but: Can you
become infinitely more sensitive to it as you explore and learn? A piano
is like this. Only some mental objects can fail this test. A good instrument
has a depth that the body can learn and the mind cannot. I believe it is
entirely possible for the mind to invent such instruments.
Hidden mechanisms in Virtual Reality are just invisible objects. While I
was developing this World, I would make the harmonic structure visible-
it looks like a bunch of notes crawling on rings and up a pole. I made it
mostly invisible for the performance as a visual design decision. There
is one part which is still visible, though, and that is the large blue ring
with tuning forks on it. Each of the tuning forks has a T-shaped thing on
the base and rings on the arms. These objects are the storage of the current
legal tonic and chords for progressions. You can see them moving as the
The CyberXylo is a mallet instrument. It's notes are taken from the tuning
forks on the blue ring, so it is always harmonious with the Rhythm Gimbal.
The mallet retains angular momentum, with some friction, when it is released.
Thus it is possible to set it spinning so that it will continue to hit the
keys of the CyberXylo on its own for a while. The spin is of poor mathematical
quality: it increments rotations instead of using quaternians. This creates
wild, unnatural spinning patterns. With practice, enthusiastic spins of
the mallet close to the keys can be a source of remarkable rhythms.
The Cybersax is the most ergonomically complex instrument. When the instrument
is grabbed, it turns to gradually become held by your hand, and tries to
avoid passing through fingers on the way. Once you are holding it, the positions
of your virtual fingers continue to respond to your physical ones, but are
adjusted to be properly placed on the sax keys. This is an example of a
"simulation of control" that is critical in the design of virtual
hand tools, especially when force-feedback is not available.
There are three musical registers, soprano, alto, and bass, located along
the main tube. Each register consists of a set of shiny sax-like keys. The
notes played by the keys come from the current set of legal notes defined
by the Rhythm Gimbal, so it will not clash with the other instruments. Once
grabbed, it is possible to slide between registers by jerking the hand towards
a targeted register. The momentum of your slide helps determine which notes
will be associated with the keys until you slide again (if, for example,
you approach the soprano register from the alto with greater force you will
choose a set of notes that are higher up in pitch). You can play freely
without dropping the horn by mistake (this was a hard quality to achieve).
In the upper register it is possible to play two melodies at once, by modulating
with the thumb. The orientation, or twist, of the horn in space controls
the timbre, mix, and other properties of the sound. Other elements of the
design include the obscene, wagging tail/mouthpiece and the throbbing bell.
The Cybersax sound and geometric construction were partly inspired by a
bizarre bamboo saxophone I have that came from Thailand. It is jointed at
the top just like the Cybersax's tail.
Computer music must use instruments which are built out of concepts of what
music is. This is a drastic departure from the "dumb" instruments
of the past. A piano doesn't know what a note is, it just vibrates when
struck. A sensitivity, and a sense of awe, at the mystery that surrounds
life is at the heart of both science and art, and instruments with mandatory
concepts built in can dull this sensitivity by providing an apparently non-mysterious
setting for activity. This can lead to "nerdy" or bland art. It
is interesting to hide oneself behind a piano, as opposed to a computer,
but only because a piano is made of resonant materials, not of concepts.
In order for computer art, or music, to work you have to be extra careful
to put people and human contact at the center of attention.
I was delighted to discover that The Sound of One Hand created an unusual
status relationship between the performer, the audience, and the technology.
The usual use of rare and expensive high technology in performance is to
create a spectacle that elevates the status of the performer. The performer
is made relatively invulnerable, while the audience is supposed to be awestruck.
This is what rock concerts and the Persian Gulf War have in common. The
Sound of One Hand creates quite a different situation. The audience watches
me contort myself in all manners as I navigate the space and handle the
virtual instruments, but I am wearing EyePhones. Five thousand people watch
me, as I display sometimes awkward poses, but I can't see them, or know
what I look like to them. I was vulnerable, and very human, despite the
technology. This created a more authentic setting for music. If you have
played music, especially improvised music, in front of an audience, you
know the kind of vulnerability I am talking about, the vulnerability that
precedes an authentic performance.
About that contorting... I am using point-flying in the performance. This
is a technique of navigating where you point where you want to go and this
causes you to fly there. I dislike point-flying in industrial applications
of VR. It requires skill and uses up your hand. I used it in this case because
I did want the un-constrained, skillful type of navigation; It allowed me
to choreograph a tour of the asteroid along with the performance. I was
completely shocked and embarrassed when I got lost in my own World during
one of the performances!
Another human element of the piece is its physicality. The Sound of One
Hand is in the tradition of the Theremin in that the interface is primarily
physical instead of mental. Although the instruments were made of information,
the music was primarily made of gesture.
The equipment I used was, for the most part, not state-of-the-art, but about
a year out of date. The synthesizers and Head Mounted Display were '92 models,
but the graphics engine, tracker and DataGlove were all older. I think you
have to actively avoid using the latest gear, in doing art, to avoid getting
caught up in technology for its own sake.
(The HMD I used deserve to be noted, since the performance will probably
turn to be the last time it will have been seen. I used a prototype XVR
EyePhone from VPL, my favorite HMD thus far, which is apparently not going
into production. Sigh.)
The software was quite current, however. The piece was written entirely
in Body Electric, a visual programming language for Virtual Reality. I am
extremely fond of this software working environment, which was designed
primarily by Chuck Blanchard. You hook up visual diagrams to control what
happens in the Virtual world and see the effect immediately. All the music
and physics was done in "B.E."; I could never have made this thing
in "C". Visually and sculpturally, the World took advantage of
every trick available for real time rendering, including radiosity, fog,
texture mapping, environment mapping, and morphing. The color "flaking"
effect results from a bug seen when the color of hardware fog on the SGI
is gradually changed (I set up a very slow-moving bouncing ball in the cylinder
of red/green/blue color space as a chooser for the fog color). I sculpted
all the parts of the world except for the illuminated skeletal hand that
sprouts from the asteroid wall, which is from a Magnetic Resonance scan
of a patient's hand taken at the Veterans Administration hospital in Palo
Alto. It was originally used in research on surgical simulation. Just for
the record, the asteroid is hollow, and about twelve feet in diameter, although
the dense fog makes it look and feel immense. It has a big crack in the
side, through which a pair of fireflies constantly frolic, and along with
the skeletal hand, it has big red ginger plant growing inside, and also
a few spotlights. The instruments are generally kept inside. During the
performance I spin the Gimbal at one point and fly out through the crack
for a while to let the audience see how lonely the asteroid is, surrounded
by absolute void.
The sounds were created on two sampler/synthesizers. I decided not to use
the wonderful 3D sound capabilities of Virtual Reality, since they are intended
primarily for headphone use, and I didn't want the audience to be trying
to hear something.
In many ways, The Sound of One Hand was a bigger leap into the unknown than
all of the weird "experimental" performances I had been involved
in New York in the late 70's. I had absolutely no idea if the piece would
take on a mood or a meaning or if the audience would find the experience
comprehensible. The performance turned out to be a cheerful, therapeutic
event for me. It was a sort of a technological blues, a bleak work that
I could play happily. It was a chance to work on a purely creative project
with the VPL family, a chance to treat all of VPL's stuff as a given set
of (reliable!) raw materials instead of as work to do, a chance to practice
what I preach about virtual tool design, a chance to use VR just for beauty,
and a chance to be musical in front of my ridiculously political professional
peer community. It was also a celebration of not having to run VPL anymore.
The audiences were incredibly responsive, and I didn't hear anyone describe
the piece as a demo; It was experienced as music. We had a blast putting
on this performance. Hope the tape gives you some idea.
This article originally appeared in the 10th Anniversary issue of the Whole
Go back to Jaron's