My pal Gregory Perez wrote me this poem on his iPhone for my birthday yesterday. Thanks, man!
A stash of mustache ashes
Flees into Shanghai breeze and trafficAlgorithmic angles
Connecting noise to signalEvery sine surges in wavelengths
Captured alive in self-made CagesThe truth of where this sound begins
Is likely found where Ben has been
Background
Breaking New Ground is an installation I did in collaboration with Jutta Friedrichs of MÜ Furniture to represent 100% Design at Shanghai International Creative Industry Week. It ran from October 16-21, 2008, way up on Lingshi Rd., just north of Shanghai Circus World. My job was the audio component, a sound installation of six discrete channels synthesized in real-time and deployed algorithmically.
Jutta quit her job as an award-winning product designer a few months ago to launch her own furniture company, MÜ. Her public debut was at the 100% Design Shanghai show last June (and she’s racked up something like 15 glowing magazine write-ups in the short time since). 100% Design is an annual design show based in London, and it seems to be a pretty big deal in design circles. This was the first year they mounted one in Shanghai.
Shanghai International Creative Industry Week is exactly what it sounds like. I don’t know how long they’ve been doing it, but last year’s was held at a big, renovated slaughterhouse on the north side of town, super funky building, but kind of an incoherent and sprawling exhibition. This year’s event was held in a brand new complex, one of those just-add-water Shanghai developments that apparently wants to be known as “The Factory.” A month before the exhibition Jutta had to wear a hardhat and climb piles of sand to visit the space, and the morning of the show landscaping was still being installed.
The Concept
Anyway, Jutta’s pals with the 100% Design folks, and they wanted to have a big display at SICIW to remind people of their presence in Shanghai as a champion of cutting edge design, and in particular to plug next year’s exhibition. So they invited her to design their space (a large, tall cement room, 240 square meters), and she came up with a kind of arctic theme, with furniture (highlights selected from the previous 100% Design show) variously floating on or bursting up through shards of Styrofoam masquerading as ice.
For my sound component, the basic idea was wind and wind chimes, to support the artic theme. There’s also this kind of insect-flute thing that occasionally floats in over the top. When we first got the sound up and running and I looked around the room, I was very pleased at how well the sound and visuals collaborated to evoke a unique ambience that was quite distinct from the rest of the show.
The Set-up
I programmed the sound behavior in Max/MSP, and it exemplifies the maxims I’ve developed from doing game audio design for the last 12 years. The sounds are completely synthesized in real time and deployed algorithmically, to ensure that nothing ever loops. The six channels are completely independent (technically superior to Dolby Digital, since Dolby is a compressed format, and I’m sending out 6 discrete channels of uncompressed PCM audio data). The goal was to create an ambient, natural-sounding environment that’s consistent yet unpredictable, analogous to what you might hear sitting on a bench in a park (if the park was in Antarctica).
The sound was generated in Max, came out of a MOTU UltraLite sound card, ran through a Yamaha MG16/6FX mixer (not strictly necessary, but handy for testing and setting up), then out to some big BAL AP1600 amps, then out to the speakers. There were actually 12 speakers, with each channel output from the computer routed to two different speakers. The speakers used were these small, black Bose speakers with no model numbers, suspended along a ledge that ran around the perimeter of the 240 square meter room, about 3 or 4 meters off the ground, a perfect height for this piece. Except for the sound card (which I had to buy at the last minute), the rest of the equipment was rented.
What’s Going On
Harmony
Though it’s not immediately apparent, there is a firm harmonic underpinning to the work. It’s all in just intonation (i.e., small number frequency ratios, no temperament). There’s a fundamental frequency (a low A=110 Hz) that changes on a random timer, something like every 1-2 minutes. It can pick from among 5 different pitch multipliers (1, 1.125, 1.25, 1.375, 1.5), which correspond basically to scale degrees 1, 2, 3, a sharp 4 (not represented on a piano keyboard, but corresponding to the 11th harmonic), and 5. There is a constant low wind drone (filtered white noise) that constantly plays this pitch, doubled at the octave, to kind of anchor the rest of the sound.
All other sounds are multiples of this pitch, and when it changes, it really refreshes the whole piece. I first discovered this effect in 2000/2001, when I was developing a DirectMusic score for a cancelled Xbox project called Jonny Drama (making extensive use of VBScripting to try to get some interesting juxtapositions to arise from asynchronous music deployment). It feels kind of like you’re cleaning out your ear after you’ve become accustomed to hearing one harmonic center for a long period of time.
Wind
There are six independent wind generators, one for each channel of audio output. The synthesis couldn’t be simpler: a noise generator and a resonant band-pass filter (noise~ and reson~). I use a random walk (drunk) to pick harmonics of the fundamental frequency (6th-16th), changing on a random timer (and the timer itself uses a random walk, so sometimes changes are more frequent than others, creating a drunk envelope). Low frequencies are weighted to be more likely than higher frequencies. The first version I did had the pitch constantly changing, but Jutta felt (and I agreed) that the effect was too creepy, a kind of ghost-like moaning, so I decided to have each wind generator swoop to a new pitch, then sit there for a while, with all pitches being multiples of a fundamental, making the whole piece very stable and consonant, while still evoking wind that blows whithersoever it will.
I spent way too much time on a subtle function that allows the wind to swoop to a new pitch in a more natural way. Rather than simply sliding from one pitch to another, it picks a new pitch somewhere close to the desired pitch, then gradually hones in, kind of like a pendulum coming to rest. It’s hardly perceptible in the final piece, but I’m happier knowing it’s there. When the wind’s pitch is swooping, I open up the Q on the filter a bit, so it’s a noisier sound, becoming more pitch-focused only when there’s an important pitch to sound.
Chimes
The chimes are also super simple. When I started work on this piece, I began by trying to develop a software synthesizer modeled on my beloved Roland JP-8000 (still recovering from a power surge at the 2006 Ubisoft company party), but I quickly realized this was not the best investment of my limited time, so I stopped with the basics: simple synthesis objects, amplitude envelope, filter. So in the end all chimes are just filtered triangle waves.
The chime deployment is really the interesting part. There are 3 chime generators, each associated with a pair of wind generators. They’re on random timers to decide when to start. When they start, they pick one of the associated wind generators, and from then on the density is linked to the wind’s pitch (perceived velocity). They have a drunk envelope (i.e., drunk target and drunk time to get there) that tells them whether to get louder or quieter, and if they get quiet beyond a certain threshold they turn themselves off. The chimes are where you can really appreciate the “cleaning out the ear” effect I described earlier, if the fundamental multiplier changes while the chimes are playing.
Insect-flute
The insect-flute is probably the least successful element of the piece. It felt like it needed another layer, but deciding exactly what that layer should be was a bit tricky, especially given the relatively short time frame. Jutta had this concept of “bees in a bag,” to create a feeling of percolating excitement bubbling over, with one bee every now and then escaping, which is a rich concept and could probably be a piece unto itself, but in this implementation, the sound is a little too similar to wind, so it’s kind of in this middle ground that’s not quite merging with the other sounds into a new aggregate, yet not quite putting them in some kind of interesting contrast. I had the idea of something really solid and statuesque that the other sounds would kind of waft around, but it was hard to fit that into this ambient conception; it would be fun to do a variation as a concert piece with an acoustic trumpet or something.
Anyway, bees in a bag instantly suggested granular synthesis, which I’ve been doing a lot of lately, so I created some drunk envelopes to constrain the various parameters and put limits on it, so that if it gets below a certain threshold it will turn itself off, but if it gets above a certain threshold, the length of the grains increases, the pitch and volume variation narrows, and it bursts into song! The “songs” are based on Markov chains, derived from some short melodies I wrote for this piece. This was the last element of the piece I added, and it caused me a lot of grief to debug (and I can lay a small part of the blame on a bug in Max 5.02 that was fixed in 5.05). In my final implementation, I feel the Markov chains still sound too random; I’ve just scratched the surface of how they can be used, and plan to explore it further in subsequent pieces (using second order chains, coordinating melody and rhythm, lots of stuff to try).
Also, I was using a sample of me blowing on a bottle of Qindao beer as a basis for granular synthesis, and the sound is too similar to the wind blowing; it would have been nice to have something more contrasting.
Observations
First thing I noticed from doing this project is that I need a new computer. This program brought my 4-year old Pentium 4 single core laptop to its knees with frequent audio dropouts, but the new laptop volunteered by one of the organizers to use at the exhibition didn’t even break a sweat, averaging around 20-30% of the CPU.
I also confirmed once again that you can’t just send an email and expect your technical needs to be covered. Despite stressing numerous times that we needed a computer and 6-channel sound card, no computer was to be found when the contracted audio team arrived to set up. As far as I can tell, it is only through sheer luck that they happened to be setting up the 12 speakers in pairs to accommodate 6 discrete sends (although I had originally requested 8). The organizer eventually found a laptop, but I had to go out and locate and buy a new sound card for this show at my own expense (and thanks to my former Ubisoft colleague Zhang Lei for helping me track down the card on short notice). Next time I’ll do better to insist that I talk with the audio contractors myself in advance, or even better, supply my own equipment and let them rent it from me instead. At least I was wary enough to start on-site set-up two days earlier than planned, to prepare for just such an eventuality.
I was surprised that it was a bit of a tough sell to do an ambient installation; towards the beginning, folks kept wanting me to turn it up. I think that’s maybe just what everyone in Shanghai is used to hearing, ugly sounds blaring out of store fronts. Subtly’s a hard sell, but from my experience in games (and singing in choirs and going to bars and in many other contexts), volume doesn’t guarantee excitement or interest or anything, really, other than volume; after a while, you just tune it out. A well-made piece doesn’t need to keep trying to draw attention to itself. In the end, I think people came to appreciate what the piece was about, and visitors to the exhibition seemed to like it.
Future Directions
There’s a bunch of other stuff I would have liked to do in this piece, but there wasn’t enough time. But I think a sign of a good piece is that it points you in a clear direction for future work.
In general, I want to work towards more coordinated systems. I think that’s the next challenge, once you’ve got some interesting dynamic behaviors going on: adding more layers and coordinating them. There was some coordination going on already (chime density linked to wind velocity), but there could have been more (e.g., connecting chime and insect-flute start/stop times to some aspect of wind behavior instead of random timers). I also would have liked to make the 6 wind generators work together in kind of coordinated network, to get the feeling of gusts of wind moving through a space.
The insect-flute thing might have been more interesting if there were several layers of it going on, coordinated to do chorale-like things, or even counterpoint. I want to experiment with that more, but adding two more layers of processor-intensive granular synthesis would have been way beyond my laptop’s capabilities, so I’d have to use a different synthesis technique.
It would have been nice to have a more sophisticated synthesizer for my chimes; FM in particular would be an obvious choice for such sounds. I’ll keep working on my virtual JP-8000.
As I mentioned, I barely scratched the surface with Markov chains, but I’m planning to explore them in detail in a subsequent work.
I developed a small crush on Theresa Hak Kyung Cha about a year and a half ago in Los Angeles. While I was in town to kick off our monumental dialog recording sessions for EndWar, I checked out a show called WACK! Art and the Feminist Revolution at the Geffen Contemporary satellite of the Museum of Contemporary Art, down in Little Tokyo.
It was not a great show, which is often true when the message is more important than the work itself. The only pieces to captivate me were two small black and white videos by Theresa Hak Kyung Cha. They were very simple, very lo-fi (dating, as they did, from the 70’s), just some plain images with written and spoken text in English, French, and Korean. They felt very personal and intimate, like I was being let in on a secret, or even sneaking a peek at a diary.
I did a little more research when I got back home and I found out that she died very young, only 30 or 31, murdered a few days after her only book was published in 1982. So I ordered the book, entitled Dictee, to find out more about her work and her world. The book dives deeper than the video pieces I saw, and while it’s much less crush-inducing, it’s notable for a number of reasons.
I want to call it Dictée, but all over the book the title is written without the accent, so Dictee it is. Like the video works, the book mixes French and English and just a few words of Korean (rendered either in Roman letters or in Chinese characters, as used to be the standard for official communication in Korea). From my years of French lessons, I can testify that a dictée is an oral test, during which the teacher reads some text, and the students have to transcribe it as accurately as possible. This act embodies two major themes of the book, memory and language. Language is inextricably linked to identity, and the act of expressing a memory in language and recording it inevitably alters it.
Dictee is broad in scope, using the nine Greek muses to represent the work’s primary divisions, and at other times reflecting aspects of Christian rite. It addresses Korean culture from a national level as well as a personal perspective. The work serves as a biography of several women, not only of the author herself, but also her mother and the Korean national martyr Yu Guan Soon, among others. The French lesson that opens the book builds in resonance later on, as the narrative turns to Koreans living in Japanese-occupied Manchuria, who were forbidden from speaking their native tongue.
While the book is fascinating to me as a student of Asian culture, the aspects that intrigued me most were structural, which is not what I had expected when I first cracked the cover. The book is a multimedia collage, weaving different kinds of text together with images, including photographs, a map, a diagram of the vocal tract, a still from Carl Theodor Dreyer’s 1928 silent film La Passion de Jeanne d’Arc, and copies of letters. At times, Ms. Cha treats the two adjacent pages as two streams of information proceeding at the same pace. Sometimes it’s English and French, sometimes it’s narrative and commentary, and other times the relationship between the streams is more oblique.
The most distinctive feature of the book, and one of the most intriguing ideas, is also the thing that makes it a real slog at times. Much of it is written in a kind of perpetual present tense, evoking a steady state with no forward impetus; this is a real trick to pull off in writing, as reading is an inherently linear activity. But by fragmenting sentences and repeating the same idea with only minute variations of text, Ms. Cha at times succeeds in achieving a sort of constant incredulity, as though she never wants to give the reader time to grow too comfortable with the idea being presented. It’s not hard to imagine why she’d want to do this, when the subject is personal or national subjugation; this specialized writing style never lets the reader lapse into complacent acceptance, keeping the shock and indignation ever fresh. But the problem with trying to make each word a revelation is that after a while, after a while they all start to sound the same.
It did occur to me a few times while reading that this effect might be well served by a non-linear musical setting. I could imagine shuffling up pieces of the text, deploying them in real time, creating a kind of indefinite, almost devotional space, dedicated to rumination and memory, and allowing for unforeseen juxtapositions to emerge through multiple streams of sound.
Then again, I kind of have musical states on the brain a lot these days, so perhaps I’m just finding in the work the kinds of ideas that are already on my mind. But that’s kind of what everyone does, isn’t it?
There are many ways to lead a life in music, and life is probably too short to experientially determine which works best. The one I’ve kind of fallen into is what could be labeled the “commercial” route. I’ve spent the last 12 years designing audio for videogames full time, squeezing in more “personal” or “artistic” projects wherever I could around the edges.
Early on, I would sometimes experience anxiety, worrying that I was prostituting my art in the crass and commercial games business, and accusative looks would occasionally be directed my way from other, more artistic quarters as well. But as my understanding of the medium grew, I came to realize what a fascinating world I had stumbled into.
When I started out in games, I had the idea that it might serve as a stepping stone into film down the road, but now I would consider that a step in the wrong direction. Whereas film is codified and calcified and highly competitive, games are full of fresh challenges and opportunities as technology evolves, genres proliferate and diversify, and the very language of the medium continues to be defined.
Not only that, but working in a high-tech corporate environment provided unwitting training in other less glamorous yet useful skills, such as team management, scheduling, budgeting, and general IT savvy, not to mention the invaluable experience of contributing to successful, long-term projects that require lots of people to work together. So I’m not knocking the games industry.
Nonetheless, from early on I had the idea that working a regular day job in the videogame trenches would at some point reward me by providing the wherewithal to take some time off and devote myself exclusively to my own projects. And as my artistic pursuits grew ever more closely in line with my professional pursuits (non-linear structures, algorithmic processes, real-time sound synthesis, etc.), I longed for a period of pure research, during which I could explore these ideas freely, without time constraints, competing tasks, or other practical considerations.
So during the inordinately long time I spent on my last game (Tom Clancy’s EndWar, which I served as audio director for the past 3.5 years), as the side projects piled up, I began to formulate a more concrete exit plan. About two years ago I started laying some extra cash aside, with the idea that I would leave Ubisoft after EndWar was completed and spend a sabbatical year in China devoted exclusively to developing my own work.
And now that EndWar is finished, I have actuated my plan. Though I’ll continue to take on small tasks here and there (doing more freelance writing, maybe a bit of consulting), I’ve freed up the bulk of my time for personal, artistic pursuits. Welcome to my sabbatical!
I’ve formulated a mission statement to guide my activities this year: to apply the techniques I’ve been developing to structure non-linear sound for videogames over the past 12 years in a broader cultural context. I’m convinced there’s some vital work to do at the nexus of videogames, music composition, sound installation, and digital art, so I plan to poke around this area and see what connections I can find. The goal is to try out new ideas, with the agility to iterate rapidly and follow up on the good ones, while developing relevant skills to help me better tackle problems as they emerge, all of the things a good sabbatical should be about.
When the year is up, I’ll reevaluate and see what makes sense as a next step. I may go back to doing the kind of work I was doing before with renewed vigor, enthusiasm, and perspective. Or I might continue trying to push at these ideas from the non-commercial side for a while. It might even be possible that the virgin soil of gaming is receptive to a new kind of organism that wouldn’t force the choice, but could represent a new, viable structure for the dissemination of serious art. I’ll let you know in a year.
After finishing up the second half of Taal last night, I attempted once more to watch Curse of the Golden Flower 满城尽带黄金甲, yet once more I was confounded by a faulty DVD. So to sate my Jay Chou 周杰伦 craving, I popped in his 2007 World Tour concert DVD, unquestionably the least exciting selection out of my copious media haul from Beijing’s Blue Line store three weeks ago.
And I’ve got to say, it’s not a notable improvement over his previous concert DVD from 2004’s Incomparable tour. Most egregious fault: no Dolby Digital. Even Jolin’s 蔡依林 imaginatively titled “Live Concert” DVD from 2005 had Dolby Digital, though the constant subwoofer pummeling makes it nearly unlistenable. The set list on the new Jay DVD reminds me of the Shanghai stop of the Incomparable tour that I caught live a while back: following an opening flourish, there was a long stretch of very similar sounding ballads in the middle of the show, and he only made an effort to rouse the crowd with some peppier material towards the very end. There were some curious omissions, too: no 东风破, no 七里想, no 简单爱, no 夜曲. The performance overall seemed pretty lackadaisical (although it could be argued that this is simply his very calculated and well-worn style). He even flubbed the words of a few songs.
For surround sound, Curse of the Golden Flower was turning out to be much more titillating, before my Xbox gave up on the DVD. I loved hearing the imperial time-tellers scurrying about the imperial chambers intoning the hour in a flurry of bells. Even Taal was more imaginatively mixed, if none too subtly, fully Dolby Digital, with delayed vocal lines and hollerbacks bouncing around the rear speakers.
If Mr. Chou seeks to dominate in the global marketplace, I am afraid that simple stereo sound is not going to cut it.