Envisioning Synaesthesia

Synaesthesia is a genetic condition that causes me to see sound and hear light. It also causes me to work with math, language, and music as patterns of colors that interact in predictable ways. This is the essay form of the talk I gave at TEDxSalem on this, as well as a demonstration of a performance of classical music on a guitar that demonstrates what the music looks like to me.

When I was a child, it briefly my responsibility to help my twin sister with her math homework. That scene went something like this:

"No, look, yellow plus yellow is red; yellow plus green is brown!"

It made no sense to her (or to anyone else), but was plainly evident to me. I wasn't asked to help much past that.

I grew up in a semi-rural part of North Texas—nobody had any idea what was going on. I would speak of the colors of people's voices, the sound of sunlight or stars, or the sweeping panoramic hues of a symphony crackling in on a late-night radio station. Some people called it the work of angels, others demons, and others just thought I was crazy or making things up to get attention. Needless to say, I learned to keep quiet about it.

I'd cope by spending most of my time indoors or only venturing outside with sunglasses until I learned to lip-read. And I stuck with what I'm good at: math, language, and music. To me, these three are all structured systems of colors that interact in predictable ways. Numbers and letters all have colors assigned to them based on their values, shapes, and pronunciations, every sound has a color, and every color, a sound.

So, I wrote my first AI at age 12, spoke five languages before finishing High School, and played half a dozen instruments in various bands—all without being able to read sheet music. Which I still can't do: all of the notes are the same shape, so they're the same color.What got me through ten years of choir was my memory: it's near-photographic for anything I've seen/heard since vision and hearing are the same sense to me.

I'd stare blankly at my sheet music while making noises that looked to me like the ones I heard coming from the piano.

It wasn't until I was in my mid-twenties that I learned this has a name: Polymodal synaesthesia. It's a genetic trait by which stimulation of one sense causes a response in another one. It's closely related to Autism and to Multiple Sclerosis, both of which I also have. With Synaesthesia, the memory, language, vision, and hearing centers of my brain all grew very tightly together.

Vanderbilt university flew me out to gather data and test whether I was making things up. They did, and I'm not, for the record. These tests mostly measured predictable response timings to things similar to Stroop tests—those things where "blue" is written in red, and "yellow" in black that always give me a headache.

So, I tracked down Dr. Sean Day, who was then the head of the American Synaesthesia Association, and invited him to dinner. I got in my car and drove straight from Fort Worth, Texas to Dayton, Ohio in one shot. As soon as I saw him, I showed him my laptop: I'd taped colored paper over each key, corresponding to the color of the letter underneath. I could type faster and more accurately that way. And he pointed as my school bus yellow L key and said "why, that's the same color as the sound of a French Horn for me!"

And I cried as I learned I wasn't broken, and I wasn't alone.

For me, the sound of lights are based mostly on their color and brightness. Brighter lights are louder, and higher-frequency lights are higher pitched. Direct sunlight sounds like squealing brakes, and the night sky has a sound not unlike a rocky stream.

Going the other way, for the vision of sound, both the place and the color of a sound come from its frequency. Low sounds appear down and to my left, and mostly yellow. As the pitch gets higher frequency, it moves counter-clockwise through my peripheral vision through oranges (bass guitars), reds and purples (human voices), blues (like a violin), greens (like a flute), and back to yellows (mostly formant harmonics). When something is too low to hear, it goes out of my periphery down and to the left. And when it's too high to hear, it gets too small to see.

I've often wanted to share this with people, to show them what it's like. A few years ago, I wrote a simulator, an Android app called "Superpowers," that you can download from my site, danne.stayskal.com. With a camera phone and headphones, you can walk around and see the world similarly to how I do. But this interface is disconnected from anything I'm really passionate about sharing.

Last year, at Burning Man, the idea came to me to modify an instrument of mine to show what the music I play looks like to me. And here, a year later, after three or four weekends building it out of spare parts I had laying around, that simulator is working.

Just like you savor a meal by how it looks, its texture, and its taste, I enjoy music based on what it sounds like and what it looks like. And, while I could sit here and try to describe what classical music looks like, now--I can show you, and we can broaden our experiences of music together. Let's combine the senses we all already have.

And here, you can finally see what music looks like to me.

Thank you.

Written by Danne Stayskal on 2014-09-13