My music visualizer and mashup code is starting to gel. Its one of those classic technologies that starts in the lab, driven by a fascinated researcher, who keeps plowing ahead. I started this in 2007 trying to learn Flash by writing a version of Milkdrop. For those old enough to remember, Milkdrop was the visualizer for Winamp, the music player many used to play Napster-downloaded content.
Yet in 2007 Flash 9 had very little market penetration. Laptops chugged at the intense mathematics of a visualizer running in a virtual machine.
Fast forward 18 months. Flash 10 is out, Flash 9 is entering the land of the 90th percentile deployment, and many laptops are shipping with fast, dual processors and 4GB of memory. The compilers have also improved (as have my Flash skills, thanks to great coaching from designers like Jesse Freeman).
What frustrated me, though, was the real lack of easy support for pictures. In the midst of slide.com and Animoto, pretty little waveforms and textures were interesting but they didn’t mesmerize. I tried slapping an image onto the waveform distortion fields. It sort of worked. It still felt lame.
I kept experimenting. Last night I tried something new. What if we replaced the core waveform with a frame for a picture, then texturemapped photographs and their transitions into this space? For fun I also encoded a few dozen of the standard wipe effects from the Society of Motion Picture and Television Engineers. [ed: Funny, I didn't even know wipes could be standardized. Apparently so!]
Next, I took the picture and layered on top a waveform, a motion vector field, and a 2d mesh. From there I hooked into an FFT analyzer for music with some additions for beat detection. With a bit of tweaking I came up with this preset for animation:
-
zoom=min(0.9*zoom+0.1*amp,6)*(1+sin(time*3.14)*0.25); zoom_exp=min(2,0.9*zoom_exp + 0.1*amp);
-
dx = 0.5*(sin(time*3.1415/4)+1);
This preset picks a zoom around 1:1, zooming in when the overall sound amplitude “amp” increases, up to 6:1. I then modulate this a bit with a sine waveform so that the picture doesn’t jump around as much. Next, I kick up a “zoom exponent” or zoom_exp. This exponent adds extra oomph to the zoom, by taking the radial distance from a pixel to the center of the image, squaring (or cubing, or..) it in tune with the sound levels. This gives us a nice distortion effect popularized by Ryan Geiss in Milkdrop. Finally, I move the image around in the x direction by changing “dx.” I tweaked a few more parameters so that the motion vector field becomes brighter and more intense with louder sound, and the crawling waveform dances in tune to the music, appearing now and then when the images shrink. Its a digital roach!
As I started to wrap this up into something I could post online, I needed a playlist, a media player, controls, and more. Great coders integrate.
In the last few days I’ve spent some time on longtailvideo.com. I really liked their business model. Its open source GPL. You can use the source for free, but you need a commercial license to resell or generate cash. The JW player is popular, too, and jw’s code is uber-clean.
I ripped out my code that was starting to deal with playlists and media playing. Sigh. I could do it, but why? Then, I began to wire up a plugin using his API. I’m still having trouble getting my playlists to show up in his little box that floats underneath the player. I think the .fla file I downloaded is missing some key elements on the “playlist” button.
Nevertheless, I’ve put the player here for your amusement. I’ve mashed up the flickr interesting images with a Project Playlist. I run these media clips through a proxy to skate around cross-domain issues during this demo. The proxy is like sipping through a straw. My apologies for the error messages and slow start while your browser cache fills up. That’ll be gone in the production version.
