Scott builds Software

A mesh and Lomo filter on the iPhone

I recently “discovered” OpenGL Shading Language on the iPhone, or GLSL for short. Its incredible. The iPhone chipset compiles a C-like language on the fly, producing byte codes for the GPU. These in turn are processed in two phases at interactive frame rates:

1. Vertex Shader – Compute where a pixel is drawn
2. Fragment Shader – Compute what’s drawn for each pixel

Readers will recall my experiments with Milkdrop, flash visualization, and image morphing math. The OpenGL Shading Language takes it to a whole new level, stuffed into a large, orange, 740-page reference guide.

I wondered if I could build a real-time visualizer on the iPhone using GLSL, applying a surface mesh to a live data feed, all at interactive rates. This could be the groundwork for visualizing massive data clouds we’re all building these days.

Sure enough, you can.

I’m working on an app tentatively titled, “FunHouse Mirror.” That’s probably the best way to explain what happens. Yes, there are other copies in the App Store. What makes mine unique is that the filters are infinite and programmable.

Inspired by Milkdrop, I built an engine for morphing video instead of audio. Instead of presets, the app downloads shaders, compiles them, shows me the compiler & linker results in a popup window. If no errors are found, the engine installs the vertex and fragment shaders. Then the magic happens.

The engine senses the compass, accelerometer, orientation, GPS location and time. It computes several waveforms on demand. It grabs the raw bytes from the forward facing camera as a data feed. These are passed into the visualizer, which changes the visual effect like Milkdrop based on ever-changing data. It’s stunning.

The result is live, interactive, morphing video. I also find that it’s a wonderful tool for testing out shader ideas, learning how math shapes surfaces, figuring out the best way to visualize a data set. The engine takes care of all the headache of sampling data, working with OpenGL, compiling shaders, building a mesh, establishing textures, interacting with the user, displaying views, all that rot.

Instead, you can write a simple shader like this.

/*
 * Roaming fisheye effect
 *
 */

vec4 FishEye()
{
        float dt = (f_parms[TIME] - 0.0*floor(f_parms[TIME]))*3.14;
        float cx = 0.4+3.0*(f_parms[ROLL]-0.08);
        float cy = 0.4+6.0*(f_parms[PITCH]+0.85);
        vec2 ray = v_texCoord-vec2(cx,cy);
        float r = length(ray);
        float theta = atan(ray.y,ray.x);
        float rs = (r*r)/0.7;
        vec2 pixel = vec2(cx+rs*cos(theta), cy+rs*sin(theta+dt*0.0));

        return texture2D(campic, pixel);
}

void main()
{
        gl_FragColor = FishEye();
}

The shader is tested and installed with a single tap. I’m also building up a library of functions for representing Photoshop effects and convolutions, in real time, with shaders.

Filed under:
Apps, iphone by Scott