E15: Visual Effects and Texture Management
I’ve been spending a lot of time on implementing E15’s texture management. We wanted to have a way to upload textures asynchronously (lazy load) so that when we apply filters to the bitmaps, we don’t have the whole process blocking. This means we need threading, and I went through some rough waters with OpenGL and Cocoa threads.
The design is pretty simple. Whenever a new texture needs to be created, the bitmap is initially split into a series of tiles, then uploaded with mipmaps onto the GPU. For images that require filtering (with Core Image filters) we create a new thread that will create CIImage instances and applies a series of CIFilters. When the image is processed, the CIImage is converted back into bitmap, then sent to a singleton texture manager which periodically uploads the queue on the main thread. There were some confusing implementation details, and I’ll write them down after I describe the result.
So far, I implemented a couple different effects. The top picture shows blurring. Elements look blurry when viewed from far away, but becomes focused as you move close. Using this idea of zoomable user interface, I can show different information on the same quad depending on how far away the camera is. Here’s some captures showing various web pages.
…and here’s the obligatory Facebook shot
Now for some implementation notes. For starters, we need to remember that we need to run all OpenGL calls on the main thread. We will have to spawn a new thread for each texture we want to apply textures to, and have a singleton TextureUploader that will upload textures on the main thread. The real pain is to workout how the images should be constructed and processed when in a secondary thread.
I’ve tried a few different ways, and most of them resulted in unexpected crashes. The one I settled with is the following. During awakeOnNib, a static CIContext is created with the main CGContextRef:
CGContextRef cgContext = [[NSGraphicsContext currentContext] graphicsPort];
_drawingContext = [[CIContext contextWithCGContext:cgContext options:nil] retain];
This is the CIContext in which all the CIImages will be rendered to a CGImageRef. This method seems to be thread friendly.
In order to lazily create the filtered image, and uploaded; the initial bitmap texture that is uploaded is first converted to a CIImage. This image is sent to a new thread, which applies the CIFilters. The finished image will then be converted as a CGImageRef and drawn to a bitmap. The bitmaps are sent to a singleton class TextureUploader, which will upload textures every 500ms if there are any pending uploads. To convert CIImage to CGImageRef, just use this:
- (CGImageRef)cgImageCreateFromCIImage:(CIImage *)cimg fromRect:(CGRect)rect
{
CGImageRef cgImage;
cgImage = [_drawingContext createCGImage:cimg fromRect:rect];
if (cgImage == NULL) {
NSLog(@”failed to create cgImage”);
}
return cgImage;
}
This all seems simple, but it took a lot of trial and error. I hate threads…
