Beginning Core Image in iOS 6

This post is also available in: Chinese (Simplified), French

Note from Ray: This is the eighth iOS 6 tutorial in the iOS 6 Feast! In this tutorial, you’re updating one of our older tutorials to iOS 6 so it’s fully up-to-date with the latest features like the new Core Image filters in iOS 6.

Parts of this tutorial come from Jake Gundersen‘s three Core Image chapters in iOS 5 by Tutorials and iOS 6 by Tutorials. Enjoy!

This is a blog post by iOS Tutorial Team member Jacob Gundersen, an indie game developer who runs the Indie Ambitions blog. Check out his latest app – Factor Samurai!

Core Image is a powerful framework that lets you easily apply filters to images, such as modifying the vibrance, hue, or exposure. It uses the GPU (or CPU, user definable) to process the image data and is very fast. Fast enough to do real time processing of video frames!

Core Image filters can stacked together to apply multiple effects to an image or video frame at once. When multiple filters are stacked together they are efficient because they create a modified single filter that is applied to the image, instead of processing the image through each filter, one at a time.

Each filter has it’s own parameters and can be queried in code to provide information about the filter, it’s purpose, and input parameters. The system can also be queried to find out what filters are available. At this time, only a subset of the Core Image filters available on the Mac are available on iOS. However, as more become available the API can be used to discover the new filter attributes.

In this tutorial, you will get hands-on experience playing around with Core Image. You’ll apply a few different filters, and you’ll see how easy it is to apply cool effects to images in real time!

Core Image Overview

Before you get started, let’s discuss some of the most important classes in the Core Image framework:

CIContext. All of the processing of a core image is done in a CIContext. This is somewhat similar to a Core Graphics or OpenGL context.
CIImage. This class hold the image data. It can be creating from a UIImage, from an image file, or from pixel data.
CIFilter. The filter class has a dictionary that defines the attributes of the particular filter that it represents. Examples of filters are vibrance filters, color inversion filters, cropping filters, and much more.

You’ll be using each of these classes as you create your project.

Getting Started

Open up Xcode and create a new project with the iOS\Application\Single View Application template. Enter CoreImageFun for the Product Name, select iPhone for the device family, and make sure that Use Storyboards and Use Automatic Reference Counting are checked (but leave the other checkboxes unchecked).

First things first, let’s add the Core Image framework. On the Mac this is part of the QuartzCore framework, but on iOS it’s a standalone framework. Go to the project container in the file view on the left hand side. Choose the Build Phases tab, expand the Link Binaries with Library group and press the +. Navigate to the CoreImage framework and double-click on it.

Second, download the resources for this tutorial, add the included image.png to your project. Done with setup!

Next open MainStoryboard.storyboard, drag an image view into the view controller, and set its mode to Aspect Fit. The position and dimensions should roughly match the following image:

Also, open the Assistant Editor, make sure it’s displaying ViewController.h, and control-drag from the UIImageView to below the @interface. Set the Connection to Outlet, name it imageView, and click Connect.

Compile and run just to make sure everything is good so far – you should just see an empty screen. The initial setup is complete – now onto Core Image!

Basic Image Filtering

You’re going to get started by simply running your image through a CIFilter and displaying it on the screen.

Every time you want to apply a CIFilter to an image you need to do four things:

Create a CIImage object. CIImage has the following initialization methods: imageWithURL:, imageWithData:, imageWithCVPixelBuffer:, and imageWithBitmapData:bytesPerRow:size:format:colorSpace:. You’ll most likely be working with imageWithURL: most of the time.
Create a CIContext. A CIContext can be CPU or GPU based. A CIContext can be reused, so you needn’t create it over and over, but you will always need one when outputting the CIImage object.
Create a CIFilter. When you create the filter, you configure a number of properties on it that depend on the filter you’re using.
Get the filter output. The filter gives you an output image as a CIImage – you can convert this to a UIImage using the CIContext, as you’ll see below.

Let’s see how this works. Add the following code to ViewController.m inside viewDidLoad:

// 1
NSString *filePath =[[NSBundle mainBundle] pathForResource:@"image" ofType:@"png"];
NSURL *fileNameAndPath = [NSURL fileURLWithPath:filePath];// 2
CIImage *beginImage =[CIImage imageWithContentsOfURL:fileNameAndPath];// 3
CIFilter *filter = [CIFilter filterWithName:@"CISepiaTone"keysAndValues: kCIInputImageKey, beginImage,@"inputIntensity", @0.8, nil];
CIImage *outputImage = [filter outputImage];// 4
UIImage *newImage = [UIImage imageWithCIImage:outputImage];
self.imageView.image = newImage;

Let’s go over this section by section:

The first two lines create an NSURL object that holds the path to your image file.
Next you create your CIImage with the imageWithContentsOfURL method.
Next you’ll create your CIFilter object. A CIFilter constructor takes the name of the filter, and a dictionary that specifies the keys and values for that filter. Each filter will have its own unique keys and set of valid values.

The CISepiaTone filter takes only two values, the KCIInputImageKey (a CIImage) and the @”inputIntensity”, a float value, wrapped in an NSNumber (using the new literal syntax), between 0 and 1. Here you give that value 0.8. Most of the filters have default values that will be used if no values are supplied. One exception is the CIImage, this must be provided as there is no default.

Getting a CIImage back out of a filter is easy. You just use the outputImage property.
Once you have an output CIImage, you will need to convert it into a UIImage. New in iOS 6 is the UIImage method +imageWithCIImage: method. This method creates a UIImage from a CIImage. Once we’ve converted it to a UIImage, you just display it in the image view you added earlier.

Compile and run the project, and you’l see your image filtered by the sepia tone filter. Congratulations, you have successfully used CIImage and CIFilters!

Putting It Into Context

Before you move forward, there’s an optimization that you should know about.

I mentioned earlier that you need a CIContext in order to perform a CIFilter, yet there’s no mention of this object in the above example. It turns out that the the UIImage method you called (imageWithCIImage) does all the work for you. It creates a CIContext and uses it to perform the work of filtering the image. This makes using the Core Image API very easy.

There is one major drawback – it creates a new CIContext every time it’s used. CIContexts are meant to be reusable to increase performance. If you want to use a slider to update the filter value, like you’ll be doing in this tutorial, creating new CIContexts each time you change the filter would be way too slow.

Let’s do this properly. Delete the code you added to viewDidLoad and replace it with the following:

CIImage *beginImage =[CIImage imageWithContentsOfURL:fileNameAndPath];// 1
CIContext *context = [CIContext contextWithOptions:nil];CIFilter *filter = [CIFilter filterWithName:@"CISepiaTone"keysAndValues: kCIInputImageKey, beginImage,@"inputIntensity", @0.8, nil];
CIImage *outputImage = [filter outputImage];// 2
CGImageRef cgimg =[context createCGImage:outputImage fromRect:[outputImage extent]];// 3
UIImage *newImage = [UIImage imageWithCGImage:cgimg];
self.imageView.image = newImage;// 4
CGImageRelease(cgimg);

Again, let’s go over this section by section.

Here you set up the CIContext object. The CIContext constructor takes an NSDictionary that specifies options including the color format and whether the context should run on the CPU or GPU. For this app, the default values are fine and so you pass in nil for that argument.
Here you use a method on the context object to draw a CGImage. Calling the createCGImage:fromRect: on the context with the supplied CIImage will produce a CGImageRef.
Next, you use UIImage +imageWithCGImage to create a UIImage from the CGImage.
Finally, release the CGImageRef. CGImage is a C API that requires that you do your own memory management, even with ARC.

Compile and run, and make sure it works just as before.

In this example, adding the CIContext creation and handling that yourself doesn’t make too much difference. But in the next section, you’ll see why this is important performance, as you implement the ability to change the filter dynamically!

Changing Filter Values

This is great, but this is just the beginning of what you can do with Core Image filters. Lets add a slider and set it up so you can adjust the image settings in real time.

Open MainStoryboard.storyboard and drag a slider in below the image view like so:

Make sure the Assistant Editor is visible and displaying ViewController.h, then control-drag from the slider down below the @interface. Set the Connection to Action, the name toamountSliderValueChanged, make sure that the Event is set to Value Changed, and click Connect.

While you’re at it let’s connect the slider to an outlet as well. Again control-drag from the slider down below the @interface, but this time set the Connection to Outlet, the name to amountSlider, and clickConnect.

Every time the slider changes, you need to redo the image filter with a different value. However, you don’t want to redo the whole process, that would be very inefficient and would take too long. You’ll need to change a few things in your class so that you hold on to some of the objects you create in your viewDidLoad method.

The biggest thing you want to do is reuse the CIContext whenever you need to use it. If you recreate it each time, your program will run very slow. The other things you can hold onto are the CIFilter and the CIImage that holds your beginning image. You’ll need a new CIImage for every output, but the image you start with will stay constant.

You need to add some instance variables to accomplish this task.

Add the following three instance variables to your private @implementation in ViewController.m:

@implementation ViewController {CIContext *context;CIFilter *filter;CIImage *beginImage;
}

Also, change the variables in your viewDidLoad method so they use the instance variables instead of declaring new local variables:

beginImage = [CIImage imageWithContentsOfURL:fileNameAndPath];
context = [CIContext contextWithOptions:nil];filter = [CIFilter filterWithName:@"CISepiaTone" keysAndValues:kCIInputImageKey, beginImage, @"inputIntensity", @0.8, nil];

Now you’ll implement the changeValue method. What you’ll be doing in this method is altering the value of the @”inputIntensity” key in your CIFilter dictionary. Once we’ve altered this value you’ll need to repeat a few steps:

Get the output CIImage from the CIFilter.
Convert the CIImage to a CGImageRef.
Convert the CGImageRef to a UIImage, and display it in the image view.

So replace the amountSliderValueChanged: method with the following:

- (IBAction)amountSliderValueChanged:(UISlider *)slider {float slideValue = slider.value;[filter setValue:@(slideValue)forKey:@"inputIntensity"];CIImage *outputImage = [filter outputImage];CGImageRef cgimg = [context createCGImage:outputImagefromRect:[outputImage extent]];UIImage *newImage = [UIImage imageWithCGImage:cgimg];self.imageView.image = newImage;CGImageRelease(cgimg);
}

You’ll notice that you’ve changed the variable type from (id)sender to (UISlider *)sender in the method definition. You know you’ll only be using this method to retrieve values from your UISlider, so you can go ahead and make this change. If we’d left it as (id), we’d need to cast it to a UISlider or the next line would throw an error. Make sure that the method declaration in the header file matches the changes we’ve made here.

You retrieve the float value from the slider. Your slider is set to the default values – min 0, max 0, default 0.5. These happen to be the right values for this CIFilter, how convenient!

The CIFilter has methods that will allow us to set the values for the different keys in its dictionary. Here, you’re just setting the @”inputIntensity” to an NSNumber object with a float value of whatever you get from your slider.

The rest of the code should look familiar, as it follows the same logic as your viewDidLoad method. You’re going to be using this code over and over again. From now on, you’ll use the changeSlider method to render the output of a CIFilter to your UIImageView.

Compile and run, and you should have a functioning live slider that will alter the sepia value for your image in real time!

Getting Photos from the Photo Album

Now that you can change the values of the filter on the fly, things are starting to get real interesting! But what if you don’t care for this image of flowers? Let’s set up a UIImagePickerController so you can get pictures from out of the photo album and into your program so you can play with them.

You need to create a button that will bring up the photo album view, so open up ViewController.xib and drag in a button to the right of the slider and label it “Photo Album”.

Then make sure the Assistant Editor is visible and displaying ViewController.h, then control-drag from the button down below the @interface. Set the Connection to Action, the name to loadPhoto, make sure that the Event is set to Touch Up Inside, and click Connect.

Next switch to ViewController.m, and implement the loadPhoto method as follows:

- (IBAction)loadPhoto:(id)sender {UIImagePickerController *pickerC = [[UIImagePickerController alloc] init];pickerC.delegate = self;[self presentViewController:pickerC animated:YES completion:nil];
}

The first line of code instantiates a new UIImagePickerController. You then set the delegate of the image picker to self (our ViewController).

You get a warning here. You need to setup your ViewController as an UIImagePickerControllerDelegate and UINaviationControllerDelegate and then implement the methods in that delegates protocol.

Still in ViewController.m, change the class extension as follows:

@interface ViewController () <UIImagePickerControllerDelegate, UINavigationBarDelegate>
@end

Now implement the following two methods:

- (void)imagePickerController:(UIImagePickerController *)picker didFinishPickingMediaWithInfo:(NSDictionary *)info {[self dismissViewControllerAnimated:YES completion:nil];NSLog(@"%@", info);
}- (void)imagePickerControllerDidCancel:(UIImagePickerController *)picker {[self dismissViewControllerAnimated:YES completion:nil];
}

In both cases, you dismiss the UIPickerController. That’s the delegate’s job, if you don’t do it there, then you just stare at the image picker forever!

The first method isn’t completed yet – it’s just a placeholder to log out some information about chosen image. The cancel method just gets rid of the picker controller, and is fine as-is.

Compile and run and tap the button, and it will bring up the image picker with the photos in your photo album. If you are running this in the simulator, you probably won’t get any photos. On the simulator or on a device without a camera, you can use Safari to save images to your photo album. Open safari, find an image, tap and hold, and you’ll get a dialog to save that image. Next time you run your app, you’ll have it!

Here’s what you should see in the console after you’ve selected an image (something like this):

2012-09-20 17:30:52.561 CoreImageFun[3766:c07] {UIImagePickerControllerMediaType = "public.image";UIImagePickerControllerOriginalImage = "";UIImagePickerControllerReferenceURL = "assets-library://asset/asset.JPG?id=253312C6-A454-45B4-A9DA-649126A76CA5&ext=JPG";
}

Note that it has an entry in the dictionary for the “original image” selected by the user. This is what you want to pull out and filter!

Now that we’ve got a way to select an image, how do you set your CIImage beganImage to use that image?

Simple, just change the delegate method to look like this:

- (void)imagePickerController:(UIImagePickerController *)pickerdidFinishPickingMediaWithInfo:(NSDictionary *)info {[self dismissViewControllerAnimated:YES completion:nil];UIImage *gotImage =[info objectForKey:UIImagePickerControllerOriginalImage];beginImage = [CIImage imageWithCGImage:gotImage.CGImage];[filter setValue:beginImage forKey:kCIInputImageKey];[self amountSliderValueChanged:self.amountSlider];
}

You need to create a new CIImage from your selected photo. You can get the UIImage representation of the photo by finding it in the dictionary of values, under the UIImagePickerControllerOriginalImage key constant. Note it’s better to use a constant rather than a hardcoded string, because Apple could change the name of the key in the future. For a full list of key constants, see the UIImagePickerController Delegate Protocol Reference.

You need to convert this into a CIImage, but you don’t have a method to convert a UIImage into a CIImage. However, you do have [CIImage imageWithCGImage:] method. You can get a CIImage from your UIImage by calling UIImage.CGImage, so you do exactly that!

You then set the key in the filter dictionary so that the input image is your new CIImage you just created.

The last line may seem odd. Remember how I pointed out that the code in the changeValue ran the filter with the latest value and updated the image view with the result?

Well you need to do that again, so you can just call the changeValue method. Even though the slider value hasn’t changed, you can still use that method’s code to get the job done. You could break that code into it’s own method, and if you were going to be working with more complexity, you would to avoid confusion. But, in this case your purpose here is served using the changeValue method. You pass in the amountSlider so that it has the correct value to use.

Compile and run, and now you’ll be able to update the image from your photo album!

What if you create the perfect sepia image, how do you hold on to it? You could take a screenshot, but you’re not that ghetto! Let’s learn how to save your photos back to the photo album.

Saving to Photo Album

To save to the photo album, you need to use the AssetsLibrary framework. To add it to your project, go to the project container, choose the Build Phases tab, expand the Link Binaries with Libraries group and click the + button. Find the AssetsLibrary framework, and add it.

Then add the following #import statement to the top of ViewController.m:

#import <AssetsLibrary/AssetsLibrary.h>

One thing you should know is that when you save a photo to the album, it’s a process that could continue even after you leave the app.

This could be a problem as the GPU stops whatever it’s doinng when you switch from one app to another. If the photo isn’t finished being saved, it won’t be there when you go looking for it later!

The solution to this is to use the CPU CIRendering context. The default is the GPU, and the GPU is much faster. You can create a second CIContext just for the purpose of saving this file.

Let’s add a new button to your app that will let us save the photo you are currently modifying with all the changes we’ve made. Open MainStoryboard add a new button labeled “Save to Album”:

Then connect it to a new savePhoto: method, like you did last time.

Then switch to ViewController.m and implement the method as follows:

- (IBAction)savePhoto:(id)sender {// 1CIImage *saveToSave = [filter outputImage];// 2CIContext *softwareContext = [CIContextcontextWithOptions:@{kCIContextUseSoftwareRenderer : @(YES)} ];// 3CGImageRef cgImg = [softwareContext createCGImage:saveToSavefromRect:[saveToSave extent]];// 4ALAssetsLibrary* library = [[ALAssetsLibrary alloc] init];[library writeImageToSavedPhotosAlbum:cgImgmetadata:[saveToSave properties]completionBlock:^(NSURL *assetURL, NSError *error) {// 5CGImageRelease(cgImg);}];
}

In this code block you:

Get the CIImage output from the filter.
Create a new, software based CIContext
Generate the CGImageRef.
Save the CGImageRef to the photo library.
Release the CGImage. That last step happens in a callback block so that it only fires after you’re done using it.

Compile and run the app (remember, on an actual device since you’re using software rendering), and now you can save that “perfect image” to your photo library so it’s preserved forever!

What About Image Metadata?

Let’s talk about image metadata for a moment. Image files taken on mobile phones have a variety of data associated with them, such as GPS coordinates, image format, and orientation. Specifically orientation is something that you need to preserve. The process of loading into a CIImage, rendering to a CGImage, and converting to a UIImage strips the metadata from the image. In order to preserve orientation, you’ll need to record it and then put it back into the UIImage.

Start by adding a new private instance variable to ViewController.m:

@implementation ViewController {CIContext *context;CIFilter *filter;CIImage *beginImage;UIImageOrientation orientation; // New!
}

Next, set the value when you load the image from the photo library in the -imagePickerController: didFinishPickingMediaWithInfo: method. Add the following line before the “beginImage = [CIImage imageWithCGImage:gotImage.CGImage]” line:

orientation = gotImage.imageOrientation;

Finally, alter the line in amountSliderChanged: creates the UIImage that you set to the imageView object:

UIImage *newImage = [UIImage imageWithCGImage:cgimg scale:1.0 orientation:orientation];

Now, if you take a picture taken in something other than the default orientation, it will be preserved.

What Other Filters are Available?

The CIFilter API has 130 filters on the Mac OS plus the ability to create custom filters. In iOS 6, it has 93 or more. Currently there isn’t a way to build custom filters on the iOS platform, but it’s possible that it will come.

In order to find out what filters are available, you can use the [CIFilter filterNamesInCategory:kCICategoryBuiltIn] method. This method will return an array of filter names. In addition, each filter has an attributes method that will return a dictionary containing information about that filter. This information includes the filter’s name, the kinds of inputs the filter takes, the default and acceptable values for the inputs, and the filter’s category.

Let’s put together a method for your class that will print all the information for all the currently available filters to the log. Add this method right above viewDidLoad:


-(void)logAllFilters {NSArray *properties = [CIFilter filterNamesInCategory:kCICategoryBuiltIn];NSLog(@"%@", properties);for (NSString *filterName in properties) {CIFilter *fltr = [CIFilter filterWithName:filterName];NSLog(@"%@", [fltr attributes]);}
}

This method simply gets the arrary of filters from the filterNamesInCategory method. It prints the list of names first. Then, for each name in the list, it creates that filter and logs the attributes dictionary from that filter.

Then call this method at the end of viewDidLoad:

[self logAllFilters];

You will see the following in the log output:

Wow, that’s a lot of filters!

More Intricate Filter Chains

Now that we’ve looked at all the filters that are available on the iOS 6 platform, it’s time to create a more intricate filter chain. In order to do this, you’ll create a dedicated method to process the CIImage. It will take in a CIImage, filter it, and return a CIImage. Add the following method:

-(CIImage *)oldPhoto:(CIImage *)img withAmount:(float)intensity {// 1CIFilter *sepia = [CIFilter filterWithName:@"CISepiaTone"];[sepia setValue:img forKey:kCIInputImageKey];[sepia setValue:@(intensity) forKey:@"inputIntensity"];// 2CIFilter *random = [CIFilter filterWithName:@"CIRandomGenerator"];// 3CIFilter *lighten = [CIFilter filterWithName:@"CIColorControls"];[lighten setValue:random.outputImage forKey:kCIInputImageKey];[lighten setValue:@(1 - intensity) forKey:@"inputBrightness"];[lighten setValue:@0.0 forKey:@"inputSaturation"];// 4CIImage *croppedImage = [lighten.outputImage imageByCroppingToRect:[beginImage extent]];// 5CIFilter *composite = [CIFilter filterWithName:@"CIHardLightBlendMode"];[composite setValue:sepia.outputImage forKey:kCIInputImageKey];[composite setValue:croppedImage forKey:kCIInputBackgroundImageKey];// 6CIFilter *vignette = [CIFilter filterWithName:@"CIVignette"];[vignette setValue:composite.outputImage forKey:kCIInputImageKey];[vignette setValue:@(intensity * 2) forKey:@"inputIntensity"];[vignette setValue:@(intensity * 30) forKey:@"inputRadius"];// 7return vignette.outputImage;
}

Let’s go over this section by section:

In section one you set up the sepia filter the same way you did in the simpler scenario. You’re passing in the float in the method to set the intensity of the sepia. This value will be provided by the slider.
In the second section you set up a filter that is new to iOS 6 (though not new on the Mac). The random filter creates a random noise pattern, it looks like this:

It doesn’t take any parameters. You’ll use this noise pattern to add texture to your final old photo look.
In section three, you are altering the output of the random noise generator. You want to change it to grey and lighten it up a little bit so the effect is less dramatic. You’ll notice that the input image key is set to the .outputImage property of the random filter. This is a convenient way to chain the output of one filter into the input of the next.
The fourth section you make use of a convenient method on CIImage. The imageByCroppingToRect method takes an output CIImage and crops it to the provided rect. In this case, you need to crop the output of the CIRandomGenerator filter because it is infinite. As a generated CIImage, goes on infinitely. If you don’t crop it at some point, you’ll get an error saying that the filters have ‘an infinte extent’. CIImages don’t actually contain data, they describe it. It’s not until you call a method on the CIContext that data is actually processed.
In section five you are combining the output of the sepia filter with the output of the alter CIRandom filter. This filter does the exact same operation as the ‘Hard Light’ setting does in a photoshop layer. Most of (if not all, I’m not sure) the options in photoshop are available in Core Image.
In the sixth section, you run a vignette filter on this composited output that darkens the edges of the photo. You’re using the value from the slider to set the radius and intensity of this effect.
Finally, you return the output of the last filter.

That’s all for this filter. You can get an idea of how complex these filter chains may become. By combining Core Image filters into these kinds of chains, you can achieve endless different effects.

The next thing to do is implement this method in amountSliderValueChanged:. Change these two lines:

[filter setValue:@(slideValue) forKey:@"inputIntensity"];
CIImage *outputImage = [filter outputImage];

To this one line:

CIImage *outputImage = [self oldPhoto:beginImage withAmount:slideValue];

This just replaces the previous method of setting the outputImage variable to your new method. You pass in the slider value for the intensity and you use the beginImage, which you set in the viewDidLoad method as the input CIImage. Build and run now and you should get a more refined old photo effect.

That noise could probably be more subtle, but I’ll leave that experiment to you, dear reader. Now you have the power of Core Image. Go crazy!

Where To Go From Here?

Here is the example project with all of the code from the above tutorial.

That about covers the basics of using Core Image filters. It’s a pretty handy technique, and you should be able to use it to apply some neat filters to images quite quickly!

If you want to learn more about Core Image, you can learn some advanced photo manipulation techniques in iOS 5 by Tutorials. Our new book iOS 6 by Tutorials also has a very awesome chapter on how to use Core Image with AVFoundation to create a live video recording app that filters the video in real time.

If you have any questions or comments on this tutorial or Core Image in general, please join the forum discussion below!

This is a blog post by iOS Tutorial Team member Jacob Gundersen, an indie game developer and co-founder of Third Rail Games. Check out his latest app – Factor Samurai!