Keepers of the Lazy Guns

Quite a few things have vied for my attention this week. I ploughed through 2 books, namely The Girl who Played With Fire a book on Obj-C. I’m now reading The Girl who Kicked the Hornets Next and Living Next Door to the God of Love (which I feel obliged to read to complete the authors set). Dragon Age has lured me back in and Awakenings was duly completed, then Darkspawn Chronicles fell and Leliana’s Song was started, which I hope is good as she ended up being my main’s partner in the first part. This made me think. What game has vied the most for my time? I could only think of two. Dragon Age and Fallout 3. I know Mass Effect 2 is probably a distant 3rd along with some close run compadres. It turns out that Dragon Age, combining play times for all expansions thus far + alts, comes to 127 hours. With Leliana’s song to go. Fallout 3 combined play time according to Steam is 105 hours. Bloody Hell!

In other news the iPhone project has come along reasonably well. I’ve rounded off all three panes of the UI (Add New, View Saved, Settings) and started on the custom views required for each type of task under Saved. I’ve found an expansion to opencv called cvblobslib which looks interesting although an initial run against a picture using the iphone 3G’s camera comsumed 3GB of ram in the emulator. Way outside of the 40MB Max (during opencv) and 2-3MB in standard use target I have for the iPhone app. Aside from that I realised again that with regards to computer vision less is so very very much more. To this end I re-jigged how I operate. Regardless of whether I land on a best method of blobs or contours I tweaked my original process from:

1. Smooth
2. Flatten
3. Threshold
4. Find Contours
5. Draw Largest

To this:

1. Create a copy of the image scaled down to 1024 if image is larger
2. Keep a copy of the scaling factor required
3. Smooth
4. Flatten
5. Threshold
6. Find Contours or Find blobs
7. Draw Largest scaling up discovered co-ordinates if found

This has several benefits, the two obvious ones being it uses far less ram and it’s much much faster for the same result. As a result the blob method becomes viable. The less obvious benefit is it’s actually more accurate. With higher resolution photographs the grain of the paper and reflections etc. all add detail to the image that clouds what you are trying to ‘see’. Thus scaling down, much like the smoothing, discards this unnecessary detail. By working on a copy when I need to access higher quality information (say for OCR or letting the user highlight an area to save for future direct reference in the case of complex text or image based clues) I can simply scale up the co-ordinates and pluck out the original data.

Alongside this I’ve got the hang of using the Instruments tool provided with XCode to track down memory leaks and the app is now 100% leak free. Which should save time later before it gets even MORE complicated.

Also Hough Transforms. I hope. Please god work even though I do not understand you fully.

Patrick in the comments asked for a bit more info on using cvblobs, so here it is:

Ok here’s what I hope are the relevant bits. Under build settings and ‘Other Linker Flags’ I have set:

-lstdc++ -lz $(SRCROOT)/opencv_simulator/lib/libcv.a $(SRCROOT)/opencv_simulator/lib/libcxcore.a $(SRCROOT)/opencv_simulator/lib/libcvaux.a

Where opencv_simulator is a directory containing my OpenCV libs built for the simulator, e.g. debug on and built for i386 and not arm. libcvaux.a is the file containing the cvblobs stuff.

The code snippet is:

			/* Use opencv expansion library for BLOB detection */
			CBlobResult blobs;
			CBlob *currentBlob;
			CBlob biggestBlob;

			// find non-white blobs in thresholded image
			blobs = CBlobResult( working, NULL, 0 );
			// exclude the ones smaller than last value in func.
			blobs.Filter( blobs, B_EXCLUDE, CBlobGetArea(), B_LESS, 10000 );

			// Having filtered above to get rid of the minnows get the big boy.
			blobs.GetNthBlob( CBlobGetArea(), 0, biggestBlob );

			// Create image for display
			temp = cvCreateImage(cvGetSize(working), IPL_DEPTH_8U, 4);
			// display filtered blobs
			cvMerge( working, working, working, NULL, temp );
			swapIplImage(&temp, &working);
			cvReleaseImage(&temp);

			for (int i = 0; i < blobs.GetNumBlobs(); i++ )
			{
				currentBlob = blobs.GetBlob(i);
				currentBlob->FillBlob( working, CV_RGB(255,0,0));
			}

			biggestBlob.FillBlob( working, CV_RGB(0,0,255) );

			// Adding all the contours to the animation
			display = [self UIImageFromIplImage:working];
			if(display) {
				steps++;
				[animation addObject:display];
				self.image = display;
			}

			cvReleaseImage(&working);

In the code above the variables are:

IplImage *temp, working;
UIImage *display;
NSMutableArray *animation;

In the code the variable working contains the initial input, which at that point has had the following operations performed already:

- Scaled to a lower resolution maintaining aspect ratio.
- Gaussian smoothed n times.
- Flattened to 1 channel (2 colours), e.g. IPL_DEPTH_8U as required by a lot of the cv functions.
- Adaptive Threshold applied with cvAdaptiveThreshold.

The final steps are involved in displaying some of the output back to the iPhones display for me, so I can see what blobs it’s detected etc. Hence the use of a NSMutableArray of UIImages which I go on to make an animation in a UIImageView. The use of cvMerge, IIRC, is to convert the 8U image back into a full image by simply merging it three times (by passing in the working image as each channel).

Don’t take any of the above actions as gospel, the only real requirement is to have an IPL_DEPTH_8U image which is what the functions require as input. I do the above to clear out noise, but my success was limited and I moved onto another project. Although I will revisit this in the future.

It’s worth noting here that the whole project was experimental for me, and in the settings for my app I had a switch that flipped between cvblobs and the standard cvFindContours routines. The cvBlobs code was far cleaner and shorter, however I never really settled on either as being better… in part because a lot of my input data was poor. E.g. noisy due to light conditions and the limitations of the iphones camera. I eventually found that reducing the size of the images and a certain amount of blurring helped but I never got it working as consistently as the Sudoku Grab app that’s in the store and it was a lot slower to process images.

HTH