There are some things I wanted to guarantee, which are probably not applicable to many so I’ll document them here:

- The application has gone through a few object model changes and I wanted to support people that may have skipped versions.
- The application’s data store location has changed on one occasion, when I moved the store out of Documents and into the Library directory due to adding iTunes document sharing.

Before we look at the code we’ll have a look at a rough outline of the procedure, in handy bullet points, step is 4 is the one that took some figuring out:

1. Run everything following this asynchronously.
2. Determine location of any existing stores, if not default to initial data set.
3. Check to see if application supports iCloud, e.g. is running iOS 5.
4. Configure store options if it does support iCloud.
5. Lock the persistent store coordinator.
6. If there is a cloud data set already, simply add it.
7. If there isn’t DON’T add it, the migrate function adds it.
8. Unlock persistent store coordinator.
9. Send messages out so that view controllers that need to can redo their fetches.

A few caveats about the following, it’s not tested aside from ensuring it does migrate data into the store. I’ve not yet verified that the data then careers across the cloud to another device but I’ll update the post as soon as I do.

The code isn’t optimal, but that should be obvious! The method should be called where in your code you’d normally add the store. A lot of the framework, minus the migrations stuff, is based on some example iCloud code.
Also note that the if block with the “//Migrate old store to new AND migrate into cloud” comment is devoid of code. In that block I would need to update the old store to match the new one and THEN migrate the store as I’m using my own migrations code which also allows importing of new data and a few other tweaks, which I think I’ve previously blogged.

CLOUD_NAME,STORE_NAME are #defines. DLog is a #define macro to NSLog that is blank in production environments.

- (void)aSynchronouslyAddPersistentStore {
  NSString *cloudPath = [[self applicationLibraryDirectory] stringByAppendingPathComponent:CLOUD_NAME];
  NSString *preCloudPath = [[self applicationLibraryDirectory] stringByAppendingPathComponent:STORE_NAME];
  NSString *defaultStorePath = [[NSBundle mainBundle]
                                pathForResource:@"InitialData" ofType:@"sqlite"];
  NSString *oldStorePath = [[self applicationDocumentsDirectory] stringByAppendingPathComponent:STORE_NAME];

  // do this asynchronously since if this is the first time this particular device is syncing with preexisting
  // iCloud content it may take a long long time to download
  dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
    NSFileManager *fileManager = [NSFileManager defaultManager];
    NSURL *storeUrl = [NSURL fileURLWithPath:cloudPath];
    NSURL *migrateUrl = nil;

    /**
     Find out which store has the data we need to migrate,
     if any.
     */
    if (![fileManager fileExistsAtPath:cloudPath]) {
      if ([fileManager fileExistsAtPath:preCloudPath]) {
        // Migrate old application data.
        migrateUrl = [NSURL fileURLWithPath:preCloudPath];
      } else if ([fileManager fileExistsAtPath:oldStorePath]) {
        // Migrate old store to new AND migrate into cloud.
      } else if ([fileManager fileExistsAtPath:defaultStorePath]) {
        // Migrate (copy) in initial recipe data set.
        migrateUrl = [NSURL fileURLWithPath:defaultStorePath];
      }
    }

    // this needs to match the entitlements and provisioning profile
    NSURL *cloudURL = nil;
    if([fileManager respondsToSelector:@selector(URLForUbiquityContainerIdentifier:)])
    {
      cloudURL = [fileManager URLForUbiquityContainerIdentifier:nil];
      NSString* coreDataCloudContent = [[cloudURL path] stringByAppendingPathComponent:@"chefsbook_v14b3"];
      cloudURL = [NSURL fileURLWithPath:coreDataCloudContent];
      DLog(@"cloudURL : %@", cloudURL);
    }

    //  The API to turn on Core Data iCloud support here.
    NSDictionary *options = nil;
    if (cloudURL) {
      options = [NSDictionary dictionaryWithObjectsAndKeys:
                 @"com.thelostsouls.chefsbook", NSPersistentStoreUbiquitousContentNameKey,
                 cloudURL, NSPersistentStoreUbiquitousContentURLKey,
                 [NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
                 [NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption
                 ,nil];
    } 

    NSError *error = nil;
    [persistentStoreCoordinator_ lock];
    DLog(@"Persistent Store ****LOCKED****");

    /**
     If I migrate url is set don't just add, migrate!
     Otherwise proceed as planned.
     */
    if (migrateUrl) {
      NSPersistentStore *srcPS = [persistentStoreCoordinator_ addPersistentStoreWithType:NSSQLiteStoreType
                                                configuration:nil
                                                          URL:migrateUrl
                                                      options:nil
                                                        error:&error];
      if (![persistentStoreCoordinator_ migratePersistentStore:srcPS
                                                         toURL:storeUrl
                                                       options:options
                                                      withType:NSSQLiteStoreType
                                                         error:&error]) {
        DLog(@"Error migrating data: %@, %@", error, [error userInfo]);
        abort();
      }
    }
    else
    {
      if (![persistentStoreCoordinator_ addPersistentStoreWithType:NSSQLiteStoreType
                                                     configuration:nil
                                                               URL:storeUrl
                                                           options:options
                                                             error:&error]) {
        DLog(@"Error adding persistent store: %@, %@", error, [error userInfo]);
        abort();
      }
    }
    [persistentStoreCoordinator_ unlock];
    DLog(@"Persistent Store ****UNLOCKED****");

    // tell the UI on the main thread we finally added the store and then
    // post a custom notification to make your views do whatever they need to such as tell their
    // NSFetchedResultsController to -performFetch again now there is a real store
    dispatch_async(dispatch_get_main_queue(), ^{
      DLog(@"asynchronously added persistent store!");
      [[NSNotificationCenter defaultCenter]
       postNotificationName:AppDelegateSharedPSCAddedStore
       object:self userInfo:nil];
    });
  });

}

So there you go.

Still a worry are scenarios where iCloud support is enabled/disabled and/or the data deleted from the cloud via the settings menu. Hopefully nobody would do that and leave iCloud enabled or they’d end up with an empty application and no recovery. That’s also the reason I’m still supporting and improving the code that saves the docs out to iTunes.

Another thought just occurred to me. The following scenario needs serious consideration:

1. User starts app with iCloud support, initial data set loaded.
2. User proceeds to add a handful of data.
3. User installs Chef’s Book on second device with iCloud enabled for app.
4. App starts and copies in default data PLUS receiving iCloud data.

Will it fail, magically know it’s the same (eminently probable), or duplicate data?

Time to find out!

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Well I found out, it hangs permanently. Thanks to some helpful people on the apple forums I know why. So the above code is only partially correct. What needs to be done is to also query data on the cloud using NSMetadataQuery and examine the results, if there is cloud data you should use that and not add a persistent store in the normal way (I guess?). What I now need to find out, having successfully been able to query the cloud and find LOTS of data on there from my previous tests, is how you should add that data.

Then I’m going to rewrite the entire core data start up code from the bottom up.

So to sort through it all I wrote a small bash script. First argument in quotes is the backup/source and the second argument in quotes is the destination. It uses imagemagicks identify command and will copy the files to:

destination/camera model/date taken/name.count.jpg

count will iterate up if the file exists, if something goes wrong it will not copy the file but will print an F instead of a dot and write a log to extractPhotos.err.

#!/bin/bash
SOURCE="$1"
DEST="$2"
cd "$SOURCE"

function getSafeFilename {
    local SAFE="$1"
    local COUNT=1
    while [ -e "$SAFE" ]
    do
        local DIR=$(dirname "$1")
        local BASE=$(basename "$1")
        local NAME=${BASE%.*}
        SAFE="$DIR/$NAME.$COUNT.JPG"
        ((COUNT++))
    done

    echo $SAFE
}

# Identify all JPG images that are not thumbnails.
find . -iname *.jpg | grep -iv thumb | while read -r jpeg
do
    # Identify Model of Camera
    MODEL=$(identify -format "%[EXIF:Model]" "$jpeg" | sed 's/ *$//g')
    # Identify Date taken YYYY-MM-DD
    DATETAKEN=$(identify -format "%[EXIF:DateTime]" "$jpeg")
    DATEDAY=$(echo "$DATETAKEN" | cut -d' ' -f1)
    # Get base photo name
    PHOTO=$(basename "$jpeg")

    # Set camera model if undefined
    len=${#MODEL}
    if [[ "$len" -lt "1" ]]; then
        MODEL="Unknown"
    fi

    # Make destination directory
    DESTDIR="$DEST/$MODEL/$DATEDAY"
    if [ ! -d "$DESTDIR" ]; then
        mkdir -p "$DESTDIR"
    fi

    # Check for existing picture in that directory, if so, modify destination name, iterate if nec. (move to function)
    SAFEFILE=$(getSafeFilename "$DEST/$MODEL/$DATEDAY/$PHOTO")

    # Copy file!
    cp -n "$jpeg" "$SAFEFILE" && echo -n "." || (echo -n "F"; echo "Failed: $jpeg to $SAFEFILE" >> extractPhotos.err)

    # End
    unset MODEL
    unset DATETAKEN
    unset DATEDAY
    unset DESTDIR
    unset SAFEFILE
    unset len
# End
done

Jagged Alliance 2 – Mac Static

The binary is i386 & X86_64 but not PowerPC at the moment.

Always run your unit tests folks.

Easy to fix though and on the upside all of this code and unit tests means development is much faster now.  So I’m well on the way to iTunes document synchronisation support.  I’ve had to relocate a plist file and the sqlite database out of documents and into library otherwise they show up in iTunes.  So all you people that write core data tutorials putting sqlite in the documents folder… stop it!  It’s not a good idea use NSLibraryDirectory instead of NSDocumentDirectory when calling NSSearchPathForDirectoriesInDomains.

On another note, to determine what files were new in the documents folder I thought I’d use file info, like dates etc.  Figuring that when the file is written to the iPhone by iTunes it would write a new file but it doesn’t… it preserves the host machines modified and creation dates:

      NSDictionary *fileInfo = [[NSFileManager defaultManager] attributesOfItemAtPath:filePath
                                                                                error:nil];
      NSDate *lastModified = [fileInfo valueForKey:NSFileModificationDate];

(gdb) po fileInfo
{
    NSFileCreationDate = "2011-05-09 11:30:53 +0000";
    NSFileExtensionHidden = 0;
    NSFileGroupOwnerAccountID = 501;
    NSFileGroupOwnerAccountName = mobile;
    NSFileModificationDate = "2011-05-09 11:30:53 +0000";
    NSFileOwnerAccountID = 501;
    NSFileOwnerAccountName = mobile;
    NSFilePosixPermissions = 420;
    NSFileProtectionKey = NSFileProtectionNone;
    NSFileReferenceCount = 1;
    NSFileSize = 1788192;
    NSFileSystemFileNumber = 2608135;
    NSFileSystemNumber = 234881027;
    NSFileType = NSFileTypeRegular;
}

A shame but totally understandable as the file is not being modified or created as such. Looks like I’ll need to track a bit more metadata myself or waste time import/exporting everything every time. Err… no.

Nom nom nom

Recipe Below

Marshmallow Teacakes

Servings : 10

Preparation Time : 45 mins

Cooking Time : 10 mins

Ingredients

For The Biscuit Base

110g PlainFlour
1tsp. BakingPowder
1tsp. Bicarbonate Of Soda
Pinch Salt
60g Unsalted Butter (softened)
60g Caster Sugar
2 Egg Yolks
1/2tsp. Pure Vanilla Extract
1tbsp.Double Cream

For The Marshmallow Filling

2 Egg Whites
100g Caster Sugar
1/2tsp. Pure Vanilla Extract
Pinch Salt
1tbsp. Plus 1tsp. Golden Syrup

For The Topping

250g Good Quality Milk Chocolate

Method

1. To start set your oven to 180c/350f/gas mark 4 then grease and line two large baking trays. Separate your eggs into small bowls or cups, keeping the White for marshmallow and the yolk for the biscuit base. Due to the number of bowls you use along the way in this recipe I would recommended washing up as you go.
2. Sift the flour, salt, baking powder and bicarbonate of soda into a small bowl.
3. Place the softened butter, softening in the microwave for 10 seconds is acceptable, and caster sugar into a large bowl and cream together until it’s pale. Add the egg yolks and vanilla extract to the mix
4. Mix well then beat in the tablespoon of double cream. 5. Add the sifted flour mixture to the bowl.
6. Mix together until just combined, I used a palette knife for this stage scraping along the bottom to unstick any buttery bits. Be careful not to over handle the dough.
7. Placetablespoonsizeddollopsofdoughontoyour bakingtraysabout2″/5cmapartfromeachotherandany edge then bake for 10 minutes or until the edges of the biscuits brown.
8. When you remove the biscuits from the oven place them on a rack to let them cool.
9. Whilst the biscuits are baking start to prepare the marshmallow filling. Put all the ingredients in a heatproof glass or metal bowl onto a pan of boiling water.
10. Whisk by hand until the mixture becomes slightly frothy, opaque and the sugar has dissolved. Remove the bowl from the heat and whisk with the electric whisk until it will hold a peak like meringue and has turned White.
11. Now spoon or pipe this mixture onto your now cooled biscuit bases. I did this by hand, if you have a piping bag with a large plain tip you will find that preferable I expect.
12. Now in a heatproof bowl again over boiling water add half of the chocolate stirring constantly until it has melted then remove from the heat.
13. Chop the remainder of the chocolate into small pieces and add to the melted chocolate but don’t stir. After 5-10 minutes you should find it has melted after a quick stir together. This lets the chocolate cool down to a reasonable but still molten temperature.
14. Now the messy bit. Arrange your biscuits on one wire rack with one of the lined baking trays underneath and start to spoon the chocolate over the biscuits. You’ll want to ration the chocolate carefully here to ensure you cover all of them. I lifted the rack onto the other baking tray and recovered the lost chocolate with a palette knife using the baking sheet as a chocolate funnel.

Viewing the MySQL dump import progress.

Small linux utility called ‘bar’ which provides a progress bar for any program that can use a pipe for input…. e.g.:

bar if=foo.sql | mysql -u user -p database

Nice… dd esq CLI by the look of it.

UPDATE: After seeing some discussions about this issue and people saying it’s how XCode pushes out the updates I tested a bit more. XCode only pushes out files that have changed to ensure a quick testing cycle, it does not currently delete files. To confirm this I built an ad-hoc ipa file and installed that on my phone with iTunes. The files are indeed deleted. Therefore this is potentially a bug with XCode, it should delete files that are no longer in the bundle even if it isn’t pushing out a full update like iTunes does. It can take shortcuts, sure it has to I mean imagine testing something like Inifinity Blade any other way?, but should really ape the real world as much as possible and therefore should delete files.
End of

Of course you shouldn’t delete data models from your application after it’s deployed as you cannot dictate that somebody will upgrade to each release of your app. If the data model really was deleted from your bundle the user would be stuck as the software could not open the database to migrate.

There’s an argument to be made there, I am sure, for not using Core Data on simple or incredibly dynamic projects but that could be one for another day.

I do like the code I have got though and will not be abandoning it as it’s lightweight in terms of memory. Much more so than the other options. The use of NSOperations pushed into a queue as the migrations procede gives it remarkable power and flexibility too. There is plenty that can be done inside the migration loop to ensure it proceeds forwards and not backwards.