Backing up Steam with CWRsync

This may or may not be of use to other people, and seeing as I’ve not written a blog post for ages, I thought I’d post it here. Before demolishing my PC last year and building a new one once I found myself gainfully employed once again I was working on something to back up Steam (and indeed image my PC). I never got the hang of using windows ghost copies from batch scripts to image the windows drive out but I did come up with a bash script to copy steam off to a USB or eSATA drive:


@ECHO OFF

SETLOCAL

REM ** CUSTOMIZE ** Specify where to find rsync and related files (C:\CWRSYNC)
SET CWRSYNCHOME="D:\Program Files (x86)\cwRsync"

REM Set CYGWIN variable to 'nontsec'. That makes sure that permissions
REM on your windows machine are not updated as a side effect of cygwin
REM operations.
SET CYGWIN=nontsec

REM Set HOME variable to your windows home directory. That makes sure
REM that ssh command creates known_hosts in a directory you have access.
SET HOME=%HOMEDRIVE%%HOMEPATH%

REM Make cwRsync home as a part of system PATH to find required DLLs
SET CWOLDPATH=%PATH%
SET PATH=%CWRSYNCHOME%\BIN;%PATH%

REM rsync
REM -r recursive
REM -v verbose
REM -P progress
REM -k transform hard link to dir on dest.
REM -L transform symlink to dir on dest.
REM -u update, probably a winblows timestamp fix to stop it copying every file every time.
sc stop MsMpSvc
taskkill /f /im msmpeng.exe
REM rsync --delete -rvuP --exclude='Google' -exclude='Temporary Internet Files' --exclude='Local Low' --exclude='Backup' --exclude='Temp' --exclude='*Cache*' /cygdrive/c/Users /cygdrive/h/
rsync -rvukPL --exclude='httpcache' --exclude='htmlcache' --exclude='overlayhtmlcache' /cygdrive/d/Games/Steam /cygdrive/g/Games/PC/
sc start MsMpSvc

Hopefully it’s of use to someone. If you’re not comfortable with the sc stops and taskkill commands then get rid of them, I just found (especially when copying the windows users directory) that it interfered and made the rsync slow to a crawl).

In other news there’s err not a lot. If there was I’d post it here. 😛

Actually that’s not true. I’ve written tonnes and tonnes of code over the last six months some of which I’d love to post but never seemed to get around to it. I’ll try and post more some. Some of things I’ve been wrestling with have been:

– Clustering Algorithms for geo location data.
– MKMapKit with transparent overlays and mad exc_bad_access errors buried deep within Map Kit/iOS. Which are mostly solved, but it still crashes occassionally even with only 1 overlay. Notably on zooming out.
– OpenGL ES, briefly last week. I’m hoping to do more fun coding in this area and have even asked my nephew to design a game. That should be amusing. 😀 It will be very very weird.
– Last.fms API, I want a basic no frills scrobbling app for the iPhone that lets me use the including Music.app. It turns out it’s non trivial.

There will definitely be a post on the latter. NSNotifications from the music app are particularly interesting when your app is in the background. They go missing and there’s not hard and fast rule it would appear on how long they are kept for or under what conditions they are dropped. A rule of thumb would appear to be that you receive about 3 track change notifications and that’s it. So you can’t receive information for a whole album with the app backgrounded. Whilst tackling this supposedly obvious ‘why the hell hasn’t anyone done this already?’ app I’ve realised why the apps that are out there either:

– Don’t work.
– Use a built in player.
– Build horrible caches of your play history each time (iScrobble I’m looking at you).

Sadly the latter was particularly good when it worked but now appears to not receive updates and recent changes to iOS have rendered it useless. I can only assume because it caches it’s data in a directory that is now cleared out by the OS.

On the Chef’s Book count I’ve been wanting to sneak the time in to polish off the recipe search and update the graphics and UI. It just never seems to happen. 🙁 I’ll do it, I just don’t know when.

Posted in Games, Programming | Tagged , , , , , , | Comments Off on Backing up Steam with CWRsync

Progress with adding iCloud to Chef’s Book.

There are some interesting challenges with incorporating iCloud support, especially into an existing application. One, which I’ve been working through, is that you can no longer copy in a pre-existing store for the purposes of loading data. Such a move appears to work but does not upload anything to the cloud, the cloud store will only see future changes and not have the initial data set.
The solution to this is to use NSPersistentStoreCordinator’s migratePersistentStore:toURL:options:withType:error method. However I found the documentation somewhat lacking in clarity so here is the solution that now appears to be working for me.

There are some things I wanted to guarantee, which are probably not applicable to many so I’ll document them here:

– The application has gone through a few object model changes and I wanted to support people that may have skipped versions.
– The application’s data store location has changed on one occasion, when I moved the store out of Documents and into the Library directory due to adding iTunes document sharing.

Before we look at the code we’ll have a look at a rough outline of the procedure, in handy bullet points, step is 4 is the one that took some figuring out:

  1. Run everything following this asynchronously.
  2. Determine location of any existing stores, if not default to initial data set.
  3. Check to see if application supports iCloud, e.g. is running iOS 5.
  4. Configure store options if it does support iCloud.
  5. Lock the persistent store coordinator.
  6. If there is a cloud data set already, simply add it.
  7. If there isn’t DON’T add it, the migrate function adds it.
  8. Unlock persistent store coordinator.
  9. Send messages out so that view controllers that need to can redo their fetches.

A few caveats about the following, it’s not tested aside from ensuring it does migrate data into the store. I’ve not yet verified that the data then careers across the cloud to another device but I’ll update the post as soon as I do.

The code isn’t optimal, but that should be obvious! 😉 The method should be called where in your code you’d normally add the store. A lot of the framework, minus the migrations stuff, is based on some example iCloud code.
Also note that the if block with the “//Migrate old store to new AND migrate into cloud” comment is devoid of code. In that block I would need to update the old store to match the new one and THEN migrate the store as I’m using my own migrations code which also allows importing of new data and a few other tweaks, which I think I’ve previously blogged.

CLOUD_NAME,STORE_NAME are #defines. DLog is a #define macro to NSLog that is blank in production environments.

- (void)aSynchronouslyAddPersistentStore {
  NSString *cloudPath = [[self applicationLibraryDirectory] stringByAppendingPathComponent:CLOUD_NAME];
  NSString *preCloudPath = [[self applicationLibraryDirectory] stringByAppendingPathComponent:STORE_NAME];
  NSString *defaultStorePath = [[NSBundle mainBundle] 
                                pathForResource:@"InitialData" ofType:@"sqlite"];
  NSString *oldStorePath = [[self applicationDocumentsDirectory] stringByAppendingPathComponent:STORE_NAME];
  
  
  // do this asynchronously since if this is the first time this particular device is syncing with preexisting
  // iCloud content it may take a long long time to download
  dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
    NSFileManager *fileManager = [NSFileManager defaultManager];
    NSURL *storeUrl = [NSURL fileURLWithPath:cloudPath];
    NSURL *migrateUrl = nil;
    
    /** 
     Find out which store has the data we need to migrate,
     if any.
     */
    if (![fileManager fileExistsAtPath:cloudPath]) {
      if ([fileManager fileExistsAtPath:preCloudPath]) {
        // Migrate old application data.
        migrateUrl = [NSURL fileURLWithPath:preCloudPath];
      } else if ([fileManager fileExistsAtPath:oldStorePath]) {
        // Migrate old store to new AND migrate into cloud.
      } else if ([fileManager fileExistsAtPath:defaultStorePath]) {
        // Migrate (copy) in initial recipe data set.
        migrateUrl = [NSURL fileURLWithPath:defaultStorePath];
      }
    }

    // this needs to match the entitlements and provisioning profile
    NSURL *cloudURL = nil;
    if([fileManager respondsToSelector:@selector(URLForUbiquityContainerIdentifier:)])
    {
      cloudURL = [fileManager URLForUbiquityContainerIdentifier:nil];
      NSString* coreDataCloudContent = [[cloudURL path] stringByAppendingPathComponent:@"chefsbook_v14b3"];
      cloudURL = [NSURL fileURLWithPath:coreDataCloudContent];
      DLog(@"cloudURL : %@", cloudURL);
    }
    
    //  The API to turn on Core Data iCloud support here.
    NSDictionary *options = nil;
    if (cloudURL) {
      options = [NSDictionary dictionaryWithObjectsAndKeys:
                 @"com.thelostsouls.chefsbook", NSPersistentStoreUbiquitousContentNameKey, 
                 cloudURL, NSPersistentStoreUbiquitousContentURLKey, 
                 [NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption, 
                 [NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption
                 ,nil];
    } 
    
    NSError *error = nil;
    [persistentStoreCoordinator_ lock];
    DLog(@"Persistent Store ****LOCKED****");
    
    /**
     If I migrate url is set don't just add, migrate!
     Otherwise proceed as planned.
     */
    if (migrateUrl) {
      NSPersistentStore *srcPS = [persistentStoreCoordinator_ addPersistentStoreWithType:NSSQLiteStoreType
                                                configuration:nil
                                                          URL:migrateUrl
                                                      options:nil
                                                        error:&error];
      if (![persistentStoreCoordinator_ migratePersistentStore:srcPS
                                                         toURL:storeUrl
                                                       options:options
                                                      withType:NSSQLiteStoreType 
                                                         error:&error]) {
        DLog(@"Error migrating data: %@, %@", error, [error userInfo]);
        abort();
      }
    }
    else
    {
      if (![persistentStoreCoordinator_ addPersistentStoreWithType:NSSQLiteStoreType 
                                                     configuration:nil 
                                                               URL:storeUrl 
                                                           options:options 
                                                             error:&error]) {
        DLog(@"Error adding persistent store: %@, %@", error, [error userInfo]);
        abort();
      }    
    }
    [persistentStoreCoordinator_ unlock];
    DLog(@"Persistent Store ****UNLOCKED****");
    
    // tell the UI on the main thread we finally added the store and then
    // post a custom notification to make your views do whatever they need to such as tell their
    // NSFetchedResultsController to -performFetch again now there is a real store
    dispatch_async(dispatch_get_main_queue(), ^{
      DLog(@"asynchronously added persistent store!");
      [[NSNotificationCenter defaultCenter] 
       postNotificationName:AppDelegateSharedPSCAddedStore
       object:self userInfo:nil];
    });
  });

}

So there you go.

Still a worry are scenarios where iCloud support is enabled/disabled and/or the data deleted from the cloud via the settings menu. Hopefully nobody would do that and leave iCloud enabled or they’d end up with an empty application and no recovery. That’s also the reason I’m still supporting and improving the code that saves the docs out to iTunes.

Another thought just occurred to me. The following scenario needs serious consideration:

  1. User starts app with iCloud support, initial data set loaded.
  2. User proceeds to add a handful of data.
  3. User installs Chef’s Book on second device with iCloud enabled for app.
  4. App starts and copies in default data PLUS receiving iCloud data.

Will it fail, magically know it’s the same (eminently probable), or duplicate data?

Time to find out!

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Well I found out, it hangs permanently. Thanks to some helpful people on the apple forums I know why. So the above code is only partially correct. What needs to be done is to also query data on the cloud using NSMetadataQuery and examine the results, if there is cloud data you should use that and not add a persistent store in the normal way (I guess?). What I now need to find out, having successfully been able to query the cloud and find LOTS of data on there from my previous tests, is how you should add that data.

Then I’m going to rewrite the entire core data start up code from the bottom up.

Posted in Chef's Book, Programming | Tagged , , , , , , | 15 Comments

Fun with bash and old photo libraries.

It recently dawned on me I had huge swathes of my photo collection entirely gone. Something like 3 years that were nowhere to be found on flickr or my main PC or Mac. I’m not sure how this gap appeared but I eventually found the files in a backup of my Mac from quite some time ago (I keep everything until I really need the space). Problem was it was in an old Aperture directory structure and I really just wanted to get the pictures out and sort through them from scratch. I’d used different cameras, some of the pictures weren’t mine etc. etc.

So to sort through it all I wrote a small bash script. First argument in quotes is the backup/source and the second argument in quotes is the destination. It uses imagemagicks identify command and will copy the files to:

destination/camera model/date taken/name.count.jpg

count will iterate up if the file exists, if something goes wrong it will not copy the file but will print an F instead of a dot and write a log to extractPhotos.err.


#!/bin/bash
SOURCE="$1"
DEST="$2"
cd "$SOURCE"

function getSafeFilename {
    local SAFE="$1"
    local COUNT=1
    while [ -e "$SAFE" ] 
    do
        local DIR=$(dirname "$1")
        local BASE=$(basename "$1")
        local NAME=${BASE%.*}
        SAFE="$DIR/$NAME.$COUNT.JPG"
        ((COUNT++))
    done

    echo $SAFE
}

# Identify all JPG images that are not thumbnails.
find . -iname *.jpg | grep -iv thumb | while read -r jpeg 
do
    # Identify Model of Camera
    MODEL=$(identify -format "%[EXIF:Model]" "$jpeg" | sed 's/ *$//g')
    # Identify Date taken YYYY-MM-DD
    DATETAKEN=$(identify -format "%[EXIF:DateTime]" "$jpeg")
    DATEDAY=$(echo "$DATETAKEN" | cut -d' ' -f1)
    # Get base photo name
    PHOTO=$(basename "$jpeg")

    # Set camera model if undefined
    len=${#MODEL}
    if [[ "$len" -lt "1" ]]; then
        MODEL="Unknown"
    fi

    # Make destination directory
    DESTDIR="$DEST/$MODEL/$DATEDAY"
    if [ ! -d "$DESTDIR" ]; then
        mkdir -p "$DESTDIR"
    fi

    # Check for existing picture in that directory, if so, modify destination name, iterate if nec. (move to function)
    SAFEFILE=$(getSafeFilename "$DEST/$MODEL/$DATEDAY/$PHOTO")

    # Copy file!
    cp -n "$jpeg" "$SAFEFILE" && echo -n "." || (echo -n "F"; echo "Failed: $jpeg to $SAFEFILE" >> extractPhotos.err)

    # End
    unset MODEL
    unset DATETAKEN
    unset DATEDAY
    unset DESTDIR
    unset SAFEFILE
    unset len
# End
done
Posted in Computer, Photography, Programming | Tagged , , , , , | Comments Off on Fun with bash and old photo libraries.

Has it been nearly a month?

It’s odd, I can’t believe it’s been that long since the last post. There hasn’t really been anything to write about, no trials with code, no revelations, just fairly steady progress. In other news I’ve finished one playthrough of The Witcher 2, which is absolutely blinding. A more recent discovery is the ancient but still utterly amazing Jagged Alliance 2. Go buy, gog have it on sale this weekend!

Posted in Miscellany | Tagged , , , , | Comments Off on Has it been nearly a month?

Jagged Alliance 2 – the static universal binary.

Using a lot of cues from elsewhere I’ve put together a DMG containing JA2 statically linked to the necessary SDL library. It contains the data for the 800×600 mod my Mythrell and still requires the game data files. More information inside:

Jagged Alliance 2 – Mac Static

The binary is i386 & X86_64 but not PowerPC at the moment.

Posted in Miscellany | Comments Off on Jagged Alliance 2 – the static universal binary.

The MD5 checksum bug identified.

Happily it’s easy to fix and exactly what I thought it was. When retrieving values from an NSManaged object like so:

NSString *foo = recipe.value;

or

NSString *foo = [recipe valueForKey:@"value"];

If the value is not present you get a nil. Messages to nil go nowhere but do not crash, so if value should have been an NSString the following is OK:

[foo isEqualToString:@"bar"]

However if the object you are retrieving from is an NSDictionary you will get an NSNull. NSNull is a class and a valid object so sending it messages it does not understand will fail:

NSString *foo = [dictionary valueForKey:@"value"];
[foo isEqualToString:@"bar"]; // CRASH

Seeing as I need to compute the md5 sum for a dictionary based recipe and an managed object based one and the sums need to be the same the code that produces the string to hash must produce identical results. In the case of a null value the code to produce the key to hash from the recipe managed object inserted this:

(null)

The code to produce it from the dictionary simply crashed. So I changed it in both the dictionary and the managed object routines to check:

if ([object isEqualToClass:[NSNull class]]) {
object = @"";
}

Which meant the keys changed and I didn’t update the initial data set to reflect that as I didn’t run OCUnit and thus didn’t get bashed upside the head that something had changed.

Always run your unit tests folks.

Posted in Chef's Book, Programming | Tagged , , , , | Comments Off on The MD5 checksum bug identified.

Implementing iTunes syncing and a lesson to always run unit tests.

So the latest version of my main app is in the app store and…. it has a minor minor bug.  In a rush to eliminate a highly unlikely, in fact to pre-empt, a bug I updated some code to check for the possibility of NSNull being assigned to an NSString.  For some reason this changed the md5 sum assigned to two recipes (Shepherd’s Pie and Seafood Linguine Carbonara) which means they will be duplicated for every user I expect.  Shame.  Not life ending but annoying.  I submitted the update metadata around 21:20 on Friday, the app at around 21:57.  I fixed the bug at 21:45.  I didn’t run the OCUnit tests which would have spotted the md5 checksum changing when it should not have.

Easy to fix though and on the upside all of this code and unit tests means development is much faster now.  So I’m well on the way to iTunes document synchronisation support.  I’ve had to relocate a plist file and the sqlite database out of documents and into library otherwise they show up in iTunes.  So all you people that write core data tutorials putting sqlite in the documents folder… stop it!  It’s not a good idea use NSLibraryDirectory instead of NSDocumentDirectory when calling NSSearchPathForDirectoriesInDomains.

On another note, to determine what files were new in the documents folder I thought I’d use file info, like dates etc.  Figuring that when the file is written to the iPhone by iTunes it would write a new file but it doesn’t… it preserves the host machines modified and creation dates:

      NSDictionary *fileInfo = [[NSFileManager defaultManager] attributesOfItemAtPath:filePath
                                                                                error:nil];
      NSDate *lastModified = [fileInfo valueForKey:NSFileModificationDate];

(gdb) po fileInfo
{
    NSFileCreationDate = "2011-05-09 11:30:53 +0000";
    NSFileExtensionHidden = 0;
    NSFileGroupOwnerAccountID = 501;
    NSFileGroupOwnerAccountName = mobile;
    NSFileModificationDate = "2011-05-09 11:30:53 +0000";
    NSFileOwnerAccountID = 501;
    NSFileOwnerAccountName = mobile;
    NSFilePosixPermissions = 420;
    NSFileProtectionKey = NSFileProtectionNone;
    NSFileReferenceCount = 1;
    NSFileSize = 1788192;
    NSFileSystemFileNumber = 2608135;
    NSFileSystemNumber = 234881027;
    NSFileType = NSFileTypeRegular;
}

A shame but totally understandable as the file is not being modified or created as such. Looks like I’ll need to track a bit more metadata myself or waste time import/exporting everything every time. Err… no.

Posted in App Store, Chef's Book, Programming | Tagged , , , | Comments Off on Implementing iTunes syncing and a lesson to always run unit tests.

Roger Hargreaves 76th Birthday – Google Logos

Google have celebrated Roger Hargreaves 76th birthday with a whole stack of Mr. Men logos. As far as I can discern from the javascript there are 16 of them, after the cut. Continue reading

Posted in Art, Miscellany | Tagged , , , , | Comments Off on Roger Hargreaves 76th Birthday – Google Logos

Recipe for Tunnock Cakes (Marshmallow Teacakes)

I cooked this yesterday, just a little to late to make it into the current build of the app. Either way no photos here just the basic ingredients and steps, after entering everything into the app the easiest way to just post the recipe here was to cut and paste from an app generated PDF:

Finished Tunnock

Nom nom nom

Recipe Below Continue reading

Posted in Cooking | Tagged , , , | 2 Comments

Viewing the MySQL dump import progress

A really useful linux shell hint from someone elses blog… stumbled across browsing recommended in google reader:

Viewing the MySQL dump import progress.

Small linux utility called ‘bar’ which provides a progress bar for any program that can use a pipe for input…. e.g.:

bar if=foo.sql | mysql -u user -p database

Nice… dd esq CLI by the look of it.

Posted in Computer | Tagged , , , | Comments Off on Viewing the MySQL dump import progress