04 September 2011

iPhoto and durable photo management


When managing your digital photos, there are three things you really should do:
  • Make backups of your backups of your backups.  These are your photos, don't mess about here - make backups regularly and store one of your backups someplace remote.
  • Use JPG for your file formats.  If you end up with a camera saving in some goofy format either convert to JPG or get a new camera.  JPG is like mp3 is for music - it'll be a durable photo format that will be around for a very long time and is supported by lots of tools.
  • Don't use software to organize your photos for you.  Use a simple directory format.  Software and their proprietary organization approaches will come and go but simple directories and folders will be around for a long time to come.  The following blog article is on this last point.
In 1999 when I bought my HP Photosmart C30 I started organizing my photos in a simple directory structure: Photos/YYYY/YYYYMMDD.  Something like this:


Over the years I would try whatever photo management software came with new cameras or emerging opensource packages.  l always regretted it as the photos would end up being hidden inside of some database and difficult to extract.  I would end up maintaining photos in two locations - the photo software and my simple archive.  That is, until I would get rid of the software and proprietary database files.

All that changed when I drank the Apple, iPhone and iPhoto koolaid when the iPhone came out.  The iPhoto software hits the magic "good enough" point Apple does so well and the integration with the iPhone Just Worked.  For the first few months I maintained iPhone photos in two locations - iPhoto and my simple archive.  But as I started using iPhoto more and more, using it to maintain photo albums, I just got lazy and one day stopped copying files from iPhoto into the archive.

I did continue to use my simple archive for regular camera photos but of course over time I started using the phone more and more for photos and conventional cameras only for special occasions like holidays.  I would pull in other camera pictures from the my simple archive into iPhoto to create albums.

I kept telling myself I could extract the photos any time - you can enter the iPhoto database structure as a filesystem (~/Pictures/iPhoto Library - "Show Package Contents"), similar to my simple archive approach.  iPhoto's File->Export function was there of course, but it would not properly set file modified date meaning it was more difficult to get into the right folder without looking at header data in the JPG.  Meanwhile, time goes on, and nothing lasts forever, maybe not even iPhoto.

That brings us to today and here's my strategy:  I've accepted that iPhoto will hold the "master" version of all iPhone (and more recently Samsung Galaxy s2 photos) and I'll occasionally extract these photos from iPhoto and fold them into my simple archives.

Here is how I did it:

1. Within iPhoto, Command-F search all your Photos for "iPhone".  Assuming you haven't otherwise marked your photos (title, tags, event names, ...) with "iPhone", iPhoto gives you a subset of your Photos created with the iPhone.

2. Within iPhoto, use File-Export to save these photos to the filesystem.  They'll be saved as one big set in whatever directory you specify.  I named the files with prefix "iphone" and selected the option to sequentially number them.  That way I can always go searching in my simple archive for iPhone photos by filename if I need to.

3. Use MacPort.org's "port" command to install jhead.  Run "jhead -ft *.JPG *.jpg" at the shell to correct all the modified dates so your files are date/time stamped with the date/time the picture was taken.  I also had a handful of .PNG (screen captures) and .MOV (movies) files and I just left these dates as they were.  Probably wrong, but I only had a few.

4.  I wrote a small shell script to organze the extracted pictures by date to match my simple archive format.  Here it is:


# orgpix.sh - organize iPhoto exported photos by date 
# Files will be moved to into a directory structure like this: 
#
# YYYY 
#    YYYYMMDD
#    YYYYMMDD 
#    ... 
# YYYY 
# ... 

# Place this file in the directory containing all the files you want 
# to organize and run it from there. 


ls -1 *.jpg | while read fn; do 
   eval set `stat -f "%Sa" -t "%Y %m %d" "$fn"` 
   export year=$1 
   export month=$2 
   export day=$3 
   echo "$fn : $year $month $day" 
   mkdir $year 2> /dev/null 
   mkdir ${year}/"${year}${month}${day}" 2> /dev/null 
   mv "$fn" ${year}/"${year}${month}${day}" 
done

5. Used cp to merge the newly organized files and directories into my main photo archive:

cp -Rvpn 2008/ /Path/to/Photo/Archives/2008

(I was tentative.  I ran it for each year by hand as I wanted to make sure it was working properly by checking a few file and directory creations as I went.  This could be scripted as well.)

So there you go.  All photos taken by the iPhone extracted from iPhoto and merged into my simple but very durable photo archives structure.

I repeated the above to extract my Samsung Galaxy s2 (Command-F search for "GT-I9100") photos as well.

Postnote: If you can't subset out the photos taken by the iPhone within iPhoto, you could just extract everything and use jhead's EXIF header reading to determine what camera took a photo.

5 comments:

  1. I am using shotwell on ubuntu, it does most of the stuff similar to this automatically. Organise by year, month, date and removing duplicates etc

    ReplyDelete
  2. A few additions to the original blog entry:

    1) Add some text to the date named directory to remind you where you were or what was going on that day. For example name a directory "20120620 Family Reunion in Sacramento". I started doing this a few years ago and am gradually filling out past years when I'm looking for an older photo.

    2) Dropbox! Dropbox has recently added photo uploading. Automatic over wireless for Android phones and plug-in required for Apple iOS iStuff and camera memory cards. Works great. Upload the better photos from Dropbox into your photo management/editing software (e.g., iPhoto or shotwell) then drag your photos out of Dropbox into your main photo repository to clean out your Dropbox photos folder. Having all your photos consolidate onto a single folder in Dropbox as a staging area is exactly how photo management should work. It's like Apple's Photo Stream, but not so proprietary.

    ReplyDelete
  3. A few months have gone by since my move to Dropbox. I'm satisfied its working as it should. I therefore did a final run at making sure everything in iPhoto was also in my main photo archive. A few gotchas and suggestions:
    - I didn't know where I'd stopped with my last extract which had been over a year ago. I ran a "find . -name \*iphone\* -print | sort" across all files and folders to find the latest photo from the last extract. It's lucky I'd prefixed all extracted photo filenames with "iphone" else this would have been more painful. I suggest using a clear and unambiguous filename prefix whenever you extract photos. Don't use the original filename - it's a mess if you have multiple devices and people in the same folder set.
    - I had to manually determine where I'd starting using Dropbox as described above. Dropbox has a solid default file naming convention making this easy.
    - Be careful how you extract files. I originally suggested above "iPhone", but that doesn't include iPad pictures for example. I suggest using "Kind" of "Original" when you extract, don't accidentally convert other formats (like those PNG screen snaps) into JPGs. Keep in mind any other (non-Apple) phones or SD cards whose photos you uploaded straight into iPhoto and perhaps didn't remember to also put into your photo archive. Keep in mind "movies" if you have any.
    Good luck!

    ReplyDelete
  4. A follow-up and tip regarding photo management and dropbox:
    - I've continued to use Dropbox's Camera Upload function to use Dropbox as a central convergence point for all photos I take on all devices. Their file naming choice for imported files is brilliant simplicity (e.g., "2014-04-18 16.41.34.jpg") which allows all device photos to end up in the same spot (Dropbox/Camera Uploads). I've had an occasional name collision (i.e., a photo taken at the exact same date/time on two devices) but Dropbox handles this by altering the name of the second and subsequent collision.
    - By regularly clearing out files from Dropbox's "Camera Uploads" folder to my archive I'm able to stay well under Dropbox's "free" storage limits. Dropbox generally remembers what it has uploaded and only copy/uploads files once even if it seems them again.

    Now for the tip: If you receive photo files from others, you'll probably receive them in a variety of forms with various naming conventions (quite possibly not Dropbox's "YYYY-MM-DD HH.MM.SS" format). Here is a trick to have Dropbox import them into your "Camera Uploads" folder:
    - Using a spare USB stick or SD card, create a folder in the root (top) folder: "DCIM/100ABCDE".
    - Copy all of the photos you've received into the folder you've created
    - (I assume you have Dropbox's camera upload enabled)
    - Unplug the SD card or USB stick and then plug the drive back in. Dropbox should see the photos and offer to import them.

    So now when you have a party or family reunion you can collect photos from everyone and converge them into your photo archive using Dropbox to import and rename them to a consistent format.

    ReplyDelete
  5. Speaking of photo management and dropbox, I just happened to connect Google Glass to an OS X laptop running dropbox and Glass comes up as a "camera" from Dropbox's point of view. Same for OS X's Image Capture application. That's great news in that you don't have to mess about with sync and cross-over hell of photos from Google photo management to use the same "dropbox convergence" strategy for all your devices, including Glass originated photos.
    (Speculation: this implies that Glass is supporting the PTP or "Picture Transfer Protocol" over USB - excellent)

    ReplyDelete