Short Story:
I had several years of photos that I needed to identify and remove the duplicate. Instead of manually combing through 12,000 (read Long Story below) and before carpal tunnel set in, I needed a script to help me out. My situation may or may not be unique, so this script may not work 100% out-of-the-box for you, but it should get you started.
This script will identify duplicate photos in your iPhoto library and mark them with a comment (keyword) of “duplicate”. It will not delete anything.
To use:
- Download and unzip the script
- Double-click the script to open in Script Editor
- Go into iPhoto and select a group of photos you want to compare
- Switch back to Script Editor and run the script
- Don’t Touch Anything! Just let the script finish, it could take a while if you are comparing a lot of photos
- After the script is done, go back into iphoto and search for “duplicate”
- You can highlight all the duplicates and delete them or move them some place safe
Photos are considered a duplicate if:
- both heights match
- both widths match
- the photo date in iPhoto match, this is typically the EXIF creation date
There are no error checks in this script and it presents no interface except an alert when it’s done. If you need help, just post a comment below and I’ll do my best.
Long Story:
About a year ago I was editing down my iPhoto library of about 6000 images, just gitting rid of those out-of-focus shots and the ones of my wife’s feet (a curiously large number of these). After a long night of editing, the next morning I awoke to start again, but when I ran iPhoto there was nothing in the library.
It was all gone!
I couldn’t find anything anywhere. Could I restore from a backup? Ooh nooo. I had erased my backup drive the day before in preparation for moving the unwanted photos onto the backup drive and then making a new backup of my iPhoto Library. So I had no backup.
Not really funny. These were all the shots of my boys being born, first steps, first birthdays, first everything. I was up sh*t creek and it put a serious hurt in my stomach. At least I knew what to do: do nothing on the computer, boot from the Mac OS X install DVD and use Disk Utility to make a byte-for-byte copy of my internal hard disk. I could use this disk image to recover the images, hopefully.
So I tried several image recovery utilities and finally settled on PhotoRescue for Mac. I mounted the disk image of my internal disk and set PhotoRescue to the task. About 9 hours later (not a typo), PhotoRescue gave me several folders of recovered JPEGs, TIFFs, GIFs and PNGs. I tossed all but the JPEGs. I felt a little better at this point.
But when I looked in the JPEG folder there was over 12,000 images! Huh? Well, PhotoRescue does not discriminate, it recovers ALL images, including thumnails, web graphics, pron (you’ve been warned). Frankly, it was unbelievable and overwhelming.
So I set about dividing the images into folder that I knew were junk images and ones that I may want to keep. First, I eliminated everything below about 120K. I knew that my oldest digital camera was around 3M pixels and it saved a file that was typically > 200K so those images below 120K were most likely thumbnails and web images. That cut my stack almost in half.
Next I looked for images > 3M. These were corrupted image files that while they looked ok in Preview, I knew there was no way a 1200×1600 images was 40M. Just a consequence of PhotoRescue’s recovery routine. I can live with that, believe me. So I tossed everything > 3M because my current 6M pixel camera images are under 2M in size.
This left me with about 6,000 images that I imported into a new iPhoto Library. From the looks of it, all my images were there! What a relief, but the bad news was nothing was rotated properly, and there were many, many duplicates. Thousands of duplicates to be exact. After I rotated all the images so that I could view them properly, I set about removing the duplicates.
The good news about removing the duplicates was that they were fairly easy to spot. When I imported all the recovered images into iPhoto it apparently used the EXIF data data to date stamp each photo instead of using the photo file’s creation date, which was set by PhotoRescue to the day I performed the recovery. So all my photo’s were dated properly, I just had to look at each photo that matches (they were sorted by date) the one next to it and delete one of them.
A closer look at the duplicate photos revealed that while they had the same height, width and date/time, they varied in size. I was not able to determine why the file sizes varied as the images themselves looked identical, but my best guess is that the size difference came about from iPhoto’s insistence that when you rotate an image iPhoto considers this an “edit” and makes a copy of the original and add’s some iPhoto specific data (no verification on this though). So hey, if you are going to keep one, why not keep the smaller of the image files? So that’s what I was doing.
After hours and days of removing duplicates, I decided there has to be a better way. A bit of searching for “applescript iphoto remove duplicates” let me to Brattoo Propaganda Software’s Duplicate Annihilator. I tried the demo and it works very well. But there was one thing I wanted to do that Duplicate Annihilator could not, and that is mark the larger of the duplicate files. Duplicate Annihilator marks duplicate files by date/time which I am sure is what most people want to do. So definitely check it out.
So Duplicate Annihilator minor missing feature led me to write my own AppleScript to do pretty much the same. The script is pretty simple and requires no additional libraries or command line voodoo. But I will say that coding in Ruby for the past year-and-a-half really reminds my why I don’t like AppleScript. AS gets the job done, but it’s so much more work, frankly it’s confusing, and if you don’t do it often it’s a lot of work getting your head around AS’s nomenclature.
For you fellow rubists, there is rb-appscript which would have made my pain a little easier, but it relies on ruby and having the rb-appscript gem installed and that would be too much for most casual Mac users. So AppleScript won this round, but only because I knew I wanted to share the script for others.
Good luck to all you photo recoverers. I’ve been down your road before.
Tags: applescript, iphoto
November 27th, 2008 at 3:56 am
Karl,
Thanks for your script.
I have developed your script a little further. How would you like me to acknowledge your work as I plan to attach GPL license to it?
Phil
November 27th, 2008 at 9:27 am
@Phil:
Oh, I’m not picky. I think just a link to this blog post is fine.
Thanks for asking.
Also, I think if you added another comment with a link to your page, that would be helpful to anyone who views this post.
November 30th, 2008 at 1:56 pm
Here is a link to my version of your script as well as a script to clear existing comments:
http://philatwarrimoo.blogspot.com/2008/11/iphoto-script-to-tag-duplicates.html
December 19th, 2008 at 4:09 pm
Great Script, worked great for me, thanks!
January 13th, 2009 at 11:59 am
“it could take a while” … How long for an iPhoto library with about 10,000 photos?
January 13th, 2009 at 12:12 pm
@Tony:
Wow, I have no idea how long it would take, but I would imagine it would take hours, maybe upto an entire day. It also depends on the computer (how fast it is).
My recommendation would be to break it down into groups of 250. Run it the group of 250 to get an idea of how long it takes.
When I have run it on groups of ~175 it took about 10 minutes (but I didn’t sit and watch it) on a PBG4 (a bit on the slow side).
Let us know how it goes!
Karl
March 31st, 2009 at 4:54 pm
Good scripts thanks all.
however the date usually doesn’t work especially when u have transfered and manipulated photos several times. I recomend using metadata which are not changed whatever happens. In addition, and it is important as applescript is slow, you ll reduce the nbr of tests to just this one as there are little chances are that 2 photos can have be taken at the EXACT same time – Especially is there is only one camera in the house!
June 22nd, 2009 at 10:46 pm
Thank you so much for putting this up. I just reformatted my computer ,but when I reintroduced my iphoto library it duplicated itself 4x. Your script saved me a chunk of sanity.
July 7th, 2009 at 1:28 pm
HELP! I ran your script on my album and now when I click on an image iPhoto only shows an empty placeholder with a question mark in it. the Thumbnails in the list looks alright though. Please tell me what to do!
September 16th, 2009 at 7:36 am
Nice script! I ran it on a collection of about 17,000 items on an Intel Mac mini, which took about seven hours – just for everyone’s orientation. It found both duplicate photos and videos.
September 24th, 2009 at 9:44 pm
Your script sounds perfect, but I’m concerned about using it because there is no answer to Panic!’s question. Is there a solution for that issue?
September 24th, 2009 at 9:57 pm
@ocayd:
Well, I replied to panic but got nothing back. I suspect that he had iPhoto Library corruption issues before he ran this script. I have seen client’s iPhoto behave like that.
Keep in mind:
1. Backup first!
2. The script only adds a comment to duplicate photos, it doesn’t change anything else, so it’s very safe.
Quite a few people have used it and experienced no problems. If you backup, you know you are safe.
December 29th, 2009 at 4:12 pm
Thanks very much for the script, appreciate the sharing.
March 18th, 2010 at 10:53 am
[...] llamado iPhoto Diet que promete hacer exactamente eso. Además puedes hacerlo mediante un script en esta página o mediante éste programa de [...]
April 10th, 2010 at 9:23 pm
Script works great. – but I keep getting a “Iphoto time out” error. When I do a large # of Photos (over 1,000). Any Idea what’s causing this. Do I have my screen saver kicking in too soon. Works if I touch the iPhoto application every once in a while.
April 12th, 2010 at 8:55 am
@Todd:
You may have to process them in smaller groups. That should take care of the timeout error.
I don’t know if the screen saver makes a difference, i don’t use one. But I wouldn’t hurt to turn it off momentarily while processing the photos.