Short Story:
I had several years of photos that I needed to identify and remove the duplicate. Instead of manually combing through 12,000 (read Long Story below) and before carpal tunnel set in, I needed a script to help me out. My situation may or may not be unique, so this script may not work 100% out-of-the-box for you, but it should get you started.
This script will identify duplicate photos in your iPhoto library and mark them with a comment (keyword) of “duplicate”. It will not delete anything.
To use:
- Download and unzip the script
- Double-click the script to open in Script Editor
- Go into iPhoto and select a group of photos you want to compare
- Switch back to Script Editor and run the script
- Don’t Touch Anything! Just let the script finish, it could take a while if you are comparing a lot of photos
- After the script is done, go back into iphoto and search for “duplicate”
- You can highlight all the duplicates and delete them or move them some place safe
Photos are considered a duplicate if:
- both heights match
- both widths match
- the photo date in iPhoto match, this is typically the EXIF creation date
There are no error checks in this script and it presents no interface except an alert when it’s done. If you need help, just post a comment below and I’ll do my best.
Long Story:
About a year ago I was editing down my iPhoto library of about 6000 images, just gitting rid of those out-of-focus shots and the ones of my wife’s feet (a curiously large number of these). After a long night of editing, the next morning I awoke to start again, but when I ran iPhoto there was nothing in the library.
It was all gone!
I couldn’t find anything anywhere. Could I restore from a backup? Ooh nooo. I had erased my backup drive the day before in preparation for moving the unwanted photos onto the backup drive and then making a new backup of my iPhoto Library. So I had no backup.
Not really funny. These were all the shots of my boys being born, first steps, first birthdays, first everything. I was up sh*t creek and it put a serious hurt in my stomach. At least I knew what to do: do nothing on the computer, boot from the Mac OS X install DVD and use Disk Utility to make a byte-for-byte copy of my internal hard disk. I could use this disk image to recover the images, hopefully.
So I tried several image recovery utilities and finally settled on PhotoRescue for Mac. I mounted the disk image of my internal disk and set PhotoRescue to the task. About 9 hours later (not a typo), PhotoRescue gave me several folders of recovered JPEGs, TIFFs, GIFs and PNGs. I tossed all but the JPEGs. I felt a little better at this point.
But when I looked in the JPEG folder there was over 12,000 images! Huh? Well, PhotoRescue does not discriminate, it recovers ALL images, including thumnails, web graphics, pron (you’ve been warned). Frankly, it was unbelievable and overwhelming.
So I set about dividing the images into folder that I knew were junk images and ones that I may want to keep. First, I eliminated everything below about 120K. I knew that my oldest digital camera was around 3M pixels and it saved a file that was typically > 200K so those images below 120K were most likely thumbnails and web images. That cut my stack almost in half.
Next I looked for images > 3M. These were corrupted image files that while they looked ok in Preview, I knew there was no way a 1200×1600 images was 40M. Just a consequence of PhotoRescue’s recovery routine. I can live with that, believe me. So I tossed everything > 3M because my current 6M pixel camera images are under 2M in size.
This left me with about 6,000 images that I imported into a new iPhoto Library. From the looks of it, all my images were there! What a relief, but the bad news was nothing was rotated properly, and there were many, many duplicates. Thousands of duplicates to be exact. After I rotated all the images so that I could view them properly, I set about removing the duplicates.
The good news about removing the duplicates was that they were fairly easy to spot. When I imported all the recovered images into iPhoto it apparently used the EXIF data data to date stamp each photo instead of using the photo file’s creation date, which was set by PhotoRescue to the day I performed the recovery. So all my photo’s were dated properly, I just had to look at each photo that matches (they were sorted by date) the one next to it and delete one of them.
A closer look at the duplicate photos revealed that while they had the same height, width and date/time, they varied in size. I was not able to determine why the file sizes varied as the images themselves looked identical, but my best guess is that the size difference came about from iPhoto’s insistence that when you rotate an image iPhoto considers this an “edit” and makes a copy of the original and add’s some iPhoto specific data (no verification on this though). So hey, if you are going to keep one, why not keep the smaller of the image files? So that’s what I was doing.
After hours and days of removing duplicates, I decided there has to be a better way. A bit of searching for “applescript iphoto remove duplicates” let me to Brattoo Propaganda Software’s Duplicate Annihilator. I tried the demo and it works very well. But there was one thing I wanted to do that Duplicate Annihilator could not, and that is mark the larger of the duplicate files. Duplicate Annihilator marks duplicate files by date/time which I am sure is what most people want to do. So definitely check it out.
So Duplicate Annihilator minor missing feature led me to write my own AppleScript to do pretty much the same. The script is pretty simple and requires no additional libraries or command line voodoo. But I will say that coding in Ruby for the past year-and-a-half really reminds my why I don’t like AppleScript. AS gets the job done, but it’s so much more work, frankly it’s confusing, and if you don’t do it often it’s a lot of work getting your head around AS’s nomenclature.
For you fellow rubists, there is rb-appscript which would have made my pain a little easier, but it relies on ruby and having the rb-appscript gem installed and that would be too much for most casual Mac users. So AppleScript won this round, but only because I knew I wanted to share the script for others.
Good luck to all you photo recoverers. I’ve been down your road before.
My apologies first as this may sound silly, I’m new to Mac.
I dont understand where to place the copy of the script. Do I open ‘iphoto’ and paste it in there somewhere?
I’m really needing a step by step guide.
I’m with those for whom the script *seems* to work (says it has worked) but then the iPhoto search doesn’t turn up any photos marked “duplicate.”
OS X 10.6.8 and iPhoto 08 7.1.5 (378).
Perfect! Thank you so much! You are wonderfull
Thank you. Works perfectly!
Hello,
I ran the script and iPhoto locked up and the screen “blanked” (i.e. everything vanished apart from the outline of the iPhoto page!) Had to Force Quit iPhoto. No apparent damage, but when I search for “duplicates”, I get the blank iPhoto page.
my start up disc is full due to the fact my iphoto has duplicated everything more than twice i ran duplicate annilhator but before i could delete them all my pics had disappeared and nothing was left in iphoto very annoying they are all in folders around the mac but i cant put them back as its full what can i do its very very annoying as it stops me from doing alot of other things clearly i am totally computer illiterate and am desperately needing some help ???
Can you tell me if your script will compare file sizes or just the physical size of the image? I am having an issue with Dup Annihilator marking some jpgs as duplicates and raw files as originals, even though the file sizes are different. Ideally I would like to be able to tell it to compare extensions not just just date/exif data, name and dimensions.
I am having a problem dragging my duplicates to the trash. I emptied my trash and that did not help. I cannot indicate trash for one or a group of pictures that are highlighted in the duplicates.
Hi Rosalie,
You can only trash pictures from the photos folder,not from albums…
hope this help..
good luck
Philippe
After the script is done, go back into iphoto and search for “duplicate”
You can highlight all the duplicates and delete them or move them some place safe.
This what I have done, but cannot trash them.
Hi!
Just to thank you very much for the script!
in about 2500 duplicate photos i could only point one not being dupe.
The only plugins or other software are not cheap, so i decided to give a try.
anyway thankfully apple decided to unify the libraries from iphoto and aperture!
so i imported all my photos to aperture, so that it moved all photos to folders by year/month/day.
then this “unification” allowed me to run the script on iphoto, flag the dupes, turn to aperture and delete all versions from the dupes (flagged) to trash, then empty it also moving the referenced files to the trash.(i did not find a way to do it in iPhoto)
now, i am only left to delete dupe photos with (1) or (n) on the name.
these photos because of different dimensions (thumbnails?) were obviously not detected by your script.
So thanks again Karl!
Thank you, worked perfectly
The script has been running for about an hour with no results. It just says “Running…” Is this normal? Thanks,
Erin
I am using your iPhoto script to identify duplicates. I keep getting message about apple event timing out. Working if I select a few photos to do but have 7,000 in lib
What am I doing wrong to get this message. I can leave it on all night to run if needed.
Hi there, thank you for the script. I need a little help…I’m getting an error because a message on iphoto is popping up saying it cannot find the original, and this in turn stops the script. “error “No result was returned from some part of this expression.” number -2763″ Can you help me?
perfect.. thanx a lot
I’ve been surfing on-line more than 3 hours today, but I never found any fascinating article like yours. It’s beautiful worth sufficient for me. In my opinion, if all web owners and bloggers made good content as you probably did, the internet can be much more helpful than ever before.
4 years later and the script still rocks – works like a charm and saves you $$$ – does what duplicate finder apps from the app store do for free.
not the fastes but works 100%
Thanks – Daddy rocks.
Thank you so much. Sometimes the simplest solutions are the best.
Well done and cheers!
-Jeff-
how long should this actually take. Im running it on 12000 images and its been going for like 3 hours. LOL =)
ok so once I have all the duplicates. Do I just delete them all? and it still leaves a copy in my photo’s? OR is this the original plus the duplicate in the folder?
One caveat. If you import both RAW and JPEG your script will consider dupes between them.
Here’s another caveat. When you use image path you’re actually getting the path of the preview file and not the original. You need to use original path to get the actual master file.
Hello Karl.
I have been using my iMac for about 5 years now, recently gave my iMac to my girlfriend and bought a PC. Trying to set up the PC with photos along with various backups i have made over the last few years has resulted in me having up to 6 duplicates in some cases! So i resolved myself to deleting every single photo individually! I had had enough of that after about 3 hours and so started to search online for a solution. Found this script and it has worked!!! Not only does this script mark the duplicate photos with the keyword “duplicate” but when i search in iPhoto for the keyword “duplicate” it only shows me the duplicated photos and leaves the original photo out from the search so i can easily delete thousands of duplicate photos!!
Thank you so much for being so kind as to share this with anyone who needs it. You are a kind genius!!
Thank you again
I have this error message? “iPhoto got an error: AppleEvent timed out.” number -1712 BUt it said that it had finished, I put in “duplicate” in the search box but nothing happened?
I got the same message but when i clicked search in the bottom of iPhoto and typed in duplicate, I had almost 1500 items show up. Some were duplicates but the system deleted both copies for some reason. I didn’t need the pictures anyhow but still.
You deserve some kind of Apple’s wage.
Thank you a lot! With this script you gave me more than 1 hour to play with my kids! Wonderful!
After using Iphoto AppleScript to remove duplicates I lost everything in images please help!
hi! looks interesting. trying it as i type. wouldu happen to have one for thumbnails?
this is self-less love for others… it’s Jesus Kindness… agape style even!
and a very awesome modern example of it.
thank you.
I’m not a programmer, but know enough to do serious damage.
Tried this script out on a few photos that I knew had duplicates. The script put “Duplicate” in the DESCRIPTION field under Information. I would prefer that “Duplicate” was added to the Keywords, because I didn’t want to over-write any existing Information.
So Searched around and found some references to iPhoto Keywords and tried modifying your script with:
FROM set comment of thisPhoto to “duplicate”
TO assign keyword of thisPhoto to “Duplicate”
Before I ran the script, I defined “Duplicate” as a Keyword in iPhoto.
Got an error for “to” (something about expected ‘given’, ‘with’, ‘without’ …etc). Don’t know enough to get any further.
If there is a quick response to this problem, please let me know. Otherwise, just drop this thread.
Thanks, Nik