Skip to main content

Find Duplicate Photos and Videos

Last month I received a warning that there's almost no more free space left for my Google account, so I decided to delete all files in Google Photos.

But before that, I downloaded all files via Google Takeout. I decided to copy files to my local backup if I don't have a local copy, for example, panorama photos generated by Google Photos. However I don't know which exactly ones that I should copy.

There are ~27k files in total. I processed them in different steps, through each step I expect to find more files that I do not need to copy.

Binary Comparison

I can easily find binary identical files, by computing hashsums of all files.

Unfortunatley, only ~7k files have a binary identical local backup.

Binary Content Comparison

It is possible two files have slighly different metadata but exactly same content (i.e. video stream).

Using `ffmpeg -f md5 -` I can compute a hashsum of the content of a file.

Unforunately, only 1 file is found in this step. Quite disappointing!

Of course, I should probably skip the first step, and just use ffmpeg from the beginning.

ImageMagick Comparison

Using `compare -metric mse`, I can compute a "similarity metric" between two files. But before that I need to scale the two input files into the same resolution.

To identify "visually identical" image files, I need to set a threshold of the MSE metric. I also use other signals like EXIF datetime and filenames. For example, if two files have exactly the same filename and exif datetime, I can use a very permissive MSE threshold.

With a few rounds of tuning, eventually I found a good set of thresholds. The end result was pretty good. I found local visually identical files for ~18k files.

These files are not binary identical, because Google Photos would rescale and compress the input files.

In general ImageMagick works pretty well, but there are some issues:
- It cannot handle CR2 files, it depends on `ufraw-batch`, which is no longer available on Debian.
- It cannot handle video files well, it converts video files to something huge, which is quite inefficient.

Manual Comparison

For about ~500 files, some potential candidates (as local duplicate) are found, but I'm not sure yet they are valid or not. So I just created a HTML file that present all files and their candidates side by side.

It's proven quite helpful. Most candidates are actual duplicates, but I was able to identify the rest.

Final Step

After all the steps, there are ~500 files left. I just copied all of them to my local backup.

Other Tools

There are other tools I found during the process, but didn't use:
Labels:

Comments

Post a Comment

Popular posts from this blog

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...
2 comments
Read More »

Chasing an IO Phantom

My home server has been weird since months ago, it just becomes unresponsive occassionally. It is annoying but it happens only rarely, so normally I'd just wait or reboot it. But weeks ago I decided to get to the bottom of it. What's Wrong My system set up is: Root: SSD, LUKS + LVM + Ext4 Data: HDD, LUKS + ZFS 16GB RAM + 1GB swap Rootless dockerd The system may become unresponsive, when the IO on HDD  is persistantly high for a while. Also: Often kswapd0 has high CPU High IO on root fs (SSD) From dockerd and some containers RAM usage is high, swap usage is low It is very strange that IO on HDD can affect SSD. Note that when this happens, even stopping the IO on HDD does not always help. Usually restarting dockerd does not help, but rebooting helps. Investigation: Swap An obvious potential root cause is the swap. High CPU on kswapd0 usually means the free memory is low and the kernel is busy exchanging data between disk and swap. However, I tried the following steps, none of the...
Post a Comment
Read More »

Moving Items Along Bezier Curves with CSS Animation (Part 2: Time Warp)

This is a follow-up of my earlier article.  I realized that there is another way of achieving the same effect. This article has lots of nice examples and explanations, the basic idea is to make very simple @keyframe rules, usually just a linear movement, then use timing function to distort the time, such that the motion path becomes the desired curve. I'd like to call it the "time warp" hack. Demo See the Pen Interactive cubic Bezier curve + CSS animation by Lu Wang ( @coolwanglu ) on CodePen . How does it work? Recall that a cubic Bezier curve is defined by this formula : \[B(t) = (1-t)^3P_0+3(1-t)^2tP_1+3(1-t)t^2P_2+t^3P_3,\ 0 \le t \le 1.\] In the 2D case, \(B(t)\) has two coordinates, \(x(t)\) and \(y(t)\). Define \(x_i\) to the be x coordinate of \(P_i\), then we have: \[x(t) = (1-t)^3x_0+3(1-t)^2tx_1+3(1-t)t^2x_2+t^3x_3,\ 0 \le t \le 1.\] So, for our animated element, we want to make sure that the x coordiante (i.e. the "left" CSS property) is \(...
3 comments
Read More »