WangLu's Notes

My home server has been weird since months ago, it just becomes unresponsive occassionally. It is annoying but it happens only rarely, so normally I'd just wait or reboot it. But weeks ago I decided to get to the bottom of it. What's Wrong My system set up is: Root: SSD, LUKS + LVM + Ext4 Data: HDD, LUKS + ZFS 16GB RAM + 1GB swap Rootless dockerd The system may become unresponsive, when the IO on HDD  is persistantly high for a while. Also: Often kswapd0 has high CPU High IO on root fs (SSD) From dockerd and some containers RAM usage is high, swap usage is low It is very strange that IO on HDD can affect SSD. Note that when this happens, even stopping the IO on HDD does not always help. Usually restarting dockerd does not help, but rebooting helps. Investigation: Swap An obvious potential root cause is the swap. High CPU on kswapd0 usually means the free memory is low and the kernel is busy exchanging data between disk and swap. However, I tried the following steps, none of the

Last month I received a warning that there's almost no more free space left for my Google account, so I decided to delete all files in Google Photos. But before that, I downloaded all files via Google Takeout. I decided to copy files to my local backup if I don't have a local copy, for example, panorama photos generated by Google Photos. However I don't know which exactly ones that I should copy. There are ~27k files in total. I processed them in different steps, through each step I expect to find more files that I do not need to copy. Binary Comparison I can easily find binary identical files, by computing hashsums of all files. Unfortunatley, only ~7k files have a binary identical local backup. Binary Content Comparison It is possible two files have slighly different metadata but exactly same content (i.e. video stream). Using `ffmpeg -f md5 -` I can compute a hashsum of the content of a file. Unforunately, only 1 file is found in this step. Quite disappointing! Of course