Skip to main content

Posts

Chasing an IO Phantom

My home server has been weird since months ago, it just becomes unresponsive occassionally. It is annoying but it happens only rarely, so normally I'd just wait or reboot it. But weeks ago I decided to get to the bottom of it. What's Wrong My system set up is: Root: SSD, LUKS + LVM + Ext4 Data: HDD, LUKS + ZFS 16GB RAM + 1GB swap Rootless dockerd The system may become unresponsive, when the IO on HDD  is persistantly high for a while. Also: Often kswapd0 has high CPU High IO on root fs (SSD) From dockerd and some containers RAM usage is high, swap usage is low It is very strange that IO on HDD can affect SSD. Note that when this happens, even stopping the IO on HDD does not always help. Usually restarting dockerd does not help, but rebooting helps. Investigation: Swap An obvious potential root cause is the swap. High CPU on kswapd0 usually means the free memory is low and the kernel is busy exchanging data between disk and swap. However, I tried the following steps, none of the
Recent posts

Find Duplicate Photos and Videos

Last month I received a warning that there's almost no more free space left for my Google account, so I decided to delete all files in Google Photos. But before that, I downloaded all files via Google Takeout. I decided to copy files to my local backup if I don't have a local copy, for example, panorama photos generated by Google Photos. However I don't know which exactly ones that I should copy. There are ~27k files in total. I processed them in different steps, through each step I expect to find more files that I do not need to copy. Binary Comparison I can easily find binary identical files, by computing hashsums of all files. Unfortunatley, only ~7k files have a binary identical local backup. Binary Content Comparison It is possible two files have slighly different metadata but exactly same content (i.e. video stream). Using `ffmpeg -f md5 -` I can compute a hashsum of the content of a file. Unforunately, only 1 file is found in this step. Quite disappointing! Of course

Dates in Leaked Passwords

I have been playing with data from haveibeenpwned.com , especially I was exploring 8-digit passwords that look like dates. I checked all dates between 19000101 and 20231231, where - The date must be in the YYYYMMDD format - Some invalid dates are also included, e.g. 19120231 Further, by default I only analyzed valid  dates after 19240101. By Year 1986 is the most popular year. It roughly forms a normal distribution. The distribution might review the distribution of the ages of the users (of the leaked services). By Month & Day It is clear that October, November and December are more popular than other months, but I think this is is because people may use 1912312 rather than 19120312. We can also see a pattern for each month, especially 1st is more popular than other days. Other than those, the passwords are roughly evenly distributed among all days. The most popular dates are: 0101, #=81516 1010, #=79997 1212, #=71724 1111, #=62065 1001, #=61109 Seems that they are just nice &