WangLu's Notes

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now. I am setting up a maintenance pipeline for my virtual machines . The pipeline has two main routines: The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs. There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week. Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup pro...

Last month I received a warning that there's almost no more free space left for my Google account, so I decided to delete all files in Google Photos. But before that, I downloaded all files via Google Takeout. I decided to copy files to my local backup if I don't have a local copy, for example, panorama photos generated by Google Photos. However I don't know which exactly ones that I should copy. There are ~27k files in total. I processed them in different steps, through each step I expect to find more files that I do not need to copy. Binary Comparison I can easily find binary identical files, by computing hashsums of all files. Unfortunatley, only ~7k files have a binary identical local backup. Binary Content Comparison It is possible two files have slighly different metadata but exactly same content (i.e. video stream). Using `ffmpeg -f md5 -` I can compute a hashsum of the content of a file. Unforunately, only 1 file is found in this step. Quite disappointing! Of course...

最近忽然想试试restic，看看几年后有什么变化。 实际上之前做数据备份计划时大致比较了rclone和restic，经过简单测试以后决定用rclone。当时对restic印象一般，主要考虑两点： 1. restic是专有格式，没有程序文件的话无法访问备份的数据。 2. 之前测试restic的时候莫名其妙备份仓库数据有损坏。 不过用了rclone几年以后发现第一条对rclone的加密仓库也成立。 测试数据 我挑选了一些数据，主要是图片和视频，分别用restic和rclone备份。 数据包含121191个文件，总共3.869TiB。由于ZFS压缩，实际占用磁盘3.6TiB。 测试环境 原始数据保存在一台Ubuntu 22.04.3的机器上，文件系统使用ZFS，启用zstd压缩，没有开启去重。 备份机器是一台群晖的机器，文件系统使用ext4. 两台机器用千兆网连接。在源机器上通过sftp访问备份机器。 restic备份 restic版本 restic 0.16.3 compiled with go1.21.6 on linux/amd64 备份仓库用默认参数创建。 最终是用了42个半小时结束。restic报告是3.655 TiB added to repo, 3.531 TiB stored on disk。备份仓库里最后有220568个文件。 我对结果比较惊讶，因为按这个结果看数据里有大概200GiB的重复数据块，我没想到有这么多。 不过最后占用磁盘大小跟原始文件在ZFS上的占用大小差不多，我也没想到。按理说已经去掉了200GiB文件，并且ZFS和restic我都用的zstd默认压缩率（也许它们的默认压缩率差很多？），我本来以为restic的仓库要再小点。 效率方面感觉42个多小时也太慢了点。不过我没找到性能瓶颈，很奇怪： - 两台机器CPU占用率都不高 - 源机器上CPU有8核，但是restic基本用不到200% - restic要调用ssh访问sftp，ssh大概也占用10%CPU - 网络数据量大概50MB/s，这感觉很低 - 磁盘也没有充分利用。ZFS那边我见过几百MB/s的速度，而群晖那边也显示磁盘使用率在一半左右。 所以最后我也没搞清楚原因。 还有一个问题，restic经常跑跑停停，有时看似完全休眠了一般，CPU，磁盘，网络完全没有动静，...

Around 2013, every year I'd burn all my important data into a single DVD, that is 4.7GB. Nowadays I have ~5TB data, and I don't even bother optimizing 5GB data. Background I realized that it is the time to consider backup. I guess I have seen enough signs. The NAS shows that the disks are quite full. I just happened to see article and videos about data backup. I found corrupted data in my old DVDs. I realized that most of my important data are not properly backed up. I have a few scripts that manage different files, which might contains bugs. The goal is to have good coverage under acceptable cost. The Plan All my data are categorized into 4 classes. Class 1: Most Important + Frequently Accessed Rougly ~50GB in total. Average file size is ~5MB.  Examples include official documents, my artworks and source code. The plan: sync into multpile locations to maximize robustness. Sometimes I chooes a smaller subset when I don't have enough free space. In case some copies are down/...