2024-02-12

Live Migrate Ubuntu 22.04.3 to Debian 12.5 on Raspberry Pi 4

I had a Ubuntu 22.04.3 on a Raspberry Pi 4. Recently I decided to switch it to Debian 12.5 in-place.

Mostly I was following this script: https://github.com/alexmyczko/autoexec.bat/blob/master/config.sys/ubuntu-deluxe 

It worked surprisingly well, however, the system was eventually broken (unsurprisingly) near the end, when I tried to upgrade the kernel.


I tried to fix initramfs by copying dtbs files, which seemed to work. Some references:

https://qiita.com/takasan/items/ef93be9e9d3f791eee66

https://bugs.launchpad.net/ubuntu/+source/flash-kernel/+bug/2012750


But the kernel did not load after reboot. Reverting /boot/config.txt didn't help.

Then I downloaded a Debian image and replaced only the boot partition of the micro SD card. This time the kernel was able to boot, but it couldn't load the filesystem.


I modified cmdline.txt, replaced "root=LABEL=RASPIROOT" with "root=/dev/mmcblk1p2", such that the kernel was able to load the filesystem. But a new error appeared: Cannot open access to console, the root account is locked.


At this point I found the process no longer fun, because it was such a pain to modify anything in the boot partition (power off Raspberry Pi, unplug the micro SD card and plug it into a PC, edit, unplug the micro SD card and plug it into Raspberry Pi, power on Raspberry Pi).


Eventually I just installed formatted the micro SD card, installed the Debian image and reconfigured the system. It was actually not slower than the in-place process.


I'd the say the ubuntu-deluxe script works pretty well. Most of the time I was just dealing with the difference between both distos (e.g. config files). Later I learned that the Ubuntu and Debian images used different methods for booting up Raspberry Pi. 

So theoretically it is possible to migrate from Ubuntu to Debian inplace. In fact there is a debtakeover script, which allows migrating to Debian from many other distros. On the other hand, normally it might make more sense to just reinstall the system.

2024-01-24

测试rclone和restic

最近忽然想试试restic,看看几年后有什么变化。

实际上之前做数据备份计划时大致比较了rclone和restic,经过简单测试以后决定用rclone。当时对restic印象一般,主要考虑两点:

1. restic是专有格式,没有程序文件的话无法访问备份的数据。

2. 之前测试restic的时候莫名其妙备份仓库数据有损坏。

不过用了rclone几年以后发现第一条对rclone的加密仓库也成立。


测试数据

我挑选了一些数据,主要是图片和视频,分别用restic和rclone备份。

数据包含121191个文件,总共3.869TiB。由于ZFS压缩,实际占用磁盘3.6TiB。


测试环境

原始数据保存在一台Ubuntu 22.04.3的机器上,文件系统使用ZFS,启用zstd压缩,没有开启去重。

备份机器是一台群晖的机器,文件系统使用ext4.

两台机器用千兆网连接。在源机器上通过sftp访问备份机器。



restic备份

restic版本 restic 0.16.3 compiled with go1.21.6 on linux/amd64
备份仓库用默认参数创建。



最终是用了42个半小时结束。restic报告是3.655 TiB added to repo, 3.531 TiB stored on disk。备份仓库里最后有220568个文件。



我对结果比较惊讶,因为按这个结果看数据里有大概200GiB的重复数据块,我没想到有这么多。
不过最后占用磁盘大小跟原始文件在ZFS上的占用大小差不多,我也没想到。按理说已经去掉了200GiB文件,并且ZFS和restic我都用的zstd默认压缩率(也许它们的默认压缩率差很多?),我本来以为restic的仓库要再小点。


效率方面感觉42个多小时也太慢了点。不过我没找到性能瓶颈,很奇怪:


- 两台机器CPU占用率都不高
- 源机器上CPU有8核,但是restic基本用不到200%
- restic要调用ssh访问sftp,ssh大概也占用10%CPU
- 网络数据量大概50MB/s,这感觉很低
- 磁盘也没有充分利用。ZFS那边我见过几百MB/s的速度,而群晖那边也显示磁盘使用率在一半左右。


所以最后我也没搞清楚原因。


还有一个问题,restic经常跑跑停停,有时看似完全休眠了一般,CPU,磁盘,网络完全没有动静,不知道什么情况。



rclone 备份

rclone版本是 v1.65.0

启用了数据加密和文件名加密。

由于有一些文件名过长,ext4那边会报错,所以我去掉了一些数据,最后是53356个文件,总共2.44TiB。

rclone最终花了15小时44分传输完毕。

由于这个运行的比较快,中间我也没看系统占用率。



比较

平均算下来restic处理速度大概27MB/s,rclone处理速度大概45MB/s。

restic比rclone慢点是正常的,但是我觉得这两个整体都偏慢,至少我认为网络应该是瓶颈,应该能跑满才对。

(更新1:后来又做了一番测试,感觉是群晖那边CPU是瓶颈,处理SSH的加密比较吃力)

(更新2:虽然也有CPU的问题,但我发现每隔一段时间源服务器防火墙会屏蔽备份服务器的ip一分钟,这个大概就是效率低下的原因了。得检查一下防火墙哪里有问题)

(更新3:查到了restic关于sftp的bug,用rclone做中继后可以跑满带宽了)

rclone加密不能很好的应对长文件名,这个很头疼。此外,一些网盘(比如OneDrive)对路径总长也有限制。

restic支持压缩很好,rclone的压缩还是实验功能



总结

感觉restic还是有很多亮点的,尤其是了解了rclone的缺点之后。

接下来考虑一下哪些地方适合用上restic。

2024-01-20

整理了一下RSS和Podcast

之前Google Reader挂了之后我基本一直用Feedly,后来也开始用Google Now看新闻,用Google Podcast听Podcast。


这几天决定切换到自己的服务器上,有若干原因:

  • 越来越不喜欢Google Now的自动推送。感觉我的眼界越来越窄。
  • Feedly经常遇到收费功能。
  • Google Podcast要关闭了。
  • 我正好也有服务器了

RSS抓取用的Tiny Tiny RSS,这个之前就搭建好了,作为Feedly的备份。最近升级发现docker compose文件有若干变化,数据库版本也变了,稍微折腾了一下。

类似的还有几个

  • Miniflux
  • FreshRSS
  • NewBlur
不过这些其实都大同小异。主要Tiny Tiny RSS导入导出备份都挺方便。


手机端用的Feedme,直接支持Tiny Tiny RSS的API,也可以放Podcast


我的RSS源都很老了,有好多都没法用了,Feedly也不提示我。发现了两个不错的RSS网站

  • https://docs.rsshub.app/
  • https://feedx.net/

Podcast回头也得整理一下,不过目前问题不大。

2023-10-30

Code Study Notes: Sphere Eversion

The Video


A Bit Background

Thanks to Youtube's algorithms, I got to watch this video again after many years. The video, made in 1994, explains the sphere eversion problem and visualizes one solution: Thurston's corrugations. More information can be found in the following links: 1, 2, 3. There is also an HD version.

The animations in the video are really fascinating. I figured it'd be a good excercise to recreate it with Blender, especially with geometry nodes. However I immediately got stuck after 20 minutes, because I had no clue how everything works.

Luckily I found a port of the original source code. (The original source code seems no longer available). The author also created a nice video. I'll refer this version as  mathIsART's version. I also found another port, which I'll refer to as McGuffin's version.

Mostly I studied mathIsART's version, and occasionally used McGuffin's version as reference. The code was really fun to read, and I learned a lot.  Here I'd like to summarize my learnings.

The Sphere

The most important thing I learned, is to parameterize everything. This allows a precise definition of the surface, and it also makes it easy to animate.

Each point on a sphere can be define by two angles: the latitude and the longitude. In mathIsART's  code, phi describes the latitude, and theta describes the longtitude. 

In blender, I'd start with a plane where both x and y values are in [0, 1], x and y will be mapped to phi and theta respectively. A subdivide modifer will give enough grid points and a set of geometry nodes can easily map the plane to the stripe. Clearly I only need to build a 1/8 sphere at the north hemisphere, the rest can be obtained by rotating the surface.


The Corrugation

A jellyfish!


At each point we can compute the local partial derivatives, actually we only care about the directions, i.e. east and north, from which we can also compute the up direction, i.e. normal.

McGuffin's version implemented 2-order jets while mathIsART implemented only 1-order jets. Since it is annoying to add lots of math nodes in Blender, I only use plain vectors unless the derivatives are absolutely needed. After several rounds of simplification, it seems that this is the only place where we need the derivatives. 

Corrugation is achieved by adding local offsets to each point, mathIsART's version uses all three directions, but I only used up and east. The offset on each direction is determined by a function of theta. The screenshot below shows the function for the up (or down) direction.

From mathIsART's code I can clearly see the trace of programming in 1990's. There are no complicated functions, yet specific curves/functions are obtained by multiple layers of interpolations. Today I got the luxury of the curve widget and instant rendering. I cannot imagine how I could implement it in 1995.



The size of the corrugation is a function of phi. Note that there is hardly any corrugation near the north pole. The piece looks like a squid:



The Push

A tabacco pipe!


This is one of the most inspiring things I learned from the code. At first I had no clue how to build this shape nicely, but the implementation was so short and elegant. It is a linear interpolation between two shapes, a positive sphere (actually a ellipsoid) and a negative sphere. The interpolation factor is a function of phi. Near the north pole (now being pushed to the south) it is the negative sphere and near the equator it is the positive sphere. 

This is actually how Bezier curves work, but I never realized that we can do the same for 3d surfaces!

The Twist

A cobra!


The twist is no longer a mystery, once I learned about the interpolation trick: we rotate the shape along different axes near the equator and near the north pole, and everything in the middle is an interpolation.

However there is still an interesting & confusing piece. Each point on the equator is locally rotated along with the axis that passes through that point and the sphere center. This could be quite counterintuitive at the first glance: does it mean the points are not moved at all? This is actually true if there is no corrugation, but the corrugation gives a local offset to the east or the west, the offset is a function of theta: size * sin(4 * PI * theta), where size is determined by phi.

When the offset is zero, the point is stable during the twist. There are 4 stable points on each stripe:


Interestingly, consider a point p1 on the equator that is near a stable point p2. Suppose p1 and p2 are respectively determined by theta1 and theta2, and let dt = theta1-theta2. 
Now we have  |p1-p2| ~= R*sin(dt) ~= R*dt
Because p2 is a stable point, sin (4 * PI * theta2) = 0, therefore the offset at p1 is:
size * sin(4 * PI * theta1) ~= size * 4 * PI * dt.
So the offset at p1 is linear to the distance between p1 and p2. This gives an illusion that p1 is rotating along with p2's axis (towards center), while p1 is actually rotating along its own axis (after applying the offset).

The Conclusion

Now the rest of the animation is more or less straightforward, just controlling all the parameters and add more interpolations.

Before making this animation, I only heard about "geometry nodes are powerful" but I have never tried them myself. I can confirm that they are indeed powerful and very easy to use. Thanks to Blender I can greatly simplify mathIsART's code:

- I don't need to worry about rendering at all. Blender takes care of everything.
- I don't need to implement something like "rotate along an axis", because there are already geometry nodes or modifiers for that.
- I can easily construct a function of a desired shape (e.g. using the curve widget), instead of trying to manually craft a magic function.
- Sometimes grid points phi (or theta) will not distribute evenly if a function is a applied. While 100% sure, I think mathIsART's code sometimes tries to migitate that by remapping phi, kind of like a timing function. However, I just ignore it and add more grid points. (Later I learned this is called arclength reparameterization)

By making this animation, not ony did I learned about parameterization, but I also refreshed my understanding of partial derivatives. What a journey!

Appendix: More about the Problem

There are many other solutions to this problem. Here is another elegant one: video, info, demo.

However, unfortunatlely I am not able to fully understand what's going on at the moment. Maybe I will try to take another look later.