Skip to main content

测试rclone和restic

最近忽然想试试restic，看看几年后有什么变化。

实际上之前做数据备份计划时大致比较了rclone和restic，经过简单测试以后决定用rclone。当时对restic印象一般，主要考虑两点：

1. restic是专有格式，没有程序文件的话无法访问备份的数据。

2. 之前测试restic的时候莫名其妙备份仓库数据有损坏。

不过用了rclone几年以后发现第一条对rclone的加密仓库也成立。


测试数据

我挑选了一些数据，主要是图片和视频，分别用restic和rclone备份。

数据包含121191个文件，总共3.869TiB。由于ZFS压缩，实际占用磁盘3.6TiB。


测试环境

原始数据保存在一台Ubuntu 22.04.3的机器上，文件系统使用ZFS，启用zstd压缩，没有开启去重。

备份机器是一台群晖的机器，文件系统使用ext4.

两台机器用千兆网连接。在源机器上通过sftp访问备份机器。



restic备份

restic版本 restic 0.16.3 compiled with go1.21.6 on linux/amd64
备份仓库用默认参数创建。



最终是用了42个半小时结束。restic报告是3.655 TiB added to repo, 3.531 TiB stored on disk。备份仓库里最后有220568个文件。



我对结果比较惊讶，因为按这个结果看数据里有大概200GiB的重复数据块，我没想到有这么多。
不过最后占用磁盘大小跟原始文件在ZFS上的占用大小差不多，我也没想到。按理说已经去掉了200GiB文件，并且ZFS和restic我都用的zstd默认压缩率（也许它们的默认压缩率差很多？），我本来以为restic的仓库要再小点。


效率方面感觉42个多小时也太慢了点。不过我没找到性能瓶颈，很奇怪：


- 两台机器CPU占用率都不高
- 源机器上CPU有8核，但是restic基本用不到200%
- restic要调用ssh访问sftp，ssh大概也占用10%CPU
- 网络数据量大概50MB/s，这感觉很低
- 磁盘也没有充分利用。ZFS那边我见过几百MB/s的速度，而群晖那边也显示磁盘使用率在一半左右。


所以最后我也没搞清楚原因。


还有一个问题，restic经常跑跑停停，有时看似完全休眠了一般，CPU，磁盘，网络完全没有动静，不知道什么情况。



rclone 备份

rclone版本是 v1.65.0

启用了数据加密和文件名加密。

由于有一些文件名过长，ext4那边会报错，所以我去掉了一些数据，最后是53356个文件，总共2.44TiB。

rclone最终花了15小时44分传输完毕。

由于这个运行的比较快，中间我也没看系统占用率。



比较

平均算下来restic处理速度大概27MB/s，rclone处理速度大概45MB/s。

restic比rclone慢点是正常的，但是我觉得这两个整体都偏慢，至少我认为网络应该是瓶颈，应该能跑满才对。

（更新1：后来又做了一番测试，感觉是群晖那边CPU是瓶颈，处理SSH的加密比较吃力）

（更新2：虽然也有CPU的问题，但我发现每隔一段时间源服务器防火墙会屏蔽备份服务器的ip一分钟，这个大概就是效率低下的原因了。得检查一下防火墙哪里有问题）

（更新3：查到了restic关于sftp的bug，用rclone做中继后可以跑满带宽了）

rclone加密不能很好的应对长文件名，这个很头疼。此外，一些网盘（比如OneDrive）对路径总长也有限制。

restic支持压缩很好，rclone的压缩还是实验功能



总结

感觉restic还是有很多亮点的，尤其是了解了rclone的缺点之后。

接下来考虑一下哪些地方适合用上restic。

Labels:

Comments

Post a Comment

Popular posts from this blog

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...
2 comments
Read More »

[转] UTF-8 and Unicode FAQ for Unix/Linux

这几天,这个东西把我搞得很头疼 而且这篇文章好像太大了,blogger自己的发布系统不能发 只好用mail了 //原文 http://www.cl.cam.ac.uk/~mgk25/unicode.html UTF-8 and Unicode FAQ for Unix/Linux by Markus Kuhn This text is a very comprehensive one-stop information resource on how you can use Unicode/UTF-8 on POSIX systems (Linux, Unix). You will find here both introductory information for every user, as well as detailed references for the experienced developer. Unicode has started to replace ASCII, ISO 8859 and EUC at all levels. It enables users to handle not only practically any script and language used on this planet, it also supports a comprehensive set of mathematical and technical symbols to simplify scientific information exchange. With the UTF-8 encoding, Unicode can be used in a convenient and backwards compatible way in environments that were designed entirely around ASCII, like Unix. UTF-8 is the way in which Unicode is used under Unix, Linux, and similar systems. It is now time to make sure that you are well familiar ...
Post a Comment
Read More »

Moving Items Along Bezier Curves with CSS Animation (Part 2: Time Warp)

This is a follow-up of my earlier article.  I realized that there is another way of achieving the same effect. This article has lots of nice examples and explanations, the basic idea is to make very simple @keyframe rules, usually just a linear movement, then use timing function to distort the time, such that the motion path becomes the desired curve. I'd like to call it the "time warp" hack. Demo See the Pen Interactive cubic Bezier curve + CSS animation by Lu Wang ( @coolwanglu ) on CodePen . How does it work? Recall that a cubic Bezier curve is defined by this formula : \[B(t) = (1-t)^3P_0+3(1-t)^2tP_1+3(1-t)t^2P_2+t^3P_3,\ 0 \le t \le 1.\] In the 2D case, \(B(t)\) has two coordinates, \(x(t)\) and \(y(t)\). Define \(x_i\) to the be x coordinate of \(P_i\), then we have: \[x(t) = (1-t)^3x_0+3(1-t)^2tx_1+3(1-t)t^2x_2+t^3x_3,\ 0 \le t \le 1.\] So, for our animated element, we want to make sure that the x coordiante (i.e. the "left" CSS property) is \(...
3 comments
Read More »