Fast BackInTime through network

Registered by spade

How to optimize BackInTime through network

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

My environnment is Linux Mepis kernel 2.6.32-1-mepis64-smp, but some parts of experience may be adapted with another environnment.
The remote backup disk is mounted on the PC to backup as a NFS drive.

Suddenly the backup of a PC to the harddisk of another one became very slow : 90 minutes instead of 5 minutes. Neither CPU, neither bandwidth is heavily used, buth the backup drive is heavily used, which could shorten its lifetime.
So I investigated to find the reasons of this change.

1/ I noticed yhat the LED on the router (Netgear RP614 used as a switch) corresponding to the client PC was yellow, instead of green. The meaning of this colour is that ethernet bandwidth is 10 Mbits/s instead of 100 Mbits/s. It was impossible to detect that with internet surfing, because the ISP bandwidth is about 10 Mbits/s. I checked twice this new limited speed with copying a big file through NFS and with the following command :
sudo mii-tool -v
The command "mii-tool" is availbale with the package "mii-tool".
To fix that, I used the following command to change the NIC settings:
sudo mii-tool -r
or
sudo ethtool -s eth0 speed 100 duplex full autoneg on
The choice of the autonegociation parameter is done after a test without it. But without autonegociation, I get ~40-50 Mbits/s, instead of ~90 Mbits/s. Maybe this setting is the best with my hardware and could be different with another one. I did not find any setting to change on the router side.
The NIC settings look to be stored in the NIC itself, and remain even if the PC si powered off (the NIC is still powered on). For some reason, those settings had changed on my NIC.

2/ After that change, the backup duration became 30 minutes, which is better but not enough. The only difference between a real backup and a copy of a big file is that the backup often consists in creating a lot of small files, which are hard links. I confirmed that with checking the system log file (filtered with the key word "backintime") : I can see that the timestamp of the task following "create hard links" is 30 minutes after this one. It looks like instead of writing many files per block, each file writing needs a disk access.
I found that the use of the parameter "async" at 2 locations could fix that:
- in the file /etc/fstab of the PC where the backup hard disk is embedded:
/dev/sdc3 /mnt/Backup ext3 users,exec,relatime,async 1 2
- in the file /etc/fstab of the PC to backup :
192.168.1.65:/mnt/Backup/ /mnt/Backup nfs rw,backup,hard,intr,async 0 0
or, if you use autofs, in the file /etc/auto.nfs :
Backup -fstype=nfs,rw,intr,async 192.168.1.65:/mnt/Backup/

So an almost-empty incremental backup is now 5~7 minutes.

3/ At this occasion, I asked myself about the best file system to use. EXT3 is very classical and I am very confident with it. But my backup partition is 1.3 TB large and it takes 1 hour to run a "fsck" on it, so it is not efficient enough. The other solutions are to use XFS, EXT4, ReiserFS, JFS. Reiser4 look to be abandoned and never really included in distributions kernels (it's a pity regarding performance). According to http://www.phoronix.com/forums/showthread.php?t=1765, JFS performances are lower than others. XFS is less flexible than EXT3/4, because downsizing is not possible without deletion then recreation. I had a bad experience with ReiserFS, because data look to be unrecoverable (probably like XFS). I consider this filesystem is still a candidate because I reckon that with a regular automatic fsck, it is very unprobable to loose both the data and the backup at the same time. For important and primary data, I keep EXT3.
So I made a benchmark test with a 77 GB set of files to backup on a 100 GB partition:

  EXT3 EXT4 XFS ReiserFS
1st backup 2:27:00h 2:20:00h 2:17:00h 2:14:00h
av. speed 72 Mbit/s 75 Mbit/s 77 Mbit/s 78 Mbit/s
other backups 1 min 2 min 1:30~3:30 min 1~2 min
(incremental, roughly empty)
fsck 3:30 min ~5 sec ~3 sec 1:46 min
big fsck 14:42 min
(450 Go data in a 1.26 To partition)

the backup partition is mounted as following :

EXT3:
(according to http://ubuntuforums.org/showthread.php?t=107856)
sudo tune2fs -o journal_data_writeback /dev/sdc3
/dev/sdc3 /mnt/Backup ext3 defaults,errors=remount-ro,noatime,auto,rw,dev,exec,suid,nouser,data=writeback,async 1 2

EXT4:
(according to http://blog.smartlogicsolutions.com/2009/06/04/mount-options-to-improve-ext4-file-system-performance/)
sudo tune2fs -o journal_data_writeback /dev/sdc3
/dev/sdc3 /mnt/Backup ext4 users,exec,noatime,data=writeback,errors=remount-ro,async 1 2

XFS
(according to http://everything2.com/title/Filesystem+performance+tweaking+with+XFS+on+Linux)
/dev/sdc3 /mnt/Backup xfs users,exec,noatime,nodiratime,logbufs=8,async 1 2

ReiserFS:
/dev/sdc3 /mnt/Backup reiserfs defaults,noatime,auto,rw,dev,exec,suid,nouser,data=writeback,async 1 2

Considering the fsck time as a discriminant critria, the winners are EXT4 and XFS. Considering that :
- I tried to copy manually files to the XFS partition and my computer rebooted without anything in the system log. It happened twice with small as well as big files.
- XFS is reputated to be efficient with big files but not with small files, like hard links
- Zenwalk now prefers EXT4 rather than XFS as it default file system
- EXT4 is also chosen by Ubuntu and Google,
I will use it too.

Any other information is welcomed.

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.