Physical tape drives are a precious resource in today’s fast growing data centers. If you’re running backups or copies to physical tape drives performance is key. I did some research on this topic based on a customer request. They reported that backups on Windows clients where much faster compared to Linux clients when writing directly to tape (via locally Media Agent).
When checking the device configuration in Data Protector I’ve noticed that drive where using the default segment size for LTO drives (2000). After changing the segment size from 2000 to 20000 (my own default) the performance issue was gone!
The default value 2000 is for LTO1 drives and there is a close relationship between cartridge capacity and the size/number of segments that are created during a write operation. If you’re using modern drives such as LTO5, LTO6 or even LTO7 you can store multiple times the data that fits on a LTO1 cartridge (6000 GB vs. 100 GB native).
While the performance issue was resolved it was still not clear why a Windows client does not show the same reduced performance with a segment size of 2000.
It turned out that st driver in the Linux Kernel usually needs to flush its buffer when a file mark is written. It was obvious that there is a relation so I started looking into MTWEOFI. MTWEOFI allows the st driver to preserve the content of its buffer, enabling the next file operation to start immediately.
Starting with Data Protector A.09.05 the MTWEOFI functionality can be abled in the omnirc of the Linux Media Agent host. It is mandatory that the used Linux Kernel understands MTWEOFI. This is the case for RHEL 6.5 and later as well as for SLES11 and later (have not checked SLES10).
# Enable MTWEOFI support on Linux Media Agent / Performance OB2FORCESCSI=WFM OB2IMMEDFM=1
The following performance numbers have been collected during my tests to demonstrate the relationship of segment size and write throughput on Linux using Data Protector A.09.07. I’ve been using a single 16 GB file that was backed up directly to a LTO4 drive. The drive was configured with a 256k block size (default) and 32 agent buffers.
- Without MTWEOFI the drive would stop after every 2GB of data written resulting in a poor overall performance.
- A larger segment size (20000 over 2000) will cause the drive to stop less frequently resulting in a better performance.
- With MTWEOFI support the drive would not stop at all, even if a file mark needs to be written.
108,84 MB/s – default omnirc, segment size 2000 (default)
183,91 MB/s – default omnirc, segment size 20000
188,24 MB/s – modified omnirc, segment size 2000 (default)
190,48 MB/s – modified omnirc, segment size 20000
Test environment: Internal HP LTO4 SAS drive (Firmware U64D) attached to HP Smart Array P410 (Firmware 6.64) in HP ProLiant ML350 G6 running CentOS release 6.8, 2.6.32-642.1.1.el6.x86_64.
The Data Protector online help gives additional details on segment size:
A medium is divided into data segments, catalog segments, and a header segment. Header information is stored in the header segment, which is the same size as the block size. Data is stored in data blocks of data segments. Information about each data segment is stored in the corresponding catalog segment. This information is first stored in the Media Agent memory and then written to a catalog segment on the medium as well as to the IDB that keeps information regarding which data was backed up, to which media it was backed up, how and when backup, restore, and other sessions were run, which devices, libraries, and disk arrays are configured, and so on. It is implemented with an embedded database and a collection of proprietary data files which reside on the Cell Manager. See also DC directory and Detail Catalog Binary Files (DCBF).
Segment size, measured in megabytes, is the maximum size of data segments. If you back up a large number of small files, the actual segment size can be limited by the maximum size of catalog segments. Segment size is user configurable for each device and influences performance during restore and during import of media. You can adjust the segment size while configuring a new device or when changing the device properties using the Advanced options for the device.
Optimal segment size depends on the media type used in the device and the kind of data to be backed up. The average number of segments per tape is 50. The default segment size can be calculated by dividing the native capacity of a tape by 50. The maximum catalog size is limited to a fixed number (12 MB) for all media types.
Data Protector finishes a segment when the first limit is reached. When backing up a large number of small files, the media catalog limit is reached faster, which can result in smaller segment sizes.
The segment size affects the speed of restore and of the import of media. A smaller segment size requires additional space on the media because each segment has a fast-search mark. The additional fast-search marks result in faster restores because the Media Agent can quickly locate the segment containing the restore data. On the other hand, with smaller segments, there are more catalog segments, which makes the importing of media slower. An optimal segment size depends on the media type used in the device and the kind of data backed up. The default segment size depends on the media type. The minimum value you can specify is 10.