16May/1213
Scrap the SCP. How to copy data fast using pigz and nc
You are currently browsing comments. If you would like to return to the full story, you can read the full entry here: “Scrap the SCP. How to copy data fast using pigz and nc”.
You are currently browsing comments. If you would like to return to the full story, you can read the full entry here: “Scrap the SCP. How to copy data fast using pigz and nc”.
June 13th, 2012 - 12:28
Nice post, thanks!
As you’re interested in non-secure file transfer, it would be nice to see how the good ol’ FTP fits in this comparison
Cheers
–
Kamal
June 19th, 2012 - 11:48
Thanks, Kamal.
Yes, it would be interesting to compare speeds with FTP. The trouble is – these days FTP is almost never to be found in production networks.
Theoretically, I have no doubt that FTP will be faster than SCP. Will it be faster than NCP ? Probably not, as NCP had two big things going for it: compression and parallelism, which standard FTP does not have.
July 23rd, 2012 - 21:40
very good comparison
Can you try rsync with compress.
regards
loh
October 10th, 2012 - 20:44
Thank you very much for this post. I was looking for a solution to the slow scp transferring speeds, and this works for me. The problem with scp is that it does not parallelize file transferring. If you use graphical scp/sftp clients like Filezilla or Transmit (OS X), they are very fast compared to the scp command line client but because they transfer several files simultaneously. However, the speed problem will still be there if, for example, you have a folder with lots of small files and one very big file (20GB) the speed will stall when transferring that file. You’ll notice a small improvement, as the small files will be transferred quickly, but as soon as the transfer is reduced to the single big file, you’ll encounter the usual problem.
I didn’t know about pigz and pv, so I learned a couple of things from your script that will be very useful in the future. I had used nc and tar for transfers in the past, but I didn’t know how to show the progress (speed, ETA, % done,…) or that you could use a multi-threaded compressor like pigz.
October 10th, 2012 - 20:48
Thanks, Sergio.
Glad that you could make it useful.
Cheers
November 21st, 2012 - 16:06
Hi,
nice one.
Speed is awesome when using both pigz and lzop. I first thought the script got stuck after I initialized a second file transfer to the same filesystem, but checking the target server using nmon, I found that dirty pages were still in progress of being flushed to disk and this caused the “freeze” of the second copy, as the device was 100% busy for some time. I will do some testing with tuning the target ext3 filesystem (writeback / commit) and use xfs (noatime,nodiratime,nobarrier,logbufs=8) for comparison. As well it might be worth doing some tests with dd.
Regards
Efstathios
November 26th, 2012 - 08:47
Hi Efstathios,
Very interesting, do keep us posted. It would be nice to know how much delay is caused by the file system (I had regular ext3 noatime in my tests) and whether different zippers have better timing (do zippers other than gzip have parallel counterparts or options?)
Regards
December 6th, 2012 - 15:48
Hi Maxym,
as usal, it depends on your individual situation. So it plays a big role how your i/o stack looks like and how your array is configured. At good start point is to figure out, if you are CPU or i/o bound. I normally do a simple test by setting up a tmpfs or ram disk at the target server and copying data from source to destination as a start to check how it would like without disks involved. For measurement you can user various tools. I personally can recommend nmon for linux and collectl.
Having performed this tests, on you can start to tune your system accordingly.
Our setup consists of HP Blade Server and EMC VMAX Storage.
The array is tuned for best response time (good for oltp) rather than throughput.
We use buffered ext3 on lvm with dm-multipath, so our request physically get written at block size 4k to disk at the end, when the dirty pages are flushed, which means alot of small requests to be commited into the SAN’s write cache. So now the art to tweak your system to use the file system cache efficiently, meaning we want some writes to be buffered, but not as much that the system get’s a large backlog and freezes up.
Keep in mind the following:
- Scope (Transfer as fast as possible from A to B only or also at the right destination, 1 time or iterative)
- General Transfer Strategy based on scope (Set Size, Compression Tool, Buffering, Risk Management of Data Loss on Target due to tuning)
- Schedule (place the transfer at a idle time to use the target server’s resource optimally)
In our setup we have found ourselves to be limited in sequential write performance and medium network bandwith, but having sufficient cpu and memory available. We had the requirement to transfer files always from scratch, so using netcat for initial transfer and then use rsync for deltas was not required. It is however a good starting point for a simple database backup to a remote server using alter database begin backup / rsync / end backup mode (be sure you backup properly as you can read from the various metalink notes on hot backup of databases).
In such scenarios it is important to find a proper transfer set size, as using a “read a bit” / write a bit”-strategy seems appropriate:
Example
- tar files (split at 1GB Size)
- transfer 1GB split chunk from source to destination (target system cache should be able to hold the write), while you create the next chunk at the source
- write these chunks to multiple file systems backed by different disks/luns, so you have more queues
In most cases I personally tested, I had good results with database files on lzop. Lzop suited our needs better than PIGZ. It has faster compress/decompress, that’s why ORACLE uses it in it’s 11g Avanced Compression Option for RMAN, for which you get a free special use license, if you use Oracle Secure Backup by the way.
As general starting point:
Network bandwidth: – start pigz, also try pbzip2
CPU bound: – start with lzop
I/O bound: – split into small chunks, that fit easily into fs-cache and can be handled by the backed
– Write chunks to multiple different file systems
– If possible increase queue size at the block device /sys/block/sda/queue/nr_requests
– Increase ext3 commit at target system (eg. commit=180)
– Test with ext2 or xfs as well
– On SATA, check write cache is enabled
In most cases what you can write on hold for writing to disk will be the limiting factor for most of us.
I hope this feedback was useful to anybody. I you have any corrections / found mistakes, please give feedback, as we are all willing to learn and nobody is perfect
Regards
Efstathios
March 12th, 2013 - 16:59
Great research and very good of you to share your findings. Another option if you are forced to use scp is to use a faster encryption cypher. For scp it’s the encryption that really slows it down.
For scp if you use the -c flag you can select another cypher from the default (not sure what the default cypher is but it’s not the fastest). If you select the medium weight “arcfour” cypher you can greatly increase your transfer speeds on large files (larger than 1000k).
example :
scp -rp -c arcfour source1:$PWD .
March 15th, 2013 - 10:25
Thanks Alan,
I’ll definitely consider (and test) “arcfour” option as some of my environments are “scp only”.
Maxym kharchenko
April 4th, 2013 - 10:06
Excellent write-up. I definitely appreciate this website.
Keep it up!
May 6th, 2013 - 10:53
Another option:
try using multiple rsyncs with lowest compression and lowest encryption level:
rsync –archive –delete –compress-level=1 –files-from=FILE_WITH_LISTOF_SOURCE_FILES –rsh=”ssh -c arcfour” YourSOURCE YourTarget
–files-from option is necessary if multiple sessions used, so you would not have conflicting transmissions. This requires some scripting (especially convenient with perl).
May 6th, 2013 - 11:32
Thanks Rosty, I will definitely try that.