Intermediate SQL Color Coded SQL, UNIX and Database Essays

16May/1236

Scrap the SCP. How to copy data fast using pigz and nc

Have you ever heard that the speed of the system is determined by its slowest component ? I am made painfully aware of that every time I do data migrations.

I.e. it doesn’t matter if you have 64 core systems with 100+ Gb of memory on either end if the majority of time is spent waiting for data to trickle across a slow 1 Gb network link.

Watching data trickle for hours, while the rest of the system is doing nothing is a pretty frustrating experience. But limitations breed creativity … so lately, I’ve been experimenting with several different copy techniques to see if there is any way transfer speed can be improved, perhaps using some of the idle capacity to speed things up.