Intermediate SQL Color Coded SQL, UNIX and Database Essays


Scrap the SCP. How to copy data fast using pigz and nc

Have you ever heard that the speed of the system is determined by its slowest component ? I am made painfully aware of that every time I do data migrations.

I.e. it doesn’t matter if you have 64 core systems with 100+ Gb of memory on either end if the majority of time is spent waiting for data to trickle across a slow 1 Gb network link.

Watching data trickle for hours, while the rest of the system is doing nothing is a pretty frustrating experience. But limitations breed creativity … so lately, I’ve been experimenting with several different copy techniques to see if there is any way transfer speed can be improved, perhaps using some of the idle capacity to speed things up.