Search this blog

mercredi 12 mai 2021

Create a tar.gz archive in parallel using SLURM

One-liner to submit SLURM job on 1 node to archive and compress a directory. 

sbatch -J MY_JOB_NAME -A MY_ACCOUNT -p MY_PARTITION -t 2:00:00 --mem 8G -N--mail-user=MY_EMAIL --mail-type=END --wrap "module load tools/pigz && tar -I pigz -cf OUTPUT.tar.gz PATH_TO_FOLDER_TO_ARCHIVE" 

This example sets the maximal time to 2 hours, the maximal memory to 8G, and needs pigz command to be available through a module named tools/pigz. Replace placeholders (words in capital letters, except END) by relevant values.