parallel.adoc

parallel (gnu) cheat code

Table of Contents

• inputs
  • list of files
  • No command means arguments are commands
• replacement strings
• execution
  • a bash script
• misc. options
• install
  • synology

inputs

• ::: : feed the given list, space separated.

• :::: : read inside file, line separated.

list of files

parallel echo ::: /*

Will echo /bin, /boot, /cdrom, /dev, /etc …​

command	output
parallel echo ::: 1 2 3	1 3 2
parallel echo ::: 1 2 3 ::: a b	1 a, 1 b, 2 a, 2 b, 3 b, 3 a.
parallel --link echo ::: 1 2 3 ::: a b	1 a, 2 b, 3 a.
parallel echo ::: 1 2 3 :::+ a b	1 a, 2 b.
parallel echo ::: 1 2 3 :::+ a b ::: 10 11 12	1 a 10, 1 a 11, 1 a 12, 2 b 10, 2 b 11, 2 b 12.

No command means arguments are commands

see a bash script

replacement strings

pattern	comment
{}	full path : mydir/mysubdir/myfile.myext
{.}	remove extension : mydir/mysubdir/myfile
{/}	filename: myfile.myext
{//}	dirname: mydir/mysubdir
{/.}	basename: myfile
{#}	job number in the sequence
{%}	job slot number (thread id)
{N}, {N.}, {N/}, …​	Nth parameter (can be -N from the end)
following available with --plus options
{..}, {...}	remove 2 or 3 file extensions
{##}	total number of jobs:
{+.}, {+/}, {+..}, {+...}	opposite of {.}, {/}, {..}, {...}.
{:-string}	Default value is string if the argument is empty.
{#string}, {%string}	If the argument starts/ends with string, remove it.
{:N}, {:N:L}	Substring from N (and length L).

examples

TODO

execution

option	meaning
--jobs	Number of simultaneous jobs. 0 means as many as possible.
--shuf	Shuffle job order
--interactive	ask the user if a command should be run using
--eta, --progress, --bar,	Progress information
--timeout X	stopping jobs taking more than X seconds (can also be 1d3.5h16.6m4s or 200% of the media range)
--delay X	will make sure there is at least X seconds between each
--tmux	A terminal for every job
--resume --joblog X	With a log file X, jobs can be stopped and later pickup where it left off.
--halt soon,fail=1, --halt now,fail=1	Stop pruning (or kill) jobs if one of the jobs fails
--nice X	run the jobs with a nice value X.

a bash script

export -f my_func
parallel my_func ::: 1 2 3

misc. options

parellel -k --lb  do_something ::: task_1 task_2 task_3 # immediately display job stderr/stdout one-at-a-time by jobs order
parallel --arg-sep ,, echo ,, A B C :::: def-file       # change ::: to ,,
parallel --arg-file-sep // echo ::: A B C // def-file   # change :::: to //
parallel -d _ echo :::: abc_-file                       # parse '_' separated instead lines
parallel -q echo "\"{}\"" ::: a b c                     # "a" "b" "c"

install

wget https://mirror.cyberbits.eu/gnu/parallel/parallel-latest.tar.bz2
# or from this repo
wget https://github.com/jujumo/memento/raw/main/coding/linux/parallel-latest.tar.bz2
tar -xjf parallel-latest.tar.bz2
cd parallel-20240222/
./configure
make
sudo make install
parallel --citation

synology

On Synology NAS, the opkg perl package may lead to following error:

$> parallel
Can't locate IPC/Open3.pm in @INC (you may need to install the IPC::Open3 module) (@INC contains: /opt/lib/perl5/5.28) at
...

Solution (A) Use this version of Perl instead

$> sudo opkg remove perl
$> sudo -i
$> curl -fsSL https://git.io/perl-install | bash -s ~/perl

This is not available for arm32.

Solution (B) manual install required packages

$> sudo opkg update
$> sudo opkg install perl perlbase-ipc perlbase-getopt
# check with :
$> /opt/bin/perl -MGetopt::Long -e 'print "Getopt OK\n"'
Getopt OK
$> /opt/bin/perl -MIPC::Open3 -e 'print "Open3 OK\n"'
Open3 OK