Welcome to GNU Parallel's documentation!

parallel

• NAME
• SYNOPSIS
• DESCRIPTION
  • Reader's guide
    • Tutorial
    • How-to
    • Reference
    • Design discussion
• OPTIONS
• EXAMPLES
• SPREADING BLOCKS OF DATA
• TIME POSTFIXES
• UNIT PREFIX
• QUOTING
• LIST RUNNING JOBS
• COMPLETE RUNNING JOBS BUT DO NOT START NEW JOBS
• ENVIRONMENT VARIABLES
• DEFAULT PROFILE (CONFIG FILE)
• PROFILE FILES
• EXIT STATUS
• DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES
• BUGS
  • Quoting of newline
  • Speed
    • Startup
    • Job startup
    • SSH
    • Disk access
  • --nice limits command length
  • Aliases and functions do not work
  • Database with MySQL fails randomly
• REPORTING BUGS
  • Bug dependent on environment
  • In summary
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• SEE ALSO

parallel_examples

• GNU PARALLEL EXAMPLES
  • EXAMPLE: Working as xargs -n1. Argument appending
  • EXAMPLE: Reading arguments from command line
  • EXAMPLE: Running full commands in parallel
  • EXAMPLE: Inserting multiple arguments
  • EXAMPLE: Context replace
  • EXAMPLE: Compute intensive jobs and substitution
  • EXAMPLE: Substitution and redirection
  • EXAMPLE: Composed commands
  • EXAMPLE: Composed command with perl replacement string
  • EXAMPLE: Composed command with multiple input sources
  • EXAMPLE: Match parts of input source
  • EXAMPLE: Replacement fields from CSV file with headers
  • EXAMPLE: Calling Bash functions
  • EXAMPLE: Function tester
  • EXAMPLE: Identify few failing jobs
  • EXAMPLE: Continously show the latest line of output
  • EXAMPLE: Log rotate
  • EXAMPLE: Simple network scanner
  • EXAMPLE: Removing file extension when processing files
  • EXAMPLE: Replacing parts of file names
  • EXAMPLE: Removing strings from the argument
  • EXAMPLE: Download 24 images for each of the past 30 days
  • EXAMPLE: Download world map from NASA
  • EXAMPLE: Download Apollo-11 images from NASA using jq
  • EXAMPLE: Download video playlist in parallel
  • EXAMPLE: Prepend last modified date (ISO8601) to file name
  • EXAMPLE: Save output in ISO8601 dirs
  • EXAMPLE: Digital clock with "blinking" :
  • EXAMPLE: Aggregating content of files
  • EXAMPLE: Breadth first parallel web crawler/mirrorer
  • EXAMPLE: Process files from a tar file while unpacking
  • EXAMPLE: Rewriting a for-loop and a while-read-loop
  • EXAMPLE: Rewriting nested for-loops
  • EXAMPLE: Finding the lowest difference between files
  • EXAMPLE: for-loops with column names
  • EXAMPLE: All combinations in a list
  • EXAMPLE: From a to b and b to c
  • EXAMPLE: Count the differences between all files in a dir
  • EXAMPLE: Speeding up fast jobs
  • EXAMPLE: Using shell variables
  • EXAMPLE: Group output lines
  • EXAMPLE: Tag output lines
  • EXAMPLE: Colorize output
  • EXAMPLE: Keep order of output same as order of input
  • EXAMPLE: Keep order, but make job 1 output fast
  • EXAMPLE: Parallel grep
  • EXAMPLE: Grepping n lines for m regular expressions.
    • Limiting factor: RAM
    • Limiting factor: CPU
    • Bigger problem
  • EXAMPLE: Using remote computers
  • EXAMPLE: Transferring of files
  • EXAMPLE: Advanced file transfer
  • EXAMPLE: Distributing work to local and remote computers
  • EXAMPLE: Running the same command on remote computers
  • EXAMPLE: Running 'sudo' on remote computers
  • EXAMPLE: Using remote computers behind NAT wall
    • No jumphost, but port forwards
    • No jumphost, no port forwards
  • EXAMPLE: Use sshpass with ssh
  • EXAMPLE: Use outrun instead of ssh
  • EXAMPLE: Slurm cluster
  • EXAMPLE: Parallelizing rsync
  • EXAMPLE: Use multiple inputs in one command
  • EXAMPLE: Use a table as input
  • EXAMPLE: Output to database
  • EXAMPLE: Output to CSV-file for R
  • EXAMPLE: Use XML as input
  • EXAMPLE: Run the same command 10 times
  • EXAMPLE: Working as cat | sh. Resource inexpensive jobs and evaluation
  • EXAMPLE: Call program with FASTA sequence
  • EXAMPLE: Call program with interleaved FASTQ records
  • EXAMPLE: Processing a big file using more CPUs
  • EXAMPLE: Grouping input lines
  • EXAMPLE: Running more than 250 jobs workaround
  • EXAMPLE: Working as mutex and counting semaphore
  • EXAMPLE: Mutex for a script
  • EXAMPLE: Start editor with file names from stdin (standard input)
  • EXAMPLE: Running sudo
  • EXAMPLE: Run ping in parallel
  • EXAMPLE: GNU Parallel as queue system/batch manager
  • EXAMPLE: GNU Parallel as dir processor
  • EXAMPLE: Locate the missing package
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• SEE ALSO

sem

• NAME
• SYNOPSIS
• DESCRIPTION
• OPTIONS
• UNDERSTANDING A SEMAPHORE
• EXAMPLE: Gzipping *.log
• EXAMPLE: Protecting pod2html from itself
• BUGS
• REPORTING BUGS
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• SEE ALSO

env_parallel

• NAME
• SYNOPSIS
• DESCRIPTION
  • Environment space
• OPTIONS
• SUPPORTED SHELLS
  • Ash
    • Installation
    • Supported use
  • Bash
    • Installation
    • Supported use
    • BUGS
  • csh
    • Installation
    • Supported use
  • Dash
    • Installation
    • Supported use
  • fish
    • Installation
    • Supported use
  • ksh
    • Installation
    • Supported use
  • mksh
    • Installation
    • Supported use
  • pdksh
    • Installation
    • Supported use
  • sh
    • Installation
    • Supported use
  • tcsh
    • Installation
    • Supported use
  • Zsh
    • Installation
    • Supported use
• EXIT STATUS
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• SEE ALSO

parset

• NAME
• SYNOPSIS
• DESCRIPTION
• OPTIONS
• SUPPORTED SHELLS
  • Bash/Zsh/Ksh/Mksh
    • Examples
    • Installation
  • ash/dash (FreeBSD's /bin/sh)
    • Examples
    • Installation
• EXIT STATUS
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• SEE ALSO

parsort

• NAME
• SYNOPSIS
• DESCRIPTION
• OPTIONS
• EXAMPLE
• PERFORMANCE
• AUTHOR
• LICENSE
• DEPENDENCIES
• SEE ALSO

parallel_design

• Design of GNU Parallel
  • One file program
  • Choice of programming language
  • Old Perl style
  • Scalability up and down
  • Exponentially back off
  • Shell compatibility
  • env_parallel
    • env_parallel.*
    • env_parallel.bash / env_parallel.sh / env_parallel.ash / env_parallel.dash / env_parallel.zsh / env_parallel.ksh / env_parallel.mksh
    • env_parallel.csh
    • env_parallel.fish
  • parset (supported in sh, ash, dash, bash, zsh, ksh, mksh)
  • Job slots
  • Rsync protocol version
  • Compression
  • Wrapping
  • Convenience options --nice --basefile --transfer --return --cleanup --tmux --group --compress --cat --fifo --workdir --tag --tagstring
  • --shard
  • Shell shock
  • The remote system wrapper
    • Make quoting unnecessary by hex encoding everything
    • Ctrl-C and standard error (stderr)
    • --nice
    • Setting $PARALLEL_TMP
    • The wrapper
  • Transferring of variables and functions
  • Base64 encoded bzip2
  • Which shell to use
  • Always running commands in a shell
  • Quoting
  • --pipepart vs. --pipe
  • --block-size adjustment
  • Automatic --block-size computation
  • --jobs and --onall
  • --shuf
  • --csv
  • Buffering on disk
    • Partly buffering in memory
    • Comparing to buffering in memory
  • Disk full
  • Memory usage
  • Argument separators ::: :::: :::+ ::::+
  • Perl replacement strings, {= =}, and --rpl
  • Test suite
  • Median run time
  • Error messages and warnings
  • Determine number of CPUs
  • Computation of load
  • Killing jobs
  • SQL interface
  • Logo
  • Citation notice
• Ideas for new design
  • Multiple processes working together
  • --rrs on remote using a perl wrapper
• Historical decisions
  • --tollef

parallel_tutorial

• GNU Parallel Tutorial
  • Reader's guide
• Prerequisites
• Input sources
  • A single input source
  • Multiple input sources
    • Linking arguments from input sources
  • Changing the argument separator.
  • Changing the argument delimiter
  • End-of-file value for input source
  • Skipping empty lines
• Building the command line
  • No command means arguments are commands
  • Replacement strings
    • The 7 predefined replacement strings
    • Changing the replacement strings
    • Perl expression replacement string
    • Positional replacement strings
    • Positional perl expression replacement string
    • Input from columns
    • Header defined replacement strings
    • More pre-defined replacement strings with --plus
    • Dynamic replacement strings with --plus
  • More than one argument
  • Quoting
  • Trimming space
  • Respecting the shell
• Controlling the output
  • Saving output into files
• Controlling the execution
  • Number of simultaneous jobs
  • Shuffle job order
  • Interactivity
  • A terminal for every job
  • Timing
  • Progress information
  • Termination
    • Unconditional termination
    • Termination dependent on job status
    • Termination signals (advanced)
  • Limiting the resources
• Remote execution
  • Sshlogin
  • Transferring files
  • Working dir
  • Avoid overloading sshd
  • Ignore hosts that are down
  • Running the same commands on all hosts
  • Transferring environment variables and functions
  • Showing what is actually run
• Saving output to shell variables (advanced)
• Saving to an SQL base (advanced)
  • CSV as SQL base
  • DBURL as table
  • Using multiple workers
• --pipe
  • Chunk size
  • Records
  • Fixed length records
  • Record separators
  • Header
  • --pipepart
• Shebang
  • Input data and parallel command in the same file
  • Parallelizing existing scripts
• Semaphore
  • Mutex
  • Counting semaphore
  • Timeout
• Informational
• Profiles
• Spread the word

parallel_alternatives

• NAME
• DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES
  • SUMMARY LEGEND
    • Inputs
    • Manipulation of input
    • Outputs
    • Execution
    • Remote execution
    • Semaphore
    • Legend
  • GNU Parallel
  • DIFFERENCES BETWEEN xargs AND GNU Parallel
  • DIFFERENCES BETWEEN find -exec AND GNU Parallel
  • DIFFERENCES BETWEEN make -j AND GNU Parallel
  • DIFFERENCES BETWEEN ppss AND GNU Parallel
    • EXAMPLES FROM ppss MANUAL
  • DIFFERENCES BETWEEN pexec AND GNU Parallel
    • EXAMPLES FROM pexec MANUAL
  • DIFFERENCES BETWEEN xjobs AND GNU Parallel
    • EXAMPLES FROM xjobs MANUAL
  • DIFFERENCES BETWEEN prll AND GNU Parallel
    • EXAMPLES FROM prll's MANUAL
  • DIFFERENCES BETWEEN dxargs AND GNU Parallel
  • DIFFERENCES BETWEEN mdm/middleman AND GNU Parallel
    • EXAMPLES FROM middleman's WEBSITE
  • DIFFERENCES BETWEEN xapply AND GNU Parallel
    • EXAMPLES FROM xapply's MANUAL
  • DIFFERENCES BETWEEN AIX apply AND GNU Parallel
    • EXAMPLES FROM IBM's KNOWLEDGE CENTER
  • DIFFERENCES BETWEEN paexec AND GNU Parallel
    • EXAMPLES FROM man paexec
    • EXAMPLES FROM presentation/paexec.tex
    • EXAMPLES FROM paexec's example dir
  • DIFFERENCES BETWEEN map(sitaramc) AND GNU Parallel
    • EXAMPLES FROM map's WEBSITE
  • DIFFERENCES BETWEEN ladon AND GNU Parallel
    • EXAMPLES FROM ladon MANUAL
  • DIFFERENCES BETWEEN jobflow AND GNU Parallel
    • EXAMPLES FROM jobflow README
  • DIFFERENCES BETWEEN gargs AND GNU Parallel
    • EXAMPLES FROM gargs WEBSITE
  • DIFFERENCES BETWEEN orgalorg AND GNU Parallel
  • DIFFERENCES BETWEEN Rust parallel(mmstick) AND GNU Parallel
  • DIFFERENCES BETWEEN parallelion AND GNU Parallel
    • EXAMPLES FROM parallelion
  • DIFFERENCES BETWEEN Rush AND GNU Parallel
    • EXAMPLES FROM rush's WEBSITE
    • Other rush features
  • DIFFERENCES BETWEEN ClusterSSH AND GNU Parallel
  • DIFFERENCES BETWEEN coshell AND GNU Parallel
  • DIFFERENCES BETWEEN spread AND GNU Parallel
  • DIFFERENCES BETWEEN pyargs AND GNU Parallel
    • Examples
  • DIFFERENCES BETWEEN concurrently AND GNU Parallel
  • DIFFERENCES BETWEEN map(soveran) AND GNU Parallel
    • EXAMPLES FROM maps WEBSITE
  • DIFFERENCES BETWEEN loop AND GNU Parallel
    • EXAMPLES FROM loop's WEBSITE
  • DIFFERENCES BETWEEN lorikeet AND GNU Parallel
  • DIFFERENCES BETWEEN spp AND GNU Parallel
  • DIFFERENCES BETWEEN paral AND GNU Parallel
    • EXAMPLES FROM README.markdown
  • DIFFERENCES BETWEEN concurr AND GNU Parallel
    • EXAMPLES FROM README.md
  • DIFFERENCES BETWEEN lesser-parallel AND GNU Parallel
  • DIFFERENCES BETWEEN npm-parallel AND GNU Parallel
  • DIFFERENCES BETWEEN machma AND GNU Parallel
    • EXAMPLES FROM README.md
  • DIFFERENCES BETWEEN interlace AND GNU Parallel
    • EXAMPLES FROM interlace's WEBSITE
  • DIFFERENCES BETWEEN otonvm Parallel AND GNU Parallel
  • DIFFERENCES BETWEEN k-bx par AND GNU Parallel
  • DIFFERENCES BETWEEN parallelshell AND GNU Parallel
  • DIFFERENCES BETWEEN shell-executor AND GNU Parallel
  • DIFFERENCES BETWEEN non-GNU par AND GNU Parallel
  • DIFFERENCES BETWEEN fd AND GNU Parallel
  • DIFFERENCES BETWEEN lateral AND GNU Parallel
  • DIFFERENCES BETWEEN with-this AND GNU Parallel
  • DIFFERENCES BETWEEN Tollef's parallel (moreutils) AND GNU Parallel
    • EXAMPLES FROM Tollef's parallel MANUAL
  • DIFFERENCES BETWEEN rargs AND GNU Parallel
    • EXAMPLES FROM rargs MANUAL
  • DIFFERENCES BETWEEN threader AND GNU Parallel
  • DIFFERENCES BETWEEN runp AND GNU Parallel
    • Examples as GNU Parallel
  • DIFFERENCES BETWEEN papply AND GNU Parallel
    • Examples as GNU Parallel
  • DIFFERENCES BETWEEN async AND GNU Parallel
    • Examples as GNU Parallel
  • DIFFERENCES BETWEEN pardi AND GNU Parallel
    • EXAMPLES FROM pardi test.sh
  • DIFFERENCES BETWEEN bthread AND GNU Parallel
  • DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel
    • EXAMPLES FROM simple_gpu_scheduler MANUAL
  • DIFFERENCES BETWEEN parasweep AND GNU Parallel
  • DIFFERENCES BETWEEN parallel-bash(2021) AND GNU Parallel
    • EXAMPLES FROM parallel-bash
  • DIFFERENCES BETWEEN parallel-bash(2024) AND GNU Parallel
    • EXAMPLES FROM parallel-bash
  • DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel
  • DIFFERENCES BETWEEN spawntool AND GNU Parallel
  • DIFFERENCES BETWEEN go-pssh AND GNU Parallel
    • EXAMPLES FROM go-pssh
  • DIFFERENCES BETWEEN go-parallel AND GNU Parallel
    • EXAMPLES FROM go-parallel
  • DIFFERENCES BETWEEN p AND GNU Parallel
    • EXAMPLES FROM p
  • DIFFERENCES BETWEEN senechal AND GNU Parallel
    • EXAMPLES FROM seneschal README.md
  • DIFFERENCES BETWEEN async AND GNU Parallel
    • EXAMPLES FROM async
  • DIFFERENCES BETWEEN tandem AND GNU Parallel
    • EXAMPLES FROM Demo
    • EXAMPLES FROM tandem -h
    • EXAMPLES FROM tandem's readme.md
  • DIFFERENCES BETWEEN rust-parallel(aaronriekenberg) AND GNU Parallel
    • EXAMPLES FROM rust-parallel's README.md
  • DIFFERENCES BETWEEN parallelium AND GNU Parallel
    • EXAMPLES
  • DIFFERENCES BETWEEN forkrun AND GNU Parallel
    • EXAMPLES
  • DIFFERENCES BETWEEN parallel-sh AND GNU Parallel
    • EXAMPLES
  • DIFFERENCES BETWEEN bash-parallel AND GNU Parallel
  • DIFFERENCES BETWEEN PaSH AND GNU Parallel
  • DIFFERENCES BETWEEN korovkin-parallel AND GNU Parallel
  • DIFFERENCES BETWEEN xe AND GNU Parallel
    • EXAMPLES
  • DIFFERENCES BETWEEN sp AND GNU Parallel
    • EXAMPLES
  • DIFFERENCES BETWEEN repeater AND GNU Parallel
    • Memory use and run time for large output
    • EXAMPLES
  • DIFFERENCES BETWEEN parallelize AND GNU Parallel
    • EXAMPLES
  • Todo
• TESTING OTHER TOOLS
  • MIX: Output mixes
  • STDERRMERGE: Stderr is merged with stdout
  • RAM: Output limited by RAM
  • DISKFULL: Incomplete data if /tmp runs full
  • CLEANUP: Leaving tmp files at unexpected death
  • SPCCHAR: Dealing badly with special file names.
  • COMPOSED: Composed commands do not work
  • ONEREP: Only one replacement string allowed
  • INPUTSIZE: Length of input should not be limited
  • NUMWORDS: Speed depends on number of words
  • 4GB: Output with a line > 4GB should be OK
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• SEE ALSO

parcat

• NAME
• SYNOPSIS
• DESCRIPTION
• OPTIONS
• EXAMPLES
  • Simple line buffered output
• REPORTING BUGS
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• SEE ALSO

niceload

• NAME
• SYNOPSIS
• DESCRIPTION
• OPTIONS
• EXAMPLE: See niceload in action
• EXAMPLE: Run updatedb
• EXAMPLE: Run rsync
• EXAMPLE: Ensure enough disk cache
• ENVIRONMENT VARIABLES
• EXIT STATUS
• REPORTING BUGS
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• SEE ALSO

sql

• NAME
• SYNOPSIS
• DESCRIPTION
• DBURL
• EXAMPLES
  • Get an interactive prompt
  • Run a query
  • Copy a PostgreSQL database
  • Empty all tables in a MySQL database
  • Drop all tables in a PostgreSQL database
  • Run as a script
  • Use --colsep to process multiple columns
  • Retry if the connection fails
  • Get info about the running database system
• REPORTING BUGS
• AUTHOR
• LICENSE
  • Documentation license I
  • Documentation license II
• DEPENDENCIES
• FILES
• SEE ALSO

Indices and tables

• Index

• Module Index

• Search Page