A brilliant way to parallize your commands

The solution I’m taking about here comes from stackoverflow.com. The original question is :

I have a list/queue of 200 commands that I need to run in a shell on a Linux server. I only want to have a maximum of 10 processes running (from the queue) at once. Some processes will take a few seconds to complete, other processes will take much longer. When a process finishes I want the next command to be “popped” from the queue and executed.

Solution: write a makefile with the following content:

all: usera userb userc....
imapsync usera
imapsync userb

Then just run

make -j 10 -f makefile

I think the answer is really brilliant as it not only makes use of available tools that are quite mature, but also solves the problem fast with little effort. If the situation is getting more complicated, the solution can still be easily adjusted. xargs is also a good solution btw.

  1. #1 by Francesco R. on September 5, 2011 - 1:33 am

    also, less precise but kinda working:

    for u in $USERS ; do
    if [[ $(jobs | wc -l) -lt 10 ]] ; then
    echo ${u} ; imapsync “${u}” &> “${u}”.log &

    • #2 by qiaomuf on September 5, 2011 - 9:01 am

      True. But when some subprocess(not all) finished their work, the main process is still waiting. So it’s not exactly what the op demanded.

      • #3 by Francesco R. on September 5, 2011 - 6:15 pm

        I’m aware of that, my problem was that other syntax/solutions are difficult to remember, now that I’m aware of GNU/parallel I’ll probably survive to the effort to remember that 😉

  2. #4 by Donnie Berkholz on September 5, 2011 - 8:41 am

    GNU Parallel is what I use.

    • #5 by qiaomuf on September 5, 2011 - 8:54 am

      Yeah that’s a powerful one.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: