Skip to main content

并行运行多进程的Bash脚本 | A srcipt for running processes in parallel in Bash

原文:http://pebblesinthesand.wordpress.com/2008/05/22/a-srcipt-for-running-processes-in-parallel-in-bash/

技巧是控制了最大并发数目,当前并发数过大时,等待有进程退出才开始下一个。

我现在正在写的脚本里正需要这个,借鉴过来


#!/bin/bash
NUM=0
QUEUE=""
MAX_NPROC=2 # default
REPLACE_CMD=0 # no replacement by default
USAGE="A simple wrapper for running processes in parallel.
Usage: `basename $0` [-h] [-r] [-j nb_jobs] command arg_list
-h Shows this help
-r Replace asterix * in the command string with argument
-j nb_jobs Set number of simultanious jobs [2]
Examples:
`basename $0` somecommand arg1 arg2 arg3
`basename $0` -j 3 \"somecommand -r -p\" arg1 arg2 arg3
`basename $0` -j 6 -r \"convert -scale 50% * small/small_*\" *.jpg"

function queue {
QUEUE="$QUEUE $1"

NUM=$(($NUM+1))
}

function regeneratequeue {
OLDREQUEUE=$QUEUE
QUEUE=""
NUM=0
for PID in $OLDREQUEUE
do
if [ -d /proc/$PID ] ; then
QUEUE="$QUEUE $PID"
NUM=$(($NUM+1))
fi
done
}

function checkqueue {
OLDCHQUEUE=$QUEUE
for PID in $OLDCHQUEUE
do
if [ ! -d /proc/$PID ] ; then
regeneratequeue # at least one PID has finished
break
fi
done
}

# parse command line
if [ $# -eq 0 ]; then # must be at least one arg
echo "$USAGE" >&2
exit 1
fi

while getopts j:rh OPT; do # "j:" waits for an argument "h" doesnt
case $OPT in
h) echo "$USAGE"

exit 0 ;;
j) MAX_NPROC=$OPTARG ;;
r) REPLACE_CMD=1 ;;
\?) # getopts issues an error message
echo "$USAGE" >&2
exit 1 ;;
esac
done

# Main program
echo Using $MAX_NPROC parallel threads
shift `expr $OPTIND - 1` # shift input args, ignore processed args
COMMAND=$1
shift

for INS in $* # for the rest of the arguments
do
# DEFINE COMMAND
if [ $REPLACE_CMD -eq 1 ]; then
CMD=${COMMAND//"*"/$INS}
else
CMD="$COMMAND $INS" #append args
fi
echo "Running $CMD"

$CMD &
# DEFINE COMMAND END

PID=$!
queue $PID

while [ $NUM -ge $MAX_NPROC ]; do
checkqueue
sleep 0.4
done
done
wait # wait for all processes to finish before exit






Original posted at: http://pebblesinthesand.wordpress.com/2008/05/22/a-srcipt-for-running-processes-in-parallel-in-bash/

This script will run multiple process in background concurrently, while the maximum number of running processed is under control. When there are already too many running processes, a new one will wait until one process finished.

I just need this trick in my project, so I copied it here.

(See codes above)

Comments

Popular posts from this blog

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...

Qubes OS: First Impressions

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand. My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project. Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into the...

ESP32S3: Flash Encryption and Secure Boot

Flash encryption and secure boot are useful security features for ESP32S3 chip. While not perfect, they definitely make it harder to extract the secrets in the chip. However, it is tricky to enable both features at the same time. The topic is actually discussed in the official documentation: ESP32S3 Security Features Security Features Enablement Workflows Especially, the second one mentioned it is recommended to enable flash encryption before secure boot. But I still find the documentation confusing. In the end I was able to successfully enable both, here's my findings. My Understanding After my adventure, here's what I think could have worked. WARNING, this is untested. Follow  Security Features Enablement Workflows : Burn all the keys, as long as their purpose eFuses and read/write protections Burn other security eFuses, but DO NOT burn ENABLE_SECURITY_DOWNLOAD in the middle, which is mentined at the end of the instruction for both flash encryption and secure boot. Burn...