Package nq: Information

These small utilities allow creating very lightweight job queue
systems which require no setup, maintenance, supervision, or any
long-running processes.

The intended purpose is ad-hoc queuing of command lines (e.g., for
building several targets of a Makefile, downloading multiple files one
at a time, running benchmarks in several configurations, or simply as
a glorified nohup). But as any good Unix tool, it can be abused for
whatever you like.

Job order is enforced by a timestamp nq gets immediately when
started.  Synchronization happens on file-system level.  Timer
resolution is milliseconds.  No sub-second file system time stamps are
required.  Polling is not used.  Exclusive execution is maintained
strictly.

Enforcing job order works like this:
- every job has a flock(2)ed output file, ala ,TIMESTAMP.PID
- every job starts only after all earlier flock(2)ed files are unlocked
- Why flock(2)? Because it locks the file handle, which is shared
  across exec(2) with the child process (the actual job), and it will
  unlock when the file is closed (usually when the job terminates).

You enqueue (get it?) new jobs using nq CMDLINE....  The job ID is
output (unless suppressed using -q) and nq detaches immediately,
running the job in the background.  STDOUT and STDERR are redirected
into the log file.

nq tries hard (but does not guarantee) to ensure the log file of the
currently running job has +x bit set.  Thus you can use ls -F to get
a quick overview of the state of your queue.

The "file extension" of the log file is actually the PID, so you can
kill jobs easily.  Before the job is started, it is the PID of nq,
so you can cancel a queued job by killing it as well.

Due to the initial exec line in the log files, you can resubmit a
job by executing it as a shell command file (i.e. running sh $jobid).

You can wait for jobs to finish using nq -w, possibly listing job
IDs you want to wait for; the default is all of them.  Likewise, you
can test if there are jobs which need to be waited upon using -t.

By default, job IDs are per-directory, but you can set $NQDIR to put
them elsewhere.  Creating nq wrappers setting $NQDIR to provide
different queues for different purposes is encouraged.

All these operations take worst-case quadratic time in the amount of
lock files produced, so you should clean them regularly.

Package nq: Information

Last changed