1 .TH swarm 1 "14 Feb 2013" "version 1.0"
3 swarm - Tool for parallelising shell scripts
8 allows one to pass commands to simultaneously running instances of a shell such as
10 .BR Swarm " can work with local shells and can also start shells on remote hosts."
13 .BR swarm " is to speed up the execution of shell scripts which involve a large number of time consuming tasks, where the order in which the tasks are completed is unimportant.
18 Set the number of shells for
21 By default, one shell is spawned for each CPU found by a call to
25 Set the type of shell which will be spawned. The default is
29 Treat command as a single line of input. This option is not very useful unless there is already an instance of
33 option running in the current directory.
43 .BI -l " [file][:level]"
44 Set the log file and optionally level.
46 .BR swarm " logs to stderr.
48 The levels are: 0 (errors), 1 (warnings), 2 (notices), 3 (info).
49 Each level will include messages of a lower level.
52 .BR swarm " logs to stderr, at level 2.
54 Any errors occuring in locally running shells will also be sent to the log file.
56 .BR Note: " If swarm runs as a daemon, stderr (and hence all log messages) detached
58 .BR -l " option is used.
60 .BR Note: " Remote shells will log messages and stderr to ~/.swarm.slave.log
64 Sets the port number to use for connections to remote hosts.
66 By default, swarm does not use a fixed port; the port numbers for connections are chosen dynamically. Initially swarm uses ssh (port 22) to start remote swarm slave instances, passing a port to connect to as an argument. Subsequent ports for shell processes are sent to the slave swarm instance over this connection, which connects to the ports and starts the shell processes.
71 as a daemon in the current directory.
73 will background itself by using
74 .BR fork "(2), and attach stdin to a " fifo "(7).
75 The daemon will continue to run regardless of any EOF sent to the input
76 .BR fifo "(7) until it receives an EXIT directive."
79 .BR -o " option is also used, the daemon's stdout will be redirected to /dev/null".
82 .BR -l " option is also used, the daemon's stderr (and all log messages) will be redirected to /dev/null".
84 Subsequent instances of
85 .BR swarm " invoked with the " -c " option or a script file in the same directory as the daemon will not start their own shells, and will send the command to the daemon's input
86 .BR fifo "(7). Certain directives will affect the behaviour of the 'wrapping' swarm. The daemon
87 .BR does " " not " send any output to the wrapping instance of " swarm ".
89 Subsequent instances of
90 .BR swarm " invoked without the " -c " option or a script file in the same directory will report an error.
92 It is of course possible to write commands directly to the daemon's input fifo using
93 .BR echo "(1) or any other program. Using
94 .BR swarm " is preferred, because it will check input for directives that will affect its behaviour.
96 .BR Note: " The 'daemon' is not really an official Unix style daemon, but 'daemon' sounds much cooler than 'background process'.
101 .BR -c " option was not supplied, a single argument remaining after option processing is assumed to name a file containing valid shell commands and
102 .BR swarm " directives. If the
103 .BR -c " option was also supplied, or there was more than a single additional argument, " swarm " will report an error and exit
107 treats each line of input as a seperate command to send to a shell.
108 By default, each command is sent to the first available shell that isn't currently executing a command.
113 will output the results of completed shell commands in the order of completion.
115 puts a prefix '%d: ' in front of the output to indicate the order in which commands were received.
117 It is also possible to direct the output of commands to specified files using the OUTPUT directive. This is probably more useful for scripting purposes.
120 A directive is any line that starts and ends with a '#' character.
122 If a line starts with a '#' but does not contain a second '#', it is treated as a comment.
124 If a line contains characters
126 the second '#' character, it is treated as a SHELL SPECIFIC COMMAND,
131 understands several directives which can be used to control its behaviour at runtime. Any unrecognised directive is treated as a comment.
142 is supplied, the output of all commands received before the next OUTPUT directive will be sent to the file.
143 The file is created if it didn't exist, and overwritten if it did exist.
147 contains a '%d' format string, the output of commands will be sent to individual files, where the filenames are created from a call to
149 using the commands number as a format argument, and
151 as the format string.
155 is not supplied, the output of commands received before the next OUTPUT directive will not be saved to any file.
159 .BR Swarm " will continue to read input and create tasks, but will not actually send any commands until all currently executing commands are finished.
163 This directive is intended for use with the
164 .BR -c " option. If a shell script starts a swarm daemon, running swarm -c '#BARRIER BLOCK#' allows the script to block until the swarm daemon finishes all tasks."
166 .BR WARNING: " Sending BARRIER BLOCK to a swarm daemon
168 using a wrapping instance of
169 .BR swarm " will cause the swarm daemon to hang after it has completed all tasks.
171 For a non-daemon instance of swarm, this directive is essentially identical to BARRIER.
174 .BI ABSORB " host[:name] [processes]
176 will start and connect to shells on the remote host and name them accordingly. An exec'd
177 .BR ssh "(1) is used to start the remote shells."
179 The default number of
181 is equal to the number of CPUs on the remote host.
187 .BR Swarm " will start instances of " ssh "(1) and use remote port forwarding to secure the connections.
190 .BI ABSORB UNSECURE host[:name] [processes]
191 .B There are security issues associated with the use of this directive
193 This directive is the same as ABSORB, except no
194 .BR ssh "(1) instances will be spawned for remote port forwarding; all data will be sent unencrypted."
196 Obviously using unencrypted connections is faster, but dangerous.
200 .SH SHELL SPECIFIC COMMANDS
201 By default, any command is sent to the first available shell.
203 A line containing two '#' characters followed by a command will be sent to any shells with names matching a POSIX regex between the '#' characters.
205 To send the command only to the first available shell with name matching the regex, the regex should be followed with ' &'.
207 Shells are normally named according to the host on which they are running, and the order they were spawned. The format is 'host:X' where X is an integer greater or equal to zero. Shells running locally are called 'local:X'. Using the ABSORB directive, it is possible to give remote shells a name that is not the same as the hostname of the remote host running the shell.
210 To print the names of all shells, run:
213 To print the name of the first available shell, run:
217 To run a command only on the shell called 'local:0':
220 .SH EXITED SHELLS / SIGNAL HANDLING
221 .BR Swarm " detects when locally running shell exits (using a handler for " SIGCHLD "). If the shell exits normally, regardless of error code,
222 .BR swarm " will create a new shell to replace it. If the shell exits because of a signal that caused it to terminate,
223 .BR swarm " will send " SIGTERM " to itself, which will cause itself (and as a result all other shells) to exit.
225 If a remote shell exits, for whatever reason,
226 .BR swarm " will create a new shell to replace it. " Swarm " has no knowledge of and will not react to signals sent to remote shells on the remote host.
229 .BR swarm " itself exits for whatever reason, it will terminate all local and remote shells.
232 .B Never allow ssh access from accross a public internet to a host with swarm installed.
234 When the ABSORB UNSECURE directive is read,
238 to start slaves processes on the remote host.
242 connections to the master. Because commands are sent as plain text over the network,
243 The remote hosts become vulnerable to "man in the middle" attacks.
245 Never use ssh keys with empty passphrases, and restrict access to hosts with
246 .BR swarm "installed.
250 is very recently developed and therefore probably has a shit load of bugs. Probably related to buffering.
252 Do not place whitespace after a directive, or it will be treated as a host specific command.
254 The stderr of remote shells will always be lost to the void.
256 Using things that aren't shells with the
257 .BR -s " option won't work, because " swarm " automatically adds extra commands for the shells.
259 Writing three bell characters in a row will cause
260 .BR swarm " to break, because it looks for three bell characters to signal the end of output from each command."
262 .BR Swarm " does try to be nice and tell shells to 'exit', but usually this doesn't work and it sends them a " kill "(2)."
264 There are probably bugs with parsing, like assuming you never use ':' as part of a filename or a name for a shell, etc.
265 Just don't do it and it won't break.
271 .BR bash "(1), " ssh "(1)."