Remote Machines over SSH

--host runs an ironflock command on another machine over SSH instead of on this one. You use it to manage devices from your laptop, a Mac included, without installing ironflock on them. Given more than once, or with a hosts file, it runs the command on many machines at once.

ironflock --host [user@]machine <command> [flags]
ironflock --host [user@]machine --host [user@]machine ... <command> [flags]
ironflock --hosts <file> [--parallel <n>] <command> [flags]

Every command but completion runs on another machine this way. --host, --hosts and --parallel are global flags: they may stand before or after the command’s name.

One Machine

--host [user@]machine runs the same command with the same flags on that machine. What it prints comes back as it is, stdout to stdout and stderr to stderr, and so does its exit code.

$ ironflock --host pi@device-17 version
ironflock 2.0.2 linux/arm64
$ ironflock --host pi@device-17 agent remove --yes --json 2>/dev/null
{"action":"removed","removed":[...]}

What the Other Machine Needs

• Linux on amd64, arm64 or 32-bit ARM (v5 to v7). Another system ends with exit code 3.
• An SSH server that runs commands. No scp or SFTP is needed.
• A POSIX login shell: sh, bash, dash or zsh. csh, tcsh and fish end with exit code 3 and a message that says so.
• A root login, or sudo, for the commands that need root (see Root on the Other Machine).

The machine does not need ironflock installed: each run brings its own binary.

SSH Settings

--host uses the ssh of this machine, so your ~/.ssh/config, keys, the SSH agent and jump hosts work as they do with ssh. It turns off what the command does not need: agent and X11 forwarding, port forwardings, a RemoteCommand and a LocalCommand.

--host takes [user@]machine and nothing else. A port or other options go into ~/.ssh/config:

Host device-*
    User pi
    Port 2222
    StrictHostKeyChecking accept-new

With this, ironflock --host device-17 status logs in as pi on port 2222, and ssh accepts the host key of a device it does not know yet. A value of --host that starts with - is refused, and so is one with other characters than letters, digits, ., _, -, and the : and % of an IPv6 address.

One SSH connection serves the whole run when the path of its control socket is short enough; the socket is made under the temporary folder, else under /tmp. Then ssh asks for a password once. Otherwise every step of the run opens its own connection, and ssh asks again for each.

A device that ironflock agent install or IronFlockOS set up has an SSH user for the project owner, who logs in with the key of the device’s .flock file. SSH Access shows the user name and how to take the key out of the file.

What Happens on the Machine

1. The binary. The machine gets the binary that runs here when it has the same platform. Otherwise ironflock downloads the binary for the machine’s platform from <download base>/cli/<os>/<arch>/<version>/ironflock and checks it here before it uses it: its signature (.asc) against the IronFlock release key that this binary carries, and its build information, which has to name the program ironflock, the machine’s platform and the same source revision as this binary. The download is kept in your cache folder (~/.cache/ironflock on Linux, ~/Library/Caches/ironflock on a Mac) and checked again on each use.
2. The folder. The binary goes into a new folder that only the login can read (mktemp -d), under ${TMPDIR:-/tmp} on the machine, or under the login’s home when a program cannot run in the temporary folder (a file system mounted noexec). When it can run in neither, the command ends with exit code 3 and names both. Before it runs anything from the folder, the machine checks that the folder and the binary belong to the login and are not symbolic links, and that the binary is the version and platform expected.
3. The files. The files named in the command’s flags go along (see Files Named in Flags).
4. The command. It runs as the login, or through sudo (see Root on the Other Machine).
5. The clean-up. When the command ends, the machine removes the folder with the binary and the copies: also after a failure, after a lost connection, and after the signals HUP, INT, QUIT, TERM, PIPE, USR1 and USR2.

--host leaves no ironflock on the machine, also after agent install. Go on with --host from here, or install ironflock on the machine with the install line, which without arguments only installs it. self-update over --host updates that /usr/local/bin/ironflock of the machine, not the binary of the run (ironflock self-update).

A folder ironflock.* stays on the machine in two cases:

• After a power loss or a kill -9 on the machine while the command runs. The folder holds the binary and the copies.
• When the connection is lost after the copy and before the command started, or the run is interrupted here while the step that starts the command connects. The folder holds the binary, and the copies too when you ran the command at a terminal.

A copy of a .flock file holds the device’s secret, a copy of a passphrase file the WiFi passphrase, a copy of a password file the proxy password. Only the login and root can read them. Remove such a folder by hand.

Files Named in Flags

Each file is read here, and a relative path is relative to this machine’s working folder. A passphrase or password file may be a pipe (/dev/stdin), which is read once. The content goes to the machine on the standard input of the SSH connection. Neither a file nor its size is on a command line, here or on the machine, where every user can read command lines. The machine checks each size against the largest the command takes, and then that all of it arrived.

On the machine, the copy is <folder>/1/device.flock: in a folder of its own (mode 0700) in the run’s folder, with the name it has here and mode 0600. The command there gets that path, for example --config=/tmp/ironflock.k3Xf9QpL2m/1/device.flock, so an error about what the file holds names the copy.

These end the command on this machine:

• Exit code 1 — a file that is not a regular file (a passphrase or password file may also be a pipe), or one larger than the command takes: 5 MiB for a .flock file, 4096 bytes for a passphrase or password file.
• Exit code 2 — a file name with a control character.
• Exit code 2 — --image-file and --output of os flash. --host copies no image to the machine and brings no file back. Run that command here without --host, or on the machine itself. Without --image-file, the machine downloads the image itself.

Root on the Other Machine

• A command that needs root runs through sudo when the login is not root. sudo without a password and without a terminal works anywhere.
• version, os images and os drives run as the login.
• status and doctor read more with root. They run as root when the login is root or may use sudo without a password and without a terminal. Otherwise they run as the login, without asking, and their hint names the command with --host root@machine.
• agent logs reads a file of root’s and runs through sudo.
• When sudo asks for a password, or its configuration wants a terminal (requiretty), it needs a terminal on the machine. That needs a terminal on stdin, stdout and stderr here. Without one, the command ends with exit code 3 before anything is copied. Then log in as root, let the login use sudo without a password and without a terminal, or run the command at a terminal without redirection.
• A sudo that refuses when the command is about to run ends with exit code 3 as well.

What the command there says to run next names a command for this side, without sudo, as --host gets root itself: ironflock --host pi@device-17 agent start.

Terminal, Questions and Output

• With a terminal here, the command gets one on the machine too, so it can ask its questions. Its stdout and stderr then come back together, on your terminal.
• Without a terminal here (a pipe, --json into a program, CI), stdout and stderr stay apart, and the command asks nothing: --yes confirms. What the login’s files on the machine print, such as a .bashrc that echoes, is dropped from stdout, so that stdout carries the command’s result alone.
• --yes, --json and the command’s flags go along. The download base goes along as --download-base: the flag, or IRONFLOCK_DL_BASE of this machine. A download base with a login (https://user:[email protected]/dl) ends with exit code 2 before any machine is reached, as on the machine it would be on a command line that every user there can read.
• --help, help, a group without a command and completion are answered here: what they print is the same on every machine.
• What the machine sends on stdout before the command runs (its answers and the text of the login’s files) is read up to 64 KiB. More ends the run with exit code 1 before the command runs there.

Ctrl-C and Lost Connections

Ctrl-C here reaches the command there as Ctrl-C at its own terminal would, and a lost connection reaches it as a hangup, with and without a terminal here. After Ctrl-C, --host waits up to 15 seconds for the machine to say that the command there ended, with its exit code (130 when Ctrl-C ended it). When the machine does not say so in time, or the connection ends first, --host ends with 130 and says that the command may still run there. A run whose command had not started when Ctrl-C came does not start it.

os install over --host has rules of its own: once its question is answered, a lost connection no longer stops it, and the command waits here for the machine to come back as IronFlockOS; see ironflock os install.

Exit Codes and Errors

The exit codes are the command’s (Exit Codes). --host adds these:

A failure of --host itself is an error like any other: an Error: line on stderr, or with --json the error object on stdout. The one exception: when the command’s output had come through already, that output is the result, and the error is the Error: line on stderr alone, also with --json.

Several Machines

--host given more than once, or --hosts <file>, runs the command on each machine, as --host runs it on one. The machines of --host come first, then those of the file. A hosts file with one machine in it runs as several machines do.

$ ironflock --host pi@vm1 --host pi@vm2 --host pi@vm3 status
pi@vm2: Agent:         installed in /opt/ironflock/device-agent
pi@vm2:   Version:     1.6.9
pi@vm2:   Service:     running (ironflock-device-agent.service)
pi@vm3: ssh: connect to host vm3 port 22: Connection timed out
pi@vm3: Error: the SSH connection to pi@vm3 failed
pi@vm1: Agent:         installed in /opt/ironflock/device-agent
...
Summary of "status" on 3 machines, exit code 255: 2 done, 1 ssh.
  pi@vm1  done
  pi@vm2  done
  pi@vm3  255 ssh: the SSH connection to pi@vm3 failed (ssh: connect to host vm3 port 22: Connection timed out)

The Hosts File

The file names one [user@]machine per line. A # starts a comment, which runs to the end of the line, and empty lines are left out. The file may also be a pipe, such as --hosts <(...), and holds at most 1 MiB.

# Packaging line 3
pi@device-17
pi@device-18
pi@device-19   # the spare unit

# Show the state of every machine in the file, 20 at a time
ironflock --hosts line3.txt --parallel 20 status
 
# Update the agent on every machine in the file; agent update asks, so --yes is needed
ironflock --hosts line3.txt agent update --yes
 
# Give the devices of a site one proxy account; the password comes from a file
ironflock --hosts site-a.txt agent configure --https-proxy http://[email protected]:3128 --proxy-password-file proxy-password.txt --yes

These end with exit code 2 before anything runs, and the error names the file and the line:

• a line with more than one word;
• a machine that --host would refuse;
• a machine named twice. Twice is also the same machine with two users, as the command would run on it twice at once. An address and a name of one machine are not recognized as the same.

Parallel Runs

At most --parallel machines run at once: 10 without the flag, from 1 to 100. They start in the order given, and the next one starts when one ends. --parallel without several machines ends with exit code 2.

SSH Without Questions

No machine gets a terminal, and ssh asks nothing (BatchMode=yes). These fail on that machine with exit code 255 and ssh’s own words, and the other machines run on:

• a login that wants a password;
• a key whose passphrase is not in the SSH agent;
• a host key that ssh does not know yet.

Connect once with ssh to accept a new host key, or set StrictHostKeyChecking accept-new for those machines in ~/.ssh/config, as in the example above. ssh gives up connecting after 30 seconds (ConnectTimeout=30, whatever ~/.ssh/config says), so a machine that is switched off is reported then. A sudo there that wants a password or a terminal ends with exit code 3 on that machine; status and doctor run as the login then.

Output of Several Machines

• Each line a machine writes comes whole, with <host>: in front: its stdout on stdout, its stderr on stderr. The machines’ lines mix in the order they come.
• A last line without its newline gets one. A line longer than 64 KiB comes in parts of 64 KiB, each with the host in front.
• Control characters other than the tab come escaped (\x1b, \r), in the lines and in the summary, so that no machine moves the cursor or writes over another machine’s line. A carriage return at the end of a line is dropped.
• A run that fails here adds its Error: line with the host in front, as --host writes it for one machine.
• At the end, stderr gets a summary: the exit code and how many machines ended how, then a line for each machine in the order given. The line says done, or the exit code, its kind and why: the error here (for 255 with ssh’s own words) or the command’s Error: line. A machine that did not confirm the end of its command after Ctrl-C is interrupted, may still run.

Exit Code of Several Machines

The command ends with the worst exit code of the machines: 0 when every machine ended with 0. Otherwise, from the worst on:

1. 130 — interrupted, here or on a machine.
2. 1 — the command failed on a machine, which may have left it half changed, whatever code it ended with there.
3. 255 — a machine was not reached, or the connection broke.
4. 3 — a machine was not in the state the command needs, or cannot run it.
5. 4 — a question was declined.

Exit code 2 is a wrong command line here, as for every command.

JSON Lines

With --json, stdout gets one line for each machine when its command has ended, in the order they end, and nothing else. A program reads the lines as they come.

{"host":"pi@vm1","state":"done","exit_code":0,"result":{...}}

stderr has the machines’ lines, among them a command’s own Error: line where it writes one with --json (agent update), and the summary. The errors of the runs here are in the lines on stdout only.

# The machines where status did not end with 0
ironflock --hosts line3.txt status --json 2>/dev/null | jq -r 'select(.state != "done") | .host'

Ctrl-C on Several Machines

Ctrl-C, SIGTERM or SIGHUP ends the runs as it ends a run on one machine: each command there gets it, and each machine has 15 seconds to confirm that its command ended. A machine that does not is may_still_run. The machines whose turn has not come run nothing. The summary lists them as not started, and the command ends with 130.

Which Commands Run on Several Machines

A flag that names a file on this machine is refused with several machines, as the file would go to every machine: -c of doctor and of agent configure names one device. The exception is --proxy-password-file of agent configure, for one proxy account for the machines of a site.

A command line that does not run on several machines ends with exit code 2 and says why, before the hosts file is read and before any machine is reached.