SSH incantations

I use ssh all the time, for shell access on remote machines, to transfer files, to carry VNC sessions, and to use different institutional affiliations to get at paywalled things. Over the years I have developed a particular way of setting it up that minimizes the pain involved in doing all of the above things. Several times in the past months I have tried to explain the whole setup to someone, so I think that's a sign it's time to post it here. I'll try to post something about, I don't know, fluffy kittens, boating on the Thames, seaborgium hexacarbonyl, or something soon for those of you who for some reason expected an interesting blog.

Mostly I'm going to post .ssh/config fragments with explanations. But there are a few things to set up first. And while some (all?) of this can be made to work on most machines, other bits might only work on a Linux machine with a relatively modern software environment. Also, of course, there are man pages for the ssh config stuff.

First key step: set up public key authentication. At least one of the places I log in to has turned it off after a break-in took advantage of people's passwordless ssh keys to break into other places; so don't make your ssh key passwordless. The ssh folks have taken other measures to reduce the likelihood of such attacks, but passwordless keys are just asking for trouble and anyway they're not needed.

Making an ssh key is something you basically set up just once, on your home machine (laptop, desktop you sit in front of; I suppose you might have several such). Basically you just run ssh-keygen and provide a decent password when it asks. This makes a private/public key pair that live under .ssh/.

Actually using that key should be done using some kind of ssh agent program; the original ssh-agent does work, but if you have a reasonable environment you shouldn't have to do anything: when you log in, or the first time you need the ssh key, some program (GNOME keyring maybe?) should pop up and ask you for the password, then hold the key in (locked) memory until it's time to forget it (probably until your machine shuts down). If this doesn't work, your desktop environment hasn't left the nineties and you should do something about that.

Getting your public key to the places you want to log in to is pretty easy these days: to get your key to a machine remote, just run ssh-copy-id remote. This pulls the public key from the agent, logs into remote, and puts it in .ssh/authorized_keys. If you can't log in to remote in one step, you can put the key in authorized_keys by hand but we'll try to fix the log-in-in-one-step thing next.

Most of the rest of the steps here are things you put in your .ssh/config file; specifically, you should probably have an entry in there for every host (or group of hosts) you log in to regularly. Lightly fictionalized, it should look like this (which I'll explain below) though you might not want every part for every machine:

Host portal
User myuserid
HostName portal.example.com
ServerAliveInterval 10
ServerAliveCountMax 3
ControlMaster auto
ControlPath ~/.ssh/control-%r@%h:%p
ControlPersist 1h
DynamicForward 9880

Host box*
User myuserid
ProxyCommand ssh portal -a -x -e none -W %h:%p
ForwardX11 yes
ForwardAgent yes
ServerAliveInterval 10
ServerAliveCountMax 3
ControlMaster auto
ControlPath ~/.ssh/control-%r@%h:%p
ControlPersist 1h
LocalForward 9888 127.0.0.1:9888
LocalForward 9889 127.0.0.1:9889

[Edited to add: it was a bit confusing, so I put in a separate stanza for the portal machine. I hope this clarifies what goes where, and I hope it's obvious what you need to change for your case. -A]

The first bit:

Host box*

specifies which host this applies to; you can use wildcards if you want this to apply to many unimaginatively-named hosts (box123, box124, box125...), though wildcards mean you don't get tab-completion on the command line. This doesn't need to be (probably shouldn't be) a fully qualified domain name; just the machine name you want to type. If it's a machine you can get to directly from the Internet, you need to tell ssh the full name somewhere; you can do that as:

Host portal
HostName portal.example.com

Otherwise, if you're logging in through a portal host (that is, there is a machine you can log in to, and you can log in from there to the machine you actually want) you can just use a one-word name that works on the portal host, and assuming you do the ProxyCommand thing (see below) it'll just work.

The userid is obvious; this is your username on the machine you are logging in to:

User myuserid

The next bit is slightly magical, and exactly how you use it will depend on how the machines are set up. But most of the places I log in to allow ssh connections from the outer world only to a portal machine (one hopes that this machine is more secure than the ones we actually use), and then connections from the portal machine to internal machines. The idea here is to save you having to constantly log in twice, start shells on the portal machine, or any of that manual rigmarole. So:

ProxyCommand ssh portal -a -x -e none -W %h:%p

What this says is, to connect to host box*, instead of just opening a network connection to port 22, run this command instead. And this command is an ssh to the host portal (which presumably has its own entry in .ssh/config) with various things disabled, but which instead of starting a shell just tells the ssh server on portal to open a network connection to host box* port 22. Then the ssh connection you just started uses stdin and stdout of the proxy command to communicate with box*. Doing things this way has a couple of advantages. The most obvious is that you don't need to manually connect to portal; it's started automatically. If you need a password to get in to portal (say it doesn't allow public keys), you'll be prompted. If you've done the ControlMaster things I describe below in the .ssh/config entry for portal, then the connection to portal will be reused for any connection that needs to go through portal, and you won't be prompted for a password for any of those (regardless of what authentication portal wants; you're already authenticated). Finally, it's a bit theoretical, but all portal ever sees of your traffic to box* is the encrypted stream, so you don't have to trust it quite as much.

Once logged in to the remote machine, I often want to do more than just run text programs; if I want to pop up the occasional plot window it's helpful to turn on X forwarding:

ForwardX11 yes

Honestly I don't use this as much as I used to, because I have other approaches (see below), but it comes in handy once in a while. It's a bit of a security risk, though, because being able to talk to your X server gives the remote machine a great deal of power over the machine you're sitting in front of. X and ssh try to limit this by classifying some X connections as trusted and others not (so the others can't snoop keystrokes, say) and forwarding only the untrusted connections; if you need to forward the trusted ones too there's an option for that.

Another option that comes in handy from time to time is agent forwarding

ForwardAgent yes

Again this is insecure if you don't trust the remote machine - it can use your agent connection to log in other places pretending to be you - but it comes in really handy if you want to (say) rsync from one server to another: that connection can use the private key sitting on the machine you're sitting in front of.

These two are largely holdovers from older days, but if you want to notice dead connections while you're not using them (as opposed to hoping they have come back before you need them again) you can put these in:

ServerAliveInterval 10
ServerAliveCountMax 3

Also some servers will drop your connection if it sits idle too long, and this can sometimes help keep that from happening. Sometimes not.

Now, these are really handy. I actually wrote about these back when they were more limited, and called them "almost great"; now they're actually great. Basically they let you transparently re-use ssh connections to a machine, to avoid re-authenticating every time. One machine at a major tech university even used to lock you out if you connected too often, say by running several rsync jobs in a script. So:

ControlMaster auto
ControlPath ~/.ssh/control-%r@%h:%p
ControlPersist 1h

These have the advantage that the ssh client process stays alive to handle port forwarding (see below) as well, even if you shut down the initial shell.

I use the ipython notebook to do a lot of work on remote machines (see upcoming post); the notebook server stays running indefinitely, and I just connect to it with a web browser - an interface designed to work with limited connections that sometimes drop out. But to have any kind of security (and access behind portal machines) I have to use port forwarding, so that port (say) 9888 on my machine connects to port 9888 on the server. This is a little annoying, because I have to allocate and remember port numbers by hand, and those port numbers need to be different for every machine I want to do this on, but it does work:

LocalForward 9888 127.0.0.1:9888
LocalForward 9889 127.0.0.1:9889

Finally, I don't do this on many machines, but lots of academic papers are behind paywalls that will let you access the content only if you appear to be coming from a machine owned by a paid-up institution. If I'm trying to read the paper from home, it'll say no even if my work machine could have accessed it. So I use this:

DynamicForward 9880

This makes the ssh process act as a SOCKS proxy: clients (web browsers) on the local machine can ask ssh to open a connection to any machine on the Net for them, and the ssh server will try. Your connection then appears to come from the machine on the far end of the ssh link. This is only really useful for portal machines, and you may want to pick different port numbers, and it only works if the ssh connection is actually active, but it's really handy. You probably don't want to do all your web browsing through such a link, so I recommend getting something like ProxySwitchy; some such programs can even save you the trouble of manually switching connections by using glob patterns to decide whether or not to use a proxy for a given host. Of course using this tunneling to access sites you aren't legally allowed to access would be Bad and Wrong, but the fact that you have to have ssh access to a machine that is legally allowed to access the material (and whose identity is probably logged) limits the mischief this shortcut can do.

Here ends the .ssh/config section, but there are a couple of other ssh tricks I use regularly.

For real-time observing with several radio telescopes, I have to use their graphical interfaces, and it would be Bad if a dropped ssh connection crashed the interface while it was controlling the telescope. So I use VNC to run a session on the remote machine, and connect to that session from the machine I sit in front of. Of course, almost always the machine running the VNC session is behind a portal machine. Most VNC clients are now smart enough to set up the appropriate tunneling:

vncviewer -via vnchost :47

where vnchost is a name I could use as ssh vnchost and 47 is the number of the VNC server (I try to use a reasonably large number so I can use it everywhere without running into other people's VNC sessions).

For long-running console jobs (ipython notebook servers are a notable example, but various data processing tasks also qualify) I usually use tmux (or screen if tmux isn't available). This means if the connection drops or I have to turn my laptop off, I can just reconnect later. (Also it lets me have several in the same terminal window, which is sort of handy, though I'd sort of rather have each one in a separate terminal.) Unfortunately, the persistence of these terminal multiplexers interferes with agent and X forwarding. There are some hacks to fix that, which are all a little flaky and complicated; people are still getting the best way to do it sorted out so I'll let you google it yourself. [Edited to add: I use this scheme to handle the X forwarding, and I'm not sure where I got it but I use this in my .bash_profile to get ssh agent forwarding working:

if [ ! -z "$SSH_AUTH_SOCK" -a "$SSH_AUTH_SOCK" != "$HOME/.ssh/agent_sock" ] ; then
    unlink "$HOME/.ssh/agent_sock" 2>/dev/null
    ln -s "$SSH_AUTH_SOCK" "$HOME/.ssh/agent_sock"
    export SSH_AUTH_SOCK="$HOME/.ssh/agent_sock"
fi

The X stuff isn't perfectly reliable, but I can usually make it work by reconnecting to tmux. Sometimes I have to manually set DISPLAY; not sure why. -A]