Johannes 'heipei' Gilger

OpenSSH - Secure Networking Swiss-Army Knife

29 Apr 2015

tl;dr: Well-known yet underused OpenSSH features and their applications in building secure systems.

Introduction

My last post on this blog was about the dangers of using SSH Agent Forwarding. In that post I recommended using OpenSSH ProxyCommand as not only a workaround but actually a superior way of hopping between hosts. This feature is clearly documented and intended to be used as such, still a lot of the discussions which ensued on Reddit and Netsec showed me that there are quite a number of users who were previously unaware of these features. That’s what prompted me to do a follow-up on some additional “tricks”, even if it’s just the application of something clearly documented in the manpages. But then, not everybody reads the manpages.

This post will cover some of these features and also provide some references for further reading. I’ll start by introducing the basic features and then presenting some ways to use them. The biggest takeaway is this:

OpenSSH is a lot more than a tool to securely to connect to your VPS. Think of it as a simple, well-understood building-block for constructing secure distributed systems, for both automated and interactive applications. With the right workflow, employing OpenSSH will come to you very naturally and comfortably.
Me, April 2015

Securing OpenSSH

We start of by applying reasonable security settings to our OpenSSH workflow. In the recent months there were enough posts written on this topic, so I’ll just reference them here. It boils down to this:

The most current and comprehensive post on this topic has been written by @stribika in his blog-post Secure Secure Shell. His observations are spot-on, including the section on “System hardening”: Keep a clean system.

What I like to do on top of this is to employ some way of monitoring the SSH logs and rate-limiting connection attempts in a very crude way. Monitoring SSH connection attempts can be done by logging TCP connections to your SSH ports and then ingesting these logs into something like Logstash and Kibana. To rate-limit connection attempts I use a very simple iptables ruleset because I don’t trust systems like fail2ban enough.

iptables -A INPUT -p tcp --dport 12345 -m conntrack --ctstate NEW -m recent --set
iptables -A INPUT -p tcp --dport 12345 -m conntrack --ctstate NEW -m recent \
	--update --seconds 120 --hitcount 3 -j DROP
iptables -A INPUT -p tcp --dport 12345 -j ACCEPT

~/.ssh/config

Next we turn our attention to your ~/.ssh/config file. This file can be used to configure per-host parameters, most commonly a combination of a custom port and username.

You should consider ~/.ssh/config your authoritative and only sources about your hosts. Whenever you get access to a new host just add it to this file and don’t worry about remembering it. The Host line can be used to create aliases which are easy to remember (e.g. ams-vps instead of a raw IP).

Let’s examine a contrived example to show the options I find myself most frequently using:

1
2
3
4
5
6
7
8
9
10
11

ServerAliveInterval 10
TCPKeepAlive no

Host ams-vps
	Hostname 123.123.123.123
	Port 12345
	User root
	IdentityFile ~/.ssh/vps_key
	IdentitiesOnly yes
	LocalForward 8080 localhost:80
	DynamicForward 9090

Let’s go through these lines one by one, ignoring the “Alive” lines.

   google-chrome-stable
   --user-data-dir=$HOME/.config/google-chrome-socks
   --proxy-server=socks://localhost:9090

My ~/.ssh/config is currently ~250 lines long, just to give you an idea. The newest versions of OpenSSH even allow include statements in this file.

ControlMaster

OpenSSH ControlMaster is one of the options that I use globally in my ~/.ssh/config.

ControlMaster auto		# Auto == Create and use as needed
ControlPath /tmp/%r@%h:%p	# The sockets are stored here
ControlPersist yes		# Optional

What ControlMaster does is create (and subsequently use) a control socket for each connection to a remote server. SSH supports multiple independent “channels” which can be multiplexed over a single existing SSH connection. On the first connection to a host, OpenSSH will create a control socket in /tmp. With each subsequent connection, if a ControlMaster exists for a given user/host/port, OpenSSH will use it to create a new channel (pty or scp) within the existing SSH connection without going through the SSH handshake and shared secret agreement. The socket will be destroyed when the last SSH session disconnects.

The biggest reason for me to use ControlMaster is performance. The SSH handshake does take a while, even more so with increasing latency. While this might not be an issue for the occasional login, it is really annoying or downright prohibitive when trying to connect more frequently. The prime example here is remote scp-completion, something that zsh is capable of doing out of the box. Unless you’re on your local network, scp-completion just plain sucks without ControlMaster. The problem is even amplified when considering scenarios with more than one hop (ProxyCommand). In this case, each full SSH session establishment can take several seconds.

The ControlPersist keyword is optional. I don’t use it because I like to keep track of the open session, but software like Ansible employs it to have the control socket linger for a while after the last/initial SSH connection is closed.

ProxyCommand

I’m not going to talk about ProxyCommand at length since I mentioned its benefits in my last post. As a quick recap: ProxyCommand allows your system to connect to an otherwise inaccessible system via one or multiple intermediate “hops”. It has undeniable advantages in terms of usability and security over using something like SSH agent forwarding. The number of ways you could get pwned when not using ProxyCommand is not something to be dismissed lightly.

Suffice to say, none of the features I mentioned in this post (LocalForward, DynamicForward, ControlMaster) work if you use agent forwarding or manual hopping as opposed to using ProxyCommand. One more reason to reconsider.

~/.ssh/authorized_keys

The authorized_keys file specifies which public key is allowed to login to the current account. You’ve probably used it when adding your own public key on a remote server. The basic format of this is very simple: It takes one SSH public key per line to allow login via that key.

Some people might not know that you can actually add a number of options per key. The complete description is in man sshd, I’ll only cover the options I frequently use.

command="date",no-agent-forwarding,no-pty,no-port-forwarding,permitopen="192.168.1.1:22" ssh-rsa AAAAB3Nza

Using these options, one can use OpenSSH with a passphrase-less dedicated key for some unsupervised applications:

For a constant connection you can simply run ssh via some process supervisor like runit.

Conclusion

The intention of this post was to show that OpenSSH is much more than just a remote-login tool. It can be used for various automated applications. The nice thing about using SSH for these cases is that it is cross-platform, dead-simple to setup, test and to actually understand the security and features offered by such a solution.

Want to forward a single port with some low-bandwidth yet high-value traffic over the Internet? Why set up a complicated VPN solution when you know how to use SSH? Want to have automated backups? Use SSH! Even if you only use SSH interactively, this post might have shown you a few tricks to improve your workflow.

Having something that a user already understands and which does not introduce a new attack surface is a huge security win in my opinion. The tendency of some modern software is to just bind to localhost and omit any form of authentication or transport security. Here, SSH can be used as a security layer, even if only during development. The same is true for supposedly secure protocols which you still don’t trust entirely. Used correctly, OpenSSH is a very robust system that does authentication, authorization and proper transport security and is part of every conceivable distribution. Also consider that OpenSSH is one of the pieces of your userland toolkit which is most closely reviewed and at the same time still being actively developed by the OpenBSD community.

Going further

Topics not mentioned here include:

References

Comments

Johannes Gilger - @heipei

Atom Feed