DEBIAN_FRONTEND environment variable to noninteractive for all our connections as we regularly use apt and related commands.
(If we don't, command execution may hang as a dialog is shown and the process is waiting for user input.)
This week someone created a bug report because their apt-get commands were hanging on multiple servers.
Pretty strange, as this was the exact reason we're setting environment variables in the first place. Let's investigate!
To change environment variable for the currently running process you can use the Setenv function provided by the standard library's os package.
This can be useful when you use the os/exec package to start subprocesses.
func Setenv(key, value string) error
The ssh package provides a similar function that sets an environment variable on an SSH session.
These variables will be available for any command you execute on that same session after changing calling the Setenv function.
func (s *Session) Setenv(name, value string) error
As this "simply sets a variable" I was ignoring the errors returned by the Setenv function. After all, what could go wrong when setting a simple environment variable? I couldn't be more wrong.
When I actually checked the error object returned by Setenv it turned out not to be nil. It contained the amazingly useful message below.
ssh: setenv failed
Diving in the source code for the ssh package we can see that the request to set an environment variable is sent directly to the SSH server, there's not much in that Go code that can go wrong. Also note that it returns the same error message regardless of what went wrong when calling SendRequest.
func (s *Session) Setenv(name, value string) error {
msg := setenvRequest{
Name: name,
Value: value,
}
ok, err := s.ch.SendRequest("env", true, Marshal(&msg))
if err == nil && !ok {
err = errors.New("ssh: setenv failed")
}
return err
}
So let's take a step back and do it without Go:
[vic@local ~]$ export DEBIAN_FRONTEND=noninteractive
[vic@local ~]$ ssh -o SendEnv=DEBIAN_FRONTEND vic@remote
[vic@remote ~]# echo $DEBIAN_FRONTEND
Nothing, it simply doesn't work.
Turns out, you can't simply change environment variables in SSH sessions. The OpenSSH server denies those requests unless you've whitelisted the specific variable in your server settings. This is for security reasons: people with access to a restricted list of commands could set an environment variable like LD_PRELOAD to escape their jail. (If your user has access to a shell there are other ways to set environment variables.)
To allow changing an environment variable over SSH, change the AcceptEnv setting in your OpenSSH server configuration (typically /etc/ssh/sshd_config) to accept your environment variable:
AcceptEnv DEBIAN_FRONTEND
And now it magically works!
]]>ssh -J charmeleon.mydomain.net bulbasaur.mydomain.net
As we're connecting to Bulbasaur a lot, we may want to tell OpenSSH that we're using charmeleon as a jumphost for bulbasaur:
# ~/.ssh/config
Host bulbasaur.mydomain.net
ProxyJump charmeleon.mydomain.net
With this configuration in place, we can simply use ssh bulbasaur.mydomain.net. This also works for scp, rsync and everything else that uses your local SSH client.
On versions of OpenSSH before 7.3 the ProxyJump option is not available.
In that case you'll have to use an alternative configuration using the ProxyCommand option to specify the exact command OpenSSH has to execute to set up the tunnel.
Host bulbasaur.mydomain.net
ProxyCommand ssh charmeleon.mydomain.net -W %h:%p
Watch out when you're using wildcards in your ssh config:
Host *.mydomain.net
ProxyJump charmeleon.mydomain.net
Using ssh server.mydomain.net now causes a loop as the connection to the jumphost also matches the block and tries to use the jumphost.
An easy way to solve this is an alias for the jumphost:
Host charmeleon
HostName charmeleon.mydomain.net
Host *.mydomain.net
ProxyJump charmeleon