Login hangs for scanning account

The Proplem

I ran into this issue the other day. Tenable.sc (formerly Security Center) was reporting a hit on plugin 21745 for a Red Hat Enterprise Linux 8 (RHEL 8) system. I checked on the account used on the systems for scanning and it wasn't locked out or anything. When I tryied to SSH into the system with the credentals, it would just hang. The system logs showed "login successful". When I rebooted the system was able to login normally again, but the problem would come back eventually.

The Cause

When the Nessus scanner connects to a system, it's scanning, it makes several connections to the host. Each connection starts a TMUX session. The problem is the TMUX sessions where not being closed after the Nessus scanner disconnected from the system. It turned out that the account used for security scanning had around 2,000 TMUX sessions running.

The Fix

Add "set -g destroy-unattached on" to the /etc/tmux.conf file.

scanuser@remotesystem> sudo echo "set -g destroy-unattached on" >> /etc/tmux.conf

This will append this line "set -g destroy-unattached on" into the /etc/tmux.conf configuration file. This will auto close sessions not being actively used.

Anther Fix

Set system wide rules for TMUX on the effected systems so only the account used by the Nessus scanner will have use of the TMUX terminal multiplexer. /etc/profile.d/custom.sh

[ "$USER" != "scanuser" ]
 then if [ "$PS1" ]
   then parent=$(ps -o ppid= -p $$)
        name=$(ps -o comm= -p $parent) 
        case "$name" in (sshd|login) exec tmux
        esac 
 fi 
fi 

Defs

TMUX is an open-source terminal multiplexer for Unix type systems. Multiple terminal sessions can be accessed simultaneously by splitting the terminal into different screens. Can also detach remote sessions and reattach later, simular to what the screen application can do.
Tenable Plugin a plugin is a script deployed by the Nessus scanner to check for security vulnerabilities. In this case plugin 21745 is an info plugin, it displays info from other plugins. This plugin is triggered (displayed) whenever a login failure occurs.

Other useful links

Tmux Cheat Sheet & Quick Reference
https://tmuxcheatsheet.com/
A beginner's guide to tmux
https://www.redhat.com/sysadmin/introduction-tmux-linux

Setup SSH Keys with Agent

This guide for setting up SSH Keys with an SSH Agent for auto-logging into Linux based systems. Why set up SSH keys, why not just use your password? SSH Keys are considered more secure than using passwords to access systems, because user accounts are authenticated by the server without ever having to send your password over the network. If the passwords are not transmitted then they can't be intercepted. This works by identifying yourself to an SSH server using public-key cryptography and challenge-response authentication. When you set up an SSH agent then the agent will handle the challenge-response authentication for you.

This guide is not for installing or setting up a SSH server. You must have the sshd service running on your system in order to follow along with this guide. All the examples are take from a Red Hat system. In the following examples, earth is the name on the local system and moon is the remote system. 

Create you key pair

The ssh-keygen command will generate a public and private keypair. The keys will be stored in the users home directory by default, this is the path  ~/.ssh/. The basic command looks like this: ssh-keygen -t [dsa|rsa]  The -t sets the type of keys used. In the example below I create a rsa key pair. This command also sets the passphrase, think of this like setting a password. Make sure you remember the  passphrase because you will use this instead of the password for logging into the remote system (moon).

man@earth> ssh-keygen -t rsa
Enter file in which to save the key (/home/man/.ssh/id_rsa): Press [Enter] key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/man/.ssh/id_rsa.
Your public key has been saved in /home/man/.ssh/id_rsa.pub.
The key fingerprint is:dfhjodfnk
04:be:15:ca:1d:0a:1e:e2:a7:e5:de:98:4f:b1:a6:01

It is import to protect your private key, so don't share it. 

Note- Do not leave the passphrase blank. Doing this is a bad security practice, because this defeats the purpose of having the extra security of SSH keys. This will make the system less secure. If you have done this just rerun the command and add a passphrase. 

Install Public Key on Remote Host

You install the public SSH key by copying or appending it to the authorized_keys file on a remote host. This file is also located in the users home directory, ~/.ssh/.  For most systems you can use the ssh-copy-id command, which I cover in Method 1.  I will show a work around if the ssh-copy-id command is not available, in Method 2 & 3.

Method 1 - Use The ssh-copy-id Command

The easiest way to install the public key to a remote SSH server is use the ssh-copy-id command. To use the command type "ssh-copy-id <remote host>". This command appends the public key to the authorized_keys file on the the remote host. If the file doesn't exist it will be created. 

In the example below "moon" is the name of the remote host.

man@earth> ssh-copy-id moon
Now try logging into the machine, with "ssh 'remote-host'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

This the best way is to copy over the public key because ssh-copy-id will create .ssh/authorized_keys file if it doesn't exist. The command also appends the public key to the authorized_keys file, whereas other methods may overwrites the file. This allows the account to use keys from more than one system.

Note- This method will not work on some systems, such as Solaris 10.

Method 2 - Manually Copy the Key File

This method uses the scp command to copy the public key to the remote server. Before the ssh-copy-id  command came about, this was way it was done. The key file will fail to copy to the system if .ssh/authorized_keys doesn't exist. If this happens just login with your password and create the file and try again. The main downside to this method is that it overwrites the authorized_keys file.

man@earth> scp ~/.ssh/id_rsa.pub moon:~/.ssh/authorized_keys

Alternately you can get around this by doing this instead. The command below mimics what the ssh-copy-id command does. It creates the .ssh directory if it doesn't exist and appends the contents of the key to the authorized_keys file.

man@earth> cat ~/.ssh/id_rsa.pub | ssh man@moon "mkdir -p ~/.ssh  &&  cat   >>  ~/.ssh/authorized_keys

Method 3 - If You Automount Your Home Directory

If your home directory automounts across a lot of servers then you can just append the contents of the public key to the authorized_keys file. This method can be a lot faster then other methods. For example if you have 100 hosts you need to connect to, you just need to run the command once to connect to all of them. Instead of running 100 copy commands you just run one. Again this will only work if the hosts your connecting to automount the same home directory that the SSH kays are on.

You can copy it over with the cat command.

man@earth> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The other methods will still work if you prefer them to this one. 

Setting up the SSH Agent

At this point when you login to a remote host you now get prompted for the SSH passphrase instead of the password. To automate the login process it is recommended to setup an SSH Agent. Luckily setting up an SSH Agent is easy. The agent holds the passphrase for the user and then presets the passphrase when prompted. 

There are some considerations you need to consider when running an SSH Agent. If a GUI is installed on the system, such as gnome, then the window manager may run the agent for you. This is by far the easiest way to setup an agent. If the system window manager handles SSH keys then you get prompted with a GUI text box for a passphrase the 1st time you try to SSH to a remote host. If you enter the passphrase the GUI will run the SSH Agent for you system wide for the user currently logged into that account. This lasts until the system is rebooted or the user logs out.

You can also run the agent from within a terminal or shell. When you run an agent from the terminal it will only work from that terminal and not system wide like does if the GUI manages the agent. 

To start an SSH Agent in a terminal run the commands show below. 

man@earth> eval `ssh-agent`
man@earth> ssh-add
Enter passphrase for /home/man/.ssh/id_rsa:
Identity added: /home/man/.ssh/id_dsa (/home/man/.ssh/id_rsa)

Note- Add these commands to the .bashrc file to start an agent automatically when you open a terminal emulator. 

If you use VNC, just start your VNC server session in the same terminal you used to starting your agent. This way all your terminal emulators launched in your VNC session, will use the same agent.     

SSH Agent Management

One issue with agents is that sometimes you end up running a lot of agents. This is because the agent doesn't stop running when the terminal closes. You need to run the command below and kill any agents that you are not using.

man@earth> ps aux | grep agent

If there is more than one agent running then you should kill the additional ssh-agent.

man@earth> pkill ssh-agent

This will only kill agents owned by the user running the command in.

One way to kill your ssh-agents is to add a kill statement to the .bash_logout file.

Reference Section
Manpage ssh-copy-id