Troubleshooting SSH Connections and Using syslog

I have been struggling with ssh connections.  So, I thought I would write down some of the different troubleshooting tips that I found.

Permissions

Make sure that your permissions are correct on your home directory and your key files.  Here are the permissions that you need.

Check with this:

ls -ld $HOME $HOME/.ssh $HOME/.ssh/authorized_keys $HOME/.ssh/id_rsa

Note: make sure that you check this on your local computer and your remote computer.  The permissions should be:

chmod go-w $HOME
chmod 700 $HOME/.ssh
chmod 600 $HOME/.ssh/authorized_keys
chmod 600 $HOME/.ssh/id_rsa

SELinux

Another thing I found was that SE Linux was stopped the private keys from working.  Using dmesg, I found lines like this:

type=1400 audit(1332520527.110:51337): avc: denied { read } for pid=25240 comm="sshd" name="authorized_keys" dev=dm-5 ino=167 scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:home_root_t:s0 tclass=file

I checked the status with sestatus, and found that SELinux was enforcing.  Then, I used the following command to turn it to permissive mode:

setenforce 0

Checking again, you’ll see it is in permissive mode now:

$ sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: permissive
Mode from config file: permissive
Policy version: 24
Policy from config file: targeted

Then, when I tried connecting, I found that the key authentication and X11 forwarding worked.  Just a note: the key authentication worked with policy version 23 but not 24.

Logging on the Client

Use the “-v” parameter to ssh will provide some output as to what is wrong.  In fact, you can maximize the debugging messages with “-vvv”.  So, you can see what’s going on, you can do something like this:

ssh -vvv remoteuser@remotecomputer

Writing messages to dmesg

I read that the messages go to dmesg.  I had trouble telling if dmesg was changing.  So, I decided to see if I could add something to the log.  Then, I would know that anything after that message was new.

Here’s the command that did the trick:

sudo bash -c "echo hello world > /dev/kmsg "

Configuring Logging for SSH on the remote machine

At first, I couldn’t get any log messages out of the remote machine.  I found that I had to adjust the configuration for both syslog and sshd.

First, I change syslog to create a separate log file for sshd.  You can make this change in /etc/rsyslog.conf.  The local7.* already existed in my configuration.  I added the local6.debug line:

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# SSH specific (Added by Stephen)
local6.debug /var/log/sshd.log

Then, I had to change the sshd configuration.  You can make this change in /etc/ssh/sshd_config.  I set the SysLogFacility to “LOCAL6” to make the separate log file setting work from above.  Then, I changed the LogLevel to “DEBUG”.  I think there is also a “DEBUG3” that might provide even more.

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#SyslogFacility AUTHPRIV
#LogLevel INFO
SyslogFacility LOCAL6
LogLevel DEBUG

Finally, restart both sshd and rsyslog to make the changes take effect.

sudo service sshd restart && sudo service rsyslog restart

This is a cool trick.  If you open a shell to monitor the log, you can use the tail command to print messages to the screen as they are written:

tail -f /var/log/sshd.log

This is what helped me find my problem.

End Result/My Problem

I finally found that my problem was the permissions of the home directory.  I found this in the log file:

Authentication refused: bad ownership or modes for directory /home/remoteuser

After I changed ownership of the home directory, my key authentication worked great.

Resources

• ServerFault.com: How to add message that will be read with dmesg?
• Synology: SSHD – LogLevel?
• DISSECTION BY DAVID BLOG: SSH log file — making one, new location
• Ubuntu Documentation: SSHOpenSSHKeys
• LHN: Quick HOWTO : Ch05 : Troubleshooting Linux with syslog
• Red Hat Enterprise Linux 6: Security-Enhanced Linux User Guide Edition 2
• Bug 708986 – enforcing MLS: root (sysadm_r or secadm_r) cannot run ssh-keygen
• selinux-policy bug fix and enhancement update
• Bug 575203 – selinux denies ssh-keygen -f /root/.ssh/id_rsa when run from /etc/init.d directory