I have been struggling with ssh connections. So, I thought I would write down some of the different troubleshooting tips that I found.
Permissions
Make sure that your permissions are correct on your home directory and your key files. Here are the permissions that you need.
Check with this:
ls -ld $HOME $HOME/.ssh $HOME/.ssh/authorized_keys $HOME/.ssh/id_rsa
Note: make sure that you check this on your local computer and your remote computer. The permissions should be:
chmod go-w $HOME chmod 700 $HOME/.ssh chmod 600 $HOME/.ssh/authorized_keys chmod 600 $HOME/.ssh/id_rsa
SELinux
Another thing I found was that SE Linux was stopped the private keys from working. Using dmesg, I found lines like this:
type=1400 audit(1332520527.110:51337): avc: denied { read } for pid=25240 comm="sshd" name="authorized_keys" dev=dm-5 ino=167 scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:home_root_t:s0 tclass=file
I checked the status with sestatus, and found that SELinux was enforcing. Then, I used the following command to turn it to permissive mode:
setenforce 0
Checking again, you’ll see it is in permissive mode now:
$ sestatus SELinux status: enabled SELinuxfs mount: /selinux Current mode: permissive Mode from config file: permissive Policy version: 24 Policy from config file: targeted
Then, when I tried connecting, I found that the key authentication and X11 forwarding worked. Just a note: the key authentication worked with policy version 23 but not 24.
Logging on the Client
Use the “-v” parameter to ssh will provide some output as to what is wrong. In fact, you can maximize the debugging messages with “-vvv”. So, you can see what’s going on, you can do something like this:
ssh -vvv remoteuser@remotecomputer
Writing messages to dmesg
I read that the messages go to dmesg. I had trouble telling if dmesg was changing. So, I decided to see if I could add something to the log. Then, I would know that anything after that message was new.
Here’s the command that did the trick:
sudo bash -c "echo hello world > /dev/kmsg "
Configuring Logging for SSH on the remote machine
At first, I couldn’t get any log messages out of the remote machine. I found that I had to adjust the configuration for both syslog and sshd.
First, I change syslog to create a separate log file for sshd. You can make this change in /etc/rsyslog.conf. The local7.* already existed in my configuration. I added the local6.debug line:
# Save boot messages also to boot.log local7.* /var/log/boot.log # SSH specific (Added by Stephen) local6.debug /var/log/sshd.log
Then, I had to change the sshd configuration. You can make this change in /etc/ssh/sshd_config. I set the SysLogFacility to “LOCAL6” to make the separate log file setting work from above. Then, I changed the LogLevel to “DEBUG”. I think there is also a “DEBUG3” that might provide even more.
# Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #SyslogFacility AUTHPRIV #LogLevel INFO SyslogFacility LOCAL6 LogLevel DEBUG
Finally, restart both sshd and rsyslog to make the changes take effect.
sudo service sshd restart && sudo service rsyslog restart
This is a cool trick. If you open a shell to monitor the log, you can use the tail command to print messages to the screen as they are written:
tail -f /var/log/sshd.log
This is what helped me find my problem.
End Result/My Problem
I finally found that my problem was the permissions of the home directory. I found this in the log file:
Authentication refused: bad ownership or modes for directory /home/remoteuser
After I changed ownership of the home directory, my key authentication worked great.
Resources
- ServerFault.com: How to add message that will be read with dmesg?
- Synology: SSHD – LogLevel?
- DISSECTION BY DAVID BLOG: SSH log file — making one, new location
- Ubuntu Documentation: SSHOpenSSHKeys
- LHN: Quick HOWTO : Ch05 : Troubleshooting Linux with syslog
- Red Hat Enterprise Linux 6: Security-Enhanced Linux User Guide Edition 2
- Bug 708986 – enforcing MLS: root (sysadm_r or secadm_r) cannot run ssh-keygen
- selinux-policy bug fix and enhancement update
- Bug 575203 – selinux denies ssh-keygen -f /root/.ssh/id_rsa when run from /etc/init.d directory
Hi,
Thanks for your comments, really useful. I mounted a new partition on /home and all my permissions etc were fine but somehow .Xauthority wasn’t being created and there was a long pause in login attempts (even successful ones). setenforce 0 did the trick for me.
Cheers
Stu