SSH security is foundational to your infrastructure. Passwords are risky; public-key authentication is the standard. This guide covers the essentials for managing keys, onboarding developers safely, and hardening SSH on an EC2 instance, plus architectural defenses for production.
Part 1: The Foundation — Understanding Your .pem Key
What AWS Gives You
When you launch an EC2 instance with a key pair, you download a .pem file. This is your private key. AWS injects the matching public key into the instance’s default user account at boot.
Key Analogy
- Public key = the lock. Safe to share; installed on the server in
authorized_keys. - Private key (.pem) = the key. Secret. Never share, email, or upload publicly.
How Authentication Works
Your SSH client proves possession of the private key via a cryptographic challenge without sending the key over the network.
Golden Rule
Treat your .pem like a root-level secret. Restrict file permissions locally (chmod 400 your-key.pem) and never store it in public places.
Part 2: Securely Verifying Your Keys
Never paste private keys into online tools. Use local utilities.
Derive the public key from a .pem private key
ssh-keygen -y -f /path/to/your-key.pem
The output begins with ssh-rsa or ssh-ed25519 and should match what’s on the instance in the default user’s ~/.ssh/authorized_keys.
Non-EC2 generic Linux VM login setup
Generate a key pair if you don’t have one:
ssh-keygen -t ed25519 -C "Ali's Key"
Note: Prefer ed25519 for modern security and smaller keys. If needed, RSA 4096 is acceptable:
ssh-keygen -t rsa -b 4096 -C "Ali's Key"
Copy your public key to the server:
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@serverIP
If ssh-copy-id isn’t available:
ssh user@serverIP "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
cat ~/.ssh/id_ed25519.pub | ssh user@serverIP "cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
Log in:
ssh user@serverIP
Part 3: Onboarding a Developer — One User, One Key
Step 1: Developer generates their key pair
ssh-keygen -t ed25519
The developer keeps their private key private and sends only the public key (id_ed25519.pub).
Step 2: Admin creates a dedicated Unix user on the EC2 instance
sudo adduser Ali
On Amazon Linux 2/2023 and Ubuntu, this creates the user and home directory.
Step 3: Install the developer’s public key with correct permissions
sudo su - Ali
mkdir -p ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
echo "ssh-ed25519 AAAA...developer-public-key" >> ~/.ssh/authorized_keys
exit
Developer connects using their private key and username
ssh -i /path/to/private_key Ali@EC2_PUBLIC_IP
Note: The -i flag points to the private key (not the public key). Ensure file permission is restricted, e.g., chmod 600 /path/to/private_key.
Why not reuse ec2-user?
Using unique users provides accountability and easy revocation. The default ec2-user commonly has passwordless sudo for automation.
Cloud-init sudoers snippet
View the file that grants ec2-user passwordless sudo:
sudo cat /etc/sudoers.d/90-cloud-init-users
Typical content:
ec2-user ALL=(ALL) NOPASSWD:ALL
This modular drop-in is managed by cloud-init, which keeps system sudoers intact while enabling automation.
Part 4: Hardening SSH for Production (/etc/ssh/sshd_config)
Edit the daemon config, then restart SSH to apply. Always keep an active session while testing changes so you don’t lock yourself out.
Disable password authentication
PasswordAuthentication no
Disable root login
PermitRootLogin no
Change the default port to reduce noise
Port 2222
Remember to update the EC2 Security Group to allow the new port and remove 22 if no longer used.
Limit who can SSH
AllowUsers Ali ec2-user
Or by group:
AllowGroups ssh_users
Set idle timeout
ClientAliveInterval 900
ClientAliveCountMax 5
Disable X forwarding
X11Forwarding no
Optional but recommended
PubkeyAuthentication yes
PermitEmptyPasswords no
MaxAuthTries 3
MaxSessions 10
LoginGraceTime 30
KexAlgorithms, Ciphers, MACs: use modern defaults unless you need legacy compatibility. On modern distros, defaults are already strong.
Apply changes safely
sudo systemctl reload sshd
If reload fails, use:
sudo systemctl restart sshd
Architectural and External Security Layers
Bastion host (jump box)
Keep app and DB servers private. Expose only a hardened bastion in a public subnet. Use ProxyJump for convenience:
ssh -J ec2-user@bastion_ip Ali@private_instance_ip
Rate limiting with fail2ban
Protects against brute-force attempts by banning offending IPs based on log patterns.
- Install
fail2ban - Enable
sshdjail - Tune
bantime,findtime,maxretry
Security groups and network ACLs
Restrict SSH to trusted CIDRs (office, VPN). Avoid 0.0.0.0/0 to the extent possible. Consider AWS Network Firewall or third-party WAF/VPN solutions for layered protection.
AWS Systems Manager Session Manager (portless access)
Eliminate inbound SSH entirely.
- How it works: SSM Agent initiates an outbound connection; sessions tunnel via AWS.
- Benefits: No open ports, IAM-controlled access, session logging to CloudWatch or S3, strong auditing.
- Requirements: SSM Agent installed, IAM role with SSM permissions, VPC endpoints recommended for private subnets.
Operational Best Practices
- Keep an emergency break-glass path: either a bastion or SSM access with a tested runbook.
- Rotate and revoke keys promptly: Remove user entries from
authorized_keysand, if needed, disable or delete the user. - Enforce local key protection: passphrase-protect private keys, use
ssh-agent, and set strict file permissions. - Use IAM and SSO for SSM access: For SSH, consider short-lived certs via SSH CA (e.g., Smallstep, HashiCorp Vault, or AWS Certificate Manager for SSH via IAM roles) for scalable ephemeral access.
- Monitor auth logs: Amazon Linux/Ubuntu typically at
/var/log/secureor/var/log/auth.log. Ship to CloudWatch or a SIEM. - Backup and test your
sshd_configrollback plan: Keep the current session open; verify a second session before closing the original.
