![[michael@MBP] ~ $ ec2-ssh --profile sandpit --send-key
-L 5432:some-rds-instance:5432 ec2-user@ec2-demo1
[michael@MBP] ~ $ ec2-ssh --profile sandpit --send-key
-L 5432:some-rds-instance:5432 ec2-user@ec2-demo1
INFO: Resolved instance name 'ec2-demo1' to 'i-08c1950f615da4aa4'
INFO: Using SSH key from SSH Agent, should be as good as any.
ec2-user@ec2-demo1 ~ $
[michael@MBP] ~ $ ec2-ssh --profile sandpit --send-key
-L 5432:some-rds-instance:5432 ec2-user@ec2-demo1
INFO: Resolved instance name 'ec2-demo1' to 'i-08c1950f615da4aa4'
INFO: Using SSH key from SSH Agent, should be as good as any.
ec2-user@ec2-demo1 ~ $ w
06:41:09 up 12 days, 21:42, 1 user, load average: 0.04, 0.09, 0.08
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
ec2-user pts/0 localhost 06:41 1.00s 0.01s 0.00s w
ec2-user@ec2-demo1 ~ $
ec2-ssh
Port forwarding example](https://image.slidesharecdn.com/shutdownyourbastionhost-221109221455-c2fd1c13/85/Shutdown-that-bastion-host-16-320.jpg)
![[michael@MBP] ~ $ rsync -e "ec2-ssh --send-key" ec2-user@ec2-demo1:
[michael@MBP] ~ $ rsync -e "ec2-ssh --send-key" ec2-user@ec2-demo1:
INFO: Resolved instance name 'ec2-demo1' to 'i-08c1950f615da4aa4'
INFO: Using SSH key from SSH Agent, should be as good as any.
drwx------ 181 2022/06/25 22:17:57 .
-rw------- 11,455 2022/06/25 22:18:06 .bash_history
-rw-r--r-- 18 2020/01/16 13:56:01 .bash_logout
-rw-r--r-- 193 2020/01/16 13:56:01 .bash_profile
-rw-r--r-- 231 2020/01/16 13:56:01 .bashrc
[...]
ec2-ssh
RSync example](https://image.slidesharecdn.com/shutdownyourbastionhost-221109221455-c2fd1c13/85/Shutdown-that-bastion-host-17-320.jpg)
![AWS CLI ECS Exec
Find the ECS Cluster name
nginx-Cluster-8SqXVHC
The Terminator-style CLI (i.e. not for the rest of us)
Find the Service name
nginx-Service-gL6Cnj
Find the Task name
11f37244484b486e9e0eda9b94c
Construct the AWS CLI command line:
[michael@MBP] ~ $ aws ecs execute-command --cluster nginx-Cluster-8SqXVHC
--task 11f37244484b486e9e0eda9b94c
--container nginx --command “/bin/bash” --interactive](https://image.slidesharecdn.com/shutdownyourbastionhost-221109221455-c2fd1c13/85/Shutdown-that-bastion-host-19-320.jpg)
![ecs-session
[michael@MBP] ~ $ ecs-session --list
Cluster Group Task Cont. IP
nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj a3875d0e...7d488 nginx 172.31.107.214
nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj 11f37244...9b94c nginx 172.31.73.13
nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj fb4099e3...aba46 nginx 172.31.94.23
For the non-Terminators among us (i.e. for me and you)
[michael@MBP] ~ $ ecs-session 172.31.73.13
nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj 11f37244...9b94c nginx 172.31.73.13
Starting session with SessionId: ecs-execute-command-09c4f8dcf93299816
root@ip-172-31-73-13:/#](https://image.slidesharecdn.com/shutdownyourbastionhost-221109221455-c2fd1c13/85/Shutdown-that-bastion-host-20-320.jpg)
![ec2-serial
[michael@MBP] ~ $ ec2-serial --region us-east-1 ec2-demo1
Wait a few seconds and then press Enter for a prompt...
EC2 console serial port over SSH
[michael@MBP] ~ $ ec2-serial --region us-east-1 ec2-demo1
Wait a few seconds and then press Enter for a prompt...
Amazon Linux 2
Kernel 5.10.147-133.644.amzn2.x86_64 on an x86_64
ip-172-31-16-189 login:](https://image.slidesharecdn.com/shutdownyourbastionhost-221109221455-c2fd1c13/85/Shutdown-that-bastion-host-23-320.jpg)
![ssm-tunnel
[michael@MBP] ~ $ ssm-tunnel ec2-demo1 --route 172.31.0.0/16
[sudo] password for mludvig: ****
INFO: Local IP: 100.64.156.100 / Remote IP: 100.64.156.101
00:00:34 | In: 472.0 B @ 5.2 B/s | Out: 2.7kB @ 0.0 B/s
Your ad-hoc VPC backdoor
[michael@MBP] ~ $ ping 172.31.104.217 # ec2-demo2
PING 172.31.104.217 (172.31.104.217) 56(84) bytes of data.
64 bytes from 172.31.104.217: icmp_seq=1 ttl=254 time=90.8 ms
64 bytes from 172.31.104.217: icmp_seq=2 ttl=254 time=89.0 ms
64 bytes from 172.31.104.217: icmp_seq=3 ttl=254 time=91.4 ms
^C
--- 172.31.104.217 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 89.043/90.447/91.413/1.073 ms](https://image.slidesharecdn.com/shutdownyourbastionhost-221109221455-c2fd1c13/85/Shutdown-that-bastion-host-24-320.jpg)
Shutdown that bastion host!
- 1. There are better ways to login to your EC2 instances Shutdown that Bastion Host Michael Ludvig Principal consultant @ Servian
- 3. Direct SSH to EC2 Quick and easy. And insecure. Pros: * Easy to set up Cons: * Easy to set up insecurely * Direct internet access * Difficult access control
- 4. Bastion host For those private subnets Pros: * Easy to set up Cons: * SSH access to Bastion * Shared SSH key * Difficult access control
- 5. VPN to VPC If you know your routes Pros: * More secure Cons: * Complicated setup * Shared SSH key * Difficult access control
- 6. Shared and re-used SSH keys ● Infamous EC2 “Key Pairs” ● Hard to audit access ● Hard to revoke and replace ● No MFA Common problem with all the methods Access Control Solutions are complex to implement ● Active Directory ● Hashicorp Vault ● FreeIPA
- 7. IAM authentication ● Access tied to IAM Users ● Controlled by IAM policies ● Auditable in Cloud Trail AWS native authentication Access Control via IAM Works with… ● SSH ● EC2 SSM Terminal Session ● EC2 Serial console ● ECS containers including Fargate
- 8. EC2 Session Modern AWS-native way to login
- 9. SSM Session Modern AWS-native way to login
- 10. SSM Session Modern AWS-native way to login
- 11. EC2 Session - requirements Modern AWS-native way to login What do we need … 1. SSM Agent running on the EC2 ○ Default in recent Amazon Linux 2 and Ubuntu 18.04+ instances 2. Network access from EC2 to SSM Service ○ Public IP, NAT gateway, Proxy, or VPC endpoint ○ No need for inbound access 3. EC2 Instance Role with AmazonSSMManagedInstanceCore managed policy
- 12. SSM Session - AWS CLI example Modern AWS-native way to login Official AWS CLI way … ~ $ aws ssm start-session --target i-08c1950f615da4aa4 Starting session with SessionId: botocore-session-1665972406-0d66b7a6d75402362 sh-4.2$ exit Exiting session with sessionId: botocore-session-1665972406-0d66b7a6d75402362. … because we all remember the Instance IDs, yeah right.
- 13. ec2-session (a.k.a. ssm-session) List instances: ~ $ ec2-session --profile sandpit --list i-08c1950f615da4aa4 ec2-demo1 172.31.16.189 3.26.20.151 i-0bf539dae623eba84 ec2-demo2 172.31.104.217 # No Public IP! Select host by name, run session as ec2-user: ~ $ ec2-session ec2-demo2 --user ec2-user Select host by IP, run a command: ~ $ ec2-session 172.31.16.189 --command "sudo yum install -y tmux" An easier way in
- 14. AWS SSM Tools ~ $ pipx install aws-ssm-tools installed package aws-ssm-tools 1.5.0-beta5, Python 3.8.11 These apps are now globally available - ec2-serial - ec2-session - ec2-ssh - ecs-session - ssm-tunnel done! ✨ 🌟 ✨ Contribute on GitHub: https://github.com/mludvig/aws-ssm-tools EC2 access for humans
- 15. ec2-ssh Why? ● No need for inbound SSH access ● No need for pre-shared SSH key (aka “EC2 Key Pair”) ● Useful where ec2-session is not enough, e.g. rsync, ssh port forwarding ● Access can be audited in CloudTrail SSH connection over SSM session
- 16. [michael@MBP] ~ $ ec2-ssh --profile sandpit --send-key -L 5432:some-rds-instance:5432 ec2-user@ec2-demo1 [michael@MBP] ~ $ ec2-ssh --profile sandpit --send-key -L 5432:some-rds-instance:5432 ec2-user@ec2-demo1 INFO: Resolved instance name 'ec2-demo1' to 'i-08c1950f615da4aa4' INFO: Using SSH key from SSH Agent, should be as good as any. ec2-user@ec2-demo1 ~ $ [michael@MBP] ~ $ ec2-ssh --profile sandpit --send-key -L 5432:some-rds-instance:5432 ec2-user@ec2-demo1 INFO: Resolved instance name 'ec2-demo1' to 'i-08c1950f615da4aa4' INFO: Using SSH key from SSH Agent, should be as good as any. ec2-user@ec2-demo1 ~ $ w 06:41:09 up 12 days, 21:42, 1 user, load average: 0.04, 0.09, 0.08 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT ec2-user pts/0 localhost 06:41 1.00s 0.01s 0.00s w ec2-user@ec2-demo1 ~ $ ec2-ssh Port forwarding example
- 17. [michael@MBP] ~ $ rsync -e "ec2-ssh --send-key" ec2-user@ec2-demo1: [michael@MBP] ~ $ rsync -e "ec2-ssh --send-key" ec2-user@ec2-demo1: INFO: Resolved instance name 'ec2-demo1' to 'i-08c1950f615da4aa4' INFO: Using SSH key from SSH Agent, should be as good as any. drwx------ 181 2022/06/25 22:17:57 . -rw------- 11,455 2022/06/25 22:18:06 .bash_history -rw-r--r-- 18 2020/01/16 13:56:01 .bash_logout -rw-r--r-- 193 2020/01/16 13:56:01 .bash_profile -rw-r--r-- 231 2020/01/16 13:56:01 .bashrc [...] ec2-ssh RSync example
- 18. Login to ECS and Fargate containers Just because you can doesn’t mean you should!
- 19. AWS CLI ECS Exec Find the ECS Cluster name nginx-Cluster-8SqXVHC The Terminator-style CLI (i.e. not for the rest of us) Find the Service name nginx-Service-gL6Cnj Find the Task name 11f37244484b486e9e0eda9b94c Construct the AWS CLI command line: [michael@MBP] ~ $ aws ecs execute-command --cluster nginx-Cluster-8SqXVHC --task 11f37244484b486e9e0eda9b94c --container nginx --command “/bin/bash” --interactive
- 20. ecs-session [michael@MBP] ~ $ ecs-session --list Cluster Group Task Cont. IP nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj a3875d0e...7d488 nginx 172.31.107.214 nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj 11f37244...9b94c nginx 172.31.73.13 nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj fb4099e3...aba46 nginx 172.31.94.23 For the non-Terminators among us (i.e. for me and you) [michael@MBP] ~ $ ecs-session 172.31.73.13 nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj 11f37244...9b94c nginx 172.31.73.13 Starting session with SessionId: ecs-execute-command-09c4f8dcf93299816 root@ip-172-31-73-13:/#
- 21. ECS / Fargate Session - requirements Modern AWS-native way to login What do we need … 1. Service / Task setting in CFN or Terraform: ○ EnableExecuteCommand: true 2. Task Role with AmazonSSMManagedInstanceCore managed policy 3. Network access from the task to SSM Service ○ Public IP, NAT gateway, Proxy, or VPC endpoint ○ No need for inbound access
- 22. Virtual serial port Why? ● Works even without instance connectivity (e.g. firewall misconfiguration, boot problems) ● Boot log messages ● Last resort Why not? ● Requires password-enabled user account Welcome back to the 90’s
- 23. ec2-serial [michael@MBP] ~ $ ec2-serial --region us-east-1 ec2-demo1 Wait a few seconds and then press Enter for a prompt... EC2 console serial port over SSH [michael@MBP] ~ $ ec2-serial --region us-east-1 ec2-demo1 Wait a few seconds and then press Enter for a prompt... Amazon Linux 2 Kernel 5.10.147-133.644.amzn2.x86_64 on an x86_64 ip-172-31-16-189 login:
- 24. ssm-tunnel [michael@MBP] ~ $ ssm-tunnel ec2-demo1 --route 172.31.0.0/16 [sudo] password for mludvig: **** INFO: Local IP: 100.64.156.100 / Remote IP: 100.64.156.101 00:00:34 | In: 472.0 B @ 5.2 B/s | Out: 2.7kB @ 0.0 B/s Your ad-hoc VPC backdoor [michael@MBP] ~ $ ping 172.31.104.217 # ec2-demo2 PING 172.31.104.217 (172.31.104.217) 56(84) bytes of data. 64 bytes from 172.31.104.217: icmp_seq=1 ttl=254 time=90.8 ms 64 bytes from 172.31.104.217: icmp_seq=2 ttl=254 time=89.0 ms 64 bytes from 172.31.104.217: icmp_seq=3 ttl=254 time=91.4 ms ^C --- 172.31.104.217 ping statistics — 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 89.043/90.447/91.413/1.073 ms
- 25. Get in touch! https://www.linkedin.com/in/michaelludvig AWS SSM Tools: https://github.com/mludvig/aws-ssm-tools Blog posts about (not only) AWS SSM Tools: https://michael-ludvig.medium.com
Editor's Notes
- #4: The easiest to set up Must have a Public IP = direct internet access = larger attack vector Restrict access with Security groups
- #9: Agent running + IAM Policy +
- #10: Agent running + IAM Policy +
- #23: Must be enabled in the account Create user in the instance
- #24: There is a way with AWS-CLI + SSH but don’t make me start on how usable is that.