Ankus, bigdata deployment and orchestration framework
1 like632 views
Cloudwick developed Ankus, an open source deployment and orchestration framework for big data technologies. Ankus uses configuration files and a directed acyclic graph (DAG) approach to automate the deployment of Hadoop, HBase, Cassandra, Kafka and other big data frameworks across on-premises and cloud infrastructures. It leverages tools like Puppet, Nagios and Logstash to provision, manage and monitor clusters in an integrated manner. Ankus aims to simplify and accelerate the adoption of big data across organizations.
![Cloudwick
BIG DATA
• Big data is the term for a collection of data sets so large and complex
that it becomes difficult to process using on-hand database
management tools or traditional data processing applications.
[1]
• As of 2012, limits on the size of data sets that are feasible to process in
a reasonable amount of time were on the order of exabytes of data.
• Big data is difficult to work with using most relational database
management systems and desktop statistics and visualization packages,
requiring instead "massively parallel software running on tens, hundreds,
or even thousands of servers".](https://image.slidesharecdn.com/ankus-140625115705-phpapp01/85/Ankus-bigdata-deployment-and-orchestration-framework-3-320.jpg)
Ankus, bigdata deployment and orchestration framework
- 1. Cloudwick ANKUS Deployment and Orchestration Framework for BigData Frameworks Ashrith
- 2. Cloudwick CLOUDWICK • Motto: Empowering the deployment’s and adoption of big-data across organizations • Accelerate enterprise big data people, process and technology transformation • Dedicated resources and team’s to research and develop big-data use-cases
- 3. Cloudwick BIG DATA • Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. [1] • As of 2012, limits on the size of data sets that are feasible to process in a reasonable amount of time were on the order of exabytes of data. • Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers".
- 4. Cloudwick REASONS FOR BIG DATA • Variety - Data today comes in all types of formats. Structured, numeric data in traditional databases. Information created from line-of-business applications. Unstructured text documents, email, video, audio, stock ticker data and financial transactions. Managing, merging and governing different varieties of data is something many organizations still grapple with. • Velocity - Data is streaming in at unprecedented speed and must be dealt with in a timely manner. RFID tags, sensors and smart metering are driving the need to deal with torrents of data in near-real time. Reacting quickly enough to deal with data velocity is a challenge for most organizations. • Volume - Many factors contribute to the increase in data volume.Transaction-based data stored through the years. Unstructured data streaming in from social media. Increasing amounts of sensor and machine-to-machine data being collected. In the past, excessive data volume was a storage issue. But with decreasing storage costs, other issues emerge, including how to determine relevance within large data volumes and how to use analytics to create value from relevant data.
- 5. Cloudwick BIG DATA STATS • As of 2012, about 2.5 exabytes of data are created each day, and that number is doubling every 40 months or so. • It is estimated that Walmart collects more than 2.5 petabytes of data every hour from its customer transactions.A petabyte is one quadrillion bytes, or the equivalent of about 20 million filing cabinets’ worth of text.An exabyte is 1,000 times that amount, or one billion gigabytes.
- 6. Cloudwick TOOLS & FRAMEWORKS • Mentioned below are some of the big data management frameworks and tools • Data Platforms - where data is stored and processed in huge volumes • Distributed file system - Hadoop HDFS • Distributed processing engine - Hadoop MapReduce/YARN • Distributed real-time computation - Storm • In-memory cluster computing - Spark
- 7. Cloudwick TOOLS & FRAMEWORKS • NoSQL Databases/Data Warehouses - NoSQL (Not only SQL) database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. • Column Family Stores - Cassandra, Hbase,Accumulo • Document databases - CouchDB, MongoDB, MarkLogic • Key-value stores - Redis,Voldemort, Oracle BDB, Riak,Amazon SimpleDB,Tokyo Cabinet • Graph databases - Neo4j, OrientDB, Infinite Graph,Allegro,Virtuoso
- 8. Cloudwick TOOLS & FRAMEWORKS • BigData Search - Solr, ElasticSearch • Data Aggregation - Sqoop, Flume, Chukwa, LogStash • Distributed Messaging Queues - Kafka, RabbiMQ • Distributed coordination services - Zookeeper • Authorization & Authentication - Kerberos, LDAP • Encryption & Masking - Gazzang & DataGuise
- 9. Cloudwick WHY DEPLOYMENT & MANAGEMENTTOOL • Deploying & managing so many complex frameworks and tools could be a lot complex • Making all those frameworks & tools working together is complicated • Ankus makes it easy to provision, manage and monitor many of the big-data frameworks
- 10. Cloudwick ANKUS • Cloudwick project to accelerate big-data discovery and testing • Ankus is a deployment and orchestration tool for managing big-data frameworks • Definition (noun):An elephant goad with a sharp spike and a hook that is used to prod an elephant into motion.
- 11. Cloudwick ANKUS FEATURES • Supports deployments across on-premises and on-cloud • Cloud infrastructure supported: • AWS • Rackspace • OpenStack • Operating System compatible - Redhat, Debian • Multiple big-data frameworks compatible
- 12. Cloudwick ANKUS FEATURES (MORE…) • Supported Big-Data Frameworks • Hadoop • HBase • Hadoop EcoSystem - Hive, Pig, Oozie, Sqoop, Hue • Cassandra • Kafka • Storm • Solr
- 13. Cloudwick ANKUS FEATURES (MORE…) • Supported Deployment Modes: • Highly-available clusters • Secure clusters • Integrated monitoring • Integrated alerting • Integrated log-aggregation
- 16. Cloudwick CLOUD ENGINE • Ankus has an powerful cloud manager which natively communicates with many of the major cloud providers • Pluggable cloud providers • Manages instances & volumes across various cloud providers • Configuration based resources management • Automatically takes care of the creating volumes and attaching them to the instances
- 17. Cloudwick ORCHESTRATION ENGINE • Ankus orchestrates deployments by designing DAG’s which embeds steps to take to achieve the desired state • Nodes being hosts and their states • Edge being how nodes depend on each other to achieve the complete state of the system • Ankus leverages `net-*` and `puppet` gems to achieve a state of the system
- 18. Cloudwick DEPLOY ENGINE • Ankus uses the DAG’s built by orchestration engine and deploy components • The power behind the deploy engine is puppet modules • Ankus has wrapper around to make it protocol agnostic • `net/ssh` • `net/scp` • `net/http` • `puppet`
- 19. Cloudwick METADATA MANAGEMENT • Ankus manages metadata of every node to consistently manage the state on each node • The metadata could be store on regular files usingYAML/JSON format or on RDBMS • Metadata includes • nodes information (cpu, cores, ram, etc..) • puppet status (last_run, install, etc..) • services statuses
- 20. Cloudwick OPEN SOURCE POWER • We at cloudwick love open-source and don’t want to reinvent the wheel rather use as many existing components as possible • Ankus leverages the following open-source projects: • Puppet, MCollective, PuppetDB, Passenger • Nagios, NRPE • Ganglia & JMXTrans • LogStash, Lumberjack, Redis & ElasticSearch
- 22. Cloudwick SCREENSHOTS - CLI Fig: Command line interface of Ankus
- 23. Cloudwick SCREENSHOTS - CONFIG Fig: Sample configuration for deploying hadoop cluster in AWS
- 24. Cloudwick SCREENSHOTS - DEPLOY Fig: Ankus deployment information
- 25. Cloudwick SCREENSHOTS - INFO Fig:Ankus information overview of the cluster
- 26. Cloudwick SCREENSHOTS - DESTROY Fig: Destroy the cluster in the cloud (AWS)
- 27. Cloudwick PROJECT RESOURCES • HomePage: http://cloudwicklabs.github.io/ankus/ • Git Repository: https://github.com/ashrithr/ankus • IssueTracker: https://github.com/ashrithr/ankus/ issues