2014/7/6 The Nutanix Bible | StevenPoitras.com StevenPoitras.com
http://stevenpoitras.com/the-nutanix-bible/ 1/46
StevenPoitras.com
The Nutanix Bible
1. Intro
2. Book of Nutanix
Architecture
Converged Platform
Cluster Components
Data Structure
I/O Path Overview
How It Works
Data Protection
Data Locality
Scalable Metadata
Shadow Clones
Elastic Dedupe Engine
Networking and I/O
CVM Autopathing
Disk Balancing
Software-Defined Controller Architecture
Search
HOME THE NUTANIX BIBLE ASKSTEVE ABOUT
2014/7/6 The Nutanix Bible | StevenPoitras.com StevenPoitras.com
http://stevenpoitras.com/the-nutanix-bible/ 2/46
Storage Tiering and Prioritization
Storage Layers and Monitoring
APIs & Interfaces
Availability Domains
Snapshots & Clones - new!
Multi-Site Disaster Recovery - new!
Administration
Important Pages
Cluster Commands
NCLI
Metrics & Thresholds
Gflags
Troubleshooting
3. Book of vSphere
Architecture
How It Works
Array Offloads – VAAI
Administration
Important Pages
Command Reference
Metrics & Thresholds
Troubleshooting
4. Book of Hyper-V
Architecture
How It Works
Array Offloads – ODX
Administration
Important Pages
Command Reference
Metrics & Thresholds
Troubleshooting
5. Revisions
2014/7/6 The Nutanix Bible | StevenPoitras.com StevenPoitras.com
http://stevenpoitras.com/the-nutanix-bible/ 3/46
Intro
Welcome to The Nutanix Bible! I work the with Nutanix platform on a daily basis – trying to find issues, push its limits
as well as administer it for my production benchmarking lab. This page is being produced to serve as a living document
outlining tips and tricks used every day by myself and a variety of engineers at Nutanix. This will also include
summary items discussed as part of the Advanced Nutanix series. NOTE: This is not an official reference so tread at
your own risk!
Book of Nutanix
Architecture
Converged Platform
The Nutanix solution is a converged storage + compute solution which leverages local components and creates a
distributed platform for virtualization aka virtual computing platform. The solution is a bundled hardware + software
appliance which houses 2 (6000/7000 series) or 4 nodes (1000/2000/3000/3050 series) in a 2U footprint.
Each node runs an industry standard hypervisor (ESXi, KVM, Hyper-V currently) and the Nutanix Controller VM (CVM).
The Nutanix CVM is what runs the Nutanix software and serves all of the I/O operations for the hypervisor and all VMs
running on that host. For the Nutanix units running VMware vSphere, the SCSI controller, which manages the SSD and
HDD devices, is directly passed to the CVM leveraging VM-Direct Path (Intel VT-d). In the case of Hyper-V the storage
devices are passed through to the CVM.
Below is an example of what a typical node logically looks like:
Together, a group of Nutanix Nodes forms a distributed platform called the Nutanix Distributed Filesystem (NDFS).
NDFS appears to the hypervisor like any centralized storage array, however all of the I/Os are handled locally to provide
the highest performance. More detail on how these nodes form a distributed system can be found below.
Below is an example of how these Nutanix nodes form NDFS:
2014/7/6 The Nutanix Bible | StevenPoitras.com StevenPoitras.com
http://stevenpoitras.com/the-nutanix-bible/ 4/46
top
Cluster Components
The Nutanix platform is composed of the following high-level components:
Cassandra
Key Role: Distributed metadata store
Description: Cassandra stores and manages all of the cluster metadata in a distributed ring like manner based
upon a heavily modified Apache Cassandra. The Paxos algorithm is utilized to enforce strict consistency. This
service runs on every node in the cluster. Cassandra is accessed via an interface called Medusa.
Zookeeper
Key Role: Cluster configuration manager
Description: Zeus stores all of the cluster configuration including hosts, IPs, state, etc. and is based upon Apache
Zookeeper. This service runs on three nodes in the cluster, one of which is elected as a leader. The leader receives
all requests and forwards them to the peers. If the leader fails to respond a new leader is automatically elected.
Zookeeper is accessed via an interface called Zeus.
Stargate
2014/7/6 The Nutanix Bible | StevenPoitras.com StevenPoitras.com
http://stevenpoitras.com/the-nutanix-bible/ 5/46
Key Role: Data I/O manager
Description: Stargate is responsible for all data management and I/O operations and is the main interface from
the hypervisor (via NFS, iSCSI or SMB). This service runs on every node in the cluster in order to serve localized I/O.
Curator
Key Role: Map reduce cluster management and cleanup
Description: Curator is responsible for managing and distributing tasks throughout the cluster including disk
balancing, proactive scrubbing, and many more items. Curator runs on every node and is controlled by an elected
Curator Master who is responsible for the task and job delegation.
Prism
Key Role: UI and API
Description: Prism is the management gateway for component and administrators to configure and monitor the
Nutanix cluster. This includes Ncli, the HTML5 UI and REST API. Prism runs on every node in the cluster and uses
an elected leader like all components in the cluster.
Genesis
Key Role: Cluster component & service manager
Description: Genesis is a process which runs on each node and is responsible for any services interactions
(start/stop/etc.) as well as for the initial configuration. Genesis is a process which runs independently of the
cluster and does not require the cluster to be configured/running. The only requirement for genesis to be running
is that Zookeeper is up and running. The cluster_init and cluster_status pages are displayed by the genesis
process.
Chronos
Key Role: Job and Task scheduler
Description: Chronos is responsible for taking the jobs and tasks resulting from a Curator scan and
scheduling/throttling tasks among nodes. Chronos runs on every node and is controlled by an elected Chronos
Master who is responsible for the task and job delegation and runs on the same node as the Curator Master.
Cerebro
Key Role: Replication/DR manager
Description: Cerebro is responsible for the replication and DR capabilities of NDFS. This includes the scheduling
of snapshots, the replication to remote sites, and the site migration/failover. Cerebro runs on every node in the
Nutanix cluster and all nodes participate in replication to remote clusters/sites.
Pithos
Key Role: vDisk configuration manager
Description: Pithos is responsible for vDisk (NDFS file) configuration data. Pithos runs on every node and is built
on top of Cassandra.
top
评论0