Cohesity- How to expand the cluster, and how to remove node from the cluster-- with examples.

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

 ..........................................................................................................................................................................

Expand a Cluster

Perform the following steps before adding new nodes to the cluster.

1 Two methods are available:

Use the iris_cli.

a Use the iris_cli vlan add command to set up the non-native VLAN to be used for the node add workflow. Example:

iris_cli vlan add if-name=bond0 id=101 subnet-mask-bits=8

b Use the following command to set the non-native VLAN logical bond interface as primary. Replace vland_id with the ID of the VLAN you added.

iris_cli ip config interface-name=<bond0.vland_id> interface-role=primary

Alternatively, to configure the IP on a new node and access the node using the IP (not required if using Avahi to discover all nodes), use this command:

iris_cli ip config interface-name=<bond0.vlan_id> iface-ips=xx subnet-gateway=yy subnet-mask-bits=zz mtu=qq

2 Or use the configure_network.sh script.

a Use configure_network.sh option 10.

Location: /home/cohesity/bin/network/configure_network.sh

3 Restart the Nexus service:

sudo service nexus restart

4 Run ifconfig and ensure Avahi runs on the non-native VLAN bonded interface.

5 On any node in the existing cluster, start the node add workflow from the UI and provide cluster IPs from the configured non-native VLAN.

NOTE: If necessary, the user can configure cluster IPs and VIPs from the non-native VLAN and keep the IPMI in the native VLAN or some other subnet.

https://docs.cohesity.com/6_1_1/Web/UserGuide/index.htm#CLI/VLANTagging.htm%3FTocPath%3DCluster%2520Administration%7CNetworking%7C_____13

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Remove a Node from Cluster. 

This is a clean way. Tricky way could be failing the node and let it reconstruct in the background— given you have right redundancy settings in place, which can be checked under Storage domain Configuration.

1. Log into Cluster/node

        > iris_cli cluster status

            (It lists Node ID with IP and Serial Numbers)

        > iris_cli node rm -id=<serial number of node>

            (It will prompt cluster username (admin) and Password, followed by message—

                “Success: Node ID: <Serial Number> marked for removal successfully.”)

Note: There is no way to track removal process using cli. But if you were to logged in to Siren Page, and go to Scribe, it will show you KRemoveNode process and metadata/replica that node holds constantly decreasing. It indicates the Node is being removed. Scribe service track/manage metadata and metadata removal and data removal from owned disks from the node in question runs in parallel. However, metadata finishes quickly. Once Data gets reshuffled across other nodes, by logging into the node and running same commands as above will show a message— Node is not part of cluster, and/or password is reset to default admin password, not the one you have it changed for entire cluster.

You are Welcome :)

Cohesity: Basics of System Internals, key concepts, and Services of Cohesity Solution

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

..........................................................................................................................................................................

Problem: It is important to use common terminology while administering a solution, and its even more important to know what each service or terminology used in solution actually does in meeting what is expected of the solution. It helps connect the dots.

What are System Internals of Cohesity and what each Internal Service function is responsible for?

Magneto Service:

Magneto service is the heart of Cohesity solution, which is responsible for Cohesity Integrated Backup Software i.e. DataProtect. Magneto is a backup Engine that integrates Backup service with VMWare vCenter VADP), Oracle, SQL Server, Pure Storage, Netapp, and more.

IRIS Service:

This service provides user interface to interact with Cohesity resources by allowing access to cluster and its services. This facilitates the REST API orchestration. This service is responsible for Web GUI for system management and CLI for advanced Commands.

Gandalf Service:

This is a Cluster Health and State Manager Service. Holds Distributed lock manager and holds Binary Configuration for other processes.

Nexus Service:

This manages platform level operations, while also starts/stops OS level services. Common tasks this performs includes node discovery, cluster initialization, VIP management, and IP/DNS/NTP configuration. This service also manages NON-Disruptive-Upgrades (NDU).

Scribe Service:

This service is responsible for Distributed metadata and journaling. It uses Paxos algorithm for data consistency while holds the metadata for the file systems such as files, snapshots, segments, dedupe chunks, data location and such.

Bridge Service:

This is a main Engine Controller for I/O Operations. This exposes FS types (VIEW- SMB, NFS, and S3) to Clients. This is responsible for updating Scribe of data location, chunking bite streams, and sending to the blob store.

What is Blob?

Blob is a system object that is simply logical allocation of bricks. Bricks is an entity that can be part of which that makes up small files (<=8MB). Whereas, MegaFile(256GB+) is multiple blobs from different nodes. Therefore, here is how the hierarchy would look like:-- Inodes are smallest entity within an inode snaptree that defines file system objects, and hierarchy. Inodes point to Blobs. Blobs are part of bigger entity i.e. Bricks.

Hydra Service:

This is a front-end write cache for bridge. Once data gets written on Cohesity, it is written in native format in Hydra level, where read can be performed directly from hydra if needed. It then flushes to blob store on SSD and HDD depending on I/O profile.

Yoda Service:

Yoda is an Indexing engine, is responsible for indexing files within VM and/or other backed up object based on policy defined if to enable indexing or not. Once Backup job is completed, that’s when Yoda service kicks in.

Genie Service:

This is essentially a service to enable remote tunnel for support. This service checks some other monitoring features such as monitoring hardware hearbeat and health.

Eagle Service:

This service enable data reporting to support (call-home) and support for dark site collection/upload/analysis. It also analyses support data for install base, support and analytics in terms of statistics and alerts. This further sensors that monitor hardware and software.

Note: All the information I have used here is a cumulative information gathering from working knowledge with Field Engineers, Account System Engineers, Cohesity Documentation from support site, as well as personal understanding which I have gathered by conversation with Support tech, and hands on experience. I have tried my best to best reflect accuracy. This is Strictly for the audience who are new and/or current Cohesity users.

You Are Welcome 