Cohesity- How to expand the cluster, and how to remove node from the cluster-- with examples.

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

 ..........................................................................................................................................................................

Expand a Cluster

Perform the following steps before adding new nodes to the cluster.

1 Two methods are available:

Use the iris_cli.

a Use the iris_cli vlan add command to set up the non-native VLAN to be used for the node add workflow. Example:

iris_cli vlan add if-name=bond0 id=101 subnet-mask-bits=8

b Use the following command to set the non-native VLAN logical bond interface as primary. Replace vland_id with the ID of the VLAN you added.

iris_cli ip config interface-name=<bond0.vland_id> interface-role=primary

Alternatively, to configure the IP on a new node and access the node using the IP (not required if using Avahi to discover all nodes), use this command:

iris_cli ip config interface-name=<bond0.vlan_id> iface-ips=xx subnet-gateway=yy subnet-mask-bits=zz mtu=qq

2 Or use the configure_network.sh script.

a Use configure_network.sh option 10.

Location: /home/cohesity/bin/network/configure_network.sh

3 Restart the Nexus service:

sudo service nexus restart

4 Run ifconfig and ensure Avahi runs on the non-native VLAN bonded interface.

5 On any node in the existing cluster, start the node add workflow from the UI and provide cluster IPs from the configured non-native VLAN.

NOTE: If necessary, the user can configure cluster IPs and VIPs from the non-native VLAN and keep the IPMI in the native VLAN or some other subnet.

https://docs.cohesity.com/6_1_1/Web/UserGuide/index.htm#CLI/VLANTagging.htm%3FTocPath%3DCluster%2520Administration%7CNetworking%7C_____13

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Remove a Node from Cluster. 

This is a clean way. Tricky way could be failing the node and let it reconstruct in the background— given you have right redundancy settings in place, which can be checked under Storage domain Configuration.

1. Log into Cluster/node

        > iris_cli cluster status

            (It lists Node ID with IP and Serial Numbers)

        > iris_cli node rm -id=<serial number of node>

            (It will prompt cluster username (admin) and Password, followed by message—

                “Success: Node ID: <Serial Number> marked for removal successfully.”)

Note: There is no way to track removal process using cli. But if you were to logged in to Siren Page, and go to Scribe, it will show you KRemoveNode process and metadata/replica that node holds constantly decreasing. It indicates the Node is being removed. Scribe service track/manage metadata and metadata removal and data removal from owned disks from the node in question runs in parallel. However, metadata finishes quickly. Once Data gets reshuffled across other nodes, by logging into the node and running same commands as above will show a message— Node is not part of cluster, and/or password is reset to default admin password, not the one you have it changed for entire cluster.

You are Welcome :)

Cohesity: How to create a new Cohesity Cluster--with Examples

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

.......................................................................................................................................................................... 

This is the method used to create a new cluster using IPMI.

This one applies to 6XX models. (If you were to use it for C25xx or 4xxx model, you set value of “3”)

C6xxx uses username: admin, and Password: administrator for IPMI.

1. Console into the very first Node.

It will take you to black Screen.

[cohesity@node ~]$sh (Type sh and enter)

UserName:cohesity

Password: Cohe$1ty

(This will take you to Cluster shell) You run these below Commands

sudo ipmitool lan print 1

sudo ipmitool lan set 1 ipsrc static

sudo ipmitool lan set 1 ipaddr 10.123.123.20

sudo ipmitool lan set 1 defgw ipaddr 10.123.123.1

sudo ipmitool lan set 1 access on

2. Now that you have enabled IPMI, you can Use IP address on URL and access KVM remotely.

Once Logged in to KVM.

[cohesity@node ~]$cd bin/network

$ ls

(This will list available Scripts)

3. Select configure_network.sh Script.

[cohesity@node ~]$./configure_network.sh

(It will list 12 options. Select Option 7 to configure LACP bonding across two 10G ports on Cohesity side. You must have 10G LACP configured the same way on Switch side too).

(LACP config on Switch Side should look like this:

SwitchA

interface Ethernet1/5

description  cohesity-node1-ens802f0

switchport mode trunk

switchport trunk allowed vlan 50

switchport trunk native vlan 50

channel-group 101 mode active

mtu 9216

SwitchB:

interface Ethernet1/5

description cohesity-node1-ens802f1

switchport mode trunk

switchport trunk allowed vlan 50

switchport trunk native vlan 50

channel-group 101 mode active

mtu 9216

4. In an event BMC/IPMI Port becomes inresponsive, Log Into IPMI from another node and run this to reboot.

ipmitool -I lanplus -U admin -P administrator -H  10.123.123.20 mc reset cold

(If a IPMI interface is frozon, then you can use this to reset the IPMI using IPMI from a different node).

5. Part of ./configure_network.sh uses Node IP. You can ssh into that NODE IP (E.G.10.123.123.40)  now.

6. Once ssh into NODE IP,

[cohesity@node ~]$cat /proc/net/bonding/bond0 (This gives info on what kind of bond config is configured)

It Shows something like this.

[cohesity@node ~]$ cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

Transmit Hash Policy: layer3+4 (1)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

802.3ad info

LACP rate: slow

Min links: 0

Aggregator selection policy (ad_select): stable

…

Slave Interface: ens802f0

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: a4:bf:01:2d:7f:56

Aggregator ID: 3

Actor Churn State: none

Partner Churn State: none

Actor Churned Count: 0

Partner Churned Count: 0

7. [cohesity@node ~]$avahi-browse -tarp

(This goes out discovering all the Nodes connected in the cluster using IPV6 internal processes). If this doesn’t see any nodes, it needs to be looked at.

8. At this Stage, you can use Node IP in URL and should be able to discover all the Nodes in discovery to be able to start Creating Cohesity Cluster.

This is Interactive session, you get to assign NODE IP, VIPS, SMTP, DNS, NTP Servers.

At the end of interactive session, it gives a message notifying you that Cluster has been created, and You can use the provided URL using admin user.

https://10.123.123.40

Username: admin

Password: admin

Note: If you want to update gflags, and other things, you may at this point in time.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Validation Steps for Cluster Settings:

1. Now that Cluster Is Up, you can run this at any Node. MII Should show UP on all the nodes you have as part of the cluster.

[cohesity@node ~]$ allssh.sh 'cat /proc/net/bonding/bond0' | grep MII

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Polling Interval (ms): 100

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Polling Interval (ms): 100

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Polling Interval (ms): 100

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Polling Interval (ms): 100

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Polling Interval (ms): 100

[01;31m[KMII[m[K Status: up

[01;31m[KMII[m[K Status: up

2. [cohesity@node ~]$ allssh.sh 'cat /proc/net/bonding/bond0' | grep Mode

(This should list link aggregation mode. Mode 4 i.e. LACP is dynamic link aggregation mode)

Bonding [01;31m[KMode[m[K: IEEE 802.3ad Dynamic link aggregation

3. [cohesity@node ~]$ iris_cli node status
4. [cohesity@node ~]$ iris_cli cluster status
5. [cohesity@node ~]$ allssh.sh hostips

(This will list all nodes iPs in the cluster)

6. [cohesity@node ~]$ less logs/iris_proxy_exec.FATAL (lists any fatals related to iris service)

You are Welcome :)

Cohesity: Basics of System Internals, key concepts, and Services of Cohesity Solution

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

..........................................................................................................................................................................

Problem: It is important to use common terminology while administering a solution, and its even more important to know what each service or terminology used in solution actually does in meeting what is expected of the solution. It helps connect the dots.

What are System Internals of Cohesity and what each Internal Service function is responsible for?

Magneto Service:

Magneto service is the heart of Cohesity solution, which is responsible for Cohesity Integrated Backup Software i.e. DataProtect. Magneto is a backup Engine that integrates Backup service with VMWare vCenter VADP), Oracle, SQL Server, Pure Storage, Netapp, and more.

IRIS Service:

This service provides user interface to interact with Cohesity resources by allowing access to cluster and its services. This facilitates the REST API orchestration. This service is responsible for Web GUI for system management and CLI for advanced Commands.

Gandalf Service:

This is a Cluster Health and State Manager Service. Holds Distributed lock manager and holds Binary Configuration for other processes.

Nexus Service:

This manages platform level operations, while also starts/stops OS level services. Common tasks this performs includes node discovery, cluster initialization, VIP management, and IP/DNS/NTP configuration. This service also manages NON-Disruptive-Upgrades (NDU).

Scribe Service:

This service is responsible for Distributed metadata and journaling. It uses Paxos algorithm for data consistency while holds the metadata for the file systems such as files, snapshots, segments, dedupe chunks, data location and such.

Bridge Service:

This is a main Engine Controller for I/O Operations. This exposes FS types (VIEW- SMB, NFS, and S3) to Clients. This is responsible for updating Scribe of data location, chunking bite streams, and sending to the blob store.

What is Blob?

Blob is a system object that is simply logical allocation of bricks. Bricks is an entity that can be part of which that makes up small files (<=8MB). Whereas, MegaFile(256GB+) is multiple blobs from different nodes. Therefore, here is how the hierarchy would look like:-- Inodes are smallest entity within an inode snaptree that defines file system objects, and hierarchy. Inodes point to Blobs. Blobs are part of bigger entity i.e. Bricks.

Hydra Service:

This is a front-end write cache for bridge. Once data gets written on Cohesity, it is written in native format in Hydra level, where read can be performed directly from hydra if needed. It then flushes to blob store on SSD and HDD depending on I/O profile.

Yoda Service:

Yoda is an Indexing engine, is responsible for indexing files within VM and/or other backed up object based on policy defined if to enable indexing or not. Once Backup job is completed, that’s when Yoda service kicks in.

Genie Service:

This is essentially a service to enable remote tunnel for support. This service checks some other monitoring features such as monitoring hardware hearbeat and health.

Eagle Service:

This service enable data reporting to support (call-home) and support for dark site collection/upload/analysis. It also analyses support data for install base, support and analytics in terms of statistics and alerts. This further sensors that monitor hardware and software.

Note: All the information I have used here is a cumulative information gathering from working knowledge with Field Engineers, Account System Engineers, Cohesity Documentation from support site, as well as personal understanding which I have gathered by conversation with Support tech, and hands on experience. I have tried my best to best reflect accuracy. This is Strictly for the audience who are new and/or current Cohesity users.

You Are Welcome 