Lets Talk SAN, NAS, Cloud, & Backup: cluster

Learn Storage, Backup, Virtualization, and Cloud. AWS, GCP & AZURE.

..........................................................................................................................................................................

Expand a Cluster

Perform the following steps before adding new nodes to the cluster.

1 Two methods are available:

Use the iris_cli.

a Use the iris_cli vlan add command to set up the non-native VLAN to be used for the node add workflow. Example:

iris_cli vlan add if-name=bond0 id=101 subnet-mask-bits=8

b Use the following command to set the non-native VLAN logical bond interface as primary. Replace vland_id with the ID of the VLAN you added.

iris_cli ip config interface-name=<bond0.vland_id> interface-role=primary

Alternatively, to configure the IP on a new node and access the node using the IP (not required if using Avahi to discover all nodes), use this command:

iris_cli ip config interface-name=<bond0.vlan_id> iface-ips=xx subnet-gateway=yy subnet-mask-bits=zz mtu=qq

2 Or use the configure_network.sh script.

a Use configure_network.sh option 10.

Location: /home/cohesity/bin/network/configure_network.sh

3 Restart the Nexus service:

sudo service nexus restart

4 Run ifconfig and ensure Avahi runs on the non-native VLAN bonded interface.

5 On any node in the existing cluster, start the node add workflow from the UI and provide cluster IPs from the configured non-native VLAN.

NOTE: If necessary, the user can configure cluster IPs and VIPs from the non-native VLAN and keep the IPMI in the native VLAN or some other subnet.

https://docs.cohesity.com/6_1_1/Web/UserGuide/index.htm#CLI/VLANTagging.htm%3FTocPath%3DCluster%2520Administration%7CNetworking%7C_____13

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Remove a Node from Cluster.

This is a clean way. Tricky way could be failing the node and let it reconstruct in the background— given you have right redundancy settings in place, which can be checked under Storage domain Configuration.

Log into Cluster/node

> iris_cli cluster status

(It lists Node ID with IP and Serial Numbers)

> iris_cli node rm -id=<serial number of node>

(It will prompt cluster username (admin) and Password, followed by message—

“Success: Node ID: <Serial Number> marked for removal successfully.”)

Note: There is no way to track removal process using cli. But if you were to logged in to Siren Page, and go to Scribe, it will show you KRemoveNode process and metadata/replica that node holds constantly decreasing. It indicates the Node is being removed. Scribe service track/manage metadata and metadata removal and data removal from owned disks from the node in question runs in parallel. However, metadata finishes quickly. Once Data gets reshuffled across other nodes, by logging into the node and running same commands as above will show a message— Node is not part of cluster, and/or password is reset to default admin password, not the one you have it changed for entire cluster.

You are Welcome :)

Learn Storage, Backup, Virtualization, and Cloud. AWS, GCP & AZURE.

..........................................................................................................................................................................

This is the method used to create a new cluster using IPMI.

This one applies to 6XX models. (If you were to use it for C25xx or 4xxx model, you set value of “3”)

C6xxx uses username: admin, and Password: administrator for IPMI.

Console into the very first Node.

It will take you to black Screen.

[cohesity@node ~]$sh (Type sh and enter)

UserName:cohesity

Password: Cohe$1ty

(This will take you to Cluster shell) You run these below Commands

sudo ipmitool lan print 1

sudo ipmitool lan set 1 ipsrc static

sudo ipmitool lan set 1 ipaddr 10.123.123.20

sudo ipmitool lan set 1 defgw ipaddr 10.123.123.1

sudo ipmitool lan set 1 access on

Now that you have enabled IPMI, you can Use IP address on URL and access KVM remotely.

Once Logged in to KVM.

[cohesity@node ~]$cd bin/network

$ ls

(This will list available Scripts)

Select configure_network.sh Script.

[cohesity@node ~]$./configure_network.sh

(It will list 12 options. Select Option 7 to configure LACP bonding across two 10G ports on Cohesity side. You must have 10G LACP configured the same way on Switch side too).

(LACP config on Switch Side should look like this:

SwitchA

interface Ethernet1/5

description cohesity-node1-ens802f0

switchport mode trunk

switchport trunk allowed vlan 50

switchport trunk native vlan 50

channel-group 101 mode active

mtu 9216

SwitchB:

interface Ethernet1/5

description cohesity-node1-ens802f1

switchport mode trunk

switchport trunk allowed vlan 50

switchport trunk native vlan 50

channel-group 101 mode active

mtu 9216

In an event BMC/IPMI Port becomes inresponsive, Log Into IPMI from another node and run this to reboot.

ipmitool -I lanplus -U admin -P administrator -H 10.123.123.20 mc reset cold

(If a IPMI interface is frozon, then you can use this to reset the IPMI using IPMI from a different node).

Part of ./configure_network.sh uses Node IP. You can ssh into that NODE IP (E.G.10.123.123.40) now.

Once ssh into NODE IP,

[cohesity@node ~]$cat /proc/net/bonding/bond0 (This gives info on what kind of bond config is configured)

It Shows something like this.

[cohesity@node ~]$ cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

Transmit Hash Policy: layer3+4 (1)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

802.3ad info

LACP rate: slow

Min links: 0

Aggregator selection policy (ad_select): stable

…

Slave Interface: ens802f0

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: a4:bf:01:2d:7f:56

Aggregator ID: 3

Actor Churn State: none

Partner Churn State: none

Actor Churned Count: 0

Partner Churned Count: 0

[cohesity@node ~]$avahi-browse -tarp

(This goes out discovering all the Nodes connected in the cluster using IPV6 internal processes). If this doesn’t see any nodes, it needs to be looked at.

At this Stage, you can use Node IP in URL and should be able to discover all the Nodes in discovery to be able to start Creating Cohesity Cluster.

This is Interactive session, you get to assign NODE IP, VIPS, SMTP, DNS, NTP Servers.

At the end of interactive session, it gives a message notifying you that Cluster has been created, and You can use the provided URL using admin user.

https://10.123.123.40

Username: admin

Password: admin

Note: If you want to update gflags, and other things, you may at this point in time.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Validation Steps for Cluster Settings:

1. Now that Cluster Is Up, you can run this at any Node. MII Should show UP on all the nodes you have as part of the cluster.

[cohesity@node ~]$ allssh.sh 'cat /proc/net/bonding/bond0' | grep MII

[01;31m[KMII[m[K Status: up