Cohesity: In Azure, how to Destroy Azure Cluster nodes and repurpose them to add it to running cluster.

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

1. Stop the cluster.

2. Destroy the cluster.

3. Wipe config on freed Nodes.

4. Add Nodes to Cluster.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Code Running 6.3.1g:

[cohesity@shcstypazbk003-000d3a317381-node-1 ~]$ iris_cli cluster stop

[cohesity@shcstypazbk003-000d3a317381-node-1 ~]$ iris_cli cluster status

[cohesity@shcstypazbk003-000d3a317381-node-1 ~]$ iris_cli cluster destroy id=<Cluster_ID>

[cohesity@shcstypazbk003-000d3a317381-node-1 ~]$ iris_cli cluster

[cohesity@ClusterName--node-1 ~]$ ps -ef iris_cli

After cluster Destroy, log into individual Node and run iris_cli node status.

[cohesity@ClusterName--node-1 ~]$iris_cli node status

NODE ID                       : 123456789107

NODE IPS                      : 10.10.9.100, fe80::20d:3aff:fe31:7c6f

NODE IN CLUSTER               : false

CLUSTER ID                    : -1

CLUSTER INCARNATION ID        : -1

SOFTWARE VERSION              : 

LAST UPGRADED TIME            : 

NODE UPTIME                   : 

ACTIVE OPERATION              : 

MESSAGE                       : Node is not part of a cluster.

(If response says, node is not part of cluster, its green to go.)

(ButIf, if node says, its part of cluster, It might need to wipe out data and config manually with prepopulated script)

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

[support@azure-cohesity--node-1 bin]$ pwd

/home/cohesity/bin

[cohesity@azure-cohesity--node-1 bin]$ cd rescue/

[cohesity@azure-cohesity--node-1 rescue]$ ls

breakfix_nvme_ssd.sh  clean_node.sh  defsh.sh  erase_disk.sh  make_bootable_device.sh  reset_linux_users.sh  rollback_upgrade.sh

[cohesity@azure-cohesity--node-1 rescue]$

[cohesity@ClusterName--node-1 rescue]$ ./clean_node.sh

CLEAN NODE IS A DESTRUCTIVE OPERATION.  DO YOU WANT TO PROCEED? (Y/N): y

Cleaning...

[cohesity@ClusterName--node-1 rescue]$ reboot.sh 

RECEIVED REQUEST TO REBOOT NODE. DO YOU WANT TO PROCEED? (Y/N): y

Rebooting...

Connection to 10.249.8.135 closed by remote host.

By Now, Node will be free and not part of Cluster. 

AT this step, you can move to Node Add.

1. Log into the Cluster where you want to join new nodes.

2. iris_cli

3. admin@127.0.0.1> cluster cloud-join node-ips=10.10.9.100,10.10.9.101,10.10.9.102

(These IPs are IPs from recently destroyed Nodes.)

Monitor Nodes add work in Siren and/or GUI.

Cohesity: Basics of System Internals, key concepts, and Services of Cohesity Solution

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

..........................................................................................................................................................................

Problem: It is important to use common terminology while administering a solution, and its even more important to know what each service or terminology used in solution actually does in meeting what is expected of the solution. It helps connect the dots.

What are System Internals of Cohesity and what each Internal Service function is responsible for?

Magneto Service:

Magneto service is the heart of Cohesity solution, which is responsible for Cohesity Integrated Backup Software i.e. DataProtect. Magneto is a backup Engine that integrates Backup service with VMWare vCenter VADP), Oracle, SQL Server, Pure Storage, Netapp, and more.

IRIS Service:

This service provides user interface to interact with Cohesity resources by allowing access to cluster and its services. This facilitates the REST API orchestration. This service is responsible for Web GUI for system management and CLI for advanced Commands.

Gandalf Service:

This is a Cluster Health and State Manager Service. Holds Distributed lock manager and holds Binary Configuration for other processes.

Nexus Service:

This manages platform level operations, while also starts/stops OS level services. Common tasks this performs includes node discovery, cluster initialization, VIP management, and IP/DNS/NTP configuration. This service also manages NON-Disruptive-Upgrades (NDU).

Scribe Service:

This service is responsible for Distributed metadata and journaling. It uses Paxos algorithm for data consistency while holds the metadata for the file systems such as files, snapshots, segments, dedupe chunks, data location and such.

Bridge Service:

This is a main Engine Controller for I/O Operations. This exposes FS types (VIEW- SMB, NFS, and S3) to Clients. This is responsible for updating Scribe of data location, chunking bite streams, and sending to the blob store.

What is Blob?

Blob is a system object that is simply logical allocation of bricks. Bricks is an entity that can be part of which that makes up small files (<=8MB). Whereas, MegaFile(256GB+) is multiple blobs from different nodes. Therefore, here is how the hierarchy would look like:-- Inodes are smallest entity within an inode snaptree that defines file system objects, and hierarchy. Inodes point to Blobs. Blobs are part of bigger entity i.e. Bricks.

Hydra Service:

This is a front-end write cache for bridge. Once data gets written on Cohesity, it is written in native format in Hydra level, where read can be performed directly from hydra if needed. It then flushes to blob store on SSD and HDD depending on I/O profile.

Yoda Service:

Yoda is an Indexing engine, is responsible for indexing files within VM and/or other backed up object based on policy defined if to enable indexing or not. Once Backup job is completed, that’s when Yoda service kicks in.

Genie Service:

This is essentially a service to enable remote tunnel for support. This service checks some other monitoring features such as monitoring hardware hearbeat and health.

Eagle Service:

This service enable data reporting to support (call-home) and support for dark site collection/upload/analysis. It also analyses support data for install base, support and analytics in terms of statistics and alerts. This further sensors that monitor hardware and software.

Note: All the information I have used here is a cumulative information gathering from working knowledge with Field Engineers, Account System Engineers, Cohesity Documentation from support site, as well as personal understanding which I have gathered by conversation with Support tech, and hands on experience. I have tried my best to best reflect accuracy. This is Strictly for the audience who are new and/or current Cohesity users.

You Are Welcome 

Cohesity: How to do hot swap of failed cohesity node...

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.
............................................................................................................................................................................

How to remove a faulty Cohesity node (or hot swap) in cohesity cluster.

1. Log into a node in question by using hostname or Node IP.

ssh cohesity@cohesity-node-1

2. Issue an IPMI command to power down Chassis. It helps identifying the node visually which will have lights turned off upon execution.

cohesity-node-1::> sudo ipmitool power chassis off

(Soon you enter this, it will kill the session from that node)

3. Now remove the Node from cluster. Log into cluster and run iris_cli command.

cohesity-cluster::>iris_cli -username=admin -password=‘xxxxx’ node rm id= force=true

You can only force if the node is unreachable. i.e. since you powered it off in previous step, it let you use force.

4. Now insert the New Node into cluster, power it back on, Assign Initial IP info.  

(Log in using Cohesity user, and under its home directory, there will be three main scripts.)

(It is important to run configure_network.sh  (Located at /home/cohesity/bin/network/configure_network.sh) Script to initially assign IP, gatemasks and such before it can be added to cluster)

a. Select Option to assign networking configuration for Node Management.

b. Configure IP, and other networking info.

Note: If you are using multiple Vlans, and your node management uses non-native vlan then you have to have interface name,bond info handy. I ran into issue when, Even after assigning using script, it required manual intervention.  In some case, it also requires bond interface created, if not created that interface during part of config. So you have to create interfacde manually, modify ip address info, and restart network service, and then only you can move forward.

>>Log into Node via crash cart and run below.

:>>cat /etc/sysconfig/network-scripts/ifcfg-bond0.

DEVICE=bond0.vlanID

USERCTL=no

BOOTPROTO=none

ONBOOT=yes

VLAN=yes

IPADDR=

PREFIX=26

GATEWAY=

MTU=1500

ZONE=public

NM_CONTROLLED=no

4. Upon persistence IP assignment, you can run same  configure_network.sh  and select option to configure IPMI configuration. Apply IPMI network configuration.

5. Now new node in in system, Cluster can discover the newly inserted Node and can be added to cluster. 

(I ran into issue when it didn’t discover from Cluster UI. I did it manually by following below. This actually is independent of what version is new Node running. So in my case when new node was on low rev of DataProtect, upon addition to cluster, its OS also got updated on its own.)

:: ris_cli cluster add-nodes auto-update=true node-ids= node-ips= node-ipmi-ips=

##Just for sanity check, it is helpful if we restart he Nexus service to equally distribute VIPS, since I ran into issue last time when one node failed and second node hosted two VIPS, but didnt’ redistribute VIPS to newly added node upon replacement causing spotty backup failures. So restarting Nexus service helps.

::$ allssh.sh "sudo systemctl restart nexus"

(Rest of the things occurs in the background).

You are Welcome :)

Source: support.cohesity.com

Netapp: How to do perform Disk Erasure, Disk Clearing and Wipe Configuration on CDOT Netapp disks.

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

............................................................................................................................................................................

Depending on what scenario is applicable to you in your environment, there are two ways to run data erasure and disk configuration wipe out activity. By simply pushing the data disk to spare pool after removing it from aggregate may not suffice some of the data erasure function requirement. Disk degaussing by feeding the drive into traditional degauss machine that uses magnetic function to certify degauss might leave some data owners in a questionable position about their comfort before disk disposal. Here are two scenarios that are standard practices which Netapp supports without intervention from third party software or application that runs disks data erasure and wiping configuration.

Scenario 1 Usage: Disk Initialize

If there is flexibility to reboot the node and entire disks data needs to be wiped or configuration reset, then “disk Initialize” is the right option.

Prerequisites.

1. If the disks are part of an aggregate and/or holds volumes, volumes must be taken offline, destroyed followed by taking aggregate offline and aggregate deletion.

2. Disks must be in a spare pool, but can be owned by nodes.

3. Only root aggregate disks should be present.

Actual Action:

Step 1. Boot each node by while accessing from console or SP (if configured) and take it to the loader prompt.

Boot each node to the LOADER/CFE prompt and ensure that the below variables are set. These variables remove the cluster RDBs, CDBs and the varfs from mroot, boot device and nvram.

setenv bootarg.init.boot_clustered true

setenv bootarg.factory_init_completed true

setenv bootarg.init.clearvarfsnvram true

Step 2. Then run boot_ontap from loader prompt and while node reboots, Press CTRL + C to go to special boot menu.

Step 3. Out of 8 special boot menu option, dont select any option yet. Rather Type “wipeconfig” on each node.

Step 4. Then select Option no. 4. that says “Clean configuration and initialize all disks”

(This will prompt if you want to zero disks, reset config and install a new file system. Type “yes”)

Step 5. This will run disk initialize operation in the background which is indicated by dots (…….) fillling the screen till its done. Each and every drives gets initialized and upon completion, it will take you to a prompt where it asks if you want to create or join cluster or new filesystem.

At this time, it is safe to power down Controller head and Disk Shelves. Disk have been reset and data have been completely erased.

Scenario 2 Usage: Disk Sanitization

Data ONTAP 8.0 and earlier, the disk sanitization feature needed a disk sanitization license.

Data ONTAP 8.1 and later, Just need to enable the feature per step 1, under Actual Action below.

If you only few drives in a stack, or just one shelf from a set of stack of shelves, you cannot use scenario 1 based solution for complete data erasure as we are not going to erase data from entire array, but only from selective disks.

Disk sanitization is the process of physically obliterating data by overwriting disks with specified byte patterns or random data so that recovery of the original data becomes impossible. You use the sanitization process to ensure that no one can recover the data on the disks. This functionality is available through the nodeshell.

Pre-Requisites.

1. Disks in question must be in spare pool, but can be owned by nodes.

How Disk sanitization works?

Disk sanitization process uses three successive default or user-specified byte overwrite patterns for up to seven cycles per operation. The random overwrite pattern is repeated for each cycle.

**Sanitization contains two phase:

a. Formatting phase

b. Pattern overwrite phase

**Disk Sanitization Feature is applied at Storage system level, and once it is enabled, it cannot be disabled.

Actual Action:

Step 1. Go to nodeshell from cluster.

node::>options nodescope.reenabledoptions licensed_feature.disk_sanitization.enable

node::>options licensed_feature.disk_sanitization.enable on

Step 2.Start disk sanitize on disk or disklist.

node::> disk sanitize start disk_list

Step 3. Check disk sanitize status

node::> disk sanitize status disk_list

Step 4. After disk sanitization is complete, return the sanitized disk to spare pool, it wont automatically send disk to spare pool.

node::> disk sanitize release disk_name

Step 5. Exit from node shell and go to cluster shell.

node::> CTRL + D

Cluster::>>

Step 6. Verify disk have been properly placed on spare pool.

cluster::> storage disk show -container-type spare

By now, disk is sanitized with no data and is in hot spare pool for it be ready to be used.

**At this time, you can use the degauss machine to crush the drive**

**Some of the Industry Standards on how to run disk sanitization or data erasure procedure**

https://kb.netapp.com/app/answers/answer_view/a_id/1034565/~/how-to-use-disk-sanitize-to-meet-department-of-defense-5220.22m-

https://kb.netapp.com/app/answers/answer_view/a_id/1072424/~/how-to-perform-disk-erasure%2C-disk-clearing-and-disk-sanitization-

You are Welcome :)

Cohesity: What are chunks, Erasure Coding (EC) and Replication Factor (RF) ?

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

............................................................................................................................................................................
If you are in Cohesity domain, you will hear a lot about data resiliency, fault tolerance and distributed workload.  

• All of these whines around smallest unit of data which Cohesity calls Chunk, a form which is written into the disk.

• Chunk combined with Drive and Node level redundancy and resiliency, results into highly available resilient backup solution.

Chunk: The unit of storage data that Cohesity uses for protection. Chunk file can be considered to be a collection of pieces of data from one or more client objects (files, VMs, etc.) packaged together into a single large unit. Cohesity takes a blob of storage, which can be a collection of one or more client objects, divides it into variable-sized, deduplicated chunks, compresses and encrypts them, and puts them in a chunk file. Usually, chunks from the same large client (user) file are combined to belong to the same chunk file. This will happen in most cases when the client file or VM writes are sequential and can be stored together. There may also be several smaller client files that are not large enough to form a single chunk file, in which case chunks from such client files could be packed together to form a chunk file. 

A chunk file could be protected using either EC or RF schemes. Cohesity provides a configurable resiliency on HDDs or node failures. A single, large file could be a part of several different chunk files and will end up getting distributed evenly across all the nodes of the cluster as defined in Cohesity Erasure coding settings

Replication Factor (RF) refers to the number of replicas of a unit of data. The unit of replication is a chunk file, and a chunk file is mirrored into either one or two other nodes depending on the Replication Factor number chosen. An RF2 mechanism provides resilience against a single data unit failure, and a RF3 provides resilience against two data unit failures.

Erasure Coding (EC) refers to a scheme where a number of usable data stripe units can be protected from failures using code stripe units, which are in turn derived from the usable data stripe units. A single code stripe unit can protect against one data (or code) stripe failure, and two code stripe units can protect against two data (or code) stripe unit failures. 

You are Welcome :)

Source:https://info.cohesity.com/Cohesity-Fault-Tolerance-White-Paper.html

Cohesity: Architecture Concept and Terminology...

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

............................................................................................................................................................................
Whats up with Cohesity Architecture ?

• Uses Paxos algorithm for read consistency which is a mechanism to return read request from recently written value especially in a distributed filesystem.

• https://lamport.azurewebsites.net/pubs/paxos-simple.pdf

• Consistent Hashing to spread data across all nodes in a cluster.
• Data distribution using selected Erasure Coding (EC) or Replication Factor (RF) factor.

• Strict consistency : Non Disruptive Upgrades and operational function of non-disruptive service delivery at an event of disk or node failures i.e. strict consistency to support backup, restore, application data consistency and so on.

• SpanFS is an underlying web-scale file system which is a fully distributed filesystem which is where Cohesity Software Defined backup and recovery application “DataProtect” runs. SpanFS is what exposes NFS, SMB, and S3 Interfaces while it also manages the IO operation for all data written to or from the system.

• Distributed Lock Manager, manages concurrent access to the data repository and metadata

• Data Repository stores actual client data, such as network files, VMs, and databases ina. deduplication, compressed, and encrypted form.

• Metadata Store keeps track of all file data sitting across nodes,  Metadata store is based on Distributed Key-Value, that incorporates a fully redundant consistent, distributed NoSQL store for fast IO operations at scale.

• SnapTree is Cohesity’s builtin function that provides unlimited, frequent snapshots which  provides a distributed metadata structure  based on B+ tree concepts.

• Data Journaling: The SpanFS file system constantly looks at incoming requests and tries to estimate the IO pattern. Journal absorbs IOs and acts as write-cache which can be committed to disks later helping making data crash-consistent. It is part of the metadata and is replicated along with the File Metadata Store .

• Distributed Metadata Manager: On each node, the underlying SpanFS file system is used to write to disks. All file data is stored on the Distributed File Data Store. Distributed Metadata Manager maintains all metadata.

Pictorial Depiction Below:

You are Welcome :)

Source:https://info.cohesity.com/Cohesity-Fault-Tolerance-White-Paper.html

Cohesity: How to find running status and Cohesity agent version in multiple OS platform

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.
............................................................................................................................................................................

How to find what cohesity version is running on supported operating system

1. Windows:
c:\Program Files\Cohesity>cohesity_windows_agent_service.exe --version
(Result will show what version release is installed as wellas last commit time)

2. Linux:
/home/cohesityagent/cohesityagent/software/crux/bin/linux_agent_exec --version

/etc/init.d/cohesity-agent status

3. RHEL 5:
/usr/local/cohesity/agent/cohesity_rhel5_linux_agent_exec --version

4. AIX:
/usr/local/cohesity/agent/aix_agent_exec  --version

5. Solaris:
/usr/local/cohesity/agent/solaris_agent_exec --version

You are Welcome :)