Showing posts with label nfs. Show all posts
Showing posts with label nfs. Show all posts

Wednesday, February 12, 2020

VMWare: Restore a VM using Netapp snapshot in an NFS exported Datastore Environment

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.
............................................................................................................................................................................
Problem: Restore a VM that resides on a NFS Exported Datastore using Netapp Native Snapshot, NOT VMWare snapshot. Use case is-- A virtual disk on Windows server cannot be expanded, if that VM has snapshot on it. This method adds one more way of restoring a VM, if you don’t have solid backup in place, and/or it can be used as secondary plan for restoring a VM, should primary backup becomes unavailable.
1.    Find which ESXi host this VM resides on
a. This can be done either through vcenter GUI or through the vmware scripts
b.   Look at Each Disk of the VM,  EVERY virtual disk MUST be on the NFS to be recoverable.  
c.   In our example,  the disks are all Netapp volumes (netappdatastore2)
2.     Shutdown/power down the VM 
3.     Unregister the VM from Vcenter (Remove from Inventory) (DONT DELETE FROM DISK)
4.     Login to the ESXi host as user root,  in this case,  ESX-Server-01.domain.com
a.   ssh -l root@ ESX-Server-01.domain.com
b.   Find the netapp volume once you log in.
c.   [root@ESX-Server-01:~] df -h
Filesystem  Size  Used      Available Used%      Mounted on
NFS         20TT  15TT      5.0T      81%        /vmfs/volumes/netappdatastore2
5.    cd into that file system
[root@ESX-Server-01: cd /vmfs/volumes/netappdatastore2
   [root@ESX-Server-01 pwd
/vmfs/volumes/netappdatastore2
6.    Make sure we have a .snapshot directory and it has snapshots for the VM
7.   cd into .snapshot directory, and locate the snapshot desired based on timestamp.
root@ESX-Server-01:/vmfs/volumes/netappdatastore2] cd .snapshot/
[root@ESX-Server-01:/vmfs/volumes/netappdatastore2/.snapshot] ls -l
Snapshot.1
Snapshot.2
Snapshot.3
8.   Find the existing vm directory and move that directory into a backup directory to save the existing state
[root@ESX-Server-01:/vmfs/volumes/netappdatastore2] mv VMname VMname.backup
9.   Recover the VM directory from snapshot directory,  you MUST choose the right snapshot directory by the time frame
Cd into desired snapshot.
[root@ESX-Server-01:/vmfs/volumes/netappdatastore2/.snapshot/Snapshot.3]
[root@ESX-Server-01:/vmfs/volumes/netappdatastore2/.snapshot/snapshot.3] ls -ld VMName
10. Rsync command is recommended vs cp  (rsync has the -S option to handle sparse files more efficiently,  thin allocated vmdk files are usually sparse files - that are files with holes in the blocks).
root@ESX-Server-01:/vmfs/volumes/netappdatastore2/.snapshot/snapshot.3] rsync -av -S VMName ../../VMName
11. Re-register the vm in vcenter with the newly recovered directory
a.   Login to vcener
b.   Brower the ESXi host the VM was running on
c.   Go to Storage for the ESXi host and browse the storage file systems on the ESXi host
d.   Browse to the directory where you have the recovered VM files. 
e.   Go into that directory and register the VM. (right click on the VM definition file, and choose register). 
12.  Power back on the VM.


 You are Welcome :)

Wednesday, December 18, 2019

Netapp: xcp tool for high file count volumes copy

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.
............................................................................................................................................................................


Troubleshooting Workflow: NetApp XCP Migration Tool errorsDocument ID BR519
Answer ID 1005557
Published Date 12/12/2019
Symptom
Symptom 1
-bash: ./xcp: Permission denied
Symptom 2
xcp: ERROR: This license has expired
Symptom 3
xcp: This copy is not licensed.
Symptom 4
xcp: ERROR: XCP not activated, run 'activate' first
Symptom 5
xcp: ERROR: License file /opt/NetApp/xFiles/xcp/license not found.
Symptom 6
xcp: ERROR: Failed to activate license: Server unreachable
Symptom 7
xcp usage error: show: requires at least one server
Symptom 8
xcp: ERROR: show '10.61.73.94:/vol/nfsvol1': invalid hostname
Symptom 9
xcp usage error: scan: missing source path
Symptom 10
xcp usage error: scan: invalid path '10.61.73.94'
Symptom 11
xcp: ERROR: Catalog inaccessible: Cannot mount nfs_server:/export[:subdirectory]
Symptom 12
xcp: compare1 'n.txt': ERROR: nfs3 LOOKUP 'n.txt' in '10.61.73.113:/target': nfs3 error 2: no such file or directory
Symptom 13
xcp: ERROR: Failed to open catalog id '3': nfs3 LOOKUP '3' in '10.61.73.94:/vol/nfsvol1/catalog/indexes': nfs3 error 2: no such file or directory
Symptom 14
xcp: ERROR: Empty or invalid index '10.61.73.94:/vol/nfsvol1/catalog/indexes/2/2.index'
Symptom 15
xcp: copying 'file3': WARNING: 10.193.67.237:/ntfs/topdir_3/subdir_52/file3: nfs3 SETATTR '10.193.67.237:/ntfs/topdir_3/subdir_52/file3' mode 00777, uid 0, gid 0, size 27, atime Thu Feb 23 10:15:00 2017, mtime Tue Feb  7 11:51:54 2017: nfs3 error 1: not owner
Symptom 16
Sync fails in different areas due to following warning:
xcp: mount 10.61.73.94:/vol/nfsvol1: WARNING: This NFS server only supports 1-second timestamp granularity. This may cause sync to fail because changes will often be undetectable.
Symptom 17
XCP fails to transfer ACLs and produces the following ERROR:
ERROR failed to obtain fallback security principal "". Please check if the principal with the name "" exists on "".
Cause
The cause and procedure to be performed to resolve these issues are described in the Solution section below.

Cause #1: Improper file permissions for XCP binary.
Solution: Modify permissions by running the  "chmod 755" command. 
Ensure XCP binary can be executed by the 
root user or sudo
command on designated XCP Linux client host.

Cause #2: This error occurs if the free 90 days XCP evaluation license is expired.
Solution: Renew or obtain the new XCP license from https://xcp.netapp.com

Cause #3: XCP is not licensed.
Solution: Download and apply XCP licenses on Linux client host system.
Cause #4: XCP license is not activated
Solution: Obtain the appropriate XCP license file. Copy the XCP license to the 
/opt/NetApp/xFiles/xcp/ directory on the XCP Linux client host. Run  "xcp activate" command to activate the license.
Cause #5: XCP license is not activated.
Solution: Download the XCP license from https://xcp.netapp.com. Copy XCP license file on to the XCP Linux client host at 
/opt/NetApp/xFiles/xcp/
After copying, run "xcp activate" command to activate the XCP license.
Cause #6: License file not found at 
/opt/NetApp/xFiles/xcp/license
Solution: Register for XCP license at https://xcp.netapp.com. Download and copy XCP license file to 
/opt/NetApp/xFiles/xcp/ directory on XCP Linux client host.
Cause #7: Incomplete parameter specified for the 
xcp show command.
Solution: Re-run the command with the server name or IP address specified. Correct syntax 
./xcp show abc.nfsserver.com
Cause #8: Valid hostname was not specified for the 
./xcp show
 command.
Solution: Run the 
./xcp show
 command with valid hostname. Exact command syntax 
./xcp show 10.61.73.94
 or 
./xcp show localhost
Cause #9: Incomplete parameters specified for the 
xcp scan
 command.
Solution: To resolve this error, run the 
xcp scan
 command with the complete source NFSv3 export path. Correct syntax 
xcp scan
Cause #10: Incomplete parameter specified for the 
xcp scan command.
Solution: Run the same command again by specifying the complete NFSv3 source export path. Correct syntax 
xcp scan .
Cause #11: Catalog path is not specified in the
 xcp.ini
 configuration file.
Solution: Open a text editor on XCP Linux client host and update XCP configuration file with proper catalog location. XCP config file is located at 
/opt/NetApp/xFiles/xcp/xcp.ini
Sample entries of config file:
[root@localhost linux]# cat /opt/NetApp/xFiles/xcp/xcp.ini
# Sample xcp config:
[xcp]
catalog = 10.61.73.94:/vol/nfsvol1

Cause #12: Verify operation did not find the source file(s) on target NFS export.
Solution: Run the 
xcp sync
 command to copy the incremental updates from the source to destination.

Cause #13: XCP could not locate the specified catalog index number.
Solution: Verify the index number of the previous operation. To determine the exact catalog path, run the 
cat /opt/NetApp/xFiles/xcp/xcp.ini
 command. Catalog indexes are located in the 
:/catalog/indexes
 directory. After locating the index number, run the same command again by specifying the correct index number or name.
Cause #14: Previous copy operation was interrupted before indexing the files.
Solution: Run the 
xcp copy
 operation again by specifying the 
–newid
 option.
Interrupt the copy operation when you see the message '
indexed' 
on the console session
:
88,126 scanned, 42,058 copied, 41,469 indexed, 150 MiB in (3.60 MiB/s), 41.9 MiB out (2.88 MiB/s), 16s
Cause #15: This happens on NTFS security style volumes because XCP attempts to chown a file and cannot, as UNIX operations will fail on NTFS security style volume by default.
Solution:  Run the following command:
# chown root newfile
chown: changing ownership of ‘newfile’: Operation not permitted

In a packet trace, the following is displayed:
3380 7.642295   10.193.67.237   10.193.67.233   NFS  214  V3 SETATTR Reply (Call In 3338) Error: NFS3ERR_PERM
The error is benign, as the owner would be set from a Windows client after the fact anyway.
The following are the two ways to avoid this:
  • Change the export policy rule to ignore with the following command: export-policy rule modify -vserver DEMO -policyname default -ruleindex 1 -ntfs-unix-security-ops ignore
  • Copy to a mixed security style or UNIX security style volume. After the copy, change the security style to NTFS.
A third possible way would be for XCP not to attempt to run setattr after copying. Unable to identify a way to perform so in the current version for the copy command. Perhaps that option could go into a future release.
  • copy: Recursively copy everything from source to target
  • newid : Catalog name for a new index
  • md5: Checksum the files (also save the checksums when indexing) (default: False)
  • edupe: Include dedupe estimate in reports (see documentation for details)
  • nonames: Do not look up user and group names for file listings or reports
  • bs : read/write blocksize (default: 64k)
  • dircount : Request size for reading directories (default: 64k)
  • parallel : Maximum concurrent batch processes (default: 7)
  • noId: Disables the creation of a default index (default: False)
Cause #16: Have seen this when source volume is SLES or RHEL with ext3 file system
Solution: Issue is documented here: 1181841 XCP Sync: Timestamp granularity
  • Workaround: We recommend to have access time enabled, because xcp compares the timestamp of the files from the source and index (contains all the file metadata found on the source). And in this use-case we may have differences < 1s and xcp will not apply the changes.
  • This is only a warning, and could be ignored.
Cause #17: This may happen when:
  • The target netbios name/machine account cannot be resolved to an IP
  • The target machine cannot resolve the fallback-user or fallback-group to a SID
Solution:
  • For cause 1, add a mapping in the hosts file (c:\windows\system32\drivers\etc\hosts) for the netbios name/machine account name of the target.
  • For cause 2, select a fallback-user/group that can be resolved by the target system. This can either be a local user to the target machine, or a domain user resolvable by the target machine.
Related Links:
  • 1098120: NetApp XCP Frequently Asked Questions and Resources
  • 1015592: Triage Template - How to troubleshoot NetApp XCP Migration Tool issues
  • 1015613: How to transition 7-Mode NFSv3 exports to clustered Data ONTAP using NetApp XCP Migration Tool 1.0
XCP 1.5
XCP Best Practices TR-4808
You are Welcome :)

Wednesday, November 13, 2019

Netapp: Flexgroup volume vs. flex volume and inodes limit

Learn Storage, Backup, Virtualization,  and Cloud. AWS, GCP & AZURE.

............................................................................................................................................................................

Netapp: Flexgroup, Inodes limit, Volumes count- NFS


Flexgroup and Files Count:
NFSv3 in ONTAP uses 32-bit file IDs by default. 32-bit file IDs are limited to 2,147,483,647 maximum unsigned integers. With the two-billion inode limit in FlexVol, this value fits nicely into the architecture. 

However, because NetApp FlexGroup volumes can officially support up to 400 billion files in a single container (and theoretically, many more), the implementation of 64-bit file IDs was needed. 64-bit file IDs support up to 9,223,372,036,854,775,807 unsigned integers. The 64-bit file identifier option is off by default. 

This was by design, to make certain that legacy applications and operating systems that require 32-bit file identifiers were not unexpectedly affected by 34 NetApp ONTAP FlexGroup Volumes .

 ONTAP changes before administrators could properly evaluate their environments. Check with your application and OS vendor for their support for 64-bit file IDs before enabling them. Alternatively, create a test SVM and enable it to see how applications and clients react with 64-bit file IDs. Most modern applications and OSes can handle 64-bit file IDs without issue.

If a FlexGroup volume does not exceed two billion files, you can leave this value unchanged. However, to prevent any file ID conflicts, the inode maximum on the FlexGroup volume should also be increased to no more than 2,147,483,647.



You are Welcome :)
Source: www..netapp.com