|
Getting your Trinity Audio player ready... |
How to Troubleshoot NFS Issues in Linux?
NFS and AutoFS are heavily used in LINUX Infrastructure. Most often LINUX Administrators deal with it. In this article, we are going to discuss how to troubleshoot NFS issues in linux. Most of the Linux administrators normally aware of NFS but for those who are still not aware let us see some basics about NFS.
Table of Contents
What is NFS in Linux?
NFS stands for Network File Sharing. NFS is a protocol used by you to share files and directories with other linux client machines over the network. NFS is also known as Network File System like ext2, ext3, ext4, etc.
NFS works on Client-Server Architecture. Source Server from where files and folder share is called Master NFS Server and Client who use these resources generally referred as NFS Remote Client.
NFS gets installed with OS bundle and service remains in an inactive state. It uses rpcbind, mount, statd, quotad daemon, etc.
How to Setup NFS Server in Linux?
Under /etc/exports file sharing is done in form of mount commands. These are used with automounts for application purposes. Once exports file is updated you can start the NFS service using the below commands. It is same like Solaris /etc/dfs/dfstab.If it is not present you can simply install it using the yum tool.
How to Start/Stop/Reload/Restart NFS Service in RHEL6?
As part of troubleshooting NFS issues, you may in a situation to perform start, stop, reload restart of the service. Let’s see how to start/stop/reload/restart NFS Service in RHEL6 if needed. Please remember to perform restart/stop/start especially with agreement from the application team as it may disrupt the application if any I/O in progress on any NFS shares. While reloading of services can be performed online as it does not stop services and it forces to reread the configuration and if any addition in NFS share it makes that active too.
# service nfs start >>> Starting NFS # service nfs restart >>> Restarting NFS # service nfs reload >>> Reloading NFS # service nfs stop >>> Stoping NFS
How to Start/Stop/Reload/Restart NFS Service in RHEL7?
Precaution is almost similar as mentioned above. You can simply run systemctl command to get all services configured in the server irrespective of their status means it includes all running and stopped services as well.
# systemctl >>> To List All Services # systemctl |egrep “nfs.service” >>> To List NFS Services # systemctl start nfs.service >>> To Start NFS Service # systemctl reload nfs.service >>> To Reload NFS Service # systemctl stop nfs.service >>> To Stop NFS Service
How to Check NFS Service Status in RHEL6/RHEL7?
Below is the command sample how to check NFS Service Status in RHEL6
# service nfs status rpc.svcgssd is stopped rpc.mountd (pid 13343) is running... nfsd (pid 13358 13357 13356 13355 13354 13353 13352 13351) is running... rpc.rquotad (pid 13338) is running...
Please follow a sample of how to check NFS Service Status in RHEL7.
# systemctl status nfs.service nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled) Active: inactive (dead)
How to Check NFS Version in Linux?
nfsstat as the name suggests provides detailed statics of NFS activity along with statics it is also handy to find what NFS version is active on the server. You can use nfsstat to check NFS Version in Linux.
# nfsstat -s Server rpc stats: calls badcalls badclnt badauth xdrcall 530321 0 0 0 0 Server nfs v4: null compound 324 0% 529997 99% Server nfs v4 operations: op0-unused op1-unused op2-future access close commit 0 0% 0 0% 0 0% 3948 0% 430 0% 0 0% create delegpurge delegreturn getattr getfh link 0 0% 0 0% 116 0% 511126 31% 274414 17% 0 0% lock lockt locku lookup lookup_root nverify 0 0% 0 0% 0 0% 272793 17% 0 0% 0 0% open openattr open_conf open_dgrd putfh putpubfh 437 0% 0 0% 6 0% 0 0% 528589 33% 0 0% putrootfh read readdir readlink remove rename 1201 0% 298 0% 5563 0% 4 0% 0 0% 0 0% renew restorefh savefh secinfo setattr setcltid 174 0% 0 0% 0 0% 0 0% 0 0% 11 0% setcltidconf verify write rellockowner bc_ctl bind_conn 11 0% 0 0% 0 0% 0 0% 0 0% 0 0% exchange_id create_ses destroy_ses free_stateid getdirdeleg getdevinfo 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% getdevlist layoutcommit layoutget layoutreturn secinfononam sequence 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% set_ssv test_stateid want_deleg destroy_clid reclaim_comp 0 0% 0 0% 0 0% 0 0% 0 0%
How to Check Packet Drop by NFS in Linux?
nfsstat command in linux is really handy in finding if any packet drop is happening in both server and client machines.
Sample output is placed for reference.
# nfsstat -o net Server packet stats: packets udp tcp tcpconn 530375 0 530375 362 Client packet stats: packets udp tcp tcpconn 0 0 0 0
How to Import Export NFS Share in Linux?
exportfs command in linux used to import and export NFS shares whenever needed. When you start the NFS service NFS shares get active. If you share a new directory via NFS and already have other shares working for example servers in production services, exportfs are really handy to re-read NFS configuration and share new NFS share online without the need of restarting the NFS service.
exportfs command line syntax
# man exportfs exportfs - maintain table of exported NFS file systems /usr/sbin/exportfs [-avi] [-o options,..] [client:/path ..] /usr/sbin/exportfs -r [-v] /usr/sbin/exportfs [-av] -u [client:/path ..] /usr/sbin/exportfs [-v] /usr/sbin/exportfs -f DESCRIPTION An NFS server maintains a table of local physical file systems that are accessible to NFS clients. Each file system in this table is referred to as an exported file system, or export, for short. The exportfs command maintains the current table of exports for the NFS server. The master export table is kept in a file named /var/lib/nfs /etab. This file is read by rpc.mountd when a client sends an NFS MOUNT request. Normally the master export table is initialized with the contents of /etc/ exports and files under /etc/exports.d by invoking exportfs. -a. However, a system administrator can choose to add or delete exports without modifying /etc/exports or files under /etc/exports.d by using the exportfs command. exportfs and its partner program rpc.mountd work in one of two modes: a legacy mode which applies to 2.4 and earlier versions of the Linux kernel, and a new mode which applies to 2.6 and later versions, providing the nfsd virtual filesystem has been mounted at /proc/fs/nfsd or /proc/fs/nfs. On 2.6 kernels, if this filesystem is not mounted, the legacy mode is used. In the new mode, exportfs does not give any information to the kernel, but provides it only to rpc.mountd through the /var/lib/nfs /etab file. rpc.mountd then manages kernel requests for information about exports, as needed. In the legacy mode, exports which identify a specific host, rather than a subnet or netgroup, are entered directly into the kernels export table, as well as being written to /var/lib/nfs/etab. Further, exports listed in /var/lib/nfs/rmtab which match a non host-specific export request will cause an appropriate export entry for the host given in rmtab to be added to the kernelâs export table. OPTIONS -a Export or unexport all directories. -i Ignore the /etc/exports file and files under /etc/exports.d directory. Only default options and options given on the command line are used. -r Reexport all directories, synchronizing /var/lib/nfs/etab with /etc/exports and files under /etc/exports.d. This option removes entries in /var/lib/nfs/etab which have been deleted from /etc/exports or files under /etc/exports.d, and removes any entries from the kernel export table which are no longer valid. -u Unexport one or more directories. -f If /proc/fs/nfsd or /proc/fs/nfs is mounted, flush everything out of the kernelâs export table. Fresh entries for active clients are added to the kernelâs export table by rpc.mountd when they make their next NFS mount request. -v Be verbose. When exporting or unexporting, show whatâs going on. When displaying the current export list, also display the list of export options.
exportfs command in linux with examples
# exportfs -a This command adds all directories listed in /etc/exports updates /etc/exports.d to /var/lib/nfs/etab and pushes the resulting export entries into the kernel.
# exportfs -o insecure_locks django:/usr/tmp
To export the /usr/tmp directory to host django, allowing insecure file locking requests
from clients.
# exportfs -u django:/usr/tmp
To unexport the /usr/tmp directory
# exportfs -au
To unexport all exports listed in /etc/exports and files under /etc/exports.d.
To export the /usr/tmp directory to IPv6 link-local clients. # exportfs [fe80::]/64:/usr/tmp
USAGE NOTES
Exporting to IP networks or DNS and NIS domains does not enable clients from these groups to access NFS immediately. Rather, these sorts of exports are hints to rpc. mounted to grant any mount requests from these clients. This is usually not a problem, because any existing mounts are preserved in rmtab across reboots.
When unexporting a network or domain entry, any current exports to members of this group will be checked against the remaining valid exports and if they themselves are no longer valid they will be removed.
# exportfs -o insecure_locks django:/usr/tmp To unexport the /usr/tmp directory: # exportfs -u django:/usr/tmp To unexport all exports listed in /etc/exports and files under /etc/exports.d # exportfs -au To export the /usr/tmp directory to IPv6 link-local clients. # exportfs [fe80::]/64:/usr/tmp
Important Files Related to exportfs
/etc/exports input file listing exports, export options, and access control lists /etc/exports.d directory where extra input files are stored. Note: only files that end with .exports are used. /var/lib/nfs/etab master table of exports /var/lib/nfs/rmtab table of clients accessing serverâs exports
Sample NFS Mount Configuration
# cat /etc/exports /usr/data lnxsrv01(rw,no_root_squash,sync)
How to Check NFS Mount Details
For Local NFS Share Details # showmount -e Export list for linuxsrv101: /usr/data1 linuxsrv102.xyz.com,linuxsrv103.xyz.com For Remote NFS Share Details # showmount –e <remote server/ip>
How to Check NFS Performance Statics in Linux?
NFS service provides facility or you can say tools similar to iostat which is used to get performance statics of reading and write. nfsiostat is the utility tool you can use to get NFS Performance Statics.
nfsiostat is only used for NFS mount points on the server. It usages /proc/self/mountstats as input for providing information about the performance of NFS shares mounted on the server.
nfsiostat command in linux is used on the NFS client to check its performance when communicating with the NFS server.
You can run nfsiostat without any argument and the output will look like below.
# nfsiostat lnxsrv101:/usr/data1 mounted on /nfs101/data1: op/s rpc bklog 0.11 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.031 267.686 0 (0.0%) 5.500 5.500 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000
To understand this report better below are few important parameters to look out for.
- avg exe (ms) – The duration since the NFS client makes the RPC request to its kernel until the RPC request is completed. This includes the RTT.
- retrans – number of retransmissions.
- avg RTT (ms) – The duration since the client’s kernel sends the RPC request until the time it receives the reply.
If you have very high avg RTT(ms) and retrans both which indicates there is high latency on the network.
High latency on the network side also impacts I/O performance, as there are many timeouts. In such cases, the client will become very slow and it becomes difficult to work.
It is always recommended to have a dedicated network between the NFS server and the NFS client to ensure high throughput.
Conclusion
That`s all about how to troubleshoot nfs issues in linux as of now. I have tried to summarize as much as I recall related to NFS. You may find it useful to have most of the details in one place. If you think anything needs to be part of this article please do not hesitate to share in form of comments.
If you like our work please share and subscribe as much as you can in your friend or colleague circle and help this article reach intended users and benefit them in terms of knowledge update.
