How to Troubleshoot NFS Issues in Linux

how-to-troubleshoot-nfs-issues-in-linux
Getting your Trinity Audio player ready...

How to Troubleshoot NFS Issues in Linux?

NFS and AutoFS are heavily used in LINUX Infrastructure. Most often LINUX Administrators deal with it. In this article, we are going to discuss how to troubleshoot NFS issues in linux. Most of the Linux administrators normally aware of NFS but for those who are still not aware let us see some basics about NFS.

What is NFS in Linux?

NFS stands for Network File Sharing. NFS is a protocol used by you to share files and directories with other linux client machines over the network. NFS is also known as Network File System like ext2, ext3, ext4, etc.

NFS works on Client-Server Architecture. Source Server from where files and folder share is called Master NFS Server and Client who use these resources generally referred as NFS Remote Client.

NFS gets installed with OS bundle and service remains in an inactive state. It uses rpcbind, mount, statd, quotad daemon, etc.

How to Setup NFS Server in Linux?

Under /etc/exports file sharing is done in form of mount commands. These are used with automounts for application purposes. Once exports file is updated you can start the NFS service using the below commands. It is same like Solaris  /etc/dfs/dfstab.If it is not present you can simply install it using the yum tool.

How to Start/Stop/Reload/Restart NFS Service in RHEL6?

As part of troubleshooting NFS issues, you may in a situation to perform start, stop, reload restart of the service. Let’s see how to start/stop/reload/restart NFS Service in RHEL6 if needed. Please remember to perform restart/stop/start especially with agreement from the application team as it may disrupt the application if any I/O in progress on any NFS shares.  While reloading of services can be performed online as it does not stop services and it forces to reread the configuration and if any addition in NFS share it makes that active too.

# service nfs start       >>> Starting NFS
# service nfs restart     >>> Restarting NFS
# service nfs reload      >>> Reloading NFS
# service nfs stop        >>> Stoping NFS

How to Start/Stop/Reload/Restart NFS Service in RHEL7?

Precaution is almost similar as mentioned above. You can simply run systemctl command to get all services configured in the server irrespective of their status means it includes all running and stopped services as well.

# systemctl                        >>> To List All Services
# systemctl |egrep “nfs.service”   >>> To List NFS Services
# systemctl start nfs.service      >>> To Start NFS Service
# systemctl reload nfs.service     >>> To Reload NFS Service
# systemctl stop nfs.service      >>> To Stop NFS Service

How to Check NFS Service Status in RHEL6/RHEL7?

Below is the command sample how to check NFS Service Status in RHEL6

# service nfs status                                   
rpc.svcgssd is stopped
rpc.mountd (pid 13343) is running...
nfsd (pid 13358 13357 13356 13355 13354 13353 13352 13351) is running...
rpc.rquotad (pid 13338) is running...

Please follow a sample of how to check NFS Service Status in RHEL7.

# systemctl status nfs.service
nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
Active: inactive (dead)

How to Check NFS Version in Linux?

nfsstat as the name suggests provides detailed statics of NFS activity along with statics it is also handy to find what NFS version is active on the server. You can use nfsstat to check NFS Version in Linux.

# nfsstat -s
Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
530321     0          0          0          0
Server nfs v4:
null         compound
324       0% 529997   99%
Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit
0         0% 0         0% 0         0% 3948      0% 430       0% 0         0%
create       delegpurge   delegreturn  getattr      getfh        link
0         0% 0         0% 116       0% 511126   31% 274414   17% 0         0%
lock         lockt        locku        lookup       lookup_root  nverify
0         0% 0         0% 0         0% 272793   17% 0         0% 0         0%
open         openattr     open_conf    open_dgrd    putfh        putpubfh
437       0% 0         0% 6         0% 0         0% 528589   33% 0         0%
putrootfh    read         readdir      readlink     remove       rename
1201      0% 298       0% 5563      0% 4         0% 0         0% 0         0%
renew        restorefh    savefh       secinfo      setattr      setcltid
174       0% 0         0% 0         0% 0         0% 0         0% 11        0%
setcltidconf verify       write        rellockowner bc_ctl       bind_conn
11        0% 0         0% 0         0% 0         0% 0         0% 0         0%
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo
0         0% 0         0% 0         0% 0         0% 0         0% 0         0%
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence
0         0% 0         0% 0         0% 0         0% 0         0% 0         0%
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp
0         0% 0         0% 0         0% 0         0% 0         0%

How to Check Packet Drop by NFS in Linux?

nfsstat command in linux is really handy in finding if any packet drop is happening in both server and client machines.

Sample output is placed for reference.

# nfsstat -o net
Server packet stats:
packets    udp        tcp        tcpconn
530375     0          530375     362
Client packet stats:
packets    udp        tcp        tcpconn
0          0          0          0

How to Import Export NFS Share in Linux?

exportfs command in linux used to import and export NFS shares whenever needed. When you start the NFS service NFS shares get active. If you share a new directory via NFS and already have other shares working for example servers in production services, exportfs are really handy to re-read NFS configuration and share new NFS share online without the need of restarting the NFS service.

exportfs command line syntax

# man exportfs
exportfs - maintain table of exported NFS file systems
/usr/sbin/exportfs [-avi] [-o options,..] [client:/path ..]
/usr/sbin/exportfs -r [-v]
/usr/sbin/exportfs [-av] -u [client:/path ..]
/usr/sbin/exportfs [-v]
/usr/sbin/exportfs -f
DESCRIPTION
An  NFS server maintains a table of local physical file systems that are 
accessible to NFS clients.  Each file system in this table is referred to 
as an exported file system, or export, for short.
The exportfs command maintains the current table of exports for the NFS 
server.  The master export table  is  kept  in  a  file  named /var/lib/nfs
/etab.  This file is read by rpc.mountd when a client sends an NFS MOUNT 
request.
Normally  the master export table is initialized with the contents of /etc/
exports and files under /etc/exports.d by invoking exportfs.
-a.  However, a system administrator can choose to add or delete exports 
without modifying /etc/exports or files under  /etc/exports.d by using the 
exportfs command.
exportfs  and  its partner program rpc.mountd work in one of two modes: a 
legacy mode which applies to 2.4 and earlier versions of the Linux kernel, 
and a new mode which applies to 2.6 and later versions, providing the  nfsd
virtual  filesystem  has  been  mounted  at  /proc/fs/nfsd or /proc/fs/nfs.
  On 2.6 kernels, if this filesystem is not mounted, the legacy mode is used.
In  the  new  mode,  exportfs  does  not  give  any  information  to  the  
kernel,  but  provides  it  only  to rpc.mountd through the  /var/lib/nfs
/etab file.  rpc.mountd then manages kernel requests for information about 
exports, as needed.
In the legacy mode, exports which identify a specific host, rather than a 
subnet or netgroup, are entered directly into  the  kernels export  table,
as  well as being written to /var/lib/nfs/etab. 
Further, exports listed in /var/lib/nfs/rmtab which match a non host-specific
export request will cause an appropriate export entry for the host given in 
rmtab to be added to the kernelâs export table.
OPTIONS
-a     Export or unexport all directories.
-i     Ignore the /etc/exports file and files under /etc/exports.d directory.
Only default options and options given on  the  command   line are used.
-r     Reexport  all  directories,  synchronizing  /var/lib/nfs/etab  with  
/etc/exports  and files under /etc/exports.d.  This option removes entries 
in /var/lib/nfs/etab which have been deleted from /etc/exports or files under
/etc/exports.d, and  removes  any entries from the kernel export table which
are no longer valid.
-u     Unexport one or more directories.
-f     If  /proc/fs/nfsd  or  /proc/fs/nfs  is  mounted,  flush everything out
 of the kernelâs export table.  Fresh entries for active clients are added to 
the kernelâs export table by rpc.mountd when they make their next NFS mount request.
-v     Be verbose. When exporting or unexporting, show whatâs going on. When displaying
the current export list, also display the list of export options.

exportfs command in linux with examples

# exportfs -a
This command adds all directories listed in /etc/exports updates /etc/exports.d to 
/var/lib/nfs/etab and pushes the resulting export entries into the kernel.
# exportfs -o insecure_locks django:/usr/tmp
To export the /usr/tmp directory to host django, allowing insecure file locking requests
from clients.

# exportfs -u django:/usr/tmp
To unexport the /usr/tmp directory
# exportfs -au
To unexport all exports listed in /etc/exports and files under /etc/exports.d.
To export the /usr/tmp directory to IPv6 link-local clients.
# exportfs [fe80::]/64:/usr/tmp

USAGE NOTES

Exporting to  IP  networks or DNS and NIS domains does not enable clients from these groups to access NFS immediately.  Rather, these sorts of exports are hints to rpc. mounted to grant any mount requests from these clients.  This is usually not a  problem,  because any existing mounts are preserved in rmtab across reboots.

When unexporting a network or domain entry, any current exports to members of this group will be checked against the remaining valid exports and if they themselves are no longer valid they will be removed.

# exportfs -o insecure_locks django:/usr/tmp
To unexport the /usr/tmp directory:
# exportfs -u django:/usr/tmp
To unexport all exports listed in /etc/exports 
and files under /etc/exports.d
# exportfs -au
To export the /usr/tmp directory to IPv6 
link-local clients.
# exportfs [fe80::]/64:/usr/tmp

Important Files Related to exportfs

/etc/exports             
input file listing exports, export options, and access control lists
/etc/exports.d           
directory where extra input files are stored.
Note: only files that end with .exports are used.
/var/lib/nfs/etab
master table of exports
/var/lib/nfs/rmtab
table of clients accessing serverâs exports

Sample NFS Mount Configuration

# cat /etc/exports
/usr/data lnxsrv01(rw,no_root_squash,sync)

How to Check NFS Mount Details

For Local NFS Share Details
# showmount -e
Export list for linuxsrv101:
/usr/data1 linuxsrv102.xyz.com,linuxsrv103.xyz.com 
For Remote NFS Share Details
# showmount –e <remote server/ip>

How to Check NFS Performance Statics in Linux?

NFS service provides facility or you can say tools similar to iostat which is used to get performance statics of reading and write. nfsiostat is the utility tool you can use to get NFS Performance Statics.

nfsiostat is only used for NFS mount points on the server. It usages /proc/self/mountstats as input for providing information about the performance of NFS shares mounted on the server.

nfsiostat command in linux is used on the NFS client to check its performance when communicating with the NFS server.

You can run nfsiostat without any argument and the output will look like below.

# nfsiostat
lnxsrv101:/usr/data1 mounted on /nfs101/data1:
op/s         rpc bklog
0.11            0.00
read:    ops/s  kB/s   kB/op         retrans         avg RTT (ms)    avg exe (ms)
         0.000  0.031  267.686        0 (0.0%)           5.500           5.500
write:   ops/s    kB/s   kB/op         retrans         avg RTT (ms)    avg exe (ms)
         0.000    0.000  0.000        0 (0.0%)           0.000           0.000

To understand this report better below are few important parameters to look out for.

  • avg exe (ms) – The duration since the NFS client makes the RPC request to its kernel until the RPC request is completed. This includes the RTT.
  • retrans – number of retransmissions.
  • avg RTT (ms) – The duration since the client’s kernel sends the RPC request until the time it receives the reply.

If you have very high avg RTT(ms) and retrans both which indicates there is high latency on the network.

High latency on the network side also impacts I/O performance, as there are many timeouts. In such cases, the client will become very slow and it becomes difficult to work.

It is always recommended to have a dedicated network between the NFS server and the NFS client to ensure high throughput.

Conclusion

That`s all about how to troubleshoot nfs issues in linux as of now. I have tried to summarize as much as I recall related to NFS. You may find it useful to have most of the details in one place. If you think anything needs to be part of this article please do not hesitate to share in form of comments.

If you like our work please share and subscribe as much as you can in your friend or colleague circle and help this article reach intended users and benefit them in terms of knowledge update.

Leave a Reply