How To Force Linux Server To Run Fsck On The Next Reboot

In this tutorial, we are going to discuss how to force linux server to run fsck on the next reboot. Sometimes we have a file system that shows inconsistency or even does not get automatically or having many I/O error and running fsck is expected to eliminate the issue at hand.

There could be much reason for this behaviour of the file system. It may be because on unclean mount or server is up since long which needs fsck to run. Let us understand a little bit more on fsck.

Table of Contents

What is fsck?

“fsck” stands for File System Consistency Check . “fsck” is a command-line utility tool to perform a file system check and fix if any error encountered during checking to maintain consistency and integrity of file system. This makes fsck really important utility as linux is a collection of files and folders.

This is one of the core functionality provided by all the operating systems either server level or desktop level operating system so as in the case of Red Hat Enterprise Linux as well. Few file system types in RHEL are ext2, ext3, ext4, etc.

Why fsck?

Inconsistencies in the file system may be due to many reasons unexpected reboots due to malfunctioning of any program terminations or abnormal shutdowns or hard reboots in case it gets hung in case of physical servers etc.

When does fsck need to run?

There might be many situations when you need to run fsck manually.For instance

1. I/O error in FS while accessing files.

2. File Read Error.

3. Server is not able to boot due to corruption in any file system and it gets placed in maintenance mode.

4. Slowness in accessing the file system.

OS Provision to overcome FS inconsistencies?

“fsck” is expected to run on every boot. It runs in six phases and in some cases if the file system journal indicates the file system is clean. It gets terminated assuming all OK or it shows a message to run full fsck manually.

fsck sets tables for storing inodes and comparisons verifies the validity of fsck options, then runs through the 6 phases. After initializing its tables fsck performs operations in the below sequence.

1: Check Blocks and Sizes: In this stage fsck validates inodes for any inconsistencies.

2: Check Path-Names: In this stage fsck validates the directory.

3: Check Connectivity: In this stage fsck validates all directories associated with the file system.

4: Check Reference Counts: In this stage fsck compares link count from Phases 2 & 3, correcting if any discrepancies.

5: Check Cylinder Groups: In this stage fsck checks free blocks and the used inode maps for consistency of file system.

6: Salvage Cylinder Groups: In this stage fsck updates the tables to reflect any changes made in earlier phases.

Understanding fsck exit codes

After running fsck, it will return an exit code. These codes can be seen in fsck’s manual by running.:

# man fsck

Note: Running fsck needs root level privilege and fsck operations time depends on fs size. In case if Fs size is bigger fsck may long time. So take downtime appropriately.

You can use the dumpe2fs command to have the status of the file system. Please refer to details that all reported by this command…

# dumpe2fs /dev/loc_trd2-dg/tcakdb
dumpe2fs 1.39 (29-May-2006)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 6afc5d7d-e323-42f9-85c2-d51b76062004
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal resize_inode dir_index filetype 
needs_recovery sparse_super large_file
Default mount options: (none)
Filesystem state: clean <----------------
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 1005305856
Block count: 2010582016
Reserved block count: 100529100
Free blocks: 237857847
Free inodes: 996233045
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 544
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Thu Jul 4 06:32:34 2019
Last mount time: Sun Dec 15 09:40:41 2019
Last write time: Sun Dec 15 09:40:41 2019
Mount count: 22 <------exceed max mount count.filesystem was 
mounted for more than 20 times and this will prompt fsck on its 
filesystem.
Maximum mount count: 20 <------
Last checked: Thu Jul 4 06:32:34 2019<----------- last time fsck 
was executed
Check interval: 15552000 (6 months)
Next check after: Tue Dec 31 05:32:34 2019
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
First orphan inode: 96076811
Default directory hash: tea
Directory Hash Seed: c7b258b0-369e-4739-85e7-dd59220b49d6
Journal backup: inode blocks
Journal size: 128M

So now we understand what is fsck and how it works and why it is important. It is a normal assumption that before running fsck i/o operations must be stopped in other terms it is always better to run fsck after unmounting the file system.

How to run fsck on live systems

Basically, it is quite easy to run it by the un-mounting the file system and run fsck. Log in as root or login and take superuser permission.

# umount /usr/ctc/opt

# fsck -y /dev/ctc/opt

“-y” option is the answer yes if any question prompted while performing fsck operations. In most cases, it runs successfully and returns a 0 error code but in some cases, you may get an error. That is the case when you need to force the system to run full fsck. See the error displayed below.

Force Linux server to run fsck on the next reboot

You can force server automatic full fsck on the next reboot. There is an option where you can set the system to perform full fsck on every reboot using “tunefs” command. Be careful it may increase the boot time of the server depending upon FS sizes configured on it.

# tune2fs -c 1 /dev/sdc  

//It tells to init scripts to run fsck on sdc at every reboot.//

# tune2fs -i 1d /dev/sdc            

//It tells init script to run fsck on sdc after 1 day.//

Sometimes due to inconsistency it is also not working. Please see the above snap. In that case, you need to force the system to run full fsck only on the next reboot. This you can achieve this by creating a file in root (/) directory with the name forcefsck followed by a reboot.

This will force full “fsck” on the next reboot. “forcefsck” file gets deleted automatically after fsck completion. You can monitor activity on the console depending on the server type.

ILO for physical Linux servers,vSphere Web Console for VMWare RHEL VM or you can connect to Azure Web Portal for Linux virtual machine console access taking VM Contributor role access if your RHEL VM is hosted on AZURE.

You will see console output like below. Need to wait till completion. It will depend on the size of the file system. Once completed you are done.

Let me share one of my experiences related to this. One of our production VM in AZURE backup was failing and the error code returned in the backup log was “File read error“.

We went through all possible troubleshooting options with no success. We went ahead with full fsck on the file system after taking downtime. After completion of full fsck using the above method to run fsck on the next reboot. We restarted the backup jobs from policy manually and it works.

That,s all about fsck magic as of now. Keep Learning. If you like our contents please subscribe to our newsletter and like and comment on our social media pages and express your support which means a lot to us and keep motivating us to share more.