How to fix – PowerPath Degraded

Scenario

You came to know that PowerPath is in degraded state either via your incident mechanism or via any alerting system you have. You have to fix it so PowerPath can take care of multi-path intelligently.Veritas is being used for managing devices and multipathing.

Validation Steps

Login to server where issue reported and gain elevated privilege as super user and run below command. Location of PowerPath command varies depending upon OS. For RHEL you can use locate command to find out command location or which command in case of Solaris.

If you are using EMC Storage, SYMCLI, PowerPath in your environment it is better to set your .profile so you need not to worry about absolute path every time you want to run these commands. In general it gets installed in /usr/symcli/bin/. Veritas Volume Manager is present with PowerPath.

Normally this happens due to disks reclaimed.

# locate powermt                                for RHEL
# which  powermt                                for Solaris

Problem Identification

Run below command it will confirm if PowerPath is in degraded state or not.
$ sudo powermt display
Error: PowerPath  3072 pci@1e,600000/lpfc@3/fp@0,0   degraded     28     13       -    0  13

Solution for how to fix Powerpath in Degraded State

To troubleshoot issue please run below commands
1. $ sudo powermt display dev=all
You can use pipe to grep dead and look for “dead” device(s) – see example down
Pseudo name=emcpower8a
Symmetrix ID=000187880133
Logical device ID=0A52
state=alive; policy=SymmOpt; priority=0; queued-IOs=0
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
3072 pci@1e,600000/lpfc@3/fp@0,0 c0t5006048ACCB4B149d56s0 FA 10cA   
active  dead 
0      1
2.Verify if device(s) are not in use via below command to make sure you are dealing with right 
devices.
$ sudo vxdisk –eo alldgs list | grep –i emcpower8
3.If device not reported in step 2 you can proceed to step 3 to forcibly remove those devices.
$ sudo powermt check force             
This will remove all the dead devices.
4. Re-validate PowerPath status run validation step again you will find it in optimal state.
$ sudo powermt display
3072 pci@1e,600000/lpfc@3/fp@0,0   optimal      15      0       -     0      0

That’s all about how to fix powerpath in degraded State as now you have it in optimal state so you are good now and said issue is fixed.

We will be glad to know if it helps anyways either increasing knowledge base or you may be find it useful in fixing mentioned issue in your server infrastructure.

Request to share this if you find it meaningful.Also if interested in technical tips related to unix linux cloud and bigdata technologies please join below facebook group.