Monday, December 5, 2016

Replacing a failed disk (rootvg)

This post will describe the replacement of a failed rootvg disk.
In short the procedure is the following:
1. unmirrorvg rootvg hdisk0
(savebase -v)
2. reducevg rootvg hdisk0
3. rmdev -Rdl hdisk0
4. diag (safely remove hot swap device/drive), physically remove the old disk
5. insert the new disk (diag – hotplug task)
6. cfgmgr -vl scsi0
7. extendvg -f rootvg hdisk0
8. mirrorvg -m rootvg hdisk0
9. bosboot -ad hdisk0
10. bootlist -m normal hdisk0 hdisk1
11. bootlist -m normal -o
In more detail the procedure is described below starting with the errpt logs indicating a drive problem.
In the errpt output we can see that there is some problem with a disk:

# errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
EAA3D429   0216144412 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0216144412 U S LVDD           PHYSICAL PARTITION MARKED STALE
F7DDA124   0216144412 U H LVDD           PHYSICAL VOLUME DECLARED MISSING
52715FA5   0216144412 U H LVDD           FAILED TO WRITE VOLUME GROUP STATUS AREA
E86653C3   0216144412 P H LVDD           I/O ERROR DETECTED BY LVM
8647C4E2   0216144412 P H hdisk0         DISK OPERATION ERROR
41BF2110   0216144412 U H LVDD           MIRROR WRITE CACHE WRITE FAILED
8647C4E2   0216144412 P H hdisk0         DISK OPERATION ERROR
EAA3D429   0216144412 U S LVDD           PHYSICAL PARTITION MARKED STALE
E86653C3   0216144412 P H LVDD           I/O ERROR DETECTED BY LVM
8647C4E2   0216144412 P H hdisk0         DISK OPERATION ERROR
Lets have a detailed look at the first error:

# errpt -aj 8647C4E2
---------------------------------------------------------------------------
LABEL:          DISK_ERR3
IDENTIFIER:     8647C4E2
Date/Time:       Thu Feb 16 14:44:06 GMT 2012
Sequence Number: 3706
Machine Id:      00CF405E4C00
Node Id:         power1
Class:           H
Type:            PERM
WPAR:            Global
Resource Name:   hdisk0         
Resource Class:  disk
Resource Type:   scsd
Location:        U787F.001.DPM28WG-P1-T10-L5-L0
VPD:            
        Manufacturer................IBM   H0
        Machine Type and Model......HUS103073FL3800
        FRU Number..................03N5262    
        ROS Level and ID............52505152
        Serial Number...............
        EC Level....................H17923D  
        Part Number.................26K5573    
        Device Specific.(Z0)........000004129F00013E
        Device Specific.(Z1)........RPQR       
        Device Specific.(Z2)........0068
        Device Specific.(Z3)........06131
        Device Specific.(Z4)........0001
        Device Specific.(Z5)........22
        Device Specific.(Z6)........H17923D  
        Brand.......................H0
Description
DISK OPERATION ERROR
Probable Causes
DASD DEVICE
STORAGE DEVICE CABLE
Failure Causes
DISK DRIVE
DISK DRIVE ELECTRONICS
STORAGE DEVICE CABLE
Recommended Actions
        PERFORM PROBLEM DETERMINATION PROCEDURES
PATH ID
           0
SENSE DATA
0A05 0000 2E00 0000 0080 0000 0800 0000 0200 0800 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0117 0002
Diagnostic Analysis
Diagnostic Log sequence number: 928
Resource tested:        hdisk0
Resource Description:   16 Bit LVD SCSI Disk Drive
Location:               U787F.001.DPM28WG-P1-T10-L5-L0
SRN:                    2643-129
Description:            Error log analysis indicates a SCSI bus problem.
Possible FRUs:
    n/a              FRU: n/a                
    U787F.001.DPM28WG-P1-T10-L5-L0
                     SCSI bus problem: cables, terminators or other SCSI
                     devices
    hdisk0           FRU: 03N5262            
    U787F.001.DPM28WG-P1-T10-L5-L0
                     16 Bit LVD SCSI Disk Drive
    sisscsia0        FRU: 42R7379              U787F.001.DPM28WG-P1
                     PCI-X Dual Channel Ultra320 SCSI Adapter
 n/a              FRU: n/a                
    U787F.001.DPM28WG-P1-T10-L5-L0
                     Software
Having a look at the disk using lsvg and lspv commands indicates that the disk is in a missing state
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            missing           546         41          05..01..00..00..35
hdisk1            active            546         41          05..00..00..00..36
# lspv hdisk0
PHYSICAL VOLUME:    hdisk0                   VOLUME GROUP:     rootvg
PV IDENTIFIER:      00c0e90dce6c290a VG IDENTIFIER     00c0e90d00004c000000012ff4e24eaa
PV STATE:           missing                                   
STALE PARTITIONS:   21                       ALLOCATABLE:      yes
PP SIZE:            128 megabyte(s)          LOGICAL VOLUMES:  24
TOTAL PPs:          546 (69888 megabytes)    VG DESCRIPTORS:   1
FREE PPs:           41 (5248 megabytes)      HOT SPARE:        no
USED PPs:           505 (64640 megabytes)    MAX REQUEST:      256 kilobytes
FREE DISTRIBUTION:  05..01..00..00..35                        
USED DISTRIBUTION:  105..108..109..109..74                    
MIRROR POOL:        None                      
An attempt to read from the disk using dd failed. So it really looks like the disk died. (actually it could also be that there is a problem with the controller/cable, but in this scenario it is the disk..)
dd if=/dev/hdisk0 of=/tmp/disk0 bs=100 count=1024
dd: 0511-051 The read failed.
: There is an input or output error.
0+0 records in.
0+0 records out.
In order to replace the disk we have to do the following:
# unmirrorvg rootvg hdisk0
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c '
        as root user to clear the boot record and avoid a potential boot
        off an old boot image that may reside on the disk from which this
        logical volume is moved/removed.
0301-108 mkboot: Unable to read file blocks. Return code: -1
0516-1798 lchangevg: Cannot change quorum without losing quorum.
0516-732 chvg: Unable to change volume group rootvg.
0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform
        bosboot of system to reinitialize boot records.  Then, user must modify
        bootlist to just include:  hdisk1.
# chpv -c hdisk0
# savebase -v
saving to '/dev/hd5'
75 CuDv objects to be saved
174 CuAt objects to be saved
25 CuDep objects to be saved
39 CuVPD objects to be saved
387 CuDvDr objects to be saved
2 CuPath objects to be saved
0 CuPathAt objects to be saved
0 CuData objects to be saved
0 CuAtDef objects to be saved
Number of bytes of data to save = 38834
Compressing data
Compressed data size is = 9840
        bi_start     = 0x3600
        bi_size      = 0x1820000
        bd_size      = 0x1800000
        ram FS start = 0x8d6ca0
        ram FS size  = 0xea2902
        sba_start    = 0x1803600
        sba_size     = 0x20000
        sbd_size     = 0x2674
Checking boot image size:
        new save base byte cnt = 0x2674
Wrote 9844 bytes
Successful completion
Now, remove the disk from the VG and also remove the disk from the system using rmdev, afterwards use diag to safely remove the physical disk from the system
# reducevg rootvg hdisk0
# rmdev -Rdl hdisk0
hdisk0 deleted
#diag
Task selection > Hot plug task > SCSI and SCSI RAID Hot Plug Manager > Replace remove device.
For the safe physical insertion of the new disk also use the diag command.
Now we need the system to detect and identify the drive, put it into the VG,mirror the drive and install BOS on the BLV and set the boot order.
 
# cfgmgr -vl scsi0
# extendvg -f rootvg hdisk0
# mirrorvg -m rootvg hdisk0
0516-1126 mirrorvg: rootvg successfully mirrored, user should perform
        bosboot of system to initialize boot records.  Then, user must modify
        bootlist to include:  hdisk0 hdisk1.
# bosboot -ad hdisk0
bosboot: Boot image is 49180 512 byte blocks.
# bootlist -m normal hdisk0 hdisk1
[root@power2](/root) # bootlist -m normal -o
hdisk0 blv=hd5 pathid=0
hdisk1 blv=hd5 pathid=0
And finally check we are done.


Ratelimit callbacks suppressed

Recently I hardened some RHEL6 based machines.
During this hardening process - among the others - I did the following:
- disdabled ipv6 by "options ipv6 disable=1" in /etc/modprobe.d/hardening.conf
- added some more audit rules according to NSA guide
- stopped auditd therefore audit log events are redirected to kernel log
- filtered audit logs by following filter to separate audilt.log:
filter f_audit { match(' audit\(' value("MESSAGE")); };

After this hardening there were staerted appearing "__ratelimit: XX callbacks suppressed" messages regularly in kernel log, like following:

Oct 18 01:00:01 test1 kernel: __ratelimit: 4 callbacks suppressed
Oct 18 01:01:01 test1 kernel: __ratelimit: 192 callbacks suppressed
Oct 18 01:05:07 test1 kernel: __ratelimit: 188 callbacks suppressed

It didn't cause any problem but after a while I've started investigating (thx Cipo) what can cause this strange behaviour.

The root cause of this problem is a bit complex. Let's see them:
- disabling ipv6 module caused that some programs would like to insert it
- I've set an audit rule which logs any module instertion attempts: "-w /sbin/modprobe -p x -k modules"
- there are kernel.printk_ratelimit* kernel parameters
# cat /proc/sys/kernel/printk_ratelimit
5
# cat /proc/sys/kernel/printk_ratelimit_burst
10
This means that there is a 10 msgs/5 seconds limit in kernel logging. Exceeding this limit messages will be dropped AND "__ratelimit NumberOfDroppedMessages: callbacks suppressed" messages will be written to the kernel log.

To put the pieces of the picture together:
- my script runs curl 20 times
- curl tried to insert ipv6 module by modprobe
- modprobe tries were logged by audit (5 lines/ modprobe)
- too much audit log in short time exceeded the ratelimit

My solution:
- re-enable ipv6 by commenting  "options ipv6 disable=1" line in /etc/modprobe.d/hardening.conf
- prevent use of ipv6 calls by appending following line to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1 
(Thanks to Daniel Walsh)

These resulted that:
- ipv6 module is already inserted, so programs do not want to insert it
- ipv6 remained pseudo-disabled

After finishing this I've continued getting ratelimit messaged but not regularly.
I've  find out that Midnight Commander deletes a few temporary files and changes permissions of its config files after exiting. This process resulted more than 150 lines within one second in audit log...
My solution is that I've appended following line to the /etc/sysctl.conf file:
kernel.printk_ratelimit = 30
kernel.printk_ratelimit_burst = 200
This means that kernel accepts 200 messages/30 sec




RedHat ResoluciĆ³n
  • The messages are suppressed because some warning messages are rate limited. The kernel parameter printk_ratelimit specifies the minimum length of time between these messages (in seconds), by default we allow one every 5 seconds.
  • A value of 0 will disable rate limiting. However, this way is not a solution for the problem. If you resolve the problem, you may need to see limited messages by this way.
  1. Add the following configuration to an /etc/sysctl.conf file.
    kernel.printk_ratelimit = 0
  2. Reboot the system or execute the follwoing command.
    sysctl -p

Friday, October 21, 2016

Modify limits configuration without reboot

Changes made by ulimit command:

$ ulimit -n 4096
$ ulimit -Hn 16384

will apply only for current user and session. In order to make it permanent, you have to modify /etc/security/limits.conf by adding your limits:

* soft nofile 4096
* hard nofile 16384

However these changes won't apply for root user. In order to do so, you have to state it explicitly:

* soft nofile 4096
* hard nofile 16384
root soft nofile 4096
root hard nofile 16384

These limits will be applied after reboot.
If you want to apply changes without reboot, modify /etc/pam.d/common-session by adding this line at the end of file:
session required pam_limits.so
Upon next login you should see updated limits, you can check them (soft and hard limits):
$ ulimit -a
$ ulimit -Ha


Example:

[root@XXX~]# cat /etc/security/limits.conf |grep -v "#"
*               soft    nproc           1024
*               hard    nproc           25000


[root@XXX ~]# ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7409
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 25000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited