Monday, March 16, 2015

Clearing Single Bit Error Logs in CSTM

Here i'll post how to clear memory error log and PDT error log in hpux zx6000 and rx26000 itanium servers.This procedure need to be done after DIMM replacing, for example the server had bad DIMM/'s and you brought a new and replaced the bad one. After that PDT(Page Deallocation Table) must be clear.To be sure if you have any memory errors run: echo "selclass qualifier memory;info;wait;infolog" | cstm you will get something like that (differ from memory map):Memory Board Inventory
  
DIMM Location          Size(MB)     DIMM Location          Size(MB)
   ——————--   ——--     ——————--   ——--
   DIMM 0A                2048         DIMM 0B                2048
   DIMM 1A                2048         DIMM 1B                2048
   DIMM 2A                —-         DIMM 2B                —-
   DIMM 3A                —-         DIMM 3B                —-
   DIMM 4A                —-         DIMM 4B                —-
   DIMM 5A                —-         DIMM 5B                —-

   Total: 8192 (MB)

   ===========================================================================

Memory Error Log Summary

   DIMM Location           Error Address     Error Type  Page           Count
   ———————-  —————-  ———-  ————-  —--
   DIMM 0B                 0x12263d00        Single-Bit  0x12263        1
   RANK 0                  0x40e6042500      Multi-Bit   0x40e6042      N/A
   RANK 0                  0x40e03ebc00      Multi-Bit   0x40e03eb      N/A
   RANK 0                  0x40c60d3580      Multi-Bit   0x40c60d3      N/A
   RANK 0                  0x40b0176480      Multi-Bit   0x40b0176      N/A
   RANK 0                  0x1428e9d80       Multi-Bit   0x1428e9       N/A

above we see a problem with DIMM 0B, he need to be replaced.

                                                                Clearing procedure:
reboot the server, choose start EFI shell, in the shell type: pdt clear
if you asked a question type "yes".
Boot the OS, rerun the cstm command (see above) to be sure if your log is clear. If you'll see the next output all is ok:

Memory Error Log Summary
    The memory error log is empty.
 Page Deallocation Table (PDT)
    The Page Deallocation Table is empty.


If you still see errors you can simply recreate memory log file:

mv /var/stm/logs/os/memlog /var/stm/logs/os/memlog.old
touch /var/stm/logs/os/memlog
chmod 644 /var/stm/logs/os/memlog
chown root:root /var/stm/logs/os/memlog

I am also tried to clear log through Logtool Utility with no luck:

cstm
cstm>runutil logtool
Logtool Utility>CL
The Memory->Clear Log operation is not available on IPF systems.

Recreating log memory file always works.



Thursday, March 12, 2015

Setting Up DM-Multipath

Before setting up DM-Multipath on your system, ensure that your system has been updated and includes the device-mapper-multipath package.
Use the following procedure to set up DM-Multipath for a basic failover configuration.
  1. Edit the /etc/multipath.conf file by commenting out the following lines at the top of the file. This section of the configuration file, in its initial state, blacklists all devices. You must comment it out to enable multipathing.
    blacklist {
            devnode "*"
    }
    
    After commenting out those lines, this section appears as follows.
    # blacklist {
    #        devnode "*"
    # }
    
  2. The default settings for DM-Multipath are compiled in to the system and do not need to be explicitly set in the /etc/multipath.conf file.
    The default value of path_grouping_policy is set to failover, so in this example you do not need to change the default value. For information on changing the values in the configuration file to something other than the defaults, see Chapter 4, The DM-Multipath Configuration File.
    The initial defaults section of the configuration file configures your system that the names of the multipath devices are of the form mpathn; without this setting, the names of the multipath devices would be aliased to the WWID of the device.
  3. Save the configuration file and exit the editor.
  4. Execute the following commands:
    modprobe dm-multipath
    service multipathd start
    multipath -v2
    
    The multipath -v2 command prints out multipathed paths that show which devices are multipathed. If the command does not print anything out, ensure that all SAN connections are set up properly and the system is multipathed.
    For further information on the multipath command output, see Section 5.5, “Multipath Command Output”.
  5. Execute the following command to ensure sure that the multipath daemon starts on bootup:
    chkconfig multipathd on
    
Since the value of user_friendly_name is set to yes in the configuration file the multipath devices will be created as /dev/mapper/mpathn

Ignoring Local Disks when Generating Multipath Devices

Some machines have local SCSI cards for their internal disks. DM-Multipath is not recommended for these devices. The following procedure shows how to modify the multipath configuration file to ignore the local disks when configuring multipath.
  1. Determine which disks are the internal disks and mark them as the ones to blacklist.
    In this example, /dev/sda is the internal disk. Note that as originally configured in the default multipath configuration file, executing the multipath -v2 shows the local disk, /dev/sda, in the multipath map.
    For further information on the multipath command output, see Section 5.5, “Multipath Command Output”.
    [root@rh4cluster1 ~]# multipath -v2
    create: SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1
    [size=33 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 0:0:0:0 sda  8:0    [--------- 
    
    device-mapper ioctl cmd 9 failed: Invalid argument
    device-mapper ioctl cmd 14 failed: No such device or address
    create: 3600a0b80001327d80000006d43621677
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:0 sdb  8:16    
      \_ 3:0:0:0 sdf  8:80    
    
    create: 3600a0b80001327510000009a436215ec
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:1 sdc  8:32    
      \_ 3:0:0:1 sdg  8:96    
    
    create: 3600a0b80001327d800000070436216b3
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:2 sdd  8:48    
      \_ 3:0:0:2 sdh  8:112   
    
    create: 3600a0b80001327510000009b4362163e
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:3 sde  8:64    
      \_ 3:0:0:3 sdi  8:128
    
  2. In order to prevent the device mapper from mapping /dev/sda in its multipath maps, edit the blacklist section of the /etc/multipath.conf file to include this device. Although you could blacklist the sda device using a devnode type, that would not be safe procedure since /dev/sda is not guaranteed to be the same on reboot. To blacklist individual devices, you can blacklist using the WWID of that device.
    Note that in the output to the multipath -v2 command, the WWID of the /dev/sda device is SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1. To blacklist this device, include the following in the /etc/multipath.conf file.
    blacklist {
          wwid SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1
    }
    
  3. After you have updated the /etc/multipath.conf file, you must manually tell the multipathd daemon to reload the file. The following command reloads the updated /etc/multipath.conf file.
    service multipathd reload
    
  4. Run the following commands:
    multipath -F
    multipath -v2
    
    The local disk or disks should no longer be listed in the new multipath maps, as shown in the following example.
[root@rh4cluster1 ~]# multipath -F
[root@rh4cluster1 ~]# multipath -v2
create: 3600a0b80001327d80000006d43621677
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:0 sdb  8:16    
  \_ 3:0:0:0 sdf  8:80    

create: 3600a0b80001327510000009a436215ec
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:1 sdc  8:32    
  \_ 3:0:0:1 sdg  8:96    

create: 3600a0b80001327d800000070436216b3
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:2 sdd  8:48    
  \_ 3:0:0:2 sdh  8:112   

create: 3600a0b80001327510000009b4362163e
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:3 sde  8:64    
  \_ 3:0:0:3 sdi  8:128

Adding Devices to the Multipathing Database

By default, DM-Multipath includes support for the most common storage arrays that support DM-Multipath. The default configuration values, including supported devices, can be found in the multipath.conf.defaults file.
If you need to add a storage device that is not supported by default as a known multipath device, edit the /etc/multipath.conf file and insert the appropriate device information.
For example, to add information about the HP Open-V series the entry looks like this:
devices {
       device {
               vendor "HP"
               product "OPEN-V."
               getuid_callout "/sbin/scsi_id -g -u -p0x80 -s /block/%n"
       }
}

Wednesday, March 11, 2015

What is logrotate?
It may surprise you to learn that logrotate is a program used to rotate logs. It’s true! The system usually runs logrotate once a day, and when it runs it checks rules that can be customized on a per-directory or per-log basis.
“Log rotation” refers to the practice of archiving an application’s current log, starting a fresh log, and deleting older logs. And while we’re explaining things, a “log” is a file where an application stores information that might be useful to an administrator or developer - what it’s been doing, what errors it’s run into, that sort of thing. So logs are good, you just usually don’t want to keep a ton of them around. That’s where logrotate comes in.
The importance of log rotation
Logs are wonderful things when you want to track usage or troubleshoot an application. Unfortunately the more information that gets logged, the more disk space the log uses. Over time it can really add up.
A log left unrotated can grow to a pretty unwieldy size. Running out of disk space because of a giant log is a problem of course, but a huge log file can also slow down the process of resizing or backing up your virtual server. Another practical consideration is that it’s hard to look for a particular event if you have a million log entries to skim through. So on the whole it’s a good idea to keep log files down to a manageable size, and to prune them when they get too old to be of much use.
Fortunately logrotate makes log rotation easy.
How it works
The system runs logrotate on a schedule, usually daily. In fact, on most distributions you’ll find the script that runs logrotate daily at:

/etc/cron.daily/logrotate
Some distributions use a variation on that theme. For example, on Gentoo the logrotate script is:

/etc/cron.daily/logrotate.cron
If you want logrotate to run more often (for hourly log rotation, for example) you’ll need to look into using cron to run logrotate through a script in /etc/cron.hourly.
When logrotate runs it reads its configuration files to determine where to find the log files it needs to rotate, and to check on details like how often the files should be rotated and how many archived logs to keep.
logrotate.conf
The main logrotate configuration file is located at:

/etc/logrotate.conf
If you look inside that file you’ll see the default parameters logrotate uses when it rotates logs. The file is nicely commented, so skim it to see how things are set up. We’ll talk about several of the specific commands in that file shortly.
Note that one line reads:

include /etc/logrotate.d
That’s where we’ll find most of the application-specific configuration files.
logrotate.d
Take a look inside the directory where you’ll store application-specific log settings:

ls /etc/logrotate.d
Depending on how much you’ve installed on your server there may be no files in this directory, or there may be several. In general, applications that are installed through your package manager will also create a config file in /etc/logrotate.d.
Most likely you will at least see a config file for your syslog service, which logrotate will read when it goes to rotate the system logs. If you look inside you’ll see an entry for various system logs along with some commands similar to what you saw in logrotate.conf.
NOTE: You won’t actually see an entry for a syslog service on versions of Ubuntu older than Karmic Koala (9.10). Prior to that release the system logs were rotated by a “savelog” command run from the “/etc/cron.daily/sysklogd” script.
Inside an application file
As an example, let’s take a look at the contents of a logrotate config file that might be put in place when you install apache on a Fedora system:

/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}
We’ll look at what most of the specific directives in this file mean in a bit, but the short version is that when logrotate runs it will check for any files in /var/log/httpd that end in “log” and rotate them, so long as they aren’t empty. If it checks the httpd directory and doesn’t find any logfiles it won’t throw an error. Then it will run the command in the “postrotate/endscript” block (in this case, a command that will tell apache to restart), but only after it’s processed all the specified logs.
What you don’t see in that file are some settings you saw back in logrotate.conf. This is because the commands in logrotate.conf act as defaults for log rotation. You can specify different settings for any application where you want to override the defaults. For example, if you run a busy web server, you may want to include a “daily” command in apache’s config block so apache’s logs will rotate daily instead of the default weekly rotation.
That might be more clear if we talk about what some of the more commonly-used commands actually do in a logrotate config file. So let’s do that next.
Configuration commands
You can get a full list of commands used in logrotate configuration files by checking the man page:

man logrotate
We’ll go over more commonly-used commands here.
Remember, the config files for applications in /etc/logrotate.d inherit their defaults from the main /etc/logrotate.conf file.
Log files
A log file and its rotation behavior is defined by listing the log file (or files) followed by curly brackets. Most application configuration files will contain just one of these blocks, but it’s possible to put more than one in a file, or to add log file blocks to the main logrotate.conf file.
You can list more than one log file for a block either by using a wildcard in the name or by separating log files in the list with spaces. For example, to specify all files in the directory /var/foo that end in “.log”, as well as the file “/var/bar/log.txt”, you would set up the block like so:

/var/foo/*.log /var/bar/log.txt {
        blah blah blah
        blah blah blah redux
}
Just not with as many blahs.
Rotate count
The “rotate” command determines how many archived logs will be kept around before logrotate starts deleting the older ones. For example:

rotate 4
That command tells logrotate to keep 4 archived logs at a time. If there are already four archived logs when the log is rotated again, the oldest one (the one with “.4” at the end, usually) will be deleted to make room for the new archive.
Rotation interval
You can specify a command that will tell logrotate how often to rotate a particular log. The possible commands include:

daily
weekly
monthly
yearly
If a rotation interval is not specified the log will be rotated whenever logrotate runs (unless another condition like “size” has been set).
If you want to use a time interval other than the keywords listed here you’ll have to get clever with cron and a separate config file. For example, if you wanted to rotate a particular log file hourly, you could create a file in “/etc/cron.hourly” (you may need to create that directory too) that would contain a line like:

/usr/sbin/logrotate /etc/logrotate.hourly.conf
Then put the configuration for that hourly run of logrotate (the log file location, whether or not to compress old files, and so on) into “/etc/logrotate.hourly.conf”.
Size
You can specify a file size that logrotate will check when determining whether or not to perform a rotation by using the “size” command. The format of the command tells logrotate what units you’re using to specify the size:

size 100k
size 100M
size 100G
The first example would rotate the log if it gets larger than 100 kilobytes, the second if it’s larger than 100 megabytes, and the third if it’s over 100 gigabytes. I don’t recommend using a limit of 100G, mind you, the example just got a little out of hand there.
The size command takes priority over and replaces a rotation interval if both are set.
Compression
If you want archived logfiles to be compressed (in gzip format) you can include the following command, usually in /etc/logrotate.conf:

compress
This is normally a good idea, since log files are usually all text, and text compresses very well. You might, however, have some archived logs you don’t want compressed, but still want compression to be on by default. In those cases you can include the following command in an application-specific config:

nocompress
One more command of note in regard to compression is:

delaycompress
This command can be useful if you want the archived logs to be compressed, but not right away. With “delaycompress” active an archived log won’t be compressed until the next time the log is rotated. This can be important when you have a program that might still write to its old logfile for a time after a fresh one is rotated in. Note that “delaycompress” only works if you also have “compress” in your config.
An example of a good time to use delaycompress would be when logrotate is told to restart apache with the “graceful” or “reload” directive. Since old apache processes would not be killed until their connections are finished, they could potentially try to log more items to the old file for some time after the restart. Delaying the compression ensures that you won’t lose those extra log entries when the logs are rotated.
Postrotate
The “postrotate” script is run by logrotate each time it rotates a log specified in a config block. You’ll usually want to use this to restart an application after the log rotation so the app can switch to a new log.

postrotate
    /usr/sbin/apachectl restart > /dev/null
endscript
That “> /dev/null” bit at the end tells logrotate to pipe the command’s output to, well, nowhere. Otherwise the output of that command will be sent off to the console or the log or email or whatever, and in this case, you don’t really care about the output if everything restarted okay.
The “postrotate” command tells logrotate that the script to run will start on the next line, and the “endscript” command says that the script is done.
Sharedscripts
Normally logrotate will run the “postrotate” script every time it rotates a log. This is true for multiple logs using the same config block. So for example, a web server config block that refers to both the access log and the error log will, if it rotates both, run the “postrotate” script twice (once for each file rotated). So if both files are rotated, the web server will be restarted twice.
To keep logrotate from running that script for every log, you can include the command:

sharedscripts
That tells logrotate to wait until it’s checked all the logs for that config block before running the postrotate script. If one or both of the logs get rotated, the postrotate script still only gets run once. If none of the logs get rotated, the postrotate script won’t run at all.

Tuesday, March 10, 2015

How To Produce High CPU Load, Memory, I/O Or Stress Test Linux Server

On CentOS/RHEL
# yum install gcc gcc-c++ autoconf automake
Download the latest tarball and run configure, make, make install
# tar zxvf stress-1.0.4.tar.gz
# cd stress-1.0.4
# ./configure
# make
# make install
The binary gets installed under /usr/local/bin
To start stress run stress followed by the -c flag for load stress, -m for memory stress, -i for io and -d for HDD. For example to stress cpu execute
# stress -c 5
Execution of the command above will hog all available cpu power and create a load five times a single core would happily handle.


Similarly to stress some memory you can execute
# stress -m 512M
To know about all available flags execute
# stress --help


Change my system timezone from the command line without using redhat-config-date

In order to change the timezone of your system you will need to access the file /etc/sysconfig/clock directly:

#ZONE="America/New_York"
ZONE="GMT"
UTC=false
ARC=false

Note: If your system's BIOS has UTC set to true, then set UTC to true. If it has it set to false, set it to false. UTC in the configuration file must always reflect your BIOS settings.
In order to get the particular zone you wish to use you must associate ZONE with a file located in /usr/share/zoneinfo. It is wise to note the directory structure because if you need to set the timezone to that of Shanghai which is located in the Asia directory you will then have to set your ZONE variable to the following :

ZONE="Asia/Shanghai"

Or perhaps you need to set the timezone to that of East Brazil :

ZONE="Brazil/East"

Finally save the file /etc/sysconfig/clock and on next reboot the system will be set to the defined timezone.

For the time on the machine to reflect the change timezone we need to link the zoneinfo file to /etc/localtime. This can be done as follows :

If you are setting your timezone to "Brazil/East" link the following file to /etc/localtime :

# ln -sf /usr/share/zoneinfo/Brazil/East /etc/localtime

Now by typing the date command to display the time you should see if reflect the newly linked timezone :

# date
Thu Sep 30 10:06:23 BRT 2004


Removing a Disk from a Logical Volume

Moving Extents to Existing Physical Volumes
In this example, the logical volume is distributed across four physical volumes in the volume group myvg.

# pvs -o+pv_used
  PV         VG   Fmt  Attr PSize  PFree  Used
  /dev/sda1  myvg lvm2 a-   17.15G 12.15G  5.00G
  /dev/sdb1  myvg lvm2 a-   17.15G 12.15G  5.00G
  /dev/sdc1  myvg lvm2 a-   17.15G 12.15G  5.00G
  /dev/sdd1  myvg lvm2 a-   17.15G  2.15G 15.00G


We want to move the extents off of /dev/sdb1 so that we can remove it from the volume group.
If there are enough free extents on the other physical volumes in the volume group, you can execute the pvmove command on the device you want to remove with no other options and the extents will be distributed to the other devices.

# pvmove /dev/sdb1
  /dev/sdb1: Moved: 2.0%
 ...
  /dev/sdb1: Moved: 79.2%
 ...
  /dev/sdb1: Moved: 100.0%


After the pvmove command has finished executing, the distribution of extents is as follows:

# pvs -o+pv_used
  PV         VG   Fmt  Attr PSize  PFree  Used
  /dev/sda1  myvg lvm2 a-   17.15G  7.15G 10.00G
  /dev/sdb1  myvg lvm2 a-   17.15G 17.15G     0
  /dev/sdc1  myvg lvm2 a-   17.15G 12.15G  5.00G
  /dev/sdd1  myvg lvm2 a-   17.15G  2.15G 15.00G


Use the vgreduce command to remove the physical volume /dev/sdb1 from the volume group.

# vgreduce myvg /dev/sdb1
  Removed "/dev/sdb1" from volume group "myvg"
# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda1  myvg lvm2 a-   17.15G  7.15G
  /dev/sdb1       lvm2 --   17.15G 17.15G
  /dev/sdc1  myvg lvm2 a-   17.15G 12.15G
  /dev/sdd1  myvg lvm2 a-   17.15G  2.15G


Change the kernel parameters in HP_UX...

kctune: It is the administrative command for HP-UX kernel to view or change kernel parameters. The following information provides how to view or modify the kernel parameters.

 Viewing Kernel Parameters:
1
$usr/sbin/kctune

Modifying Kernel Parameters:
/usr/sbin/kctune <parameter name and it’s value>
Sample Output: 
1
2
3
4
5
6
7
8
9
10
mydb:/ #/usr/sbin/kctune hires_timeout_enable=1
     ==> Update the automatic 'backup' configuration first? yes
       * The automatic 'backup' configuration has been updated.
       * Future operations will update the backup without prompting.
        * The requested changes have been applied to the currently
         running configuration.
Tunable                         Value  Expression  Changes
hires_timeout_enable  (before)     0   Default     Immed
                       (now)       1   1
mydb:/ #

Viewing Specific Kernel Parameter:
/usr/sbin/kctune <parameter name >
Use the bellow command if you have HP_UX B.11.31 
1
2
3
4
mydb:/ #/usr/sbin/kctune hires_timeout_enable
Tunable               Value  Expression  Changes
hires_timeout_enable      1  1           Immed
mydb:/ #
Use the bellow command if you have HP_UX B.11.23
1
2
3
sun2:/home/oracle #sysdef | grep kctune hires_timeout_enable
maxuprc                    3686          -          3-                   -
sun2:/home/oracle #


Number of Open LV and Current LV is different for VG

On one of our systems I see that the current number of logical volumes in vg00 does not equal the number of logical volumes currently open. The host does not seem to have any unusual problems because of this.

There are 16 pairs of files in the /dev/vg00 directory, which matches the 16 "open" LVs. However, there are 18 reported as "Cur LV". 

Unfortunately, due to additions/deletions the minor numbers are all over the map ranging from 0x000001 to 0x00001c.

I had hoped to avoid the brute-force approach-- but it worked,  to give me a nice list of all the LVs, sorted in minor number order, I used

ls -l /dev/vg00 | sort +5

Then I used a manual list of missing minor numbers to build a script to create all the devices:

foreach minor in 09 0c 10 11 12 14 15 16 17 18 19 1b; do echo mknod /dev/vg00/mia_${minor} b 64 0x0000${minor}; echo mknod /dev/vg00/rmia_${minor} c 64 0x0000${minor}; done > /tmp/make-minors.sh 

Then I just ran "sh /tmp/make-minors.sh" (after verifying that it looked good)

Now "pvdisplay -v" shows me which LVs were missing but still allocated. Now for a little judicious use of "lvremove"...

Thanks for the help.

PS: Here's what I saw that was odd: [Note the Cur LV vs. Open LV]

# vgdisplay vg00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 18
Open LV 16
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 2500
VGDA 4
PE Size (Mbytes) 4
Total PE 4338
Alloc PE 3964
Free PE 374
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

PPS: It was "19" and "1b" that had been removed without removing their allocated logical extents.