Monday, December 5, 2016

Replacing a failed disk (rootvg)

This post will describe the replacement of a failed rootvg disk.
In short the procedure is the following:
1. unmirrorvg rootvg hdisk0
(savebase -v)
2. reducevg rootvg hdisk0
3. rmdev -Rdl hdisk0
4. diag (safely remove hot swap device/drive), physically remove the old disk
5. insert the new disk (diag – hotplug task)
6. cfgmgr -vl scsi0
7. extendvg -f rootvg hdisk0
8. mirrorvg -m rootvg hdisk0
9. bosboot -ad hdisk0
10. bootlist -m normal hdisk0 hdisk1
11. bootlist -m normal -o
In more detail the procedure is described below starting with the errpt logs indicating a drive problem.
In the errpt output we can see that there is some problem with a disk:

# errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
EAA3D429   0216144412 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0216144412 U S LVDD           PHYSICAL PARTITION MARKED STALE
F7DDA124   0216144412 U H LVDD           PHYSICAL VOLUME DECLARED MISSING
52715FA5   0216144412 U H LVDD           FAILED TO WRITE VOLUME GROUP STATUS AREA
E86653C3   0216144412 P H LVDD           I/O ERROR DETECTED BY LVM
8647C4E2   0216144412 P H hdisk0         DISK OPERATION ERROR
41BF2110   0216144412 U H LVDD           MIRROR WRITE CACHE WRITE FAILED
8647C4E2   0216144412 P H hdisk0         DISK OPERATION ERROR
EAA3D429   0216144412 U S LVDD           PHYSICAL PARTITION MARKED STALE
E86653C3   0216144412 P H LVDD           I/O ERROR DETECTED BY LVM
8647C4E2   0216144412 P H hdisk0         DISK OPERATION ERROR
Lets have a detailed look at the first error:

# errpt -aj 8647C4E2
---------------------------------------------------------------------------
LABEL:          DISK_ERR3
IDENTIFIER:     8647C4E2
Date/Time:       Thu Feb 16 14:44:06 GMT 2012
Sequence Number: 3706
Machine Id:      00CF405E4C00
Node Id:         power1
Class:           H
Type:            PERM
WPAR:            Global
Resource Name:   hdisk0         
Resource Class:  disk
Resource Type:   scsd
Location:        U787F.001.DPM28WG-P1-T10-L5-L0
VPD:            
        Manufacturer................IBM   H0
        Machine Type and Model......HUS103073FL3800
        FRU Number..................03N5262    
        ROS Level and ID............52505152
        Serial Number...............
        EC Level....................H17923D  
        Part Number.................26K5573    
        Device Specific.(Z0)........000004129F00013E
        Device Specific.(Z1)........RPQR       
        Device Specific.(Z2)........0068
        Device Specific.(Z3)........06131
        Device Specific.(Z4)........0001
        Device Specific.(Z5)........22
        Device Specific.(Z6)........H17923D  
        Brand.......................H0
Description
DISK OPERATION ERROR
Probable Causes
DASD DEVICE
STORAGE DEVICE CABLE
Failure Causes
DISK DRIVE
DISK DRIVE ELECTRONICS
STORAGE DEVICE CABLE
Recommended Actions
        PERFORM PROBLEM DETERMINATION PROCEDURES
PATH ID
           0
SENSE DATA
0A05 0000 2E00 0000 0080 0000 0800 0000 0200 0800 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0117 0002
Diagnostic Analysis
Diagnostic Log sequence number: 928
Resource tested:        hdisk0
Resource Description:   16 Bit LVD SCSI Disk Drive
Location:               U787F.001.DPM28WG-P1-T10-L5-L0
SRN:                    2643-129
Description:            Error log analysis indicates a SCSI bus problem.
Possible FRUs:
    n/a              FRU: n/a                
    U787F.001.DPM28WG-P1-T10-L5-L0
                     SCSI bus problem: cables, terminators or other SCSI
                     devices
    hdisk0           FRU: 03N5262            
    U787F.001.DPM28WG-P1-T10-L5-L0
                     16 Bit LVD SCSI Disk Drive
    sisscsia0        FRU: 42R7379              U787F.001.DPM28WG-P1
                     PCI-X Dual Channel Ultra320 SCSI Adapter
 n/a              FRU: n/a                
    U787F.001.DPM28WG-P1-T10-L5-L0
                     Software
Having a look at the disk using lsvg and lspv commands indicates that the disk is in a missing state
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            missing           546         41          05..01..00..00..35
hdisk1            active            546         41          05..00..00..00..36
# lspv hdisk0
PHYSICAL VOLUME:    hdisk0                   VOLUME GROUP:     rootvg
PV IDENTIFIER:      00c0e90dce6c290a VG IDENTIFIER     00c0e90d00004c000000012ff4e24eaa
PV STATE:           missing                                   
STALE PARTITIONS:   21                       ALLOCATABLE:      yes
PP SIZE:            128 megabyte(s)          LOGICAL VOLUMES:  24
TOTAL PPs:          546 (69888 megabytes)    VG DESCRIPTORS:   1
FREE PPs:           41 (5248 megabytes)      HOT SPARE:        no
USED PPs:           505 (64640 megabytes)    MAX REQUEST:      256 kilobytes
FREE DISTRIBUTION:  05..01..00..00..35                        
USED DISTRIBUTION:  105..108..109..109..74                    
MIRROR POOL:        None                      
An attempt to read from the disk using dd failed. So it really looks like the disk died. (actually it could also be that there is a problem with the controller/cable, but in this scenario it is the disk..)
dd if=/dev/hdisk0 of=/tmp/disk0 bs=100 count=1024
dd: 0511-051 The read failed.
: There is an input or output error.
0+0 records in.
0+0 records out.
In order to replace the disk we have to do the following:
# unmirrorvg rootvg hdisk0
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1734 rmlvcopy: Warning, savebase failed.  Please manually run 'savebase' before rebooting.
0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c '
        as root user to clear the boot record and avoid a potential boot
        off an old boot image that may reside on the disk from which this
        logical volume is moved/removed.
0301-108 mkboot: Unable to read file blocks. Return code: -1
0516-1798 lchangevg: Cannot change quorum without losing quorum.
0516-732 chvg: Unable to change volume group rootvg.
0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform
        bosboot of system to reinitialize boot records.  Then, user must modify
        bootlist to just include:  hdisk1.
# chpv -c hdisk0
# savebase -v
saving to '/dev/hd5'
75 CuDv objects to be saved
174 CuAt objects to be saved
25 CuDep objects to be saved
39 CuVPD objects to be saved
387 CuDvDr objects to be saved
2 CuPath objects to be saved
0 CuPathAt objects to be saved
0 CuData objects to be saved
0 CuAtDef objects to be saved
Number of bytes of data to save = 38834
Compressing data
Compressed data size is = 9840
        bi_start     = 0x3600
        bi_size      = 0x1820000
        bd_size      = 0x1800000
        ram FS start = 0x8d6ca0
        ram FS size  = 0xea2902
        sba_start    = 0x1803600
        sba_size     = 0x20000
        sbd_size     = 0x2674
Checking boot image size:
        new save base byte cnt = 0x2674
Wrote 9844 bytes
Successful completion
Now, remove the disk from the VG and also remove the disk from the system using rmdev, afterwards use diag to safely remove the physical disk from the system
# reducevg rootvg hdisk0
# rmdev -Rdl hdisk0
hdisk0 deleted
#diag
Task selection > Hot plug task > SCSI and SCSI RAID Hot Plug Manager > Replace remove device.
For the safe physical insertion of the new disk also use the diag command.
Now we need the system to detect and identify the drive, put it into the VG,mirror the drive and install BOS on the BLV and set the boot order.
 
# cfgmgr -vl scsi0
# extendvg -f rootvg hdisk0
# mirrorvg -m rootvg hdisk0
0516-1126 mirrorvg: rootvg successfully mirrored, user should perform
        bosboot of system to initialize boot records.  Then, user must modify
        bootlist to include:  hdisk0 hdisk1.
# bosboot -ad hdisk0
bosboot: Boot image is 49180 512 byte blocks.
# bootlist -m normal hdisk0 hdisk1
[root@power2](/root) # bootlist -m normal -o
hdisk0 blv=hd5 pathid=0
hdisk1 blv=hd5 pathid=0
And finally check we are done.


Ratelimit callbacks suppressed

Recently I hardened some RHEL6 based machines.
During this hardening process - among the others - I did the following:
- disdabled ipv6 by "options ipv6 disable=1" in /etc/modprobe.d/hardening.conf
- added some more audit rules according to NSA guide
- stopped auditd therefore audit log events are redirected to kernel log
- filtered audit logs by following filter to separate audilt.log:
filter f_audit { match(' audit\(' value("MESSAGE")); };

After this hardening there were staerted appearing "__ratelimit: XX callbacks suppressed" messages regularly in kernel log, like following:

Oct 18 01:00:01 test1 kernel: __ratelimit: 4 callbacks suppressed
Oct 18 01:01:01 test1 kernel: __ratelimit: 192 callbacks suppressed
Oct 18 01:05:07 test1 kernel: __ratelimit: 188 callbacks suppressed

It didn't cause any problem but after a while I've started investigating (thx Cipo) what can cause this strange behaviour.

The root cause of this problem is a bit complex. Let's see them:
- disabling ipv6 module caused that some programs would like to insert it
- I've set an audit rule which logs any module instertion attempts: "-w /sbin/modprobe -p x -k modules"
- there are kernel.printk_ratelimit* kernel parameters
# cat /proc/sys/kernel/printk_ratelimit
5
# cat /proc/sys/kernel/printk_ratelimit_burst
10
This means that there is a 10 msgs/5 seconds limit in kernel logging. Exceeding this limit messages will be dropped AND "__ratelimit NumberOfDroppedMessages: callbacks suppressed" messages will be written to the kernel log.

To put the pieces of the picture together:
- my script runs curl 20 times
- curl tried to insert ipv6 module by modprobe
- modprobe tries were logged by audit (5 lines/ modprobe)
- too much audit log in short time exceeded the ratelimit

My solution:
- re-enable ipv6 by commenting  "options ipv6 disable=1" line in /etc/modprobe.d/hardening.conf
- prevent use of ipv6 calls by appending following line to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1 
(Thanks to Daniel Walsh)

These resulted that:
- ipv6 module is already inserted, so programs do not want to insert it
- ipv6 remained pseudo-disabled

After finishing this I've continued getting ratelimit messaged but not regularly.
I've  find out that Midnight Commander deletes a few temporary files and changes permissions of its config files after exiting. This process resulted more than 150 lines within one second in audit log...
My solution is that I've appended following line to the /etc/sysctl.conf file:
kernel.printk_ratelimit = 30
kernel.printk_ratelimit_burst = 200
This means that kernel accepts 200 messages/30 sec




RedHat ResoluciĆ³n
  • The messages are suppressed because some warning messages are rate limited. The kernel parameter printk_ratelimit specifies the minimum length of time between these messages (in seconds), by default we allow one every 5 seconds.
  • A value of 0 will disable rate limiting. However, this way is not a solution for the problem. If you resolve the problem, you may need to see limited messages by this way.
  1. Add the following configuration to an /etc/sysctl.conf file.
    kernel.printk_ratelimit = 0
  2. Reboot the system or execute the follwoing command.
    sysctl -p

Friday, October 21, 2016

Modify limits configuration without reboot

Changes made by ulimit command:

$ ulimit -n 4096
$ ulimit -Hn 16384

will apply only for current user and session. In order to make it permanent, you have to modify /etc/security/limits.conf by adding your limits:

* soft nofile 4096
* hard nofile 16384

However these changes won't apply for root user. In order to do so, you have to state it explicitly:

* soft nofile 4096
* hard nofile 16384
root soft nofile 4096
root hard nofile 16384

These limits will be applied after reboot.
If you want to apply changes without reboot, modify /etc/pam.d/common-session by adding this line at the end of file:
session required pam_limits.so
Upon next login you should see updated limits, you can check them (soft and hard limits):
$ ulimit -a
$ ulimit -Ha


Example:

[root@XXX~]# cat /etc/security/limits.conf |grep -v "#"
*               soft    nproc           1024
*               hard    nproc           25000


[root@XXX ~]# ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7409
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 25000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Monday, March 16, 2015

Clearing Single Bit Error Logs in CSTM

Here i'll post how to clear memory error log and PDT error log in hpux zx6000 and rx26000 itanium servers.This procedure need to be done after DIMM replacing, for example the server had bad DIMM/'s and you brought a new and replaced the bad one. After that PDT(Page Deallocation Table) must be clear.To be sure if you have any memory errors run: echo "selclass qualifier memory;info;wait;infolog" | cstm you will get something like that (differ from memory map):Memory Board Inventory
  
DIMM Location          Size(MB)     DIMM Location          Size(MB)
   ——————--   ——--     ——————--   ——--
   DIMM 0A                2048         DIMM 0B                2048
   DIMM 1A                2048         DIMM 1B                2048
   DIMM 2A                —-         DIMM 2B                —-
   DIMM 3A                —-         DIMM 3B                —-
   DIMM 4A                —-         DIMM 4B                —-
   DIMM 5A                —-         DIMM 5B                —-

   Total: 8192 (MB)

   ===========================================================================

Memory Error Log Summary

   DIMM Location           Error Address     Error Type  Page           Count
   ———————-  —————-  ———-  ————-  —--
   DIMM 0B                 0x12263d00        Single-Bit  0x12263        1
   RANK 0                  0x40e6042500      Multi-Bit   0x40e6042      N/A
   RANK 0                  0x40e03ebc00      Multi-Bit   0x40e03eb      N/A
   RANK 0                  0x40c60d3580      Multi-Bit   0x40c60d3      N/A
   RANK 0                  0x40b0176480      Multi-Bit   0x40b0176      N/A
   RANK 0                  0x1428e9d80       Multi-Bit   0x1428e9       N/A

above we see a problem with DIMM 0B, he need to be replaced.

                                                                Clearing procedure:
reboot the server, choose start EFI shell, in the shell type: pdt clear
if you asked a question type "yes".
Boot the OS, rerun the cstm command (see above) to be sure if your log is clear. If you'll see the next output all is ok:

Memory Error Log Summary
    The memory error log is empty.
 Page Deallocation Table (PDT)
    The Page Deallocation Table is empty.


If you still see errors you can simply recreate memory log file:

mv /var/stm/logs/os/memlog /var/stm/logs/os/memlog.old
touch /var/stm/logs/os/memlog
chmod 644 /var/stm/logs/os/memlog
chown root:root /var/stm/logs/os/memlog

I am also tried to clear log through Logtool Utility with no luck:

cstm
cstm>runutil logtool
Logtool Utility>CL
The Memory->Clear Log operation is not available on IPF systems.

Recreating log memory file always works.



Thursday, March 12, 2015

Setting Up DM-Multipath

Before setting up DM-Multipath on your system, ensure that your system has been updated and includes the device-mapper-multipath package.
Use the following procedure to set up DM-Multipath for a basic failover configuration.
  1. Edit the /etc/multipath.conf file by commenting out the following lines at the top of the file. This section of the configuration file, in its initial state, blacklists all devices. You must comment it out to enable multipathing.
    blacklist {
            devnode "*"
    }
    
    After commenting out those lines, this section appears as follows.
    # blacklist {
    #        devnode "*"
    # }
    
  2. The default settings for DM-Multipath are compiled in to the system and do not need to be explicitly set in the /etc/multipath.conf file.
    The default value of path_grouping_policy is set to failover, so in this example you do not need to change the default value. For information on changing the values in the configuration file to something other than the defaults, see Chapter 4, The DM-Multipath Configuration File.
    The initial defaults section of the configuration file configures your system that the names of the multipath devices are of the form mpathn; without this setting, the names of the multipath devices would be aliased to the WWID of the device.
  3. Save the configuration file and exit the editor.
  4. Execute the following commands:
    modprobe dm-multipath
    service multipathd start
    multipath -v2
    
    The multipath -v2 command prints out multipathed paths that show which devices are multipathed. If the command does not print anything out, ensure that all SAN connections are set up properly and the system is multipathed.
    For further information on the multipath command output, see Section 5.5, “Multipath Command Output”.
  5. Execute the following command to ensure sure that the multipath daemon starts on bootup:
    chkconfig multipathd on
    
Since the value of user_friendly_name is set to yes in the configuration file the multipath devices will be created as /dev/mapper/mpathn

Ignoring Local Disks when Generating Multipath Devices

Some machines have local SCSI cards for their internal disks. DM-Multipath is not recommended for these devices. The following procedure shows how to modify the multipath configuration file to ignore the local disks when configuring multipath.
  1. Determine which disks are the internal disks and mark them as the ones to blacklist.
    In this example, /dev/sda is the internal disk. Note that as originally configured in the default multipath configuration file, executing the multipath -v2 shows the local disk, /dev/sda, in the multipath map.
    For further information on the multipath command output, see Section 5.5, “Multipath Command Output”.
    [root@rh4cluster1 ~]# multipath -v2
    create: SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1
    [size=33 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 0:0:0:0 sda  8:0    [--------- 
    
    device-mapper ioctl cmd 9 failed: Invalid argument
    device-mapper ioctl cmd 14 failed: No such device or address
    create: 3600a0b80001327d80000006d43621677
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:0 sdb  8:16    
      \_ 3:0:0:0 sdf  8:80    
    
    create: 3600a0b80001327510000009a436215ec
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:1 sdc  8:32    
      \_ 3:0:0:1 sdg  8:96    
    
    create: 3600a0b80001327d800000070436216b3
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:2 sdd  8:48    
      \_ 3:0:0:2 sdh  8:112   
    
    create: 3600a0b80001327510000009b4362163e
    [size=12 GB][features="0"][hwhandler="0"]
    \_ round-robin 0 
      \_ 2:0:0:3 sde  8:64    
      \_ 3:0:0:3 sdi  8:128
    
  2. In order to prevent the device mapper from mapping /dev/sda in its multipath maps, edit the blacklist section of the /etc/multipath.conf file to include this device. Although you could blacklist the sda device using a devnode type, that would not be safe procedure since /dev/sda is not guaranteed to be the same on reboot. To blacklist individual devices, you can blacklist using the WWID of that device.
    Note that in the output to the multipath -v2 command, the WWID of the /dev/sda device is SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1. To blacklist this device, include the following in the /etc/multipath.conf file.
    blacklist {
          wwid SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1
    }
    
  3. After you have updated the /etc/multipath.conf file, you must manually tell the multipathd daemon to reload the file. The following command reloads the updated /etc/multipath.conf file.
    service multipathd reload
    
  4. Run the following commands:
    multipath -F
    multipath -v2
    
    The local disk or disks should no longer be listed in the new multipath maps, as shown in the following example.
[root@rh4cluster1 ~]# multipath -F
[root@rh4cluster1 ~]# multipath -v2
create: 3600a0b80001327d80000006d43621677
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:0 sdb  8:16    
  \_ 3:0:0:0 sdf  8:80    

create: 3600a0b80001327510000009a436215ec
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:1 sdc  8:32    
  \_ 3:0:0:1 sdg  8:96    

create: 3600a0b80001327d800000070436216b3
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:2 sdd  8:48    
  \_ 3:0:0:2 sdh  8:112   

create: 3600a0b80001327510000009b4362163e
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
  \_ 2:0:0:3 sde  8:64    
  \_ 3:0:0:3 sdi  8:128

Adding Devices to the Multipathing Database

By default, DM-Multipath includes support for the most common storage arrays that support DM-Multipath. The default configuration values, including supported devices, can be found in the multipath.conf.defaults file.
If you need to add a storage device that is not supported by default as a known multipath device, edit the /etc/multipath.conf file and insert the appropriate device information.
For example, to add information about the HP Open-V series the entry looks like this:
devices {
       device {
               vendor "HP"
               product "OPEN-V."
               getuid_callout "/sbin/scsi_id -g -u -p0x80 -s /block/%n"
       }
}

Wednesday, March 11, 2015

What is logrotate?
It may surprise you to learn that logrotate is a program used to rotate logs. It’s true! The system usually runs logrotate once a day, and when it runs it checks rules that can be customized on a per-directory or per-log basis.
“Log rotation” refers to the practice of archiving an application’s current log, starting a fresh log, and deleting older logs. And while we’re explaining things, a “log” is a file where an application stores information that might be useful to an administrator or developer - what it’s been doing, what errors it’s run into, that sort of thing. So logs are good, you just usually don’t want to keep a ton of them around. That’s where logrotate comes in.
The importance of log rotation
Logs are wonderful things when you want to track usage or troubleshoot an application. Unfortunately the more information that gets logged, the more disk space the log uses. Over time it can really add up.
A log left unrotated can grow to a pretty unwieldy size. Running out of disk space because of a giant log is a problem of course, but a huge log file can also slow down the process of resizing or backing up your virtual server. Another practical consideration is that it’s hard to look for a particular event if you have a million log entries to skim through. So on the whole it’s a good idea to keep log files down to a manageable size, and to prune them when they get too old to be of much use.
Fortunately logrotate makes log rotation easy.
How it works
The system runs logrotate on a schedule, usually daily. In fact, on most distributions you’ll find the script that runs logrotate daily at:

/etc/cron.daily/logrotate
Some distributions use a variation on that theme. For example, on Gentoo the logrotate script is:

/etc/cron.daily/logrotate.cron
If you want logrotate to run more often (for hourly log rotation, for example) you’ll need to look into using cron to run logrotate through a script in /etc/cron.hourly.
When logrotate runs it reads its configuration files to determine where to find the log files it needs to rotate, and to check on details like how often the files should be rotated and how many archived logs to keep.
logrotate.conf
The main logrotate configuration file is located at:

/etc/logrotate.conf
If you look inside that file you’ll see the default parameters logrotate uses when it rotates logs. The file is nicely commented, so skim it to see how things are set up. We’ll talk about several of the specific commands in that file shortly.
Note that one line reads:

include /etc/logrotate.d
That’s where we’ll find most of the application-specific configuration files.
logrotate.d
Take a look inside the directory where you’ll store application-specific log settings:

ls /etc/logrotate.d
Depending on how much you’ve installed on your server there may be no files in this directory, or there may be several. In general, applications that are installed through your package manager will also create a config file in /etc/logrotate.d.
Most likely you will at least see a config file for your syslog service, which logrotate will read when it goes to rotate the system logs. If you look inside you’ll see an entry for various system logs along with some commands similar to what you saw in logrotate.conf.
NOTE: You won’t actually see an entry for a syslog service on versions of Ubuntu older than Karmic Koala (9.10). Prior to that release the system logs were rotated by a “savelog” command run from the “/etc/cron.daily/sysklogd” script.
Inside an application file
As an example, let’s take a look at the contents of a logrotate config file that might be put in place when you install apache on a Fedora system:

/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}
We’ll look at what most of the specific directives in this file mean in a bit, but the short version is that when logrotate runs it will check for any files in /var/log/httpd that end in “log” and rotate them, so long as they aren’t empty. If it checks the httpd directory and doesn’t find any logfiles it won’t throw an error. Then it will run the command in the “postrotate/endscript” block (in this case, a command that will tell apache to restart), but only after it’s processed all the specified logs.
What you don’t see in that file are some settings you saw back in logrotate.conf. This is because the commands in logrotate.conf act as defaults for log rotation. You can specify different settings for any application where you want to override the defaults. For example, if you run a busy web server, you may want to include a “daily” command in apache’s config block so apache’s logs will rotate daily instead of the default weekly rotation.
That might be more clear if we talk about what some of the more commonly-used commands actually do in a logrotate config file. So let’s do that next.
Configuration commands
You can get a full list of commands used in logrotate configuration files by checking the man page:

man logrotate
We’ll go over more commonly-used commands here.
Remember, the config files for applications in /etc/logrotate.d inherit their defaults from the main /etc/logrotate.conf file.
Log files
A log file and its rotation behavior is defined by listing the log file (or files) followed by curly brackets. Most application configuration files will contain just one of these blocks, but it’s possible to put more than one in a file, or to add log file blocks to the main logrotate.conf file.
You can list more than one log file for a block either by using a wildcard in the name or by separating log files in the list with spaces. For example, to specify all files in the directory /var/foo that end in “.log”, as well as the file “/var/bar/log.txt”, you would set up the block like so:

/var/foo/*.log /var/bar/log.txt {
        blah blah blah
        blah blah blah redux
}
Just not with as many blahs.
Rotate count
The “rotate” command determines how many archived logs will be kept around before logrotate starts deleting the older ones. For example:

rotate 4
That command tells logrotate to keep 4 archived logs at a time. If there are already four archived logs when the log is rotated again, the oldest one (the one with “.4” at the end, usually) will be deleted to make room for the new archive.
Rotation interval
You can specify a command that will tell logrotate how often to rotate a particular log. The possible commands include:

daily
weekly
monthly
yearly
If a rotation interval is not specified the log will be rotated whenever logrotate runs (unless another condition like “size” has been set).
If you want to use a time interval other than the keywords listed here you’ll have to get clever with cron and a separate config file. For example, if you wanted to rotate a particular log file hourly, you could create a file in “/etc/cron.hourly” (you may need to create that directory too) that would contain a line like:

/usr/sbin/logrotate /etc/logrotate.hourly.conf
Then put the configuration for that hourly run of logrotate (the log file location, whether or not to compress old files, and so on) into “/etc/logrotate.hourly.conf”.
Size
You can specify a file size that logrotate will check when determining whether or not to perform a rotation by using the “size” command. The format of the command tells logrotate what units you’re using to specify the size:

size 100k
size 100M
size 100G
The first example would rotate the log if it gets larger than 100 kilobytes, the second if it’s larger than 100 megabytes, and the third if it’s over 100 gigabytes. I don’t recommend using a limit of 100G, mind you, the example just got a little out of hand there.
The size command takes priority over and replaces a rotation interval if both are set.
Compression
If you want archived logfiles to be compressed (in gzip format) you can include the following command, usually in /etc/logrotate.conf:

compress
This is normally a good idea, since log files are usually all text, and text compresses very well. You might, however, have some archived logs you don’t want compressed, but still want compression to be on by default. In those cases you can include the following command in an application-specific config:

nocompress
One more command of note in regard to compression is:

delaycompress
This command can be useful if you want the archived logs to be compressed, but not right away. With “delaycompress” active an archived log won’t be compressed until the next time the log is rotated. This can be important when you have a program that might still write to its old logfile for a time after a fresh one is rotated in. Note that “delaycompress” only works if you also have “compress” in your config.
An example of a good time to use delaycompress would be when logrotate is told to restart apache with the “graceful” or “reload” directive. Since old apache processes would not be killed until their connections are finished, they could potentially try to log more items to the old file for some time after the restart. Delaying the compression ensures that you won’t lose those extra log entries when the logs are rotated.
Postrotate
The “postrotate” script is run by logrotate each time it rotates a log specified in a config block. You’ll usually want to use this to restart an application after the log rotation so the app can switch to a new log.

postrotate
    /usr/sbin/apachectl restart > /dev/null
endscript
That “> /dev/null” bit at the end tells logrotate to pipe the command’s output to, well, nowhere. Otherwise the output of that command will be sent off to the console or the log or email or whatever, and in this case, you don’t really care about the output if everything restarted okay.
The “postrotate” command tells logrotate that the script to run will start on the next line, and the “endscript” command says that the script is done.
Sharedscripts
Normally logrotate will run the “postrotate” script every time it rotates a log. This is true for multiple logs using the same config block. So for example, a web server config block that refers to both the access log and the error log will, if it rotates both, run the “postrotate” script twice (once for each file rotated). So if both files are rotated, the web server will be restarted twice.
To keep logrotate from running that script for every log, you can include the command:

sharedscripts
That tells logrotate to wait until it’s checked all the logs for that config block before running the postrotate script. If one or both of the logs get rotated, the postrotate script still only gets run once. If none of the logs get rotated, the postrotate script won’t run at all.