Troubleshooting systemd Error 203/EXEC While Starting Chronos

Recently, while trying to start the Chronos service using systemctl, I encountered the following error:

systemctl start chronos

Upon checking the status:

chronos.service: Control process exited, code=exited status=203/EXEC

---

Root Cause

The error status=203/EXEC means that systemd failed to execute the service script. Common reasons include:

1. Missing shebang (#!/bin/bash) at the top of the script.

2. Incorrect file permissions — in this case, the script had chmod 777, which is too permissive and not allowed by systemd.

---

Resolution Steps

Here’s what I did to resolve it:

1. Added the shebang to the script:

   #!/bin/bash

   Placed at the very top of /etc/rc.d/init.d/chronos.

2. Corrected the script permissions:

   chmod 755 /etc/rc.d/init.d/chronos

3. Verified that the script works manually:

   /etc/rc.d/init.d/chronos start

4. Restarted the service:

   systemctl start chronos

This time, the service started successfully and showed active (exited), which is expected behavior for SYSV-style scripts.

---

Recommendation

While the issue is resolved, consider migrating to a native systemd unit file for better process management, logging, and compatibility. Legacy init scripts work, but native units provide more control and clarity.

Conflict in Kernel Path Assignment and Device Mapping for Multipath LUN /dev/mapper/mpXXX

 

Problem Overview

The issue stemmed from a conflict with the multipath device /dev/mapper/mpathch. The kernel assigned identical paths (e.g., sdmg, sdfl, sdmd, sdft) to a new LUN, resulting in duplicate device mappings and access conflicts. This caused errors during device scans and prevented the cleanup of the mpathch device.

Errors Observed

1. During pvscan, errors were encountered when reading the device:

   
   Error reading device /dev/mapper/mpathch at 0 length 512.
   Error reading device /dev/mapper/mpathch at 0 length 4.
   Error reading device /dev/mapper/mpathch at 4096 length 4.
   

   These errors indicated that the device could not be properly accessed or read by the system, likely due to the path conflicts.

2. dmsetup info -c revealed that:

   • The device /dev/mapper/mpathch was still active with paths assigned.
   • The logical volume vgexport-lvexport was in use, blocking further actions on mpathch.

Resolution Steps

1. Checked Active Devices:

   • Used dmsetup info -c to identify active devices and locate mpathch and associated logical volumes.

   
   dmsetup info -c | grep mpathch
   dmsetup info -c | grep lv
   

2. Removed the Blocking Logical Volume:

   • Identified and removed the logical volume vgexport-lvexport, which was preventing the unmapping of mpathch.

   dmsetup remove vgexport-lvexport

3. Forcefully Removed the Multipath Device:

   • Used dmsetup remove -f to forcibly delete the mpathch device from the device-mapper layer.

   
   dmsetup remove -f mpathch
   

Validation

• Verified that mpathch and its associated paths were no longer present using:

  dmsetup info -c
  multipath -ll
  

• Confirmed the system was no longer referencing the conflicting LUN and informed the storage team for reassignment or cleanup.

• After the reload, the new LUN became visible in the system.

Conclusion

The problem was caused by duplicate kernel path assignments for a new LUN, which conflicted with existing device mappings and caused read errors during device scans. By removing the blocking logical volume and forcefully unmapping the multipath device, the issue was resolved, and the system was returned to a clean state.