Instructions for how to set up the watchdog daemon to work with IPMI's hardware watchdog
----------------------------------------------------------------------------------------
First, verify that the ipmitool utility is present on the system to allow
the watchdog timer to be turned off via the command line (which ipmitool).
This will allow the hardware watchdog timer to be turned off gracefully
should it ever become necessary. If ipmitool is not present, install
it or download the latest version from http://ipmitool.sourceforge.net and
build and install it on your system.
Next, prior to starting up the watchdog daemon, the BMC BIOS should be set
to enable the IPMI/BMC hardware watchdog timer, the OpenIPMI watchdog driver
module should be inserted with the desired configuration/startup settings,
and the watchdog daemon's configuration file should be modified to use /dev/watchdog:
1. To setup the IPMI/BMC BIOS to enable the hardware watchdog
timer, see BMC documentation. The main settings in the BMC BIOS
requiring modification to turn on the IPMI watchdog timer are:
- Set the BMC POST Watchdog to "ENABLED".
- Set the BMC POST Watchdog Timeout to "5 Minutes".
2. To insert the OpenIPMI watchdog driver module with the
desired configuration settings, two steps are necessary:
i.) Configure the OpenIPMI watchdog driver by editing the
/etc/sysconfig/ipmi configuration file:
- Set "IPMI_WATCHDOG=yes".
- Set desired options via the IPMI_WATCHDOG_OPTIONS
config entry.
EXAMPLE: 'IPMI_WATCHDOG_OPTIONS="timeout=60 start_now=1 \
preop=preop_give_data action=power_cycle pretimeout=1" '
Execute "modinfo ipmi_watchdog" for more detailed information
on the available ipmi watchdog timer options.
- Execute "service ipmi start" (the watchdog driver starts
automatically along with the other ipmi drivers).
IMPORTANT: If "start_now=1" has been set as one of the
configuration options, be sure to start up the watchdog
daemon before the BMC timer expires!
ii.) Set the OpenIPMI daemon and watchdog to start during bootup:
- chkconfig ipmi on
- chkconfig watchdog on
3. Configure the watchdog daemon by editing the
/etc/watchdog.conf configuration file:
- Uncomment the "watchdog-device = /dev/watchdog" line.
- Ensure that "realtime = yes" and "priority = 1" are set and not
commented-out.
- Uncomment the "interval" line, and set the interval to be less
than what you set the timeout option to be in the /etc/sysconfig/ipmi
file (ex "timeout=60" so you might set interval to 50).
So in the example described herein, the BMC BIOS setting is in
minutes (5), and the "interval" and ipmi_watchdog "timeout" settings
are both in seconds (50 and 60 respectively). Therefore, the BMC
hardware watchdog timer is set to expire and trigger a system power
cycle unless reset by the watchdog daemon within 5 minutes, and the
watchdog daemon will reset the timer every 60 seconds.
4. Start the Watchdog daemon:
- execute "service watchdog start"
IMPORTANT: To gracefully stop/kill the watchdog daemon, be sure
to use "service watchdog stop" (which executes "kill -s SIGTERM <pid>")
and do *not* use "kill -9 <pid>". Using "kill -9 <pid>" will cause the
daemon to be shut off without stopping the BMC's watchdog timer, thus
a system reboot will be triggered when the BMC's watchdog timer expires.
Alternately, or in case the watchdog daemon is killed "ungracefully",
you can stop the BMC timer by executing the following ipmitool utility
command before the watchdog timer expires:
# ipmitool -v raw 0x06 0x24 0x04 0x01 0x00 0x10 0x00 0x0a
----------------------------------------------------------------------
To test the watchdog after system configuration and setup:
. Use kill -9 on the watchdog daemon so it doesn't shut down the watchdog daemon
gracefully. Verify that the system gets reset after the BMC timer expires.
. Use "service watchdog stop" and verify that the watchdog daemon shuts off
the BMC watchdog timer gracefully (the system doesn't get reset).
. Set the timer on the watchdog daemon to be greater than the time set in
the BMC BIOS for system reset and verify that the system is reset.
. Set the timer on the daemon to be less than the time set in the
BMC timer and verify that the BMC watchdog is poked regularly and the
system is not reset.
. Test some of the other actions the BMC can take when the watchdog timer
goes off (see modinfo ipmi_watchdog for some other settings to try).
