Linux: HowTo: Emergency Reboot a Remote Machine

Sometimes the machine you're working on has a little oops. Maybe the reboot command has hung and the system will not shutdown or a kernel panic has occurred and although you still have shell access, there is little more you can do with the machine. The solution is simple: you need to hard reboot the machine. But you're in Phoenix and the machine is in L.A. Like any good system administrator, you have the machine hooked up to an IP-KVM (or serial over IP, if the machine is headless), but the magic SysRq keys won't send properly. So what's a sysadmin to do?

The solution is amazing simple and stems from the SysRq key command to reboot the machine. As it happens, you can send SysRq commands to a machine without using a keyboard via the proc interface.

The first step is to enable the SysRq system, if it's disabled:

echo 1 > /proc/sys/kernel/sysrq

Now you can reboot (instantly, be forewarned) if the machine is hung:

echo b > /proc/sysrq-trigger

Apparently, there's a mnemonic for the "proper" order to reboot a machine, so that interruption to the system is minimized. According to the Magic SysRq Wikipedia article, the mnemonic is "Raising Elephants Is So Utterly Boring," with each letter corresponding to the command to pass to /proc/sysrq-trigger. Running these in sequence cause the machine to...

  1. R - keyboard goes out of RAW mode.
  2. E - TErminate all processes.
  3. I - KIll all processes.
  4. S - Sync disks.
  5. U - Unmount disks and remount them read-only.
  6. B - ReBoot the machine.

reply

Emergency situations are relatively infrequent so getting free or very affordable phones will serve the needs of emergency user.
Garage Doors

If you are doing this to a

If you are doing this to a remote computer that you are lucky enough to still have a ssh connection to, do not send the terminate or kill signals to the processes or you will kill you connection before being able to reboot the system.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Copy the characters (respecting upper/lower case) from the image.