Train at Home keeps crashing or hangs overnight: the auto-restart watchdog

If you leave Train at Home running overnight, you sometimes wake up to a dead app, or to an app that is still open but has stopped doing anything. subnera has an opt-in watchdog for exactly that case, called Auto-Restart on Crash. It is deliberately cautious, because a restart is not free of consequences. This guide covers what it detects, when it stays out of the way, and what it cannot fix.

Crash versus hang

The watchdog tells two situations apart, using only what subnera can already observe: whether the official app is running, whether a main_pool worker process is still alive, and how old the last CLI log line is.

• Crash: the official app is no longer running, but its worker is still alive. The official app has a cleanup bug on quit that can leave that worker behind, so "app gone plus orphaned worker" is read as a crash. The watchdog kills the leftover worker and launches the app again.
• Hang: the app is running, but the CLI log has not been written to for more than 5 minutes. The watchdog quits the app, waits about 3 seconds, kills any leftover worker and launches it again.

If the app is gone and there is no orphaned worker, the watchdog assumes you quit it on purpose and leaves it alone.

The menu title shows off once the log has been silent for 2 minutes. The watchdog waits for 5 minutes before acting, so a short pause never triggers a restart. For how the log is read, see the queue position guide.

When it deliberately does nothing

• It is off. Auto-restart is off until you turn it on.
• The app was not recently up. The watchdog only acts if it saw the official app running within the last 2 minutes. It never starts an app you keep closed.
• You just stopped or restarted the app. After a stop or restart from the menu, or from the hub, it stays idle for 5 minutes, so it never undoes your action or races the relaunch.
• It acted recently. At least 3 minutes must pass between two actions, so a slow boot is not mistaken for a new crash.
• An official update restart is in progress. The watchdog is skipped while subnera is restarting the app to install an update.

This spacing is a fixed 3-minute minimum, not a growing delay.

The flap guard

If the watchdog would act more than 3 times within 15 minutes, it stops instead of looping. The menu then shows a warning, Watchdog stopped (flapping), with a Re-enable Watchdog action. Switching the option off and on again also starts a fresh window.

Repeated recoveries usually mean the cause is not a one-off: a Mac that goes to sleep, a network that keeps failing, or a damaged install. Look at that cause instead of re-enabling it blindly. The keep-awake guide and the troubleshooting guide cover the common ones.

What a restart costs

Restarting the official app sends a fresh registration request, and the orchestrator answers with a new queue id at the back of the queue. That is the same cost whether you restart by hand or the watchdog does it. If you were queued, you start the wait again.

So the watchdog is for real crashes and real hangs, where the miner was not doing anything useful anyway. It is not a tool for impatience, and it does not speed anything up. If your position looks stuck but the log is still moving, do nothing. The queue position guide explains why restarting while queued is the costliest mistake.

subnera makes no promise about uptime or earnings. The watchdog shortens the time a dead app stays dead, nothing more.

What it does to the official app

subnera never modifies the official app. The watchdog only does what you could do yourself: it asks the app to quit, ends leftover main_pool worker processes, and opens the installed app again in the background, hidden. It does not touch the app's files, and it makes no network connection to do any of this.

Turning it on and off

1. Open the subnera menu, then Options.
2. Click Auto-Restart on Crash. A checkmark means it is on. Click again to turn it off.

The setting is remembered across launches.

If a Mac is enrolled in subnera hub, you can also switch the watchdog on or off remotely. That works only on Macs where Allow remote control is ticked locally, because the command is refused otherwise. See the remote control guide for how consent works, and the fleet hub guide for the rest of the hub.

The related "Reap Orphaned Worker" action

When the official app is not running but a worker is still alive, the menu shows a Reap Orphaned Worker action in the official-app section, together with an "Orphaned worker still running" warning. It ends the leftover worker without relaunching anything. It does not appear while the app is running, and a hub reap command is refused in that case too, because the worker of a running app is the live miner.

With the watchdog on, you normally do not need it: the crash path reaps the worker for you.

Checklist

1. Install subnera if you have not yet: install guide.
2. Turn on Options, Auto-Restart on Crash only if the Mac runs unattended.
3. Expect a recovery after about 5 minutes of silence for a hang, and sooner for a crash.
4. Do not restart the app yourself while queued.
5. If the flapping warning appears, find the cause before re-enabling.
6. For several Macs, decide per Mac whether remote control may switch the watchdog.