Add API to restart a node of the cluster

POST /api/nodes/:nodehost/restart restarts one node of the cluster.

API

POST /api/nodes/:nodehost/restart
Authorization: Bearer <api jwt>

:nodehost is the host part of the node name or the full node name. It can be any node: the node that receives the request relays it to the target via :erpc.

How the restart works

The reply goes out first. One second later the node calls System.stop(1), and whatever supervises the process (systemd with Restart=always) starts it again. A watchdog calls System.halt(1) after 60s if the graceful stop hangs. Both are :timers, which live under the kernel and outlast the shutdown of our supervision tree.

For discussion: exit vs systemctl restart

I went with exiting instead of running systemctl restart. Notes from comparing them:

• The shutdown is the same. systemctl sends SIGTERM, the BEAM turns it into init:stop(), and System.stop/1 calls init:stop/1 directly.
• systemctl restart restarts right away and sends SIGKILL if the stop hangs (TimeoutStopSec). It needs permission to talk to systemd, and it doesn't depend on Restart=.
• Exiting needs no permissions and works the same under any supervisor. It depends on Restart=always, waits RestartSec (5s) before the start, needs our watchdog instead of the SIGKILL, and shows up in the journal as a failed exit.
• Why a non-zero status. After a clean exit systemd runs ExecStop, which in the unit is bin/server stop. That script ignores its argument and runs supavisor start, so it would boot a second instance. A non-zero exit skips ExecStop. Today it's harmless, because ExecStop runs while the node is alive, fails, and SIGTERM does the actual stop.
• I'd remove the ExecStop line from the unit (deploy/service/supavisor.service has it too).

This is from reading systemd's source and local experiments (System.stop(1) exits with status 1, a hung stop is ended by the watchdog). I haven't verified it on a real node yet. Swapping to systemctl --no-block restart only touches Ops.restart_node_local/1.

Health endpoint

After a restart we need to know the node is back, and today we can't ask about one node: /api/health and a GET on the proxy ports go through the load balancer, and metrics lag by the scrape interval and can't distinguish a restarting node from a missing one.

Proposal: GET /api/nodes, which returns every node of the cluster with its name, host, uptime and app version. The answering node collects them via :erpc, and a node that fails to answer is reported under errors, like in the pool list.

Uptime replaces a boot identifier: a restarted node shows up with an uptime lower than the time since the restart request, which tells it apart from the old process that may still answer between the reply and the exit. The list also gives callers the exact node names to use with the restart route.

_POOLER-893