cluster health monitoring - uses for system internal/read-only topics

[glycerine]

To consolidate discussion on slack about using a system-internal topic to have servers provide health and cluster group membership information:

As a nats system operator, when gnatsd-cluster membership changes, I want:

(a) nats-top to be monitoring the correct set of all servers; (see also nats-io/nats-top#31)

(b) to receive an event published on nats that gets translated into a pager duty call so I know what box to go reboot/reclone;

(c) to allow the remaining set of servers’ clients to decide upon a new master to be the one writing.

Moreover, as a nats operator, when I run nats-top, for efficiency and ease of configuration:

(d) I don’t want to have to setup TLS. I want to avoid TLS between nats-top and my local gnatsd, and I want to avoid TLS between nats-top and any other remote gnatsd server.

Slack participants pointed out prior art in #230, which is similar but we're talking here about monitoring gnatsd-to-gnatsd (intra-cluster) connections rather than client connections.

With help from the Slack discussions, I'm trying to work up pull-request that will address these. My plan is start with #230 and adjust as needed.

[glycerine]

useful answers from the slack discussion:

larry:
How would you tell the difference between a planned or unplanned server outage?

jason:
a planned outage is one where I tell pager duty to stop bothering me about it :)

ivan: [when we were discussing if we could approximate with a regular client instead of a "system-internal" client]
... as long as your health monitor knows 1 address in the cluster, it could use that to connect. On success, it can then regularly invoke Servers() to detect the list of servers in the cluster and create individual connections to those servers to check for health. Also, know that Servers() return list of URLs that clients can connect to. If each server uses 0.0.0.0 as its listen address, then the URLs that you get in your client are the resolved external IPs. So you may get more than one address if a server’s host has many interfaces. If you want to restrict to 1, then you should start your servers with -a <specific IP>. Then clients will receive only those. ...otherwise, you may create many health connections to the same server. Nothing bad per se, but you have to be aware of that.

[ColinSullivan1]

There quite a bit of good work here, and this is really appreciated. At this time, this is a feature we do not plan on implementing in the NATS server.

Parts of this would make a valuable monitoring tool outside of the server; if you are interested and implement something along the lines of using an external client monitor the health of a cluster and surface health related events, we'd certainly reference your work on our NATS.io Connectors and Utilities.

We hope you do so!