I did it at quite small scale, but within an on-prem docker swarm. It was indeed a pain because if I remember correctly the default discovery uses multicast which is not enabled on typical cloud networks or on a Swarm/Kubernetes overlay network. I looked at database pings where they'd use your RDMS for a sort of quorum mechanism, but that seemed very brittle and I got the impression it was more of a last-resort type thing.

I was able to use the kubernetes cluster driver which uses the Swarm cluster's DNS for node discovery. It was indeed quite a pain to get working, but since then has been solid as far as I know. I believe there is also a native ec2 networking driver these days, but that is not something that I explored.