Author here. I started Liftbridge after working as a core committer on NATS and NATS Streaming (as well as my experience using Kafka in the past). In particular, implementing replication in NATS Streaming and drawing from that experience/lessons learned. Happy to answer any questions!
- Written in Go rather than Java/Scala (big benefit here IMO is getting a small static binary rather than having to run a JVM)
- Doesn't rely on Zookeeper
- Is integrated with NATS (can extend NATS with Kafka-like semantics, but in the future will allow for abstracting NATS away entirely)
- Uses gRPC for its API
- Supports "wildcard topics" - e.g. a stream can listen to the topic "foo.*" and will receive messages published on "foo.bar", "foo.baz", "foo.qux", etc.
- Allows for streams to be paused and subsequently resumed automatically when published to - on the roadmap is "auto-pausing" of sparsely used streams
- Exposes an activity stream that allows you to respond to events such as streams being created, deleted, paused, etc.
One advantage Kafka currently has is its consumer groups, which are on the roadmap for Liftbridge but not implemented yet.
And for exactly these reasons I absolutely LOVE this project. Big Go user. Finding this today was a pure gift. Thank you very much for making these excellent design choices and for building this! I hope to contribute in the future.
Could you compare Liftbridge to Akka with persistence module and or streams? I'm looking for a lightweight, embeddable Kafkasque thing (esp interested in exactly once delivery) that I can use to glue various services together.
I'm not super familiar with Akka beyond high-level/actor model so I can't competently compare the two. NATS can be used to implement an actor model and has a similar philosophy to Akka core in terms of delivery semantics (i.e. no real guarantees - https://doc.akka.io/docs/akka/2.1/general/message-delivery-g...).
Looking at Akka Streams, it appears to be more in line with Liftbridge, though I'm not totally sure. Liftbridge (like Kafka), is a message log, so messages do not get acked or removed from a stream (unless removed due to retention limits or compaction). I'm not sure if that's the case with Akka Streams.
I'm curious what your needs are around exactly-once delivery. From what I can tell, Akka only supports at-least-once semantics and is of the opinion that exactly-once isn't really possible as such (which is an opinion I agree with, but that's a more nuanced discussion). https://www.lightbend.com/blog/how-akka-works-exactly-once-m...
Bear in mind that there are always limits in how guaranteed a "guaranteed delivery" actually is. "Exactly once" means that the broker has received it and the client has acknowledged that. What happens beyond the broker is outside the scope of the protocol and would have to be handled at the application level, eg using transaction ids, etc.
Not a question, but just a quick comment: Your overview page assumes familiarity with a bunch of concepts. If you could find the time, perhaps it would be helpful writing up a high-level explanation that makes none of those assumptions.
For instance, it sounds like it's a cluster-based distributed system for some clients to post messages and other clients to get notified about them through pattern matching on a standardized protocol, without any messages getting lost.
