However, Prometheus can use different storage backends. The TSDB that it comes with is horrible.
I mean, it's workable. And can store an impressive amount of data points. If you don't care about historical data or scale, it may be all you need.
However, if your scale is really large, or if you care about the data, it may not be the right solution, and you'll need something like Cortex.
For instance, Prometheus' own TSSB has no 'fsck'-like tool. From time to time, it does compaction operations. If your process (or pod in K8s) dies, you may be left with duplicate time series. And now you have to delete some (or a lot!) of your data to recover.
Prometheus documentation, last I checked, even says it is not suitable for long-term storage.
The TSDB it uses is actually pretty state of the art. I think your pain point is more that it's designed for being used on local disk, but that doesn't mean it isn't possible to store the TSDB remotely. In fact, this is exactly how Thanos works.
The docs say Prometheus is not intended for long term storage because without a remote_write configuration, all data is persisted locally, and thus you will eventually hit limits on the amount that can be stored and queried locally. However, that is a limitation on how Prometheus is designed, not how the TSDB is designed, and which can be overcome by using a remote_write adapter.
However, Prometheus can use different storage backends. The TSDB that it comes with is horrible.
I mean, it's workable. And can store an impressive amount of data points. If you don't care about historical data or scale, it may be all you need.
However, if your scale is really large, or if you care about the data, it may not be the right solution, and you'll need something like Cortex.
For instance, Prometheus' own TSSB has no 'fsck'-like tool. From time to time, it does compaction operations. If your process (or pod in K8s) dies, you may be left with duplicate time series. And now you have to delete some (or a lot!) of your data to recover.
Prometheus documentation, last I checked, even says it is not suitable for long-term storage.