Elasticsearch is amazing. It lives up to the hype. It's perfect for rolling over logs, and they have lots of documentation on how to make it work just right.
Just as an example of how awesome Elasticsearch is, you can trivially segment your storage tiers (say, SSD versus HDD) and then easily move older data to other storage, with a single command.
They have a log-specific handler called Logstash, and a dashboard system called Kibana (which is sorta neat but the UI seems a big laggy in my brief experience). Apparently some folks use Logstash/Elasticsearch to record millions and millions of events per day and ES does a great job.
If you want hosted, check out Stackify. I'm totally blown away with the product (no affiliation other than being a new user). You can send log info to them and they'll sort it all out, similar to Splunk, but not ridiculously priced and no dealing with terrible sales teams. But it gets better - they offer all sorts of ways to define app-specific data and metrics, so you can get KPIs and dashboards just adding a line or two of code here and there. It's a lot easier than running your own system, and it looks like it can make ops a ton easier.
Another hosted service is SumoLogic. I only used them for logging, but it seemed to work well enough.
At my current employer, we are currently implementing Splunk. And it's taking forever and they do charge arm and a leg for their offering. I don't mind if a good product costs money but you shouldn't need a consultant on premise just to configure your logging solution.
I manage a 2.5 TB/day Splunk cluster at my current employer and can offer a few tips for making Splunk less painful to manage:
- Make frequent visits to answers.splunk.com. It has a very active community, and I've frequently been able to type "how do I do X in Splunk" into Google and found multiple answers on Splunk Answers.
- Deployment Server. Make friends with it. In a perfect world, it should hold your configurations for all Indexers, Heavy Forwarders, and Forwarders. If you're having to populate $SPLUNK_HOME/etc/system/local/ yourself, you're doing it wrong.
- Make friends with the "splunk btool config_file_name list --debug" command. That makes it dead simple to know which configuration options a Splunk install is running. Append "| grep -v system/default" on the end of that command to filter out the defaults and you'll more easily see which of your options are being used.
- If you have the cash, attend Splunk Conf and load your schedule up with presentations. It's worth every penny.
Installing and configuring Splunk to ingest and index data is close to being dead-simple.
Splunk consultant maybe needed if you have complex enterprise deployment scenario or wish to develop really advanced apps - but configuring logging solution?
Yeah I don't get it either. Splunk is still losing money after many years. Their model seems to be "spend $2 to make $1". Compared with how arrogant the sales people are - one told me they really just don't care as there's so much demand for the product - something doesn't make sense.
It was cool software, a bit slow (this was several years back). But with things like Elasticsearch catching up release over release (even if you don't use it directly, other platforms will build on it), Splunk is no longer the totally unique thing they used to be. I can't figure out their $8bn+ market cap with revenue under 500M and costs increasing as they grow.
I always have to Google it. http://stackify.com/errors-log-management/ Dunno why it's not more discoverable. Maybe they're doing tests, because it's underpriced IMO. I've not found any service like them - maybe New Relic but that's 10x the price, last I checked. Azure has something they're pushing in their UI, too, but a brief look made it seem nontrivial to implement.
Apart from that, Stackify is $15/server/mo for the server monitoring and app metrics, IIRC.
Just as an example of how awesome Elasticsearch is, you can trivially segment your storage tiers (say, SSD versus HDD) and then easily move older data to other storage, with a single command.
They have a log-specific handler called Logstash, and a dashboard system called Kibana (which is sorta neat but the UI seems a big laggy in my brief experience). Apparently some folks use Logstash/Elasticsearch to record millions and millions of events per day and ES does a great job.
If you want hosted, check out Stackify. I'm totally blown away with the product (no affiliation other than being a new user). You can send log info to them and they'll sort it all out, similar to Splunk, but not ridiculously priced and no dealing with terrible sales teams. But it gets better - they offer all sorts of ways to define app-specific data and metrics, so you can get KPIs and dashboards just adding a line or two of code here and there. It's a lot easier than running your own system, and it looks like it can make ops a ton easier.
Another hosted service is SumoLogic. I only used them for logging, but it seemed to work well enough.