Sunday, September 29, 2013

New Project - Splunk Replacement

So... I love, love, love Splunk, but it's really freakin' expensive.  I think it's time to start looking for a free and open replacement.

Near as I can tell, we'd count as a major/large installation, size-wise.  We're talking about at least a 7-digit number of nodes contributing log events, all in active use, each generating Megabytes of data a day. Hold on a second, lemme do some math...

<thinking/>

Holy Schnikeys! If I'm not wrong in my volume estimates, I need to collect, store and index more than a Petabyte/Day if I want complete visibility into the system.  That can't be right...

In any event, for anyone following along, there's a new meme in town.  Previously I used this blog as a place to dump links for the Cassandra and DevOps work we were doing, now I'll do the same thing for the upcoming Splunk work.

Happy Reading.

Thursday, March 1, 2012

I had a conversation with my brother who runs prod ops over at <CENSORED> and we were discussing their use of EC2.  He mentioned that they don't currently use SQS to coordinate activities and instead they use a system based on a relational database for messaging and cited cost as a potential rationale.  I was curious about the break-even point so I made a little spreadsheet to summarize it.  You can find it here:

https://docs.google.com/spreadsheet/ccc?key=0Av3PKINeqTSBdF90Mk1ja2VXMExYc3JXVEJCUlY5TlE

If you assume you need 4 quadruple-high-memory instances it looks like you're better off using SQS  if your message rate is less than 8,000,000/hour. 

Wednesday, December 7, 2011

Myths Rumors Fud Hate NoSQL Cassandra vs hbase : Edward Capriolo

Myths Rumors Fud Hate NoSQL Cassandra vs hbase : Edward Capriolo:

Cassandra vs HBase | WhyNosql

Cassandra vs HBase | WhyNosql:

Adku: HBase vs Cassandra

Adku: HBase vs Cassandra:

HBase vs Cassandra: why we moved « Dominic Williams

HBase vs Cassandra: why we moved « Dominic Williams:

The Netflix Tech Blog: Introducing Curator - The Netflix ZooKeeper Library

The Netflix Tech Blog: Introducing Curator - The Netflix ZooKeeper Library:

Keeping Movies Running Amid Thunderstorms!

Keeping Movies Running Amid Thunderstorms!:

Apache Kafka

Me Likey.

Apache Kafka: