[colug-432] Any kafka users on the list ?
jim at rossberry.com
Sat Nov 22 14:41:45 EST 2014
I'd second Docker or something like it. The potential advantages from
running applications in containers are great enough that the drawbacks
will be addressed, corrected or programmed around.
That said, I still maintain that the preponderance of apps in existence
today will never run in containers (or anything like containers).
Either they will continue to exist as they are today, or new companies
will be formed that base their business model around the advantages
that containers provide and are able to out compete business models
which do not leverage those advantages and the apps will die with the
businesses which use them.
Technology has to enable the business model or its an impedance mis-
match which sucks $$ out of the bottom line.
On Sat, 22 Nov 2014, Chris Embree wrote:
> Hey Tom,
> If I were looking to start something new, I'd look hard into docker. Spark and Storm are neat if you like stream
> processing. Kafka is a niche. It needs to mature or it will be replaced.
> Good luck.
> On Fri, Nov 21, 2014 at 10:56 PM, Tom Hanlon <tom at functionalmedia.com> wrote:
> Thanks Chris,
> As a hadoop instructor of 4 years, i was looking to get into something
> more interesting then the jumble of tools, and focus on one tool.
> Kafka, storm and spark are all candidates, and perhaps docker.
> My plan is to learn one of them well enough to teach it. Kafka and
> Docker and Storm seem to be in demand, but not well served with
> training. Spark is in demand, but some training exists.
> Your summary is useful, I think I will read what documentation is out
> there, and have a look at the code and see if I like it well enough to
> keep learning.
> On Fri, Nov 21, 2014 at 9:42 PM, Chris Embree <cembree at ez-as.net> wrote:
> > 8-o So many questions.
> > List is fine, nothing secret here, just my opinions which are now
> > worth slightly less than you paid for them.... sorry.
> > We run a limited size cluster due to physical limits. That said, it's
> > anywhere from 10 -14 kafka nodes. Each w/ 2 dedicated 10k disks. GC
> > hasn't shown up as an issue so far, but it might be the culprit behind
> > a couple of anomalous issues.
> > Generally, as a cluster is seems somewhat immature. It works well
> > when it works well, otherwise things get ugly.
> > It can use JMX for monitoring, but management tools are somewhat
> > limited. One of smart guys on my team built a tool, dubbed Kurator
> > (play on ES tools) that uses some Python Kafka API's to provide some
> > insight. However, it relies heavily on Zookeeper status and Kafka
> > telling the truth. We've seen a few issues that raise doubt about
> > Kafka's agreement w/ ZK on what's real.
> > HOWEVER: Our use case is extremely abusive. We're looking for 1.2M
> > 1K transactions per second. If you are anywhere south of 100K/s
> > chances are extremely good you can construct a highly reliable Kafka
> > Cluster.
> > On the fence: We've had little luck re-allocating partitions to
> > recover from a lost node. Listing Kafka topics will show you the #
> > and nodes hosting In Sync Replicas (ISR). The re-balance feature is
> > somewhat new and lightly documented, at least at last Google. I've
> > had little luck re-syncing after a node loss.
> > Kafka is a minor part of our solution in the grand scheme of things.
> > I feel ill equipped to give a talk on the subject in any reasonable
> > depth.
> > That said, I will be speaking at Cisco Live Milan (EMEA). I'd be
> > happy to re-present at a COLUG if there is any value. The talk is a 4
> > hour tech session (I'm only 1 of 3 speakers) on the entire openSOC
> > project. My focus will be on the platform side and it may not be the
> > best fit for a LUG. It's more of a HUG topic.
> > FWIW, I absolutely hate the name Hadoop. ;)
> > I hope that helps.
> > Chris
> > On 11/21/14, Tom Hanlon <tom at functionalmedia.com> wrote:
> >> Chris,
> >> Can you talk about it on the list ? If not maybe we can send some
> >> private emails.
> >> How big is the kafka cluster ? How many events are handled?
> >> What are the details of the hiccups ? Java Garbage collection?
> >> Configuration changes ? General strangeness ?
> >> Does it provide any hooks for monitoring or managing? Nagios for
> >> monitoring ? Some api hooks for management ?
> >> Thanks,
> >> Tom
> >> On Fri, Nov 21, 2014 at 1:17 PM, Chris Embree <cembree at ez-as.net> wrote:
> >>> Sadly, yes.
> >>> We're using Kafka as the buffering queue for OpenSOC (getopensoc.com)
> >>> and while it works well when things are fine, it has significant
> >>> difficulty recovering from hiccups.
> >>> Also, there are few tools for managing it from an Admin point of view.
> >>> Deleting a topic is a non-trivial task, for example.
> >>> Chris
> >>> On 11/21/14, Tom Hanlon <tom at functionalmedia.com> wrote:
> >>>> Colug,
> >>>> Are there any kafka users on this list.
> >>>> http://kafka.apache.org/
> >>>> I am looking to dive into kafka and some use-case, war-story,
> >>>> discussion with a user would be helpful.
> >>>> If there is broader interest perhaps we can make a meeting
> >>>> presentation out of it.
> >>>> Thanks,
> >>>> Tom
> >>>> _______________________________________________
> >>>> colug-432 mailing list
> >>>> colug-432 at colug.net
> >>>> http://lists.colug.net/mailman/listinfo/colug-432
> >>> _______________________________________________
> >>> colug-432 mailing list
> >>> colug-432 at colug.net
> >>> http://lists.colug.net/mailman/listinfo/colug-432
> >> _______________________________________________
> >> colug-432 mailing list
> >> colug-432 at colug.net
> >> http://lists.colug.net/mailman/listinfo/colug-432
> > _______________________________________________
> > colug-432 mailing list
> > colug-432 at colug.net
> > http://lists.colug.net/mailman/listinfo/colug-432
> colug-432 mailing list
> colug-432 at colug.net
Jim Wildman, CISSP, RHCE jim at rossberry.com http://www.rossberry.net
"Society in every state is a blessing, but Government, even in its best
state, is a necessary evil; in its worst state, an intolerable one."
More information about the colug-432