[colug-432] Any kafka users on the list ?
Tom Hanlon
tom at functionalmedia.com
Sat Nov 22 20:06:53 EST 2014
I agree that docker is neat and the company focussed around docker has
yet to implement training.
It would be fun to focus on a technology that is more technology and
less shims and splices.
It is funny to watch the uses of cgroups and lxc and whatever else in
the kernel enables containers, and then see folks get excited about
the use, but are unaware of the underlying technology. I am no master
of linux internals, but at least when I see something like YARN, or
MESOS or docker I take a quick look under the hood.
Maybe I will run kafka in docker...;-) Just kidding.
--
Tom
On Sat, Nov 22, 2014 at 2:41 PM, Jim Wildman <jim at rossberry.com> wrote:
> I'd second Docker or something like it. The potential advantages from
> running applications in containers are great enough that the drawbacks
> will be addressed, corrected or programmed around.
>
> That said, I still maintain that the preponderance of apps in existence
> today will never run in containers (or anything like containers). Either
> they will continue to exist as they are today, or new companies
> will be formed that base their business model around the advantages
> that containers provide and are able to out compete business models
> which do not leverage those advantages and the apps will die with the
> businesses which use them.
>
> Technology has to enable the business model or its an impedance mis-
> match which sucks $$ out of the bottom line.
>
>
> On Sat, 22 Nov 2014, Chris Embree wrote:
>
>> Hey Tom,
>> If I were looking to start something new, I'd look hard into docker.
>> Spark and Storm are neat if you like stream
>> processing. Kafka is a niche. It needs to mature or it will be replaced.
>>
>> $0.02.
>>
>> Good luck.
>>
>> On Fri, Nov 21, 2014 at 10:56 PM, Tom Hanlon <tom at functionalmedia.com>
>> wrote:
>> Thanks Chris,
>>
>> As a hadoop instructor of 4 years, i was looking to get into
>> something
>> more interesting then the jumble of tools, and focus on one tool.
>>
>> Kafka, storm and spark are all candidates, and perhaps docker.
>>
>> My plan is to learn one of them well enough to teach it. Kafka and
>> Docker and Storm seem to be in demand, but not well served with
>> training. Spark is in demand, but some training exists.
>>
>> Your summary is useful, I think I will read what documentation is
>> out
>> there, and have a look at the code and see if I like it well enough
>> to
>> keep learning.
>>
>> --
>> Tom
>>
>>
>>
>> On Fri, Nov 21, 2014 at 9:42 PM, Chris Embree <cembree at ez-as.net>
>> wrote:
>> > 8-o So many questions.
>> >
>> > List is fine, nothing secret here, just my opinions which are now
>> > worth slightly less than you paid for them.... sorry.
>> >
>> > We run a limited size cluster due to physical limits. That said,
>> it's
>> > anywhere from 10 -14 kafka nodes. Each w/ 2 dedicated 10k disks.
>> GC
>> > hasn't shown up as an issue so far, but it might be the culprit
>> behind
>> > a couple of anomalous issues.
>> >
>> > Generally, as a cluster is seems somewhat immature. It works well
>> > when it works well, otherwise things get ugly.
>> >
>> > It can use JMX for monitoring, but management tools are somewhat
>> > limited. One of smart guys on my team built a tool, dubbed
>> Kurator
>> > (play on ES tools) that uses some Python Kafka API's to provide
>> some
>> > insight. However, it relies heavily on Zookeeper status and Kafka
>> > telling the truth. We've seen a few issues that raise doubt about
>> > Kafka's agreement w/ ZK on what's real.
>> >
>> > HOWEVER: Our use case is extremely abusive. We're looking for
>> 1.2M
>> > 1K transactions per second. If you are anywhere south of 100K/s
>> > chances are extremely good you can construct a highly reliable
>> Kafka
>> > Cluster.
>> >
>> > On the fence: We've had little luck re-allocating partitions to
>> > recover from a lost node. Listing Kafka topics will show you the
>> #
>> > and nodes hosting In Sync Replicas (ISR). The re-balance
>> feature is
>> > somewhat new and lightly documented, at least at last Google.
>> I've
>> > had little luck re-syncing after a node loss.
>> >
>> > Kafka is a minor part of our solution in the grand scheme of
>> things.
>> > I feel ill equipped to give a talk on the subject in any
>> reasonable
>> > depth.
>> >
>> > That said, I will be speaking at Cisco Live Milan (EMEA). I'd be
>> > happy to re-present at a COLUG if there is any value. The talk is
>> a 4
>> > hour tech session (I'm only 1 of 3 speakers) on the entire openSOC
>> > project. My focus will be on the platform side and it may not be
>> the
>> > best fit for a LUG. It's more of a HUG topic.
>> >
>> > FWIW, I absolutely hate the name Hadoop. ;)
>> >
>> > I hope that helps.
>> >
>> > Chris
>> >
>> > On 11/21/14, Tom Hanlon <tom at functionalmedia.com> wrote:
>> >> Chris,
>> >>
>> >> Can you talk about it on the list ? If not maybe we can send some
>> >> private emails.
>> >>
>> >> How big is the kafka cluster ? How many events are handled?
>> >>
>> >> What are the details of the hiccups ? Java Garbage collection?
>> >> Configuration changes ? General strangeness ?
>> >>
>> >> Does it provide any hooks for monitoring or managing? Nagios for
>> >> monitoring ? Some api hooks for management ?
>> >>
>> >> Thanks,
>> >> Tom
>> >>
>> >>
>> >> On Fri, Nov 21, 2014 at 1:17 PM, Chris Embree <cembree at ez-as.net>
>> wrote:
>> >>> Sadly, yes.
>> >>>
>> >>> We're using Kafka as the buffering queue for OpenSOC
>> (getopensoc.com)
>> >>> and while it works well when things are fine, it has significant
>> >>> difficulty recovering from hiccups.
>> >>>
>> >>> Also, there are few tools for managing it from an Admin point of
>> view.
>> >>> Deleting a topic is a non-trivial task, for example.
>> >>>
>> >>> Chris
>> >>>
>> >>> On 11/21/14, Tom Hanlon <tom at functionalmedia.com> wrote:
>> >>>> Colug,
>> >>>>
>> >>>> Are there any kafka users on this list.
>> >>>>
>> >>>> http://kafka.apache.org/
>> >>>>
>> >>>> I am looking to dive into kafka and some use-case, war-story,
>> >>>> discussion with a user would be helpful.
>> >>>>
>> >>>> If there is broader interest perhaps we can make a meeting
>> >>>> presentation out of it.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Tom
>> >>>> _______________________________________________
>> >>>> colug-432 mailing list
>> >>>> colug-432 at colug.net
>> >>>> http://lists.colug.net/mailman/listinfo/colug-432
>> >>>>
>> >>> _______________________________________________
>> >>> colug-432 mailing list
>> >>> colug-432 at colug.net
>> >>> http://lists.colug.net/mailman/listinfo/colug-432
>> >> _______________________________________________
>> >> colug-432 mailing list
>> >> colug-432 at colug.net
>> >> http://lists.colug.net/mailman/listinfo/colug-432
>> >>
>> > _______________________________________________
>> > colug-432 mailing list
>> > colug-432 at colug.net
>> > http://lists.colug.net/mailman/listinfo/colug-432
>> _______________________________________________
>> colug-432 mailing list
>> colug-432 at colug.net
>> http://lists.colug.net/mailman/listinfo/colug-432
>>
>>
>>
>>
>
> ----------------------------------------------------------------------
> Jim Wildman, CISSP, RHCE jim at rossberry.com http://www.rossberry.net
> "Society in every state is a blessing, but Government, even in its best
> state, is a necessary evil; in its worst state, an intolerable one."
> Thomas Paine
> _______________________________________________
> colug-432 mailing list
> colug-432 at colug.net
> http://lists.colug.net/mailman/listinfo/colug-432
>
More information about the colug-432
mailing list