[colug-432] named pipe vs tail -F /path/to/file

[Travis Sidelinger]

Other ideas:

1. What about having haproxy send to multiple syslog servers?  One for
logging and one for processing?

2. If rotating the log files causes it to restart, then don't rotate.
Either remove the log file or send them to /dev/null.  Let your program
that watches the pipe do the logging.

3. You could write your own syslog daemon that captures the syslog stream,
handles the logging, and does the processing of log files.  With off the
shelf modules, it's probably not that complicated.

4. Another high end method for what you are doing is to have your logging
daemon break the logs apart and insert then in a structured SQL database.
Then of course your methods for processing, alerting, reports, and rotating
will be completely different.

~Travis Sidelinger



On Thu, Aug 2, 2012 at 11:32 PM, Jeff Frontz <jeff.frontz at gmail.com> wrote:

> I'm not sure I completely understand the intent/implementation but
> wouldn't this work?
>
> while true
> do
>     processPipeContents < namedPipe
> done
>
> The alternative would be to (if I remember the FIFO semantics) write
> something that opens the FIFO in non-blocking mode and then poll/select it.
>
> Jeff
>
> On Aug 2, 2012, at 19:50, Rick Hornsby <richardjhornsby at gmail.com> wrote:
>
> > Curious about opinions/thoughts.  I have haproxy logging activity via
> > syslog.  HAproxy logs contain lots of useful information about HTTP
> > transactions (request, timing, result code, etc).  I'd like to process
> > the logs in real time, so I set up syslog to not only write the log to
> > a file, but also to a named pipe, /tmp/haproxy.log.pipe.  I figured
> > this would be the ideal way to peel off and process log data as it
> > happened (using awk, etc) - realtime.
> >
> > Except for one thing - when syslog restarts for logrotate, the pipe is
> > broken for a moment.  Anything reading from the pipe dies, because
> > that's how pipes work.  So the script that was processing the log data
> > coming off the pipe goes away.
> >
> > One alternative is to tail -F /var/log/haproxy.log |
> > /path/to/processor, but boy that just seems hackish - and not in the
> > good way.
> >
> > Is a named pipe the right tool?  Does it make sense to somehow trap
> > the signal that is causing the script to bail?  It isn't critical that
> > every single log event be processed, but we want to make a best effort
> > so that we're not losing information.
> > _______________________________________________
> > colug-432 mailing list
> > colug-432 at colug.net
> > http://lists.colug.net/mailman/listinfo/colug-432
>
> _______________________________________________
> colug-432 mailing list
> colug-432 at colug.net
> http://lists.colug.net/mailman/listinfo/colug-432
>



-- 
"A careful reading of history clearly demonstrates ...
that people don't read history carefully.”

“We can't solve problems by using the same kind of thinking we used when we
created them.”
—Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.colug.net/pipermail/colug-432/attachments/20120803/de1b0388/attachment.html