java-user@lucene.apache.org
[Top] [All Lists]

Re: Using Lucene to search log files

Subject: Re: Using Lucene to search log files
From: "Erick Erickson"
Date: Mon, 11 Dec 2006 10:23:59 -0500
Then you really want to look at the classes that do the work with filters if
you require milliseconds. You should be just fine....

On 12/11/06, abdul aleem <janaabdulaleem@xxxxxxxxx> wrote:

Many thanks to All,

well kind of puzzled because ours is a fast moving log
down to Milliseconds :( as we deal with forex on a
financial system.

Im sure there will be workarounds, actually most of
the time it is enough to search within 2 log files of
1-5MB size, coz we are more intersted in seconds and
minutes than hour as we need to get the logs between
minutes

I had regular expressions in Mind before thinking of
Lucene, but i still feel Lucene should be appropriate
to deal with this,

I will search the archives for respective issues, Also
have Lucene in Action book i guess it should help


Many thanks again,
Abdul


--- mark harwood <markharw00d@xxxxxxxxxxx> wrote:

> >>Extend QueryParser to sort this out.
>
> The latest version in SVN has changed the default
> QueryParser behaviour to use RangeFilters instead of
> RangeQuerys
>
>
>
> ----- Original Message ----
> From: Mike Streeton
> <mike.streeton@xxxxxxxxxxxxxxxxxx>
> To: java-user@xxxxxxxxxxxxxxxxx
> Sent: Monday, 11 December, 2006 1:35:47 PM
> Subject: RE: Using Lucene to search log files
>
> I would use a RangeFilter instead of using the
> default Boolean query as
> this will always break at some point with Too many
> Boolean clauses.
> Extend QueryParser to sort this out.  As far as
> extracting information
> from log files I would look at creating yourself a
> LogAnalyzer that can
> interpret the contents and indexing it
> appropriately.
>
> Hope this helps
>
> Mike
>
> www.ardentiasearch.com the home of NetSearch
> -----Original Message-----
> From: abdul aleem [mailto:janaabdulaleem@xxxxxxxxx]
> Sent: 11 December 2006 12:04
> To: java-user@xxxxxxxxxxxxxxxxx
> Subject: Using Lucene to search log files
>
> Hi All,
>
> Im a Lucene newbie,
>
>
> Requirement :
> ==============
> a) Build a log viewer tool, search log files for
> keywords and time stamp
>
> b)  files in production approx 200 logs per day and
> each log file may range from 1MB - 5MB
>
> Lucene
> ========
> We wanted to utilize Lucene's search capabilities
> especially search all 200 log files content quickly
>
> a) Search criteria:
>     i) Timestamp search: Fetch contents between any
> two timestamps
>
>    ii) Fetch log file contents for specified keyword
>
>
>
> Query
> ========
>     a ) Would greatly appreciate if some suggestions
>
>         whether Lucene will be appropriate tool for
> the requirement ??
>
>
>     b) I have tried to use SpanQuery however
> struggling to fetch entire conents e.g. (between two
> timestamps)
>
>     c) I had also looked at
> LargeScaleDateRangeProcessing in the wiki, is that a
> right approach for the requirement
>
>
>
>   Any help / suggestion would be greatly
> appreciated,
>
>
>   Many thanks in advance,
>   Abdul
>
>
>
>
>
________________________________________________________________________
> ____________
> Do you Yahoo!?
> Everyone is raving about the all-new Yahoo! Mail
> beta.
> http://new.mail.yahoo.com
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> For additional commands, e-mail:
> java-user-help@xxxxxxxxxxxxxxxxx
>
>
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> For additional commands, e-mail:
> java-user-help@xxxxxxxxxxxxxxxxx
>
>
>
>
>
>
>
>
___________________________________________________________
>
> Inbox full of spam? Get leading spam protection and
> 1GB storage with All New Yahoo! Mail.
> http://uk.docs.yahoo.com/nowyoucan.html
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> For additional commands, e-mail:
> java-user-help@xxxxxxxxxxxxxxxxx
>
>





____________________________________________________________________________________
Yahoo! Music Unlimited
Access over 1 million songs.
http://music.yahoo.com/unlimited

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx


<Prev in Thread] Current Thread [Next in Thread>