UTF-8 search engines

[Nadim Shaikli <shaikli at yahoo dot com>]

This is not so much a development note/request but more
of a practical investigate collective to raise everyone's
awareness (in case anyone needs it) - I've spent some
time on this and thought I'd share some of my findings
and opinions.

A bit of background - Arabeyes is using swish-e for its
web/mailing-list searching and since we don't have (yet)
lots of Arabic content per se (this is slated to change
soon) we never really needed UTF-8 support for indexing,
searching, etc.  I recently started looking into what
is out there for us to use which resulted in,

 + Swish-e (http://www.swish-e.org)
   - NO UTF-8 support (needs to be added ASAP)
   - Fast, VERY flexible
   - Used in many serious sites
   - Very active development

 + Namazu (http://www.namazu.org)
   - Notes UTF-8 support
   - Used by GNU's mailing-lists
   - Development stopped ?

 + Mnogosearch (http://www.mnogosearch.org)
   - Notes UTF-8 support
   - Various database storage support (mysql)
   - Used by debian.org among lots of others
   - Active development

There are heaps of other search engines but none seem to
support UTF-8 and that is the crux of my email.  I would
_love_ to see Swish-e seriously moving in that direction
(I guess I'm biased and like swish-e the most).  My various
probes in that regard got answered with "Swish-e will need
a very large scale rewrite" yet the author was open to
patches/discussions, etc (I don't think its high on his
priority list though and we need to change that by, at a
min, demanding for it).

The point here is the more options we have the better off
everyone is.  UTF-8 support doesn't mean Arabic only either
and so it would be wise to band with the various non-ASCII
(8-bit) languages to form a united front to push UTF-8
support on the most basic and useful utilities including,
in this instance, search engines.

Any thoughts/experiances/suggestions ?

Salam.

 - Nadim


__________________________________
Do you Yahoo!?
Yahoo! Search - Find what you’re looking for faster
http://search.yahoo.com