Re: cygwin's arabic support

[Nadim Shaikli <shaikli at yahoo dot com>]

--- Ahmed Ghoneim <ahmed.ghoneim - gmail.com> wrote:
> On 7/24/07, Nadim Shaikli <shaikli - yahoo.com> wrote:
> > Mr. Ghoneim/khalifa/et al, how about it ?
> 
> Sure! It's a big project, but we can rely on the expertise around
> here for support.
> 
> Personally, I don't have much experience with cygwin, as I only
> use linux or the native mingw on windows. But i understand the
> project's importance as a starting point for windows users.

Wonderful - I hope others will also join in this effort to not only
test things out and report back but to actually join in the development.
I've made some progress (well discoveries) that might get us going
sooner than I thought - yes that's reason for excitement :-)

As almost everyone is now well aware - to properly support Arabic one
needs a couple of key ingredients,

 A. UTF-8 support
 B. Bidirectional (Bidi) support
 C. Proper fonts and handling there-of

So I started looking at cygwin's UTF-8 support to see what's out there
and if others have gone down this path and to my surprise I found out
that _many_ people have in the past and that some have even submitted
patches to the cygwin project which unfortunately so-far have been
rejected not due to the concept but due to approach/code-style (more
on this later).  So to make a long story short there very much is a
working UTF-8 cygwin model as is noted here,

  http://www.okisoft.co.jp/esc/utf8-cygwin

So let me give a step-by-step instructional to note what I've done
so far (to get others moving along as well) and an action plan to
see how we can progress.

 1. Do a "normal/proper" cygwin install from cygwin.com [1] - make
    sure you install the necessary X libraries and applications so
    you can start xterms
 2. Make sure you have an Arabic UTF-8 file handy.  You can grab this
    sample from awhile back for testing purposes,
      http://arabeyes.org/~nadim/ar_utf8
 4. See the mess we have _today_ with regard to cygwin and Arabic and
    try to view the ar_utf8 file using less/more/cat/vim/emacs/etc
      % less ar_utf8
 5. Let's get cygwin to recognize utf-8, so shut everything down and
    replace 'cygwin1.dll' with Suzuki's version (.jp link above).
 6. Download arabeyes' fixed-width/mono-spaced fonts [2] and copy the
    10x21.pcf within cgywin's install directory in
      /usr/X11R6/lib/X11/fonts/misc
    (you might want to gzip it) and then run the following command
    for the applications to find the font
      % xset +fp /usr/X11R6/lib/X11/fonts/misc
 7. Now let's start an xterm that is unicode aware using our font
    (you can include these command-line options within your .Xdefaults
    if you so desire and xrdb them into affect)
      % xterm -u8 \
              -fn
-arabeyes-fixed-medium-r-normal--20-200-75-75-c-100-iso10646-1
    with this new xterm repeat step #4 and you should see Arabic
    non-joined, non-bidi'ed characters - wooohooo !!
    a. For vim (shaped results) make sure to do,
       :set encoding=utf-8
       :set arabic
 8. Celebrate and jump for joy -- we're 60% there :-)

So what needs to happen you ask ?  Well, now the fun part begins and
this will require a bit of elbow grease from our end (ie. development,
compilation, packaging, etc) and a bit of talking and communicating
with the cygwin developers (both of which we, thankfully, have
experience with within arabeyes).

Here is what I've been able to come-up with (a TODO list),

 1. We need to get various Bidi libraries running natively within
    cygwin.  Let's start with porting Fribidi [3] as that is the
    base for a lot of applications out there.  Once fribidi is
    compiled and available we should be able to see proper Bidi'ed
    Arabic files (some applications, like vim, etc do their own
    shaping so we can leverage that).  In passing, we need to
    continue asking Fribidi developers to fold-in the shaping
    functions.
 2. We need to compile/port mlterm to run natively (with fribidi
    linked-in) so we can run 'less' or 'cat' on Arabic files and
    see their actual contents - bidi and all.  mlterm brings a great
    amount of functionality to the table so its a must have.
 3. Communicate with the cygwin community on what we can do to get
    the UTF-8 patch incorporated into the cgywin main trunk (ie. for
    it to simply be offered natively with every download).  This might
    require us to recode and/or to rework the patch which might mean
    talking and working with Mr. Sukuzi if he still has interest.
 4. We'll have to ask the cygwin community to incorporate into their
    various repositories our 10x21 font as well as fribidi, mlterm and
    any other applications we deem useful for our Bidi and Arabic needs.
    Having more Arabic font options here would also be nice.
 5. We'll need to go through the various applications cygwin already
    has in their repository to Bidi'fy them (for instance fvwm, the
    window manager, should be recompiled to include fribidi as a library).
    This step is not mandatory but it nicely complete what we need to
    accomplish.

I've been asked about completion expectations before so here is my
opinion on how this will inshalla progress (given enough support,
developers and testers of course),

 Steps complete   Percentage done   Potential Effort   Time estimate
 ---------------+-----------------+------------------+---------------
        1                 70%        Easy/Moderate        1  week
        2                 80%        Easy/Moderate        2  weeks
        3                 90%          Very Hard         2-3 months
        4                 95%            Hard             1  month
        5                100%            Hard             1  month

As is always the case, the more difficult parts are the convince
jobs that need to take place but if we've done our homework right it
should be relatively easy (the more we communicate with cygwin and
the more we cater to their requests the easier all of this will be).
Keep in mind that its always hard to convince people to include
functions/files/libraries into their applications or repositories
which they, on average, have no usage for.

Hope all of this makes sense (report back if my outlined steps don't
work).  If there is interest (and I sure hope so since this now seems
more doable than ever), I can start a cygwin sub-project on arabeyes
(should we ?) to track things and to simplify newbie introductions.

Mr. Ghoneim, can you please take care of #1/#2 above ?  You have the
proper required win32 compilations experience (it will probably also
take you a bit to get cygwin's X running since you've never done it
though its rather simple).  If you'd like to talk and/or have a quickie
howto on IRC, let me know.

I'll try to send a longer email next time around ;-)

BTW: in replies do please only cite relevant portions of emails.

[1] http://cygwin.com/setup.exe
[2] http://downloads.sourceforge.net/arabeyes/ae_fonts_mono.tar.bz2
[3] http://fribidi.freedesktop.org/wiki/

Salam.

 - Nadim



       
____________________________________________________________________________________
Be a better Heartthrob. Get better relationship answers from someone who knows. Yahoo! Answers - Check it out. 
http://answers.yahoo.com/dir/?link=list&sid=396545433