[wmii] moving liblitz on

Hello there.

So, wmii-3 is out, it's features are nice, and it works more or less for most people. That's great. What has been accomplished is immense.

wmii-4's todo list is already big. Writing it was like writing the Christmas wishlist for the next year, yet before this year's christmas eve, I suppose ;) I hope it will still grow, as there is much to do yet.

There was much talk on IRC about sane apps, be it document viewers (»web browsers« too), terminals, mail clients. Also, there are issues with the upcoming tag bars and the current wmiimenu (concerning especially text input and rendering.) Thirdly, there is the issue of UTF-8.
To approach these issues, imo it is immediately neccessary to improve and even restructure liblitz. Iirc it was one goal of the development of liblitz to provide a sane gui library to application programmers. Atm, liblitz is far from that.

Now, with the beginning development of wmii-4 the discussion about the further development of liblitz has to start, as there are loads of non-trivial design decisions.

1) Liblitz should provide a total abstraction from X11 insanity. The applications should not need to access any X library calls, as long as they don't need them for explicitely interacting with the X server. This would make applications easily portable to any platform liblitz is ported to. Linux is not /the/ OS (some say so at least ;)
To accomplish this task, it is necessary to provide a full-featured window abstraction and widget model. It must be easy to use and fulfill all requirements that dwm [dynamic window management ;)] poses upon it. This is firstly: absolute values for the layout of widgets should not be needed as long as it is reasonable. It makes sense for multiline text widgets, but not for single-line text widgets and to buttons only to a limited extent. Then, widgets should be layouted in a dynamic manner too, meaning some more or less extreme geometries (which should be needed/useful for one application) should not make the widgets unusable. On the other hand, in-window frames and sub-windows shoulb be omitted, as they interfere with dwm. The window and widget model should rather encourage to use multiple windows that are layouted by the WM.

2) Text handling is central to any application. On the one hand, text is so ubiqitous that no-one really sees it as something special. On the other hand, text is something so extremely complex that there are is an infinite amount of books about handling it in computers. That's a gap that is not closed in most widget sets. Both, rendering and input lack far behind to what is self-evident on paper! This is because handling text is technically very challenging. For example, many see (or at least they try to repress the thruth they know too well ;) text as a one-to-one mapping between bytes. That is true for ASCII or latin-*, but we won't discuss here that it's not enough. What we really want is Unicode[1]. (NO, I'M SURE WE DON'T WANT TO DISCUSS THIS HERE.) Most probably UTF-8, though it is not the most performant representation form, it is possibly the most rational, at least for us.
Thus, we must handle text as a stream of variable-width characters. No direct access via C-type arrays is possible. No use of all the C standard library string functions, like it is done throughout /all of wmii's source!/
As Unicode is a 21-bit encoding, the amount of characters is gigantic, and this poses extra difficulties. There are a lot of languages that are possibly represented, and the amount of features NEEDED is somewhat overwhelming. Even simple upper-lowercase mappings or algorithms for line-breaking are not trivial anymore! One has to deal with character classes, combining characters, grapheme clusters or whatever. It is absolutely impossible to handle that in C-style arrays for an application programmer. So, liblitz or some other library has to provide that functionality. I don't know of any library that can do that and is not bloated like hell (the only library of that I know that it can do almost everything in the Unicode standard is IBM's ICU[2], the binaries are more than 5 MB large oO, and the source is the incarnation of C Preprocessor). I am convinced that this can be done better.

I began with extending liblitz by implementing the basic functionality of a text widget[3]. It is not possible to proceed without further embedding into liblitz. It is not even near to being finished, not only technically, but also and especially in design. There is also some implementing of some text handling routines. This includes the representation of the text as a gapped array for insertion/deletion of text in Θ(1) most of the time. Maybe it could be used as the default representation of text in liblitz, just for simplicity's sake, but maybe the routines can be rewritten in such a way that they easily work both with gapped and non-gapped arrays. I didn't find a way to accomplish this.

I hereby request some discussion on this topic, and especially about the code I wrote so far.

Greetings,
Denis

[1] http://www.unicode.org
[2] http://www-306.ibm.com/software/globalization/icu/
Open Source, has C- and C++-Bindings, ported to most significant platforms (that's also what Java uses internally)
sloc stats of the source:
cpp: 185897 (54.63%)
ansic: 148004 (43.49%)
sh: 3452 (1.01%)
perl: 2938 (0.86%)
sed: 14 (0.00%)
[3] http://www.ueberl33t.net/tmp/widgets_060506_1146924689.tar.bz2