Re: [dev] [st] [patch] If no Glyph Found, Do Not Keep a Font Loaded

From: Gary Allen Vollink <gary_AT_vollink.com>
Date: Wed, 21 Mar 2018 17:42:07 -0400

Notes to add to what I said above.

1) If X no longer cares if a font remains open, perhaps frc[] can be
changed to a non-array. This would allow frc to continue to be used
for the one-off glyph loads, without the extra maintenance of keeping
track of fonts already opened. This moves the issue from 'look for
fonts if I don't have a glyph loaded' to 'look for fonts every time I
have a non-default glyph', but things will actually work better. I am
NOT advocating this.

2) If the above seems bad, then first changing frc[] to close the
OLDEST alt-font might be a good start (FIFO not LIFO).

3) Not just MacOS. The font ".LastResort" is a default on MacOS, but
LastResort is available anywhere, and at any time someone could
install it on Linux.

  https://unicode.org/policies/lastresortfont_eula.html

This can be a huge problem because once a Fallback font exists within
frc[], then no new rune will ever get matched. Consider this:

default fonts = "Ubuntu Mono"
frc[0] = "LastResort"

If this happens, an attempt to print any glyph that isn't in the
default WILL be matched by LastResort. Effectively, frc[0] satisfies
every search.
Any request for a glyph that isn't present will load the LastResort
font, effectively killing all future fallback searches.

I have NO idea how to fix this without opening the vacuum of space
(wow, a lot of suck). I will note that if frc[] is not an array then
it wouldn't be checked for an already opened font (fixed)! I am still
NOT advocating this.

Fontconfig sucks.

Again, I hope I've helped illuminate the issues. If you have
questions, feel free to ask and I'll try to help.

Thank you,
Gary Allen Vollink

On Wed, Mar 21, 2018 at 4:56 PM, Gary Allen Vollink <gary_AT_vollink.com> wrote:
> Okay, you seem to be right. Before I upgraded my Ubuntu at the end of
> last year I did have this issue there, too. I can assure you that
> this /was/ a crash on Linux, and perhaps you've just had a more recent
> version of the relevant X libraries since I started on this list.
>
> I know - for certain - that loaded screen fonts are still being
> closed, but that is no longer an error condition. It just slows
> things down a tiny bit.
>
> Assuming first that you do not have a font that handles Chinese
> Extended range E, this is my way to recreate.
>
> #Start st:
> #Four default "Source Code Pro" fonts loaded (three are default medium).
> #The array frc[] is empty.
> #The first block of echo statements will fill up frc[16],
> interestingly X simply re-loads the current default font when it
> cannot find a glyph.
>
> echo -e "\xf0\xab\xa0\xa0"
> echo -e "\xf0\xab\xa0\xa1"
> echo -e "\xf0\xab\xa0\xa2"
> echo -e "\xf0\xab\xa0\xa3"
> echo -e "\xf0\xab\xa0\xa4"
> echo -e "\xf0\xab\xa0\xa5"
> echo -e "\xf0\xab\xa0\xa6"
> echo -e "\xf0\xab\xa0\xa7"
> echo -e "\xf0\xab\xa0\xa8"
> echo -e "\xf0\xab\xa0\xa9"
> echo -e "\xf0\xab\xa0\xaa"
> echo -e "\xf0\xab\xa0\xab"
> echo -e "\xf0\xab\xa0\xac"
> echo -e "\xf0\xab\xa0\xad"
> echo -e "\xf0\xab\xa0\xae"
> echo -e "\xf0\xab\xa0\xaf"
>
> #Now frc[] has 16 open copies of "Source Code Pro".
> #
> #The next steps is tricker since everyone has a different fontconfig.
> Find two glyphs that you DO have fonts for, but are not represented in
> your default font. For me, "Place of Interest" works nicely.
>
> echo -e "\xe2\x8c\x98"
>
> # Now frc[15] has closed its unused "Source Code Pro", and instead has
> opened up "Segoe UI Symbol".
> # Since THAT copy of "Source Code Pro" isn't on screen this never caused harm.
> # (it was only held open to keep track of "\xf0\xab\xa0\xaf")
> # That means, though, that printing anything new that doesn't use an
> already open font will close
> # "Segoe UI Symbol" and open that new font.
> # Have some smiley shades
>
> echo -e "\xf0\x9f\x98\x8e"
>
> I hope that makes sense.
>
> Thanks,
> Gary
>
> On Wed, Mar 21, 2018 at 3:35 PM, Hiltjo Posthuma <hiltjo_AT_codemadness.org> wrote:
>> On Wed, Mar 21, 2018 at 02:18:53PM -0400, Gary Allen Vollink wrote:
>>> This patch is over my previous simplification patch
>>> (https://lists.suckless.org/dev/1803/32601.html), though it could be
>>> reworked to go in without it.
>>>
>>> Problem: When working with text from many languages, it does not take long
>>> to run out of font slots for Runes that do not exist within any given
>>> fontconfig. Currently, each rune that is not part of a default set takes up
>>> another font slot (whether or not that rune is even found).I have found that
>>> runes in text (not emoji) have a tendency of showing up in sets, for
>>> instance a whole block of text may be in Cyrillic, which quickly takes up
>>> the 16 slots available for loaded fonts. This is fine until another rune IS
>>> found and put on screen. Whenever that happens, the next time a rune isn't
>>> found that font is closed and a new one inserted (whether or not that font
>>> includes the required glyph). Unloading an on-screen font at the next
>>> not-found rune will crash st.
>>>
>>>
>>> What I have is a partial Fix. This patch makes it take longer for this
>>> problem to manifest by making sure the limited font slots are not kept open
>>> when a glyph cannot be found. Note that this does not really help the issue
>>> on MacOS where the noglyph font will always claim a glyph (even though it's
>>> nothing but a binarybox). I haven't found a way to fix it completely (on
>>> MacOS and/or over long periods of usage on Linux) without writing something
>>> that even I think sucks. I personally got over this by /also/ adjusting the
>>> size of the frc[] array on my own build. Either way, this is what I have to
>>> offer...
>>>
>>>
>>>
>>> Patch:
>>>
>>> Track runes that are not found separately from the Fontcache, frc[], forget
>>> not-found runes in fifo, not lifo. Close an opened font if it doesn't
>>> include the required glyph (not taking up an frc[] slot).
>>>
>>>
>>> diff -U 3 -p a/x.c b/x.c
>>> --- a/x.c 2018-03-21 13:22:53.549236600 -0400
>>> +++ b/x.c 2018-03-21 13:32:00.894134500 -0400
>>> _AT_@ -3,6 +3,7 @@
>>> #include <math.h>
>>> #include <limits.h>
>>> #include <locale.h>
>>> +#include <wchar.h>
>>> #include <signal.h>
>>> #include <sys/select.h>
>>> #include <time.h>
>>> _AT_@ -219,9 +220,12 @@ enum {
>>> typedef struct {
>>> XftFont *font;
>>> int flags;
>>> - Rune unicodep;
>>> } Fontcache;
>>>
>>> +/* Runes not found array. */
>>> +static wchar_t noglyph[1024];
>>> +static int noglyphlen = 0;
>>> +
>>> /* Fontcache is an array now. A new font will be appended to the array. */
>>> static Fontcache frc[16];
>>> static int frclen = 0;
>>> _AT_@ -1145,7 +1149,7 @@ xmakeglyphfontspecs(XftGlyphFontSpec *sp
>>> float runewidth = win.cw;
>>> Rune rune;
>>> FT_UInt glyphidx;
>>> - int i, f, numspecs = 0;
>>> + int i, f, r, numspecs = 0;
>>>
>>> for (i = 0, xp = winx, yp = winy + font->ascent; i < len; ++i) {
>>> /* Fetch rune and mode for current glyph. */
>>> _AT_@ -1177,7 +1181,8 @@ xmakeglyphfontspecs(XftGlyphFontSpec *sp
>>>
>>> /* Lookup character index with default font. */
>>> glyphidx = XftCharIndex(xw.dpy, font->match, rune);
>>> - if (glyphidx) {
>>> + /* OR if already failed to find a glyph for this rune. */
>>> + if ((glyphidx) || (wmemchr(noglyph, rune, noglyphlen))) {
>>> specs[numspecs].font = font->match;
>>> specs[numspecs].glyph = glyphidx;
>>> specs[numspecs].x = (short)xp;
>>> _AT_@ -1193,11 +1198,6 @@ xmakeglyphfontspecs(XftGlyphFontSpec *sp
>>> /* Everything correct. */
>>> if (glyphidx && frc[f].flags == frcflags)
>>> break;
>>> - /* We got a default font for a not found glyph. */
>>> - if (!glyphidx && frc[f].flags == frcflags
>>> - && frc[f].unicodep == rune) {
>>> - break;
>>> - }
>>> }
>>>
>>> /* Nothing was found. Use fontconfig to find matching font.
>>> */
>>> _AT_@ -1208,7 +1208,6 @@ xmakeglyphfontspecs(XftGlyphFontSpec *sp
>>> if (frclen >= LEN(frc)) {
>>> frclen = LEN(frc) - 1;
>>> XftFontClose(xw.dpy, frc[frclen].font);
>>> - frc[frclen].unicodep = 0;
>>> }
>>>
>>> /*
>>> _AT_@ -1224,12 +1223,23 @@ xmakeglyphfontspecs(XftGlyphFontSpec *sp
>>> }
>>> frc[frclen].font = newFont.match;
>>> frc[frclen].flags = frcflags;
>>> - frc[frclen].unicodep = rune;
>>>
>>> glyphidx = XftCharIndex(xw.dpy, frc[frclen].font,
>>> rune);
>>>
>>> - f = frclen;
>>> - frclen++;
>>> + if (!glyphidx) {
>>> + XftFontClose(xw.dpy, frc[frclen].font);
>>> + if (noglyphlen >= LEN(noglyph)) {
>>> + /* Get rid of the oldest not found
>>> rune */
>>> + for (r=1; r <= LEN(noglyph); r++) {
>>> + noglyph[r-1] = noglyph[r];
>>> + }
>>> + noglyphlen = LEN(noglyph) - 1;
>>> + }
>>> + noglyph[noglyphlen++] = rune;
>>> + } else {
>>> + f = frclen;
>>> + frclen++;
>>> + }
>>> }
>>>
>>> specs[numspecs].font = frc[f].font;
>>>
>>>
>>>
>>
>> I've never had this issue and cannot reproduce it all (still). Can you
>> provide more information? Recently there are some issues reported by users
>> using MacOS and st, this makes for wasted debugging sessions: MacOS and it's
>> ecosystem is not supported.
>>
>> There are also some code-style issues in the patch (newFont -> newfont name).
>>
>> --
>> Kind regards,
>> Hiltjo
>>
Received on Wed Mar 21 2018 - 22:42:07 CET

This archive was generated by hypermail 2.3.0 : Wed Mar 21 2018 - 22:48:19 CET