Q: performance benchmarks

I have explained this here: https://github.com/JimmyLefevre/kb/blob/970609b3f7c6913c3e10929aa816dcf78deda690/kb_text_shape.h#L289

Although these are just my numbers. I would love for more people to benchmark the library and share their results.

Your'e opening the font once to shape a single string. This is the worst possible case for kbts.

But, I don't do the timing until after loading the font? I only do the timing of the text shaping? Or am I still missing something? (quite possible...)

static void TimeSegmentation(const char* name, uint32_t* codepoints, uint32_t num_codepoints)
{
    uint64_t tstart = GetTime();

    uint32_t num_glyphs = SegmentText(codepoints, num_codepoints);

    uint64_t tend = GetTime();
    float t = (tend - tstart)/1000.0f;
    printf("%s: Segmented %u codepoints into %u glyphs in %.3fms (%.3f ms / glyph)\n", name, num_codepoints, num_glyphs, (tend - tstart)/1000.0f, t/num_codepoints);
}

Also, the example ShapeText function is written in a convenient way, but, as a result, it allocates all over the place

Apart from the global glyph buffer, which are the parts I should refactor to make use of "best practices" (as opposed to "convenient example code")? Do you have another piece of code one could look at for inspiration?

Regarding the note you linked to, this line does stand out, given my different results between NotoSans-Regular.ttf and NotoSansArabic-Regulat.ttf:

Font files can be arbitrarily complex, making it difficult to predict system behavior at runtime.

However, I am still curious why the timings of the actual shaping differs so much when using the Arabic version (as seen in the OP). Does your benchmarks behave the same (4-5x faster) compared to Harfbuzz, even when using NotoSansArabic-Regulat.ttf?

You're right, sorry. I'm being scatterbrained right now.

We weren't doing binary searches for glyph tables yet, which indeed becomes a problem with large fonts.

This is fixed in https://github.com/JimmyLefevre/kb/commit/5d397d89f24b4c67be0d05d5d8422bae81e14866

Here is what I get from your test script with the new version:

Using font NotoSans-Regular.ttf
hello: Segmented 11 codepoints into 11 glyphs in 0.056ms (0.005 ms / glyph)
lorem: Segmented 309 codepoints into 309 glyphs in 0.157ms (0.001 ms / glyph)
lorem arabic: Segmented 671 codepoints into 671 glyphs in 0.193ms (0.000 ms / glyph)

Using font NotoSansArabic-Regular.ttf
hello: Segmented 11 codepoints into 11 glyphs in 0.034ms (0.003 ms / glyph)
lorem: Segmented 309 codepoints into 309 glyphs in 0.130ms (0.000 ms / glyph)
lorem arabic: Segmented 671 codepoints into 893 glyphs in 0.276ms (0.000 ms / glyph)

Your'e opening the font once to shape a single string. This is the worst possible case for kbts. The whole point of this library is that opening fonts is slow and shaping is fast. Also, the example ShapeText function is written in a convenient way, but, as a result, it allocates all over the place. The API is set up so you only have to allocate once.

Comparing only shaping time against Harfbuzz on their own test suite (+ a few of my own), this is what I get on my AMD Ryzen 9 5900HX:

We are ~4.5x faster on average.

We are now ~4.6x faster on average..!

Hi!

I was wondering if you had any available benchmark numbers comparing with libraqm/harfbuzz?

In my current due diligence, for libraqm I'm using this example (by modifying their unit test to do the same):

static void DoTest(const char* name, raqm_t* rq, FT_Face face, const uint32_t* text, uint32_t len)
{
    raqm_clear_contents(rq);

    uint64_t tstart = GetTime();

    assert (raqm_set_text(rq, text, len));
    assert (raqm_set_freetype_face (rq, face));

    if (len < 600)
      raqm_set_par_direction (rq, RAQM_DIRECTION_LTR);
    else
      raqm_set_par_direction (rq, RAQM_DIRECTION_RTL);

    assert (raqm_layout (rq));

    size_t count;
    raqm_glyph_t *glyphs = raqm_get_glyphs (rq, &count);
    uint64_t tend = GetTime();

    // printf("glyph count: %zu\n", count);
    // for (size_t i = 0; i < count; i++)
    // {
    //     printf ("gid#%d off: (%d, %d) adv: (%d, %d) idx: %d\n",
    //             glyphs[i].index,
    //             glyphs[i].x_offset,
    //             glyphs[i].y_offset,
    //             glyphs[i].x_advance,
    //             glyphs[i].y_advance,
    //             glyphs[i].cluster);
    // }

    float t = (tend - tstart)/1000.0f;
    printf("RAQM: %s: Segmented %u codepoints into %zu glyphs in %.3fms (%.4f ms / glyph)\n", name, len, count, (tend - tstart)/1000.0f, t/len);
}

And I'm using your suggested SegmentText from the Readme.md. (I've attached the test app in the zip)

The numbers I get are a fair bit slower than the raqm/harfbuzz test, so I'm curious if I'm perhaps missing something obvious in my setup? Are the numbers what you'd expect compared to your benchmarks?

Thanks for any info! /Mathias

• The test text is the first paragraph of Lorem Ipsum, 309 codepoints
• Results are milliseconds (and milliseconds/glyph)
• Font is NotoSansArabic-Regulat.ttf

EDIT: By using `NotoSans-Regulat.ttf` the timing becomes a lot faster:

Segmented 309 codepoints into 309 glyphs in 0.293ms

Still, about 2x the time from raqm/harfbuzz.

The KB test app (contains both fonts for easy testing)

bench-kb.zip

May I see the rest of the raqm numbers, for reference's sake?

You're right, sorry. I'm being scatterbrained right now.

Heh, no worries!

We weren't doing binary searches for glyph tables yet, which indeed becomes a problem with large fonts. This is fixed in https://github.com/JimmyLefevre/kb/commit/5d397d89f24b4c67be0d05d5d8422bae81e14866

Thanks! This makes me happy, as that was the main concern left on that front :)

We are now ~4.6x faster on average..!

Nice! Still being puzzled by my local benchmarks though, as one is very slightly faster, and the other is ~20% slower 🤔 But for my use case, I consider them equal enough for me to be happy. (i..e not having to consider harfbuzz et al)

Before (for completeness):

After (with your the fix):

In either case, I think the build/memory/runtime part of my due diligence favors kb_text_shape as the winner. Now I also have to finish integrating it into our engine for a final look, to actually verify it.

Thanks.

There's a large difference between my numbers and yours for Arabic, which probably comes down to me removing calls to malloc and realloc from your example code, if I had to guess?

In my last set of numbers, I've replaced it with a global buffer with room for 1000 glyphs (as I know I only need 884 of them), so it shouldn't be the allocation that does it. :/

Perhaps it's an arm vs amd/intel thing, or memory alignment etc... 🤔

So you get an idea, for Roboto-Regular.ttf, kb seems to be 3x slower than hb:

https://github.com/harfbuzz/harfbuzz/pull/5398#issuecomment-3109451500

Here is my current assessment:

https://github.com/harfbuzz/harfbuzz/discussions/5375#discussioncomment-13868590

I think what happens in your testing is this:

• kb initializes all its accelerator structures upon loading the font. Then shaping doesn't change the caches.
• hb initializes all caches lazily, upon first use; ie. first shape.

So, if you open the font with both, then measure the first shape call, then it's apples to oranges. Only second and subsequent shape calls will be apples to apples, which is what my harness does.

HI @behdad

hb initializes all caches lazily, upon first use; ie. first shape

So, to actually hide the cost of loading the font in HB, one has to shape a dummy text at that time? Will any text do, or is it dependent on the final text?

So, to actually hide the cost of loading the font in HB, one has to shape a dummy text at that time? Will any text do, or is it dependent on the final text?

That's not why we do it. For example, if you shape Arabic only, it only produces accelerators for the Arabic lookups. All other scripts' lookups remain unaccelerated. This is a huge memory and speed win with fonts that support many scripts.

Or for example, if you don't shape vertical text, the caches regarding vertical glyph advances & origin will never be allocated.

In fact, IIRC, only lookups accessed during shaping get their accelerator allocated. So, for example, if a font has many auxiliary features (ss01, ss02, dlig, etc) that are not enabled by default, they won't get their cache created, saving time and memory.

It's not to hide, it's to pay for what you use.

Thanks for the very good info! Btw, I haven't gone through all the HB docs yet, but this part should be well suited in the docs as well?

It's not to hide, it's to pay for what you use.

Well, on my end I still have to think about it. I.e I'd prefer no unnecessary hitches on the first frame displayed. I'd prefer to hide that during the loading of my content. But your info does inform my thinking some more, and I may change my setup a bit to make it possible.

Thanks for the very good info! Btw, I haven't gone through all the HB docs yet, but this part should be well suited in the docs as well?

I noted it here: https://github.com/harfbuzz/harfbuzz/discussions/5397#discussioncomment-13886102

It's not to hide, it's to pay for what you use.

Well, on my end I still have to think about it. I.e I'd prefer no unnecessary hitches on the first frame displayed. I'd prefer to hide that during the loading of my content. But your info does inform my thinking some more, and I may change my setup a bit to make it possible.

I suggest you try without any extra steps first. It's extremely lean, definitely not frame-breaking. See the numbers from the Font loading performance in the report for example. Roboto time to first use is 23us.

If you like to read more about HB caches, I wrote those down recently: https://docs.google.com/document/d/1_VgObf6Je0J8byMLsi7HCQHnKo2emGnx_ib_sHo-bt4/view

In my testing, kb is slightly faster in the heavy Arabic fonts test cases (NotoNastaliqUrdu / Amiri), but not by the margin you see. Same about Devanagari.

Can you please share exactly how you are measuring this? I tried your script from https://x.com/jimmy__lefevre/status/2001138908998910192 but it crashes since there's a bug that text is advanced twice in the kb shape() branch of the code. So it clearly cannot be the final script you used.

Even if I fix that, I still see much less of a difference. kb:

----------------------------------------------------------------------------------------------------------
Benchmark                                                                Time             CPU   Iterations
----------------------------------------------------------------------------------------------------------
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-thelittleprince.txt/ot       80.0 ms         80.0 ms            9
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-words.txt/ot                 72.1 ms         72.1 ms           10
BM_Shape/Amiri-Regular.ttf/fa-thelittleprince.txt/ot                  51.2 ms         51.2 ms           14
BM_Shape/NotoSansDevanagari-Regular.ttf/hi-words.txt/ot               27.4 ms         27.4 ms           25
BM_Shape/Roboto-Regular.ttf/en-thelittleprince.txt/ot                 28.2 ms         28.2 ms           26
BM_Shape/Roboto-Regular.ttf/en-words.txt/ot                           32.1 ms         32.1 ms           22
BM_Shape/Roboto-Regular.ttf/react-dom.txt/ot                           310 ms          310 ms            2

hb:

----------------------------------------------------------------------------------------------------------
Benchmark                                                                Time             CPU   Iterations
----------------------------------------------------------------------------------------------------------
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-thelittleprince.txt/ot       87.7 ms         87.6 ms            8
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-words.txt/ot                 98.1 ms         98.1 ms            7
BM_Shape/Amiri-Regular.ttf/fa-thelittleprince.txt/ot                  41.0 ms         41.0 ms           17
BM_Shape/NotoSansDevanagari-Regular.ttf/hi-words.txt/ot               23.4 ms         23.4 ms           30
BM_Shape/Roboto-Regular.ttf/en-thelittleprince.txt/ot                 6.11 ms         6.11 ms          114
BM_Shape/Roboto-Regular.ttf/en-words.txt/ot                           8.71 ms         8.71 ms           80
BM_Shape/Roboto-Regular.ttf/react-dom.txt/ot                          67.6 ms         67.6 ms           10

This is on a Mac M1. I'll test on AMD when I'm back home.

Also curious how you have configured and built HB.

In my testing, kb is slightly faster in the heavy Arabic fonts test cases (NotoNastaliqUrdu / Amiri), but not by the margin you see. Same about Devanagari.

Can you please share exactly how you are measuring this? I tried your script from https://x.com/jimmy__lefevre/status/2001138908998910192 but it crashes since there's a bug that text is advanced twice in the kb shape() branch of the code. So it clearly cannot be the final script you used.

Even if I fix that, I still see much less of a difference. kb:

----------------------------------------------------------------------------------------------------------
Benchmark                                                                Time             CPU   Iterations
----------------------------------------------------------------------------------------------------------
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-thelittleprince.txt/ot       80.0 ms         80.0 ms            9
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-words.txt/ot                 72.1 ms         72.1 ms           10
BM_Shape/Amiri-Regular.ttf/fa-thelittleprince.txt/ot                  51.2 ms         51.2 ms           14
BM_Shape/NotoSansDevanagari-Regular.ttf/hi-words.txt/ot               27.4 ms         27.4 ms           25
BM_Shape/Roboto-Regular.ttf/en-thelittleprince.txt/ot                 28.2 ms         28.2 ms           26
BM_Shape/Roboto-Regular.ttf/en-words.txt/ot                           32.1 ms         32.1 ms           22
BM_Shape/Roboto-Regular.ttf/react-dom.txt/ot                           310 ms          310 ms            2

hb:

----------------------------------------------------------------------------------------------------------
Benchmark                                                                Time             CPU   Iterations
----------------------------------------------------------------------------------------------------------
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-thelittleprince.txt/ot       87.7 ms         87.6 ms            8
BM_Shape/NotoNastaliqUrdu-Regular.ttf/fa-words.txt/ot                 98.1 ms         98.1 ms            7
BM_Shape/Amiri-Regular.ttf/fa-thelittleprince.txt/ot                  41.0 ms         41.0 ms           17
BM_Shape/NotoSansDevanagari-Regular.ttf/hi-words.txt/ot               23.4 ms         23.4 ms           30
BM_Shape/Roboto-Regular.ttf/en-thelittleprince.txt/ot                 6.11 ms         6.11 ms          114
BM_Shape/Roboto-Regular.ttf/en-words.txt/ot                           8.71 ms         8.71 ms           80
BM_Shape/Roboto-Regular.ttf/react-dom.txt/ot                          67.6 ms         67.6 ms           10

This is on a Mac M1. I'll test on AMD when I'm back home.

Also curious how you have configured and built HB.

That's true.

At this point, I think I've shown I'm unreliable enough at posting performance numbers that I've deleted my posts mentioning it.

Sorry for the bother =)