Motivated by this, benchmarked my embedding model vs. similar JEV-like open sourced models. Still, my embedding model performed much better. JEV might be different. Later, I can benchmark again using their production API. While doing this benchmark, I just got bored and benchmarked several INT8 quantization approaches. Based on the benchmarked result, I am now moving to int8-channel from fp32.

Moving emb to an int8-quantized embedding model made the model 3.8× smaller (556 MB → 147 MB) and cut memory after load by about 70% (868 MB → 256 MB). Each request got 1.2–3.3× faster than with fp32. Search queries, which send one text per request, gained about 1.8–2.2×, for example 43.7 ms → 20.2 ms for a short query. Search quality held up: MRR went from 0.879 to 0.882 on clean queries and dipped about one point (0.853 → 0.846) on queries with many typos. The speed ratios should be read as a direction rather than exact figures, because another benchmark was sharing the CPU during the runs.