#8
Issues/vercel-labs/lat.md/#8

feature: add local embeddings to search engine

Opennickshirobokovopened 8mo ago

Summary

Support running the search engine with locally hosted embedding models instead of remote APIs.

Problem

The search engine currently requires embeddings from third-party remote endpoints. This creates friction and limits usability in several common scenarios:

  • Users must obtain and manage external API keys
  • Outbound network access may be restricted (corporate or air-gapped environments)

Proposed Solution

Allow the search engine to run with a locally hosted embedding model.

Embedding models are relatively small (~300 MB–1 GB). HuggingFace TEI provides fast local inference with minimal configuration.

Implementation could be:

  • Add --local-search-engine flag to init command
  • Download weights and configure inference during init
  • Default to a commonly used open model
  • Allow advanced users to customize the backend
npx lat.md init --local-search-engine

Activity

6
1st18mo ago

How about we add support for a custom endpoint URL (e.g. LAT_LLM_ENDPOINT) so users can point to an OpenAI-compatible local server they already run — Ollama, llama.cpp, HuggingFace TEI, etc?

nickshirobokov8mo ago

I don't think asking users to deal with building and running local AI endpoints is a good idea.

That means they need to:

  • read about the differences between Ollama, llama.cpp, and TEI
  • go to MTEB and research which model they need
  • build the inference
  • introduce pooling for multi-vector models to make it OpenAI-compatible

At the same time, AGENTS.md and Skills provide fully local usage without any remote endpoints.

jeffmm7mo ago

I would also like this feature. While I personally wouldn't mind a custom endpoint option, I'm attempting to have my team try lat.md in our codebase, but we're a Claude Enterprise company. So I'd prefer to avoid:

  1. relying on OpenAI or Vercel API keys, because that requires getting legal involved
  2. having to help our non-power-user engineers set up ollama or llama.cpp servers locally or
  3. configuring an internal server as an endpoint and dealing with auth for it

My PR adds @huggingface/transformers as an optional dependency, and adds some local model options to make lat search trivial without API keys.

dnslin7mo ago

Is it possible to use the Embeddings api provided by other model providers such as google qwen, etc.

JoeHelbing5mo ago

How about we add support for a custom endpoint URL (e.g. LAT_LLM_ENDPOINT) so users can point to an OpenAI-compatible local server they already run — Ollama, llama.cpp, HuggingFace TEI, etc?

I actually sort of did this in a hacky way but I would MUCH prefer just having an OpenAI compatible endpoint choice directly.

I pointed lat.md at a custom embedding endpoint by routing lat search through a ~/.local/bin/lat wrapper, which exports the embedding env defaults and loads local preload shims to make lat.md’s replay provider use a llama.cpp local EmbeddingGemma endpoint and 768-dimensional schema instead.