Audio, images and AI models

Audio transcription and the short TLDR: summaries above entries are off until you turn them on: set WHISPER_URL for audio and TLDR_MODEL for summaries in .env (see Settings).

The AI engine

One localai service (pinned to v4.12.0) serves chat, embeddings, transcription and images, on CPU, CUDA, Vulkan, Intel or ROCm. It is at http://127.0.0.1:8090, localhost only, since its UI is unauthenticated unless you set LOCALAI_API_KEY. CHAT_URL, EMBED_URL and WHISPER_URL all default to it; set one only to move that role to another machine. docker compose logs -f localai shows its model downloads.

On an NVIDIA card the GPU stack uses the CUDA 13 image, which needs NVIDIA driver 580 or newer (nvidia-smi shows yours). With an older driver, set LOCALAI_IMAGE_GPU=localai/localai:v4.12.0-gpu-nvidia-cuda-12 in .env.

Three lists must agree: LOCALAI_MODELS is what the engine installs, CHAT_MODELS / EMBED_MODEL / WHISPER_MODEL are what the app asks for, and LOCALAI_EXTERNAL_BACKENDS is which runtimes it downloads. A mismatch shows as a 404 or no backends found. Models that ship as a config file under localai/ are mounted by Compose and must not also be in LOCALAI_MODELS.

Running the engine outside Compose? Bind each localai/<name>.yaml to /models/<name>.yaml:ro yourself, then check the embedder answers with a 768-length vector, since without it search quietly becomes keyword-only:

curl -s -X POST http://<engine>:8080/v1/embeddings -H 'content-type: application/json' \
  -d '{"model":"nomic-embed-text-v1.5","input":"test"}'

Image models

GPU only. Set IMAGE_URL=http://localai:8080/v1, add stablediffusion-ggml to LOCALAI_EXTERNAL_BACKENDS, list the models in IMAGE_MODELS, and download their files yourself into ./mnt/localai/models/stablediffusion-cpp/models/. (Naming them in LOCALAI_MODELS instead holds the whole engine down for the download, which was measured at over an hour.) Qwen-Image letters best but needs ~16 GB and rendered nothing on a 12 GB card; Z-Image-Turbo is ~8 GB and fits.

# Qwen-Image (Apache-2.0), ~16 GB
https://huggingface.co/QuantStack/Qwen-Image-GGUF/resolve/main/Qwen_Image-Q4_0.gguf
https://huggingface.co/QuantStack/Qwen-Image-GGUF/resolve/main/VAE/Qwen_Image-VAE.safetensors
https://huggingface.co/mradermacher/Qwen2.5-VL-7B-Instruct-GGUF/resolve/main/Qwen2.5-VL-7B-Instruct.Q4_K_M.gguf

# Z-Image-Turbo (Apache-2.0), ~8 GB. Save the third as z-image-ae.safetensors.
https://huggingface.co/leejet/Z-Image-Turbo-GGUF/resolve/main/z_image_turbo-Q4_K.gguf
https://huggingface.co/MaziyarPanahi/Qwen3-4B-GGUF/resolve/main/Qwen3-4B.Q4_K_M.gguf
https://huggingface.co/Tongyi-MAI/Z-Image-Turbo/resolve/main/vae/diffusion_pytorch_model.safetensors

Take Z-Image-Turbo's VAE from Tongyi's repo as above: the engine gallery's copy comes from a non-commercial FLUX.1 mirror.

Reranker

Set RERANK_MODEL=bge-reranker-gguf and recreate. Its config is already mounted, and the engine downloads it (636 MB, Apache-2.0) on the next start. Do not add it to LOCALAI_MODELS. Readers then get Ranker and Hybrid under their Relevance check setting.

Fitting your machine

Most machines need none of these. Each is explained beside it in .env.example.