Docs › Running your own inference
Running your own inference
ProLyt can use any AI model server that speaks the OpenAI-compatible API and supports tool calling: for example vLLM or Ollama on your own GPUs. Your questions, table structures and query results then go only to your server.
Requirements for the model server
- An OpenAI-compatible endpoint (
/v1/chat/completions). - A model with tool (function) calling, and a server started so that it emits tool calls: on vLLM that means
--enable-auto-tool-choicewith a matching--tool-call-parser. This produces no error when missed — every answer is prose with no query, so check it (section *Checking it works*). - Enough context for a question with its table descriptions and query results. Measured floor: a 8B model answers at an 8,192-token context on an 8 GB box; give the model 16k or more when the hardware allows.
Examples: ```
Ollama
ollama serve # listens on :11434, API at /v1 ollama pull <model>
vLLM
vllm serve <model> --enable-auto-tool-choice --tool-call-parser <parser> ```
Connecting it to ProLyt
The endpoint is a deployment setting in .env (the operator sets it; there is no field for it in the web UI):
`` LLM_RUNTIME=custom CUSTOM_RUNTIME_URL=http://gpu-server:11434/v1 # or :8000/v1 for vLLM DEFAULT_MODEL=<a registry model name> # the model ids are Ollama tags; override QWEN_MODEL_ID / LLAMA_MODEL_ID to what your server calls the weights ``
Then docker compose … up -d (a restart does not re-read .env). On the hosted service an operator can also give one workspace its own endpoint when the workspace is created.
Option A: same network (on-prem, simplest)
Use the server's internal address, as above. Nothing leaves your network.
Option B: private network between sites (VPN)
Connect the ProLyt server and the GPU server through your VPN (for example WireGuard) and use the GPU server's private address. Nothing is exposed to the internet.
Option C: reverse SSH tunnel (GPU behind a firewall or NAT)
When the GPU machine cannot accept incoming connections, it can open the connection itself. ProLyt runs in a Docker container, so the forwarded port must be bound to the Docker bridge address of the ProLyt server (172.17.0.1 by default), not to 127.0.0.1 — a container cannot reach the host's loopback.
On the ProLyt server, a dedicated tunnel user that can do nothing but forward that one port (/etc/ssh/sshd_config.d/70-tunnel.conf):
`` Match User tunnel-user GatewayPorts clientspecified AllowTcpForwarding remote PermitListen 172.17.0.1:11434 PermitOpen none ForceCommand /bin/false PasswordAuthentication no ``
On the GPU machine (autossh or a systemd unit so it comes back after a restart):
`` autossh -M 0 -N -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes \ -R 172.17.0.1:11434:127.0.0.1:11434 tunnel-user@prolyt-server ``
Then CUSTOM_RUNTIME_URL=http://172.17.0.1:11434/v1. Allow 172.17.0.1:11434 only from the Docker network in the server's firewall (ufw: allow from 172.18.0.0/16 to 172.17.0.1 port 11434), so nothing else on the host or the internet can reach the model. This is the exact shape the hosted service uses for a GPU laptop behind NAT.
Option D: public HTTPS endpoint
If the endpoint must be reachable over the internet, put it behind HTTPS with an API key, and allow only the ProLyt server's address.
Checking it works
- Ask a question that needs a query ("how many rows are in <table>?"). The answer must show SQL under it. An answer with no SQL on a question like that means the server is not emitting tool calls (see the vLLM flags above).
- The Inference settings panel's endpoint card shows the endpoint in use and its declared context window.
- If the endpoint is down, the question fails with a provider error and no credit is used.