samples/ollama_1 — awesome-ai-stack

Example

Ollama — a local LLM call via LiteLLM

A real, runnable mini-project. Download it and run with Docker.

View samples/ollama_1 on GitHub ↗

June 28, 2026

Ollama — a local LLM call via LiteLLM

A tiny script that calls a model running in a local Ollama server through LiteLLM — no API key, no cloud. Because it goes through LiteLLM, the same code runs against a cloud model by changing MODEL.

Configure

cd samples/ollama_1
cp .env.sample .env

First pull the model on the host — ollama pull qwen3.5:9b (or ollama run qwen3.5:9b). No API key is needed. .env sets MODEL and OLLAMA_API_BASE; in a devcontainer with DooD the container reaches the host's Ollama at http://host.docker.internal:11434 (already the default in .env.sample).

Tool calling needs the ollama_chat/ prefix (Ollama's chat endpoint), not ollama/ — and a tool-capable model.

Run with Docker

cd samples/ollama_1
docker build -t aas-ollama .
docker run --rm --env-file .env aas-ollama "Summarize Ollama in one line."

Run with Docker (in a devcontainer with DooD)

In a dev container that talks to the host Docker daemon (Docker-outside-of-Docker), the foreground docker run above often prints nothing and exits 0 — but the run itself succeeds. The script runs to completion and Docker captures all of its output; only the live attached stream drops it over the VM boundary. Run detached and follow the logs instead:

cd samples/ollama_1
docker build -t aas-ollama .
docker logs -f "$(docker run -d --env-file .env aas-ollama \
  "Summarize Ollama in one line.")"

Run locally

cd samples/ollama_1
pip install -r requirements.txt
# point at your local Ollama (no Docker): http://localhost:11434
OLLAMA_API_BASE=http://localhost:11434 python app.py "Summarize Ollama in one line."

Example run

Output varies by model and run — LLMs are non-deterministic, so the wording differs each time. Below is one run with ollama_chat/qwen3.5:9b.

Ollama allows users to easily run open-source large language models locally on
their own devices, enabling private and offline access to AI without relying on
external servers or internet connectivity.

.env.sample

# Copy this file to `.env`.
#   cp .env.sample .env
#
# No API key needed — Ollama runs locally. Pull the model first on the host:
#   ollama pull qwen3.5:9b
# Use a tool-capable model and the ollama_chat/ prefix (chat endpoint).
MODEL=ollama_chat/qwen3.5:9b

# Where the Ollama server runs:
#   Local, no Docker:           http://localhost:11434
#   Docker / DooD on the host:  http://host.docker.internal:11434
OLLAMA_API_BASE=http://host.docker.internal:11434

app.py

"""Ollama: a local LLM call routed through LiteLLM.

Calls a model running in a local Ollama server — no API key. Routing through
LiteLLM means the same code also works against cloud models by changing MODEL.

Run (with Ollama running on the host):
    docker build -t aas-ollama .
    docker run --rm --env-file .env aas-ollama "Summarize Ollama in one line."
"""

import os
import sys

from dotenv import load_dotenv
from litellm import completion

# Load .env when running locally; in Docker the config comes from --env-file.
load_dotenv()

# ollama_chat/ uses Ollama's /api/chat endpoint (supports tool calling).
MODEL = os.environ.get("MODEL", "ollama_chat/qwen3.5:9b")


def main() -> None:
    prompt = " ".join(sys.argv[1:]) or "Summarize what Ollama does in one sentence."

    resp = completion(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        api_base=os.environ.get("OLLAMA_API_BASE", "http://localhost:11434"),
    )
    print(resp.choices[0].message.content)


if __name__ == "__main__":
    main()

Dockerfile

# Minimal image for the Ollama sample.
FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

# Provide your config at run time (.env is NOT baked into the image):
#   docker run --rm --env-file .env aas-ollama "<prompt>"
ENTRYPOINT ["python", "app.py"]