samples/deepeval_1 — awesome-ai-stack

Example

DeepEval — implementation sample

A real, runnable mini-project. Download it and run with Docker.

View samples/deepeval_1 on GitHub ↗

June 28, 2026

DeepEval — implementation sample

A tiny DeepEval sample: it scores an answer with the Answer Relevancy metric (LLM-as-judge).

Configure

cd samples/deepeval_1
cp .env.sample .env
# edit .env: set OPENAI_API_KEY (used as the judge model)

Run with Docker

cd samples/deepeval_1
docker build -t aas-deepeval .
docker run --rm --env-file .env aas-deepeval

Run with Docker (in a devcontainer with DooD)

The foreground docker run may print nothing under Docker-outside-of-Docker — run detached and follow the logs:

docker logs -f "$(docker run -d --env-file .env aas-deepeval)"

Run locally

cd samples/deepeval_1
pip install -r requirements.txt
python app.py

Example run

Scores use an LLM judge, so the score and reason vary slightly run to run. One run with gpt-4o-mini:

answer_relevancy: 1.00
reason: The score is 1.00 because the response directly addresses the question about the programming language of Qdrant without any irrelevant statements.

.env.sample

# Copy to `.env` and add your OpenAI API key (used as the judge model).
#   cp .env.sample .env
OPENAI_API_KEY=

MODEL=gpt-4o-mini

app.py

"""DeepEval: score an LLM answer with a metric (LLM-as-judge).

Builds one test case and runs the Answer Relevancy metric. The judge model uses
OPENAI_API_KEY. Output is a score (0-1) with a short reason.

Run:
    docker build -t aas-deepeval .
    docker run --rm --env-file .env aas-deepeval
"""

import os

from dotenv import load_dotenv
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

load_dotenv()

MODEL = os.environ.get("MODEL", "gpt-4o-mini")


def main() -> None:
    case = LLMTestCase(
        input="What language is Qdrant written in?",
        actual_output="Qdrant is written in Rust.",
    )
    metric = AnswerRelevancyMetric(model=MODEL)
    metric.measure(case)
    print(f"answer_relevancy: {metric.score:.2f}")
    print(f"reason: {metric.reason}")


if __name__ == "__main__":
    main()

Dockerfile

FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

ENTRYPOINT ["python", "app.py"]