samples/ragas_1 — awesome-ai-stack

Example

Ragas — implementation sample

A real, runnable mini-project. Download it and run with Docker.

View samples/ragas_1 on GitHub ↗

June 28, 2026

Ragas — implementation sample

A tiny Ragas sample: it scores a RAG answer for faithfulness and response relevancy, judged by an OpenAI model + embeddings.

Configure

cd samples/ragas_1
cp .env.sample .env
# edit .env: set OPENAI_API_KEY (judge LLM + embeddings)

Run with Docker

cd samples/ragas_1
docker build -t aas-ragas .
docker run --rm --env-file .env aas-ragas

Run with Docker (in a devcontainer with DooD)

The foreground docker run may print nothing under Docker-outside-of-Docker — run detached and follow the logs:

docker logs -f "$(docker run -d --env-file .env aas-ragas)"

Run locally

cd samples/ragas_1
pip install -r requirements.txt
python app.py

Example run

Scores use an LLM judge, so they vary slightly run to run. One run with gpt-4o-mini:

{'faithfulness': 1.0000, 'answer_relevancy': 0.9701}

.env.sample

# Copy to `.env` and add your OpenAI API key (judge LLM + embeddings).
#   cp .env.sample .env
OPENAI_API_KEY=

MODEL=gpt-4o-mini

app.py

"""Ragas: evaluate a RAG answer with reference-free metrics.

Scores one (question, answer, context) sample for faithfulness and response
relevancy, judged by an OpenAI model + embeddings (OPENAI_API_KEY).

Run:
    docker build -t aas-ragas .
    docker run --rm --env-file .env aas-ragas
"""

import os

from dotenv import load_dotenv
from ragas import EvaluationDataset, evaluate
from ragas.metrics import Faithfulness, ResponseRelevancy
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

load_dotenv()

MODEL = os.environ.get("MODEL", "gpt-4o-mini")


def main() -> None:
    dataset = EvaluationDataset.from_list(
        [
            {
                "user_input": "What language is Qdrant written in?",
                "response": "Qdrant is written in Rust.",
                "retrieved_contexts": ["Qdrant is a vector database written in Rust."],
            }
        ]
    )
    llm = LangchainLLMWrapper(ChatOpenAI(model=MODEL))
    embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

    result = evaluate(
        dataset,
        metrics=[Faithfulness(llm=llm), ResponseRelevancy(llm=llm, embeddings=embeddings)],
    )
    print(result)


if __name__ == "__main__":
    main()

Dockerfile

FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

ENTRYPOINT ["python", "app.py"]

requirements.txt

ragas>=0.2,<1
# Pin: ragas still imports langchain_community.chat_models.vertexai, which was
# removed in langchain-community 0.4. Stay on 0.3.x so the import resolves.
langchain-community>=0.3,<0.4
langchain-openai>=0.2,<1
python-dotenv>=1.2,<2