Testing RAG Apps: Dual Gates for Performance and Answer Quality

Why RAG Testing Needs Two Gates

Traditional load‑testing tools only measure response time, which misses the hallucination problem where a model returns fast but fabricated answers. To fully assess a RAG service you need a performance gate (speed) and a quality gate (answer correctness).

Key Performance Metrics

TTFT (Time to First Token) : time until the first token appears on the screen.

ITL (Inter‑Token Latency) : smoothness of token streaming after generation starts.

Tokens/sec : generation speed, crucial for long answers.

p95 / p99 latency : tail‑latency that reflects real user experience.

Quality Metrics Evaluated by DeepEval

Faithfulness : whether the answer is grounded in retrieved context.

Answer Relevancy : whether the answer actually addresses the question.

Context Precision : correctness and ordering of retrieved chunks.

Context Recall : completeness of retrieved information.

When faithfulness is low but context recall is high, the retriever works but the LLM ignores the context, indicating a prompt‑engineering issue.

DeepEval Test Example

from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.models import GeminiModel

judge_model = GeminiModel(
    model="gemini-3.5-flash",
    api_key=os.getenv("GEMINI_API_KEY"),
)

faithfulness_metric = FaithfulnessMetric(threshold=0.75, model=judge_model)
answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.8, model=judge_model)

def test_jmeter_non_gui_mode_answer():
    question = "How do I run JMeter in non-GUI mode?"
    result = query_rag_app(question)
    test_case = LLMTestCase(
        input=question,
        actual_output=result["answer"],
        retrieval_context=result["retrieved_chunks"],
    )
    for metric in [faithfulness_metric, answer_relevancy_metric]:
        metric.measure(test_case)
        status = "PASS" if metric.success else "FAIL"
        print(f"[{status}] {metric.__class__.__name__}: {metric.score:.3f}")
    failed = [m for m in [faithfulness_metric, answer_relevancy_metric] if not m.success]
    if failed:
        names = ", ".join(m.__class__.__name__ for m in failed)
        raise AssertionError(f"Metrics below threshold: {names}")

The test runs with pytest and produces PASS/FAIL signals that CI/CD can consume.

k6 Load Test for TTFT

Because the xk6‑sse extension is not yet compatible with k6 v2, the article uses the built‑in http module to call the /chat/complete endpoint, which returns a full JSON response. This yields end‑to‑end latency but not true TTFT; the script estimates tokens_per_second from word count.

import http from 'k6/http';
import { Trend, Counter } from 'k6/metrics';
import { check } from 'k6';

const totalDuration = new Trend('total_duration_ms', true);
const tokensPerSecond = new Trend('tokens_per_second');
const BASE_URL = __ENV.RAG_APP_URL || 'http://localhost:8080';

export const options = {
  scenarios: {
    rag_chat: {
      executor: 'ramping-vus',
      stages: [
        { duration: '30s', target: 10 },
        { duration: '1m', target: 10 },
        { duration: '30s', target: 0 },
      ],
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<6000'],
    total_duration_ms: ['p(95)<6000'],
  },
};

export default function () {
  const startTime = Date.now();
  const res = http.post(`${BASE_URL}/chat/complete`, JSON.stringify({ query: 'How do I run JMeter in non-GUI mode?' }), {
    headers: { 'Content-Type': 'application/json' },
    timeout: '30s',
  });
  check(res, {
    'status 200': r => r.status === 200,
    'has answer': r => JSON.parse(r.body).answer !== undefined,
  });
  const duration = Date.now() - startTime;
  totalDuration.add(duration);
  const words = JSON.parse(res.body).answer.trim().split(/\s+/).length;
  tokensPerSecond.add((words / duration) * 1000);
}

The test ramps up to 10 virtual users, holds for a minute, then ramps down, checking that p95 latency stays below 6000 ms.

CI/CD Integration with GitHub Actions

Two independent jobs— performance‑gate running the k6 script and quality‑gate running the DeepEval pytest suite—are triggered on every pull request. Secrets GEMINI_API_KEY and FILE_SEARCH_STORE_NAME are injected as environment variables.

name: RAG CI
on: [pull_request]
jobs:
  performance-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Write app env file
        run: |
          cat > app/.env << EOF
          GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
          GEMINI_MODEL=gemini-3.5-flash
          FILE_SEARCH_STORE_NAME=${{ secrets.FILE_SEARCH_STORE_NAME }}
          PORT=8080
          EOF
      - name: Start RAG app
        run: docker compose up -d --build app
      - name: Wait for health
        run: |
          timeout 60 bash -c 'until curl -f http://localhost:8080/health; do sleep 2; done'
      - name: Run k6 load test
        run: docker compose --profile perf run --rm k6

  quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Write app env file
        run: |
          cat > app/.env << EOF
          GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
          GEMINI_MODEL=gemini-3.5-flash
          FILE_SEARCH_STORE_NAME=${{ secrets.FILE_SEARCH_STORE_NAME }}
          PORT=8080
          EOF
      - name: Start RAG app
        run: docker compose up -d --build app
      - name: Wait for health
        run: |
          timeout 60 bash -c 'until curl -f http://localhost:8080/health; do sleep 2; done'
      - name: Run DeepEval tests
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
        run: docker compose --profile quality run --rm deepeval

Both jobs fail the PR if latency exceeds the threshold or if any quality metric falls below its configured threshold, catching regressions before they reach reviewers.

Setting Service‑Level Objectives (SLOs)

The article suggests starting with a baseline from your own load test and then setting p95 latency targets (e.g., < 6000 ms for the demo). It also recommends tracking tail latency (p95/p99), concurrency‑scaled latency, and long‑term trends of faithfulness and relevance.

Conclusion

RAG performance testing requires two complementary gates: a classic load test enriched with LLM‑aware metrics, and a quality gate where an LLM judges answer faithfulness and relevance. Although k6 cannot yet measure true TTFT without an SSE client and DeepEval scores have consistency limits, the combined workflow is sufficient to catch both speed regressions and hallucination bugs before production.