RAG for Swiss SMEs: Setup, Eval, FADP — What You Should Know First

What RAG Actually Does

A RAG system is not a "chat with your data". It is a pipeline:

1. Chunking — your documents are split into semantically meaningful pieces.
2. Embedding — each piece gets a vector that represents its meaning.
3. Retrieval — for a question, the most relevant chunks are found via vector similarity.
4. Reranking — a second model sorts the hits by relevance to the specific question.
5. Generation — the LLM receives the question plus the retrieved chunks as context and answers with citations.

The temptation is to skip a stage ("we don't need reranking, it runs fine without it"). In practice, it is precisely the reranking stage that decides between "nice, but it hallucinates" and "reliable, with a traceable source".

When RAG Pays Off for an SME

Three criteria:

• Volume — your knowledge base has enough documents (rule of thumb: 100+) that classic full-text search already noticeably fails. At 30 documents, a well-maintained Confluence is cheaper.
• Frequency — employees repeatedly ask the same questions. If HR answers "How many vacation days do I have left?" five times a week, that is a RAG candidate. If the HR inbox sees four requests a month, probably not.
• Unambiguity — your documents are fundamentally consistent and versioned. RAG cannot magically decide between two contradicting policy versions — you have to do that beforehand.

If you clearly meet two of the three criteria, a PoC makes sense. If none, the answer "maintain a wiki and a glossary" is often cheaper and more honest.

Setup: What Actually Needs to Be Built

A production-ready RAG pipeline for an SME with 200–400 documents typically comprises:

• Document pipeline — connection to the source systems (SharePoint, Confluence, OneDrive, your own file shares), incremental updates, versioning.
• Chunking strategy — not "1,000 tokens per chunk and done", but document-type-specific. We split a policy differently from a contract.
• Embedding model — text-embedding-3-large or text-embedding-ada-002 via Azure OpenAI Switzerland are the usual defaults. Both work well for DE/FR/IT.
• Vector store — PostgreSQL with pgvector, Azure AI Search, or Qdrant. For Swiss data residency: PostgreSQL/Qdrant on-premises or the Azure Switzerland region.
• LLM layer — Claude or GPT-4 for answer generation. Both strong in DE/FR/IT, with a slight edge for Claude on Swiss politeness forms.
• Eval gold set and monitoring — see the next section.

Eval: The Three Pillars That Separate AI from Gut Feeling

A RAG pipeline without evaluation is a gut-feeling project. With evaluation, it becomes an engineering project with verifiable output. Three metrics are standard:

Concretely: we build a gold set of 50–200 questions with expected answers. On every index update, model change, or prompt adjustment, the gold set runs through — frameworks like RAGAS or TruLens automate this. If faithfulness drops below a threshold, that is a pre-production blocker, not a "we can optimise it later".

FADP, the EU AI Act, and Data Residency

Three points we explicitly clarify on every Swiss RAG project:

• Data residency — Azure OpenAI Switzerland (North/West region) keeps data in Switzerland. Microsoft does not train on your content (default off, contractually assured). Logs are retained for a limited period for abuse detection, then deleted. For even stricter requirements: an on-premise variant with open-source models (Llama, Qwen, Mistral) — with a performance trade-off we communicate openly.
• FADP-compliant retention — user queries and agent answers are logged so that access and deletion requests can be fulfilled. Sensitive topics (HR matters, health, conflicts) often get their own retention and escalation rules.
• EU AI Act — for an internal HR/IT RAG, the use case usually falls under "minimal risk", but depending on the setup also under "limited" or "high risk", with transparency and documentation obligations. Architecture reviews plan for this from the start.

Cost Reality

We discuss concrete ranges individually, because they depend heavily on document volume, model choice, the desired eval depth, and the compliance level. Three driver classes:

• Setup PoC (4–8 weeks) — eval gold set, chunking pipeline, first model setup, topic classification, integration connection.
• Running costs — token consumption (scales with usage frequency), hosting and vector store, plus internal maintenance effort (typically 0.1–0.2 FTE for document updates and quarterly eval reviews).
• A deeper compliance layer — on-premise stack, custom monitoring tools, external audits if the industry demands it.

For an SME with 50–150 employees, monthly running costs are usually well below the cost of a single full-time position. ROI, however, cannot be cleanly calculated before the PoC — you test whether the usage frequency and the time saved justify the costs.

When RAG Is Not the Right Answer

Three anti-patterns we see regularly:

Conclusion

A well-built RAG is the most honest first AI use case for many Swiss SMEs: concrete, measurable, with traceable sources, FADP-compliant. A poorly built RAG is a demo that never goes to production. The difference lies in setup discipline, eval rigour, and an honest assessment of the compliance requirements.

More detail on the concrete application: Use Case 06 — RAG over an internal knowledge base. Related applications: the compliance / contract review agent for the curated variant with a rule set. Common questions on model choice, data residency, and eval on the FAQ page.