When people talk about RAG systems, the conversation often starts with embeddings, vector databases, and prompt design. But once you move from a demo to a real production system, the hard problems change. Retrieval quality becomes inconsistent. Large documents get split in ways that destroy meaning. Multi-tenant isolation becomes a concern. Ranking gets noisy. And the final LLM response starts suffering from context placement issues that are easy to miss in small tests.

In this article, I want to walk through the architecture of a production-ready hybrid RAG system and show why deterministic chunking alone is not enough. I will also show how a dynamic hierarchical retrieval strategy can improve precision, preserve structure, and reduce the “lost in the middle” problem that appears in long-context workflows.

The goal is not just to build something that works in a notebook. The goal is to build something that survives real user data, real latency constraints, and real retrieval trade-offs.

The core problem

Most RAG pipelines begin with a simple idea:

  1. Split the document

  2. Embed the chunks

  3. Store them in a vector database

  4. Retrieve the top matches

  5. Send them to the LLM

This works reasonably well for short, clean documents. But as soon as the input gets more complex, the weaknesses appear.

Common failure modes include:

  • chunks cutting through semantic boundaries

  • duplicate or near duplicate results

  • weak retrieval on technical or structured documents

  • poor context ordering before generation

  • lost references across sections

  • tenant data leakage if filtering is weak

  • expensive reranking that is added too late

For example, if a fixed token count splits a CV, policy document, or technical specification, an important section heading may end up separated from the paragraph it explains. The model sees text, but not structure. That loss of structure reduces answer quality more than many people expect.

This is why I started treating retrieval as an architecture problem, not just a similarity search problem.

Why deterministic chunking is not enough

Deterministic chunking has one major advantage: predictability. You know exactly how text will be segmented. That is useful for debugging, indexing, and reproducibility.

But deterministic chunking has several limits:

  • It ignores document structure

  • It can split concepts across chunks,

  • It can reduce coherence in long documents,

  • It may hurt retrieval recall when queries depend on context spanning multiple sections

A chunk size of 400 tokens might work fine for one document type and fail badly for another. Even if the chunking is consistent, the meaning may not be.

This is why I prefer a hierarchical approach in which the document structure is preserved at multiple levels.

A better pattern

Instead of thinking only in chunks, think in layers:

  • document

  • section

  • subsection

  • paragraph

  • sentence

That way, retrieval can happen at the most relevant level, and the final context can be reconstructed with the surrounding structure intact.

Dynamic hierarchy as a retrieval strategy

A dynamic hierarchy means that the system does not treat every document the same way. It adapts retrieval granularity based on structure and query intent.

For example:

  • A broad query may retrieve the section-level context

  • A precise query may retrieve paragraph-level nodes

  • A comparison query may retrieve sibling sections together

  • A definition query may retrieve a short, high-confidence span

This gives the system more flexibility than a fixed chunk strategy.

Conceptual flow

Raw document
   -> structure parser
   -> hierarchical nodes
   -> embeddings at multiple levels
   -> metadata indexed by tenant and document type
   -> hybrid retrieval
   -> reranking
   -> context assembly
   -> generation

The key difference is that retrieval is no longer a flat top-k search over isolated text chunks. It becomes a structured selection problem.

Multi-tenant isolation should be designed first

If your RAG system serves multiple users or multiple clients, tenant isolation is not optional. It is a core system requirement.

One mistake I see often is treating tenant filtering as an afterthought. That is dangerous. Retrieval should never be able to cross tenant boundaries, even if similarity scores suggest it.

A practical pattern is to store tenant metadata with every indexed node and enforce filtering at query time.

Example metadata design

node = {
    "tenant_id": "tenant_123",
    "document_id": "doc_456",
    "node_id": "node_789",
    "level": "paragraph",
    "section_title": "Evaluation Strategy",
    "text": "..."
}

Then retrieval always includes tenant-aware filtering.

def query_retriever(query, tenant_id):
    return vector_index.search(
        query=query,
        filter={"tenant_id": tenant_id}
    )

This seems basic, but in practice, it is one of the most important design choices in the system.

Dense retrieval is powerful, but it is not enough on its own. Sparse retrieval still matters, especially for exact terminology, product names, identifiers, acronyms, and rare terms.

That is why I prefer hybrid retrieval.

Why hybrid search helps

  • dense retrieval captures semantic similarity

  • Sparse retrieval captures exact lexical overlap

  • Together, they improve recall and robustness

A simple method is the Reciprocal Rank Fusion.

RRF idea

If one result ranks high in both sparse and dense search, it should be boosted.

A common RRF formula is:

score(d)=i=1∑n​k+ri​(d)1​

Where:

  • ri​(d) is the rank of document d in the retrieval list i

  • k is a smoothing constant

This is elegant because it is simple, fast, and does not require score normalisation across different retrieval systems.

Example fusion logic

def reciprocal_rank_fusion(results_list, k=60):
    scores = {}

    for results in results_list:
        for rank, item in enumerate(results, start=1):
            doc_id = item["id"]
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

This gives you a stable way to combine results from different retrievers without forcing them into the same scoring scale.

Reranking is where precision improves

Even with hybrid retrieval, the first pass is usually not good enough for final generation. The top candidates often include useful but noisy results.

This is where reranking becomes valuable.

A reranker can look at the query and candidate text together and decide what is actually most relevant. This is especially useful for:

  • long technical documents

  • nested policies

  • enterprise knowledge bases

  • document collections with repetitive language

Practical reranking flow

  1. retrieve 20 to 50 candidates

  2. apply a lightweight reranker

  3. keep the top 5 to 10

  4. preserve the surrounding hierarchy where needed

  5. Send compact context to the LLM

This gives the generation step much better input quality.

The lost in the middle problem is real

Once you begin assembling longer contexts, another issue appears: the model often pays less attention to information in the middle of the prompt.

This is commonly called the lost in the middle problem.

If the most important evidence is placed in the wrong part of the context window, answer quality can drop even if the right information is technically present.

What helps

  • Put the strongest evidence near the start or end

  • group related nodes together

  • avoid unnecessary context expansion

  • Keep the final prompt structured

  • Place summary metadata before detailed text

Example context assembly

def assemble_context(top_nodes):
    ordered = sorted(top_nodes, key=lambda x: x["score"], reverse=True)
    blocks = []

    for node in ordered:
        blocks.append(f"""
[Section: {node["section_title"]}]
{node["text"]}
""".strip())

    return "\n\n".join(blocks)

This is simple, but the ordering strategy matters a lot.

A hierarchy-aware parser is worth the effort

If you only split by token count, you lose the structure of the source. A hierarchy-aware parser gives you much better retrieval control.

For example, a parser can detect:

  • headings

  • subheadings

  • lists

  • paragraphs

  • tables

  • code blocks

Then each node can be linked to its parent and children.

Example tree shape

Document
  -> Section 1
      -> Paragraph 1
      -> Paragraph 2
  -> Section 2
      -> Subsection 2.1
          -> Paragraph 3

That structure allows a retriever to surface not just the best paragraph, but the paragraph in its proper context.

Simple parser sketch

class Node:
    def __init__(self, text, level, parent_id=None):
        self.text = text
        self.level = level
        self.parent_id = parent_id
        self.children = []

def create_child(parent, text, level):
    child = Node(text=text, level=level, parent_id=id(parent))
    parent.children.append(child)
    return child

In production, the parser would be more advanced, but the principle is the same: structure should survive ingestion.

Production trade-offs matter

Every architectural decision has a cost.

Better retrieval usually means:

  • more indexing work

  • more metadata

  • more compute

  • more complexity in orchestration

Better precision may mean:

  • more reranking latency

  • smaller final context windows

  • more careful prompt assembly

Better isolation may mean:

  • stricter filters

  • more query parameters

  • less flexibility in global search

The point is not to eliminate trade-offs. The point is to make them explicit.

A system that is accurate but too slow will fail in practice. A system that is fast but weak on relevance will also fail. Production design is the art of balancing both.

Example end-to-end retrieval flow

Here is a simplified version of what a production pipeline might look like:

def answer_query(query, tenant_id):
    sparse_results = bm25_search(query, tenant_id=tenant_id)
    dense_results = vector_search(query, tenant_id=tenant_id)

    fused = reciprocal_rank_fusion([sparse_results, dense_results])
    top_candidates = fetch_documents([doc_id for doc_id, _ in fused[:20]])

    ranked = rerank(query, top_candidates)
    context = assemble_context(ranked[:5])

    prompt = f"""
Use the context below to answer the question.

Context:
{context}

Question: {query}
"""

    return llm_generate(prompt)

That is still simplified, but it captures the shape of the system:

  • retrieve from multiple methods

  • filter by tenant

  • fuse and rerank

  • preserve structure

  • generate from compact evidence

Why this approach is better in practice

This architecture gives you several advantages:

  • better retrieval quality on long documents

  • more stable behaviour across document types

  • stronger tenant isolation

  • improved answer grounding

  • more control over prompt construction

  • better resilience against noisy top-k retrieval

In other words, it helps move RAG from “works on my sample data” to “works in a real product.”

Final thought

The main lesson I have learned is that RAG quality is not just a model problem. It is a systems problem.

If you want reliable results, you need to think about:

  • structure at ingestion

  • metadata at indexing

  • filtering at query time

  • ranking before generation

  • ordering inside the prompt

Once you do that, the system starts to feel much more dependable.

Deterministic chunking is useful, but it is not the whole answer. A dynamic hierarchical retrieval design gives you more control, better precision, and a cleaner path to production.

That is the direction I find most promising for serious AI systems.