{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"15-rag-concepts-every-ai-builder-should-know-b2468","url":"https://zyvop.com/15-rag-concepts-every-ai-builder-should-know-b2468","title":"15 RAG Concepts Every AI Builder Should Know","subtitle":"A practical guide to chunking, embeddings, hybrid search, reranking, GraphRAG and evaluation, and why most bad RAG answers are retrieval problems.","tldr":"Most bad RAG answers are retrieval problems, not model problems. This guide walks through 15 essential concepts, from chunking, embeddings and vector databases to hybrid search, reranking, GraphRAG, agentic RAG and faithfulness metrics, plus a practical order for adopting them.","keywords":["RAG","Generative AI","Vector Search","LLM","AI Engineering","AI"],"entities":["Samod Alex","RAG","Generative AI","Vector Search","LLM","AI Engineering","AI","ZyVOP"],"keyTakeaways":["Large language models (LLMs) don’t know your company’s documents, and they can’t memorize everything.","Retrieval-Augmented Generation (RAG) fixes this in a simple way.","Before the model answers, you search your own documents, find the useful parts, and give them to the model along with the question."],"headings":["Words you will see","The pipeline at a glance","Part 1: Preparing your documents","1. Chunking","2. Embedding Models","3. Vector Database","4. Metadata Filtering","Part 2: Understanding the question","5. Query Rewriting","6. HyDE (Hypothetical Document Embeddings)","7. Query Decomposition","Part 3: Finding and refining context","8. Hybrid Search","9. Reranking","10. Small-to-Big Retrieval","11. GraphRAG","12. Context Compression","13. Agentic RAG","Part 4: Measuring the results","14. Faithfulness","15. Context Precision & Recall","Where to start","Final thoughts"],"outboundLinks":[],"contentText":"Large language models (LLMs) don’t know your company’s documents, and they can’t memorize everything. Retrieval-Augmented Generation (RAG) fixes this in a simple way. Before the model answers, you search your own documents, find the useful parts, and give them to the model along with the question. Think of it as an open-book exam. The model is the student, your documents are the book, and retrieval is how the student finds the right page. The idea is simple. Making it work well is harder, because a RAG system has many steps and any of them can go wrong. The infographic ends with a line worth remembering: most bad answers are retrieval problems, not model problems. If the model never sees the right page, it can’t give the right answer. This guide explains all 15 concepts in plain language, with an example for each. Words you will see LLM (large language model): the AI model that writes the answer. Token: a small piece of text, roughly three-quarters of a word. Models count text in tokens. Chunk: a small piece of a document. Vector (or embedding): a list of numbers that captures the meaning of a piece of text. Context: the text you give the model along with the question. Retrieval: finding the most useful chunks for a question. The pipeline at a glance A RAG system runs in ten steps: Ingest → Chunk → Embed → Index → Rewrite → Retrieve → Rerank → Compress → Generate → Evaluate flowchart LR subgraph Offline[\"Offline: prepare the knowledge base\"] A[\"Ingest\"] --> B[\"Chunk\"] --> C[\"Embed\"] --> D[\"Index\"] end subgraph Online[\"Online: every question\"] E[\"Rewrite\"] --> F[\"Retrieve\"] --> G[\"Rerank\"] --> H[\"Compress\"] --> I[\"Generate\"] end D --> F I --> J[\"Evaluate\"]The first four steps happen ahead of time, when you load and prepare your documents. Rewrite through Generate happen every time someone asks a question. Evaluate runs alongside both, checking quality on test questions and on real use. Generate is the final call to the LLM. It is a step in the pipeline, but it is not one of the 15 concepts. The 15 concepts are grouped by where they fit: preparing your documents, understanding the question, finding and refining context, and measuring the results. Part 1: Preparing your documents 1. Chunking Chunking means cutting long documents into small pieces so they are easy to search. A 200-page manual is too big to treat as one block. As a single item it is about everything and nothing, and sending all of it to the model would waste space. Size matters. Chunks that are too big mix many topics. Chunks that are too small lose their meaning: “It was approved in March” tells you nothing if you can’t see what was approved. The standard fix is overlap. Each chunk repeats the last couple of sentences of the one before it, so an idea at the edge isn’t cut in half. A good starting point is to split at natural breaks like headings and paragraphs, use chunks of a few hundred tokens (roughly one to three paragraphs), and add about 10–20% overlap. Then adjust based on test results. flowchart LR A[\"Long document\"] --> B[\"Split with overlap\"] B --> C[\"Chunk 1\"] B --> D[\"Chunk 2: repeats the last sentences of chunk 1\"] B --> E[\"Chunk 3: repeats the last sentences of chunk 2\"]2. Embedding Models Computers can’t compare meaning directly, so an embedding model turns each chunk of text into a list of numbers called a vector, like [0.12, -0.4, …]. Texts with similar meanings get similar numbers, so they end up close together. That is why a search for “How do I reset my password?” can find a page titled “Recovering account access”. The two share almost no words, but their meaning is close. When you choose a model, look at the languages it supports, how well it fits your topic, the size of its vectors, its speed, and its cost. One rule to remember: use the same model for your documents and your questions. If you switch models later, you have to redo all your documents. flowchart LR T1[\"Query: How do I reset my password?\"] --> M1[\"Embedding model\"] T2[\"Chunk: Recovering account access\"] --> M2[\"Embedding model\"] M1 --> V1[\"Vector: 0.12, -0.4, ...\"] M2 --> V2[\"Vector: 0.11, -0.38, ...\"] V1 <-->|\"close together, so a match\"| V23. Vector Database Once your chunks are vectors, you need a place to keep them and a fast way to search them. A vector database does both. It takes the vector for a question and returns the closest chunks, usually in milliseconds. Checking every vector one by one is fine for a small collection but too slow for millions. So these databases use an approximate nearest neighbor (ANN) index, which finds very close matches without checking everything. A popular type is HNSW (Hierarchical Navigable Small World). It arranges vectors in layers, like zooming in on a map: start with a wide view, then move closer step by step. You give up a tiny bit of accuracy for a big gain in speed. The result is the top-k matches, meaning the k closest chunks (for example, the top 5 or the top 20). flowchart LR A[\"Chunk embeddings\"] -->|store| B[(\"Vector database with ANN index: HNSW\")] Q[\"Query vector\"] --> B B --> C[\"Top-k nearest neighbors\"]4. Metadata Filtering Meaning alone isn’t enough. Your documents come from different sources, dates, teams, and permission levels. Metadata is the label attached to each chunk: where it came from, when it was written, which team owns it, and who may read it. Filtering uses those labels to shrink the search before it starts. For example: only this year’s documents, only the legal team’s folder, or only files this person is allowed to see. You attach metadata when you load documents, but the filter runs when someone asks a question. Chunks that match the filter are searched. The rest are skipped. Filters make results more relevant (no outdated policies), faster (less to search), and safer. Safety matters most. Enforce access rights in the search itself, instead of asking the model to ignore things it shouldn’t see. flowchart TD Q[\"Query + filters: source, date, team, access rights\"] --> F{\"Does the chunk metadata match?\"} F -->|Match| S[\"Search it\"] F -->|No match| K[\"Skip it\"] S --> R[\"Candidates for ranking\"]Part 2: Understanding the question 5. Query Rewriting People don’t type clean search queries. In a chat, someone might discuss a subscription plan and then ask “and the price?” A search engine has no idea what that means. Query rewriting fixes this by using an LLM to turn a vague, chatty question into a clear one. The LLM reads the chat so far and rewrites the question so it makes sense on its own. “And the price?” becomes “Pro plan price.” It costs one short extra LLM call, and it often makes a noticeable difference in chat-style apps. flowchart LR H[\"Chat history: the Pro plan was just discussed\"] --> L[\"LLM rewriter\"] Q[\"User: and the price?\"] --> L L --> R[\"Rewritten query: Pro plan price\"] R --> S[\"Search\"]6. HyDE (Hypothetical Document Embeddings) A short question and a long answer can look quite different, even when they belong together. That makes the question a weak search key. HyDE gets around this by asking the LLM to write a fake answer first. You turn that draft into a vector and search with it. The draft doesn’t need to be correct. It only needs to look like a real answer, so it lands near real documents that do. There is one risk. If the draft includes confidently wrong details, it can pull the search in the wrong direction. HyDE works best when the model knows the general topic but not your specific data. flowchart LR U[\"User question\"] --> L[\"LLM drafts a hypothetical answer\"] L --> E[\"Embed the draft\"] E --> S[\"Search by that vector\"] S --> D[\"Real documents that look like the draft\"]7. Query Decomposition Some questions are really several questions in one: “How did our sales change this quarter, and how does that compare with last year?” A single search struggles with that. Query decomposition splits a complex question like this into smaller ones, searches for each separately, and combines the findings into one answer. Each search is focused, so it is much more likely to find what it needs. flowchart TD Q[\"Complex question\"] --> A[\"Sub-question 1\"] Q --> B[\"Sub-question 2\"] A --> RA[\"Retrieve for sub-question 1\"] B --> RB[\"Retrieve for sub-question 2\"] RA --> M[\"Combine the findings\"] RB --> M M --> ANS[\"Single answer\"]Part 3: Finding and refining context 8. Hybrid Search Vector search understands meaning but can miss exact terms. Keyword search is the opposite. Hybrid search combines the two and merges the results: BM25 is a classic keyword method. It scores a document higher when it contains the words in your query, especially rare words. It works well for error codes, product names, and IDs. Vector search finds text with a similar meaning, even when the words are different. The two ranked lists are then merged using Reciprocal Rank Fusion (RRF). The raw scores from the two methods can’t be compared directly, so RRF ignores them and looks only at position. A chunk that ranks high on both lists rises to the top. flowchart LR Q[\"Query\"] --> B[\"BM25 keyword search: exact terms\"] Q --> V[\"Vector search: meaning\"] B --> R[\"Reciprocal Rank Fusion\"] V --> R R --> L[\"Fused ranked list\"]9. Reranking The first search is fast but rough. Reranking takes the shortlist and scores it again, more carefully, keeping only the best few. The tool for this is a cross-encoder, a model that reads the question and a chunk together, so it can judge how well that chunk really answers the question. It is more accurate, but too slow to run on every chunk. So the usual pattern is to cast a wide net first and then choose carefully. Retrieve plenty (say, the top 50), let the reranker pick the best few (say, the top 5), and send only those to the LLM. flowchart LR Q[\"Query\"] --> F[\"First-pass retrieval\"] F --> T[\"Top 50 chunks\"] T --> C[\"Cross-encoder reads query and chunk together\"] C --> B[\"Best 5 go to the LLM\"]10. Small-to-Big Retrieval Small chunks are good for matching because each one is focused. But they are poor for answering, because they lack surrounding context. Small-to-big retrieval gives you both, in three steps: Match the question against small chunks (the “children”). Look up the larger section each one belongs to (the “parent”). Send the whole parent section to the LLM. Search stays precise, and the model gets enough context to answer well. flowchart LR subgraph Parent[\"Parent section\"] C1[\"Child chunk 1\"] C2[\"Child chunk 2\"] C3[\"Child chunk 3\"] end Q[\"Query\"] -->|matches the small chunk| C2 Parent -->|send the whole section| L[\"LLM\"]11. GraphRAG Some questions are about relationships, not passages: “Which suppliers are connected to both of our delayed product lines?” No single chunk contains that answer. GraphRAG builds a map of how people, companies, and things connect. It reads your documents and pulls out entities (people, companies, products, places) and the relationships between them. These are stored as a knowledge graph: a network of dots (entities) joined by lines (relationships). The system can then follow connections across several entities to answer questions that plain chunk search can’t. The downside is cost. Building and maintaining the graph is expensive, so use it when relationships are central to your use case, not by default. flowchart LR subgraph Build[\"Build the graph\"] D[\"Documents\"] --> E[\"Extract entities and relationships\"] --> G[\"Knowledge graph\"] end subgraph Ask[\"Example: which suppliers connect to both delayed lines?\"] SA[\"Supplier A\"] -->|supplies| P1[\"Product line 1\"] SA -->|supplies| P2[\"Product line 2\"] SB[\"Supplier B\"] -->|supplies| P1 end G --> Ask12. Context Compression Even good chunks often have a lot of filler around the one or two sentences that matter. Context compression goes through the retrieved chunks (say, 10 of them), keeps only the useful sentences, and drops the rest. This saves cost and helps the model focus. Long inputs can hide the key facts, so shorter, cleaner context often gives better answers. Just don’t compress so hard that you cut details the answer needs. flowchart LR A[\"10 retrieved chunks\"] --> B[\"Check each sentence against the question\"] B --> C{\"Relevant?\"} C -->|Yes| K[\"Keep the line\"] C -->|No| N[\"Drop as noise\"] K --> L[\"Compact context for the LLM\"]13. Agentic RAG A standard pipeline searches once, in a fixed way. In agentic RAG, an AI agent (an LLM that can plan and take steps on its own) decides when, where, and how often to search. It plans a search and retrieves, then asks: “Do I have enough information?” If not, it searches again with a better plan. If yes, it answers. This helps with vague or multi-step questions that one search would get wrong. The cost is more waiting time, more tokens, and a system that is harder to debug. Set a limit on the number of loops, and use agentic RAG only when a simpler pipeline isn’t enough. flowchart TD Q[\"Question\"] --> P[\"Agent plans the search\"] P --> R[\"Retrieve\"] R --> J{\"Enough context?\"} J -->|Not yet| P J -->|Yes| A[\"Generate the answer\"]Part 4: Measuring the results 14. Faithfulness Faithfulness measures how many of an answer’s claims are backed up by the text you retrieved. To check it, break the answer into individual claims and compare each one with the retrieved context. A claim that appears there is grounded. A claim that doesn’t is unsupported. Unsupported claims are the hallucinations (made-up or unverifiable statements) you want to avoid. That is true even when a particular claim happens to be correct, because you can’t trace it back to a source. Faithfulness tells you whether the model stuck to the material it was given. It doesn’t tell you whether that material was any good, which is why you also need the next metric. flowchart LR A[\"Generated answer\"] --> C[\"Break into individual claims\"] C --> K{\"Supported by the retrieved context?\"} K -->|In context| G[\"Grounded\"] K -->|Not in context| U[\"Unsupported: possible hallucination\"] G --> S[\"Faithfulness = grounded claims / all claims\"] U --> S15. Context Precision & Recall These two scores judge the search step itself. Precision asks whether what we found is relevant. Recall asks whether we found everything we needed: Precision: Of the chunks we retrieved, how many are actually relevant? (Some tools also reward putting the relevant ones near the top.) Recall: Of everything the answer needed, how much did we actually retrieve? Here is an example. Say you retrieve 10 chunks and 4 are useful. Precision is 40%. If the answer needed 5 useful chunks and you found 4 of them, recall is 80%. The two pull against each other. Retrieving more chunks raises recall but usually lowers precision. Recall is also harder to measure, because you need to know the full set of relevant information. That means you need reference answers or labeled test questions. Once you measure both, you know where to look: Low recall: check chunking, embeddings, hybrid search, or query rewriting. Low precision: check reranking, metadata filtering, or compression. Good retrieval but low faithfulness: check the prompt or the generation step. That turns “the answers feel off” into a specific, fixable problem. flowchart LR Q[\"Question\"] --> R[\"Retrieved chunks\"] R --> P[\"Precision: how many retrieved chunks are relevant?\"] N[\"All relevant chunks that exist\"] --> RC[\"Recall: how many of those did we retrieve?\"] R --> RCWhere to start You don’t need all 15 concepts on day one. Here is a sensible order: Get the basics right: sensible chunking, a good embedding model, and a vector database. Build a small test set of real questions, and measure precision, recall, and faithfulness from the start. Add hybrid search and reranking. These are common upgrades that tend to add noticeable quality for relatively little complexity. Add metadata filtering for freshness and access control. Add query rewriting if your product is a chat. Try the advanced techniques (HyDE, decomposition, small-to-big, compression) where your tests show a specific gap. Consider GraphRAG and agentic RAG only when simpler tools can’t solve the problem. Final thoughts RAG isn’t one technique. It is a chain of steps, and the final answer is only as good as the weakest link. The good news is that every link can be measured and improved. So the next time your RAG system gives a bad answer, don’t blame the model first. Look at what it was given. Chances are the fix is somewhere earlier in the pipeline.","contentHash":"sha256:8cda804e1e19c48bd94a32eb830036ccaff9dd38e8b1a9a597bd5e4d012ce127","authorName":"Samod Alex","authorUrl":"https://zyvop.com/author/samod","authorSameAs":[],"category":"AI","tags":["RAG","Generative AI","Vector Search","LLM","AI Engineering"],"audience":"Software engineers and developers building applications with AI","tone":"Instructional, practical, code-first","readingTimeMinutes":11,"wordCount":2878,"faqs":null,"primaryTopic":"AI","publishedAt":"2026-11-03T09:43:04.436Z","updatedAt":"2026-11-03T09:43:04.436Z","canonicalUrl":"https://zyvop.com/15-rag-concepts-every-ai-builder-should-know-b2468"}