Query-Time Intelligence: Teaching Systems to Decide When They Actually Know

The pipeline flips

Part 1 and Part 2 laid out the core limit in traditional RAG: it makes its chunking decision at index time, then lives with it forever. That locks you into a trade-off. Optimize for one query pattern, degrade another. Nine different chunking strategies exist, and every one is a compromise in a different direction.

The fix isn't a better chunking strategy. It's making the chunking decision at the moment you actually know what the user wants, instead of guessing months in advance.

That's the core idea behind the shift from Retrieval-Augmented Generation to Retrieval Language Models. It's not a technical upgrade so much as a different idea of when a system should make its decisions.

Here's the old pipeline. Index time is smart, expensive, and happens once: analyze documents, pick a chunking strategy, generate embeddings, store them in a vector database. Query time is fast, cheap, and happens constantly: a user asks a question, you retrieve similar chunks, pass them to the LLM, generate an answer.

All the intelligence lived at index time. Query time was just lookups and generation. That made sense when embedding models were slow and expensive. It also locked every decision in before anyone knew what users would ask.

Three things changed

Between 2023 and 2025, three things moved.

LLMs got fast and cheap enough to run real logic at query time. Claude Sonnet, GPT-4, and similar models can now analyze a query, break down a complex question, and orchestrate a retrieval strategy in milliseconds.

Long-context models went mainstream. Models with context windows past 1 million tokens mean you can hand the LLM entire documents or document collections, not just pre-chunked fragments. That opens up architectures that didn't exist two years ago.

Agentic frameworks matured. LangChain, LangGraph, LlamaIndex, and DSPy moved from proof-of-concept tools to production systems with real orchestration, error handling, and optimization.

The result: systems that make chunking and retrieval decisions at query time, when they actually know what the user wants. The intelligence moved from index time to query time.

Not every query needs retrieval

The clearest example of this shift is KAIST's Adaptive-RAG, published in March 2024.[1] The insight is simple. Not every query needs retrieval.

Ask a system "What is photosynthesis?" and the LLM already knows the answer from training. Retrieval just adds latency and cost for nothing. Ask "What were the specific photosynthesis research findings from our lab in Q3 2024?" and retrieval is essential. That's not in the training data anywhere.

Traditional RAG treats every query the same: retrieve, pass to LLM, generate. Adaptive-RAG runs a small classifier that predicts query complexity and routes accordingly. Simple queries get no retrieval, just the model's own knowledge. Moderate queries get single-step retrieval. Complex queries get multi-step, iterative retrieval.

That classifier trains on real performance data. The system runs thousands of queries both with and without retrieval, measures which performs better, and learns when retrieval actually helps versus when it just adds noise and delay.

The results hold up. On benchmark datasets, Adaptive-RAG matched or beat traditional RAG while cutting unnecessary retrieval by 40%. That's not just faster. It's cheaper, and it gives cleaner answers on the simple queries that never needed a document in the first place.

Checking the work before you send it

Where Adaptive-RAG routes queries to different strategies, Self-RAG and CRAG take a different approach. They check retrieval quality at runtime and adjust.

Self-RAG uses "reflection tokens," special tokens the model generates to grade its own retrieval and generation.[8] ISREL tokens ask: is this retrieved document actually relevant to the query? ISSUP tokens ask: does this document actually support my answer? Need-retrieval tokens ask: should I even be retrieving for this query?

The system generates several candidate answers with citations, scores each for relevance and factual support, and picks the best one. It's a built-in fact-checker that runs before the answer ever reaches the user.

CRAG, Corrective RAG, adds a lightweight evaluator that grades document quality in real time.[9] High confidence in the retrieved docs, use them as-is. Medium confidence, filter and refine before using them. Low confidence, trigger a web search for better sources.

The insight underneath both: retrieval isn't a yes or no. It's not "the retrieved docs are perfect" or "retrieval failed." It's a spectrum of quality, and you can score that quality programmatically and act on it. Both approaches target a real production problem: irrelevant retrieval. Vector search returns the most semantically similar chunks, and semantic similarity doesn't guarantee relevance or accuracy. Self-RAG and CRAG add a quality gate before generation runs.

When one question hides five

Query complexity isn't only about whether to retrieve. It's also about how.

Complex questions often need multi-hop reasoning: answering a chain of sub-questions to build up to the real answer. Take "How did our Q3 server costs compare to Q2, and what drove the increase?" That needs Q3 costs, Q2 costs, the difference between them, the cost drivers in Q3, and which of those were new versus simply larger.

Traditional RAG retrieves chunks for "Q3 server costs" and hopes the rest is in there. RAP-RAG, Retrieval-Augmented Planning RAG, breaks the query into a graph of sub-questions, retrieves for each, then synthesizes the result.[3] It builds a weighted graph index that combines semantic similarity (standard vector search), structural connectivity (document hierarchies, citations, references), and temporal relationships (what precedes or follows what). An adaptive planner reads the query, picks retrieval methods based on its features, and runs a multi-step retrieval strategy. The payoff: systems that handle complex analytical queries that would trip up traditional RAG entirely.

IRCoT, Interleaving Retrieval with Chain-of-Thought, goes further still, alternating reasoning steps with retrieval steps.[24] Reason about what's needed, retrieve it, reason about what's still missing, retrieve again, repeat until the question is answered. It's closer to how a person actually researches something hard: iterative, adaptive, responsive to what you learn at each step.

Solving chunk size by not choosing one

Different query types need different chunk sizes. That was the core problem from Part 1. Mix-of-Granularity solves it by pre-chunking documents at multiple sizes at once and using a trained router to pick the right one per query.[27] Your corpus exists in three forms at the same time: small chunks (256 tokens) for factoid queries, medium chunks (512) for moderate complexity, large chunks (1,024 and up) for analytical queries. A lightweight classifier predicts the query type and routes to the right size. You pay for triple-indexing in storage. You get the right chunk size for every query instead of one compromise size for none of them.

LongRAG takes a different route, one that only long-context models make possible.[26] Instead of retrieving several small chunks, it retrieves a handful of large, compressed contexts. The compression happens at index time, using the LLM itself: a 10,000-token document compresses down to a 1,000-token summary that keeps the key facts and structure. At query time, the system retrieves 5 to 10 of these summaries, 5,000 to 10,000 tokens total, and hands the full compressed context to a long-context LLM. The model reasons across the whole retrieved context instead of stitching together fragments. None of this was practical three years ago. Passing 10,000 tokens to an LLM was too expensive to consider. Today it's routine.

That's how far the intelligence has moved from index time to query time. What's left is the part most people skip past: retrieval isn't even the whole toolset anymore. It's one tool of many, and that changes the architecture again.

← Back to Part 2: The hidden cost of re-embedding Continue to Part 4: Tools, frameworks, and the production reality of running this for real →

Works Cited

[1] Jeong, S., et al. "Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models Through Question Complexity." KAIST, arXiv, March 2024. [3] Li, X., et al. "RAP-RAG: A Retrieval-Augmented Generation Framework with Adaptive Retrieval Task Planning." MDPI, October 2025. [8] Asai, A., et al. "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." arXiv, 2024. [9] Yan, S., et al. "Corrective Retrieval Augmented Generation (CRAG)." arXiv, 2024. [24] Trivedi, H., et al. "Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT)." arXiv, 2024. [26] "LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs." arXiv, 2024. [27] "Mix-of-Granularity: Adaptive Chunk Size Selection for RAG Systems." Research Paper, 2024.