Query-Time Intelligence: Tools, Trade-offs, and the Hybrid Future

Retrieval is one tool among many

The deeper shift isn't in retrieval strategy. It's in realizing retrieval is just one tool, not the only one. Traditional RAG has exactly one move: semantic search over pre-chunked documents. Agentic RAG systems carry dozens, sometimes hundreds, of tools: databases, APIs, calculators, code interpreters, web search, specialized retrievers.

That's where Tool RAG comes in.[13] As the tool count grows, naive approaches break down. Give an LLM 100 tool descriptions in the prompt and you blow past context limits and confuse the model. The fix: apply RAG's own logic to tool selection. Instead of retrieving document chunks, retrieve tool descriptions. A user asks a question. The system runs semantic search over tool schemas, pulls the 5 to 10 most relevant tools, and lets the LLM choose only from those, not all 100. It executes the chosen tool, then generates the answer.

The research backs this up: it can triple tool invocation accuracy while cutting prompt length in half.[13] It solves the real scaling problem, how you give a model access to thousands of tools without drowning it in options.

The frameworks are production-ready now

Four frameworks now do most of the heavy lifting.[15,16,17,18,19,20]

LangChain is the construction kit, with 50,000-plus integrations. Maximum flexibility, higher overhead: about 10ms and 2.4k tokens per interaction. Best for fast prototyping and systems where breadth matters more than efficiency.[17,34]

LlamaIndex is built for data-heavy retrieval, with 150-plus data connectors. Lower overhead: about 6ms and 1.6k tokens. Best when search performance is the priority.[18,34]

LangGraph handles graph-based agent orchestration, with human-in-the-loop controls and checkpointing built in. It's become the go-to choice for enterprise agentic workflows in 2025. It handles real conditional branching: if retrieval confidence is low, trigger web search, then verify with a database query.[14,19]

DSPy focuses on automated optimization of LLM outputs. Lowest framework overhead of the four, about 3.5ms. Best where you need consistent, reproducible output.[20,34]

These frameworks absorb the orchestration work that would be brutal to build from scratch: error handling, retry logic, state management, tool chaining, result fusion.

Two pipelines, side by side

Here's the architectural difference between traditional RAG and the RLM approach, in code.

Traditional RAG:

# Index time (once)
chunks = chunk_text(document, size=512, overlap=50)
embeddings = embed(chunks)
store(embeddings, vector_db)

# Query time (repeated)
query_embedding = embed(query)
results = vector_db.similarity_search(query_embedding, k=5)
context = concatenate(results)
answer = llm.generate(context + query)

Everything here is pre-computed. Query time is just a lookup and a generation call.

The RLM, or agentic, approach:

# Query time
query_analysis = llm.analyze(query)
strategy = select_strategy(query_analysis.complexity)

if strategy == "simple":
    answer = llm.generate(query)
elif strategy == "moderate":
    chunks = adaptive_chunk(documents, query_analysis.topics)
    relevant = filter_by_relevance(chunks, query)
    answer = llm.generate(relevant + query)
else: # complex
    subqueries = llm.decompose(query)
    results = [retrieve_and_answer(sq) for sq in subqueries]
    answer = llm.synthesize(results, query)

Chunking happens at query time, matched to the actual request. Strategy selection happens at query time, based on the real query, not an average one. Filtering and refinement happen at query time, with full context.

This costs more per query. It's also more capable. You're trading compute for intelligence: the system adapts to what the user actually needs instead of guessing at index time.

What this costs you in production

Multi-agent systems, adaptive routing, iterative retrieval. All of it reads well in a paper. Here's what it costs in the real world.

Latency. Adaptive systems add 100-500ms over simple RAG. Multi-step retrieval or tool orchestration adds another 200-800ms per step. For interactive applications, that matters. For analytical queries where accuracy beats speed, it's a fair trade.

Cost. More LLM calls means more spend. Self-RAG, generating and grading several candidate answers, can cost 3 to 5 times a simple RAG query. Semantic caching helps, storing query-context-result triples for reuse, but hit rates vary a lot by application.

Failure modes. Agentic systems get stuck. The agent loops, retrying a strategy that already failed. You need an explicit way out: if retrieval confidence hasn't improved after three attempts, escalate to a human. Traditional RAG might give a bad answer, but it doesn't get stuck in a loop.

Complexity. These systems are harder to debug. With traditional RAG, a wrong answer sends you to check the retrieved chunks and fix your chunking or embeddings. With agentic systems, the failure could sit in query decomposition, strategy selection, tool invocation, result synthesis, or some mix of all four. Observability stops being optional.

What the trade-off buys you

For complex queries needing multi-hop reasoning, adaptive systems land 15-30% more accurate than traditional RAG. Where accuracy matters more than speed or cost, that trade-off is worth making. Multi-stage retrieval with contextual re-ranking showed a 15% gain in retrieval precision on legal documents.[29] Adaptive approaches tripled tool invocation accuracy while cutting prompt length in half. Clinical decision support using adaptive chunking hit 87% accuracy against a 50% baseline.[30]

The industry is settling on a hybrid answer. Traditional RAG fits high-volume, low-complexity queries, latency-sensitive applications, well-defined domains with stable query patterns, and cost-sensitive deployments. Adaptive or agentic RAG fits complex analytical queries with multi-hop reasoning, applications where accuracy is critical (medical, legal, financial), varied query patterns across domains, and use cases that justify the higher per-query cost.

The sweet spot routes between the two based on the query itself. Simple queries get fast traditional RAG. Complex queries get adaptive, multi-step processing. The routing decision stays cheap and fast. It's the same logic as web caching: simple requests hit the cache, fast and cheap; complex requests hit the database and the business logic, slower but accurate. The trick is routing most traffic down the fast path while keeping the slow, careful path for the queries that need it.

Where the market is going

The RAG market's growth from $1.85 billion in 2024 toward a projected $5 billion-plus by 2027 is being driven by enterprises running into the limits of first-generation systems.[31,32] Newer forecasts run higher still: MarketsandMarkets now projects roughly $9.9 billion by 2030, at a compound growth rate above 40%. The companies winning competitive bids aren't pitching "we have RAG" anymore. They're pitching "we have adaptive RAG that handles your complex queries."

Framework maturity is moving just as fast. LangGraph showed up in late 2024 and by early 2025 had become the default choice for enterprise agentic workflows. DSPy went from an academic project to more than 23,000 GitHub stars in under two years.[20,33] These aren't experiments anymore. They're production tools running real applications.

The stack is stabilizing around a pattern: LlamaIndex for data-heavy indexing and retrieval, LangChain or LangGraph for orchestration and agent logic, specialized embedding models like ColBERT v2 and late-chunking models, long-context LLMs (Claude, GPT-4, Gemini) for synthesis, and semantic caching to manage cost. That convergence is a sign of maturity. The "try every new technique" phase is giving way to established patterns.

Intelligence at the edge

This shift isn't unique to retrieval. It's part of a bigger pattern across AI: moving intelligence to the edge, to the point where a decision actually gets made. Five years ago, everything got pre-computed. Batch processing ran the show. Real-time was expensive and rare. Today, LLMs are fast and cheap enough to do real analysis at query time: dynamic chunking based on the actual query, task-specific adaptation to query complexity, multi-step reasoning with iterative refinement, tool orchestration across hundreds of available tools, and self-correction before results ever reach the user.

The same pattern shows up outside retrieval. Code generation moved from static templates to LLM-generated code per request. Interfaces moved from fixed layouts to AI-generated, adaptive ones. Data analysis moved from pre-built dashboards to natural-language queries. Content creation moved from templates to fully custom generation on demand.

The common thread: stop deciding things when you have the least information, at index time, design time, build time, and start deciding them when you have the most, at query time, interaction time, runtime. That takes more compute per interaction. But compute keeps getting cheaper while the value of having the right information at the right moment keeps going up. The economics favor moving the intelligence to the point of decision.

What to actually do

If you're building a RAG system today, a few things are worth knowing.

Don't start with agentic frameworks. Start with simple RAG. Learn your actual query patterns. Measure what fails. Add complexity only where it earns its cost.

Instrument everything. You can't optimize what you don't measure. Log query complexity, retrieval confidence, answer quality, latency, and cost. The data will tell you exactly where adaptive approaches pay off and where they're overkill.

Plan for change. Your chunking strategy will change. Your retrieval approach will change. Your users' needs will change. Build the system to support experimentation: A/B test strategies, measure satisfaction, iterate.

Count the full cost. Adaptive systems cost more per query but may serve fewer total queries, because better answers mean less back-and-forth. Factor in user time, support cost, and the cost of getting it wrong when accuracy actually matters.

Invest in observability early. Agentic systems are complex. You need to see which tools the system picked, why it chose a given strategy, where retrieval failed, and how its confidence moved over the course of a query. Without that visibility, debugging is close to impossible.

The bet, resolved

The future of retrieval isn't a perfect chunking strategy. It's a system smart enough to adapt its chunking, retrieval, and synthesis to the specific question in front of it. That future is already here. The frameworks are mature. The economics favor query-time intelligence. The only real question left is when to adopt these approaches, and for which part of your system.

Key takeaways

Static chunking optimizes for the wrong problem. It buys index-time efficiency at the cost of query-time accuracy.

Query complexity varies more than most systems admit. One-size-fits-all retrieval fails because not every question needs the same amount of work.

Intelligence belongs at query time, when you actually know what the user wants, not at index time, when you're guessing.

The frameworks have caught up. LangGraph, LlamaIndex, DSPy, and others are ready for production agentic workflows.

The trade-offs are real. Adaptive systems cost more per query and deliver meaningfully better accuracy on complex questions.

Hybrid wins. Route simple queries to fast traditional RAG. Route complex ones to adaptive, multi-step processing.

The market has already made its choice. Enterprises are moving from "we have RAG" to "we have intelligent, adaptive RAG."

The chunking dilemma from Part 1 was never going to be solved by finding the one perfect chunking strategy. It gets solved by systems smart enough to make that decision at the moment they actually know what they're optimizing for.

Luke Paxton. Square Mile Design, May 2026.

← Back to Part 3: Adaptive-RAG, Self-RAG, and the graph-based approaches Back to the start: Part 1, the chunking dilemma →

Works Cited

Sources for Parts 1 and 2 (chunking strategies and benchmarks):

[1] "Chunking Strategies for RAG: A Comprehensive Guide." Medium, November 2024. [2] Radhakrishnan, S., et al. "Comparative Evaluation of Advanced Chunking Techniques for Retrieval-Augmented Generation in LLMs for Clinical Decision Support." PMC - PubMed Central, 2024. [3] "Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation." arXiv, April 2025. [4] Günther, M., et al. "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models." Jina AI, 2024. [5] "Enhancing RAG System Performance Through Semantic Layout Chunking." Springer, 2024. [6] "The Rise and Evolution of RAG in 2024." RAGFlow, December 2024. [7] "Best Chunking Strategies for RAG in 2025." Firecrawl, 2025. [8] "RAG Evaluation: Best Practices and Common Pitfalls." NVIDIA Technical Blog, 2024. [9] "Chunking for RAG: Best Practices." Chroma Research, 2024. [10] "HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering." Research on Multi-hop QA with ColBERT v2, 2024. [11] "Retrieval-Augmented Generation Market Analysis and Forecast 2024-2027." Industry Market Research, 2024. [12] "Enterprise AI Adoption: RAG Integration Trends." Technology Research Report, 2024. [13] "Introducing Contextual Retrieval." Anthropic, 2024. https://www.anthropic.com/news/contextual-retrieval [14] "Contextual Chunk Embeddings: Best Practices for RAG Systems." Anthropic Technical Documentation, 2024.

Sources for Parts 3 and 4 (query-time intelligence and tool orchestration):

[1] Jeong, S., et al. "Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models Through Question Complexity." KAIST, arXiv, March 2024. [2] "The 2025 Guide to Retrieval-Augmented Generation (RAG)." Eden AI, 2025. [3] Li, X., et al. "RAP-RAG: A Retrieval-Augmented Generation Framework with Adaptive Retrieval Task Planning." MDPI, October 2025. [4] "Optimization of RAG Multi-Query Rewrite Generation Strategy Based on Markov Decision Process." ACM Digital Library, 2024. [5] "FAIR-RAG: Faithful Adaptive Iterative Refinement for RAG Systems." arXiv, October 2025. [6] "Retrieval-Augmented Generation: A Comprehensive Survey of Architectures and Techniques." arXiv, May 2025. [7] "HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks." arXiv, September 2025. [8] Asai, A., et al. "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." arXiv, 2024. [9] Yan, S., et al. "Corrective Retrieval Augmented Generation (CRAG)." arXiv, 2024. [10] "Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG." arXiv, February 2025. [11] "What is Agentic RAG." Weaviate, November 2024. https://weaviate.io/blog/what-is-agentic-rag [12] "What is Agentic RAG." IBM Research, November 2025. [13] "Tool RAG: The Next Breakthrough in Scalable AI Agents." Red Hat Developer Blog, November 2025. [14] "Build a Custom RAG Agent with LangGraph." LangChain Documentation, 2025. https://python.langchain.com/docs/tutorials/ [15] "LangChain vs LlamaIndex 2025: Complete RAG Framework Comparison." Latenode, 2025. [16] "RAG Frameworks: LangChain vs LangGraph vs LlamaIndex vs Haystack vs DSPy." AIMultiple, 2025. [17] "LangChain Documentation." LangChain, 2025. https://python.langchain.com/ [18] "LlamaIndex Documentation." LlamaIndex, 2025. https://docs.llamaindex.ai/ [19] "LangGraph Documentation." LangChain, 2025. https://langchain-ai.github.io/langgraph/ [20] "DSPy Documentation." Stanford NLP, 2024. https://github.com/stanfordnlp/dspy [21] "Instructor: Library for Structured LLM Outputs." Python/TypeScript/Ruby/Go, 2024. https://github.com/jxnl/instructor [22] "Pydantic AI Documentation." Pydantic, 2025. https://ai.pydantic.dev/ [23] "BAML: A Language for LLM Structured Outputs." BoundaryML, 2024. [24] Trivedi, H., et al. "Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT)." arXiv, 2024. [25] "KRAGEN: Knowledge Graph-Augmented Generation for Complex Question Answering." arXiv, 2024. [26] "LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs." arXiv, 2024. [27] "Mix-of-Granularity: Adaptive Chunk Size Selection for RAG Systems." Research Paper, 2024. [28] "FILCO: Filtering Context for Improved RAG Performance." arXiv, 2025. [29] "Multi-stage Retrieval Pipelines for Legal Document Analysis." Industry Case Study, 2024. [30] "Clinical Decision Support Systems Using Adaptive RAG: Performance Benchmarks." Mayo Clinic Research, 2024. [31] "Retrieval-Augmented Generation Market Forecast 2024-2027: Growth Trends and Enterprise Adoption." Market Research Report, 2024. [32] "Enterprise AI Framework Adoption Report 2025." Technology Industry Analysis, 2025. [33] "Agentic AI Frameworks 2025: Comparative Analysis." Flobotics, 2025. [34] "Framework Performance Benchmarks: Token Usage and Latency Analysis." Independent Research, 2024. [35] "Semantic Caching for LLM Applications: Implementation Patterns." Technical Documentation, 2024. [36] "Cost Optimization Strategies for Production RAG Systems." Industry White Paper, 2025.