GraphRAG: Your AI Agent Needs a Map, Not More Documents

Imagine asking an agent: if we change an active customer’s status to suspended, which systems and business processes could be affected?

Large enterprise teams try increasingly capable agents, larger context windows, more documents, more repositories, and more tools. The agent can still struggle to produce a complete answer after spending a costly number of tokens.

The information exists. That is not the problem. What is missing is the connection between the information, the topology of the systems, and the evidence behind each relationship. Without that structure, we leave the agent to reconstruct the same map every time.

A traditional RAG system might retrieve chunks containing “customer status” and “suspended”. The answer may sound reasonable, but reasonable is not enough when we carry that response forward, build on it, or make a decision.

The agent does not need twenty more chunks to rank. It needs a map.

Where RAG starts to struggle

Standard Retrieval-Augmented Generation splits documents into chunks, converts those chunks into embeddings, and retrieves the ones most similar to the question.

This works well when the answer exists in one specific place:

• What does this policy say about password expiration?
• Which API updates the customer address?
• What was decided in this architecture document?

These are direct retrieval questions. Similarity search can find text that sounds like the answer.

Enterprise questions are often relationship-bound:

• Which applications depend on these database table objects?
• What could break if we change API authentication?
• Why does customer onboarding call the legacy billing system?
• Which screens, APIs, events, and reports use the vendor_code field?

That is where a graph becomes useful.

From document chunks to connected knowledge

A knowledge graph stores entities and relationships. Entities might include:

• Applications and interfaces
• APIs and services
• Source-code methods
• Database tables
• Business capabilities
• Teams and processes
• Architecture diagrams and decisions
• Incidents, events, documents, and wikis

The relationships explain how they connect:

• A screen calls an API
• An API uses a service
• A service updates a table
• A service publishes an event
• Another service consumes that event
• A team owns the application
• An incident affected the service
• An architecture decision replaced an older decision

Instead of treating every paragraph as an independent piece of text, the graph preserves the path between them. An agent can start from “customer status”, find the related database field, follow the applications that read it, identify the APIs that update it, and trace the downstream event consumers.

This is the basic value of GraphRAG, a research direction published by Microsoft in 2024: use the graph to prepare better context before asking the model to answer.

The graph should prepare the context

GraphRAG is sometimes described as replacing vector search with a graph database. We do not see it that way.

Vector search remains useful because it can find the likely entry point when a question uses business language that does not exactly match system-defined names. Once that entry point is found, graph traversal and graph data science can expand the context with only the relevant pieces.

For example, vector search may connect “stopping a customer account” to the CustomerStatus entity. The context sent to the agent in one request can then include:

• The database definition
• The API method that changes the value
• The event published after the update
• The billing service consuming that event
• The owner of each system
• The source documents or code proving these relationships

That is more useful than filling the context window with the ten chunks that happened to contain similar words.

Do we need the source code?

A graph edge saying “Account API publishes StatusChanged” is useful, but where did it come from? The last thing you want is an LLM hallucination presented as architecture.

Every important graph entity and relationship should keep a pointer to its source. The graph helps the agent navigate; the source provides the evidence.

For the best results, context should include both:

1. The graph path that explains how the information is connected
2. The original source pointer that supports it

This makes the answer easier to verify, explain, and audit. We will write another post about why that matters.

Where do you get the graph?

The exciting thing about a knowledge graph is that almost anything can be traced and connected. The dangerous thing is exactly that as well.

Teams get excited modelling everything: systems, processes, documents, databases, servers, tickets, and meetings. Months later, they have an impressive graph and an agent that gets lost in it.

We tried a different approach: build from the questions. Identify the first use cases. If the use case is change-impact analysis, model the relationships needed to trace changes and build that first:

• UI to API
• API to its code nodes
• Code to databases and services
• Service to event
• Event to consumer
• Component to owner

Support every relationship with its source. If the use case is customer support, the graph will look different. It may connect customers, products, subscriptions, tickets, known issues, and policies.

The rule is simple: design the graph around the context the agent needs, not around everything the organization knows.

GraphRAG is not required for every question

Knowledge graphs are useful, but they add work. After building initial entities from extracted data, relationships must be validated. Systems and ownership change over time, so the graph must be recreated and re-indexed.

The graph can also become noisy when the same service appears under five different names or when an LLM extracts relationships too aggressively and follows irrelevant loops.

Regular vector RAG remains the right choice when:

• The answer is normally found in one or two documents
• The documents already have strong metadata
• Questions are mainly lookup or summarization
• The data changes too frequently to maintain useful relationships
• The additional cost and complexity are not justified

To optimize the agent tools in this case, run evaluations. That is another post.

Most mature enterprise agents will eventually use both.

Give the agent choices

An agent should not traverse the graph for every question. A practical setup gives the retrieval router several options:

• Vector search for semantically similar text
• Graph traversal for connected context
• Full-text or code search for exact identifiers
• Existing APIs for current operational data
• SQL for structured facts and calculations
• Scripts or shell commands when the task genuinely requires them

If the user asks what the application password policy says, use document retrieval and validate or ask a clarifying question if needed.

If they ask which applications would be affected by changing the identity provider, query the graph.

If they ask how many failed logins happened yesterday, query operational data and enforce the required access controls.

The intelligence is not only in the model. It is also in selecting the correct retrieval method and preparing the smallest useful context for the task.

Security must travel through the graph

Connected context can expose information a user would never find through normal document search. A user who can access one application document should not automatically see connected security incidents, customer records, or restricted architecture details.

Permissions must be applied during retrieval, before the context reaches the model. Filtering the final answer is too late because sensitive information has already entered the prompt.

This becomes especially important when the graph connects data across systems with different access models. The graph should understand not only how information is connected, but also who is allowed to follow each connection.

Entity tagging, permission boundaries, and subgraphs are some ways to enforce that rule.

The real goal

GraphRAG turns scattered enterprise information into something an AI agent can navigate and use as meaningful context.

If your agent answers direct questions but still misses the bigger picture, reach out. We can help straighten that out.