You built the RAG pipeline. Chunking is tuned, embeddings are solid, retrieval precision looks good on your eval set. The agent still gets things wrong in production, and the failures are strange.
It answers a question about the release process by citing a document from a team that does not own releases. It cannot find a policy that definitely exists because the policy is filed under an acronym nobody outside one team uses. It gives an answer that is correct in general and wrong for your company.
These are not retrieval quality failures. They are structural limits of similarity search, and you cannot tune your way out of them.
Three failure modes vector search cannot fix
Relationships are not semantic neighbors
Vector search answers one question: which passages are semantically close to this query? That is genuinely useful and completely insufficient for anything requiring multi-hop reasoning over structure.
Consider the query: "Who needs to approve this before it ships?"
The correct answer depends on the item type, which team owns it, that team's reporting structure, which policy applies at that stage, and who currently holds the approving role. That is a traversal across five relationship types. Embedding similarity gives you documents that talk about approval, which is not the same thing and does not compose.
You can chunk more aggressively and add metadata filters. What you cannot do is retrieve a relationship that was never stored as a relationship, and flat vector stores do not store relationships.
Every organization speaks a private dialect
Your company has a term for its requirements documents. It is probably an acronym. It probably collides with another expansion of the same acronym used by a different team in the same building.
"SRD" means Software Requirements Document on one team and System Reference Design on another. Both teams are certain their meaning is the obvious one. Neither has documented the distinction, because to them there is nothing to document.
Embedding models are trained on public corpora. They encode general-language semantics. They have no way to know that in your organization, an SRD from the platform team requires security review and an SRD from the hardware team does not.
The failure mode is subtle and bad. A query using the canonical term returns nothing, because your evidence is all filed under the local one. Or worse, it returns evidence from the wrong team's usage and the agent produces a confidently incorrect answer with a real citation attached.
Similarity has no notion of truth
Vector search retrieves what is similar. It has no opinion on whether the retrieved content is correct, current, or authoritative.
Your corpus contains a policy document from 2023, a chat message where someone proposed changing that policy, a meeting note where the change was rejected, and a Slack thread where two people implemented it anyway. All four are semantically close to a query about the policy. Nothing in the embedding space tells you which one governs.
Agents that take real actions cannot operate on retrieval alone. They need a layer that establishes what is actually true, and that layer requires human judgment somewhere in it.
The architecture that addresses all three
Here is how Dactic's pipeline is structured. The sequence matters more than any individual component.
Investigator sync → Noise Filter → PII redaction → EvidenceItems
→ Footprint Analyzer → Detective interview → Cross-Examination
→ Org Lexicon → Playbook Generator → Champion Review
→ grounded agent context
Observe before you ask
The Investigator is a crawler that ingests approved resources across connected platforms: Gmail labels, Chat spaces, Drive folders, Calendars, monday.com boards. It builds an observable footprint of the organization. People, tools, channels, board structures, recurring processes, volumes.
Critically, this runs before any employee is interviewed. The point is not to skip the interview. It is to make the interview specific.
Two filters run before storage. A rule-based keyword pass, then an LLM pass using a cost-efficient model, discarding low-signal noise before anything gets written. Documents get cleansed to token-efficient text with branding, layout artifacts, and image noise stripped.
PII redaction happens before embedding generation and before any graph write. Not after. Redaction is logged with counts and categories only, never raw values. If PII enters the index, it is in the index, so the ordering here is load-bearing rather than cosmetic.
Mine candidate terms, then interview about those
The Footprint Analyzer mines candidate terms from the collected evidence: recurring file and title tokens, acronyms, board and column labels, channel names. Each candidate is scored by frequency, ambiguity, and centrality, producing a ranked list of terms that need human confirmation. Acronym collisions get flagged explicitly.
This ranked list drives interview question generation. The Detective conducts a conversational interview over a WebSocket, targeting roughly seven minutes of active answer time for the role discovery stage.
Stage one captures role, team, reporting lines, core responsibilities, recurring tasks, primary tools, key handoffs, and the artifacts the employee creates. Reporting lines and handoffs persist as structured graph relationships, not free text, because they seed people and dependency edges you will traverse later.
Stage two asks only about terms actually observed in that employee's connected sources. No generic vocabulary quizzing. A question looks like: when your team says "SRD," which do you mean, with disambiguation options and provenance showing how many evidence items the term was drawn from.
Interview state persists server-side, so a dropped connection resumes at the last answered turn rather than restarting.
Normalize with a confidence cascade
The Org Lexicon resolves each term to a canonical concept through four steps in order:
- Exact alias match
- Embedding similarity against canonical centroids
- LLM disambiguation using a cost-efficient model
- Below confidence threshold, queue for human confirmation Each entry stores the canonical concept, alias, owner role, confidence score, provenance, and status of confirmed, inferred, or needs review.
Two design decisions here are worth stealing regardless of what you are building.
Normalization is additive. The evidence node stores both rawLabel and the resolved canonicalType plus aliasId. The original label is never discarded. This means a query phrased canonically returns evidence filed under org-specific vocabulary through lexicon expansion, and generated output renders in the org's own language with the canonical term as metadata. You get retrieval recall without losing the local dialect that makes output feel correct to the reader.
Conflicts are flagged, never auto-resolved. When two employees contradict each other on a mapping, the conflict routes to human reconciliation. Silently picking a winner produces a knowledge base that is internally consistent and wrong, which is harder to debug than one that is openly uncertain.
Re-mining on the 30-day refresh re-scores existing aliases and flags drift.
Traverse, do not just match
Retrieval runs multi-hop graph traversal rather than pure vector match. The target data layer moves from PostgreSQL with pgvector to AWS Neptune with Bedrock-generated embeddings as multi-tenant scale requires.
A query for a policy point returns connected evidence plus related employees and dependencies within N hops, in a structured consumable format. Queries expand through the lexicon in both directions.
Gate on human approval
The Playbook Generator synthesizes a versioned playbook from indexed evidence. Every policy point cites at least one originating evidence item and renders in the team's lexicon. Playbooks are created in PENDING_REVIEW.
A Champion reviews each policy against its cited evidence and approves, edits, requests changes, or sends back. Nothing in review grounds a live agent. Inferred lexicon terms sit in needs review until confirmed.
This is the layer that gives you ground truth, and there is no version of it that removes the human.
From approved context to deployed agent
Agent generation takes an approved profile and emits a platform-agnostic bundle: persona, objective, procedure, decision rules, communication, guardrails, escalation, parameters, termination, skills, knowledge, tools, triggers, permissions, model.
The bundle renders per target. Today that target is monday.com's native Agents API. A Dactic-hosted path over Amazon Bedrock AgentCore is planned as the second, and the abstraction exists so adding it does not require re-authoring anything already deployed.
A pre-deploy validator blocks deployment on any spec violation. All required instruction sections present, every dynamic value bound to a named parameter, at least one board and one trigger present, permission scope declared and minimal, web search state explicit, context in supported formats with no PII or raw logs, version and timestamp stamps present.
Determinism is enforced at the artifact level. Generated instructions name exact boards, columns, statuses, and recipients. Never "decide as appropriate." An agent given latitude to decide will decide differently across runs, which makes it unauditable and unusable for anything consequential.
Deployment is idempotent. Re-runs update the existing agent by ID rather than creating duplicates, which is what makes the 30-day refresh cycle safe to automate.
Two execution modes govern behavior. Goal agents run until a designated outcome, then stop. Loop agents run recurringly until halted, and ship with mandatory max-iteration caps, cost ceilings, and automatic pause-and-notify on anomaly. Anomaly detection does not rely on a human noticing a spend spike.
Model selection is an evaluation problem
Model choice per pipeline stage is decided by a formal evaluation program rather than by preference. Candidate model and prompt pairs are screened against hand-built gold test companies under frozen, identical conditions before anything ships.
High-volume build-time roles run on cost-efficient models chosen where they match frontier accuracy at a fraction of the cost. Judgment-heavy stages, such as applying a Champion's review corrections, route to a frontier model where it demonstrably outperforms cheaper options.
Model IDs are version-pinned and the regression suite re-runs on provider updates. Silent behavior change from an upstream model update is a real failure mode and pinning is the only defense.
Each agent gets a complexity tier with a target credits-per-run budget assigned before generation, which bounds model choice and cadence. Post-deploy, token usage is metered per agent and per organization, with estimated versus actual reconciled. Agents exceeding tier budget are treated as re-engineering signals.
What to take from this
If you are building agent infrastructure, three things here generalize.
Store relationships as relationships. You cannot retrieve structure you never modeled. Retrofitting a graph onto a vector store after you discover you need multi-hop reasoning is substantially more painful than starting with one.
Handle organizational vocabulary explicitly. Every enterprise deployment hits this, usually late, usually as a mysterious retrieval failure. Additive normalization that preserves the raw label costs little and prevents a class of bug that is genuinely hard to diagnose.
Put the human gate before grounding, not after output. Reviewing agent output means catching errors one at a time forever. Reviewing the knowledge base means catching each error once, before it propagates into every agent that depends on it.
The last one is the load-bearing insight. Everything else is implementation.