[P0] CRITICAL: Agentic Flow is Non-Functional - Orchestrator v3_llm Missing Entity Tools and Proper Prompts #93

New issue

Closed

opened 2026-06-10 03:42:28 +00:00 by Mark · 0 comments

Mark commented

2026-06-10 03:42:28 +00:00

Owner

🔴 CRITICAL: Agentic Flow is Completely Broken

Summary

The "active" orchestrator (orchestrator_v3_llm.py) is missing all entity creation tools and has fundamentally weak prompts. The LLM can only call modules but cannot create structured entities (organizations, people, findings, relationships). This makes the entire agentic flow non-functional.

Root Cause Analysis

Problem 1: Missing Tools

The v3_llm orchestrator ONLY provides the run_module tool:

# orchestrator_v3_llm.py line 182
tools = [build_run_module_tool(allowed_modules=allowed_modules)]

But the ORIGINAL orchestrator.py provided ALL tools:

# orchestrator.py lines 492-496
tools = (
    [build_run_module_tool(allowed_modules=allowed_modules)]
    + entity_tools_list  # create_or_update_org, create_or_update_person, etc.
    + get_all_evidence_tools()  # get_evidence_artifact_chunk
)

Missing tools:

create_or_update_org - Create organizations
create_or_update_person - Create persons
create_lead - Create investigation leads
create_finding - Create evidence-backed findings
create_relationship_edge - Create entity relationships
create_evidence_snippet - Create citeable evidence snippets
get_evidence_artifact_chunk - Retrieve evidence for analysis

Problem 2: Weak System Prompts

The v3_llm prompt (lines 455-473) is missing the detailed workflow guidance from the original orchestrator.py.

Problem 3: No Anti-Hallucination Safeguards

The v3_llm has no guidance on:

Only creating entities from actual evidence
Proper citation of evidence IDs
Distinguishing real data from metadata

Impact

LLM can gather data but cannot create entities - It runs 50 modules but all raw data is lost
No structured intelligence output - No organizations, people, relationships in database
Investigations are useless - Even successful "50 module calls" produce no actionable intel
Broken feedback loop - Cannot use discovered entities to inform next module selection

Required Fix

Step 1: Add All Tools Back

In orchestrator_v3_llm.py, change tools list to include entity and evidence tools.

Step 2: Restore Strong System Prompts

Restore the detailed prompts from orchestrator.py (lines 537-623).

Step 3: Wire Up Entity Tool Execution

The v3_llm orchestrator needs handlers for all entity tools.

Step 4: Use entity_module_map

The entity_module_map.py has 1000+ lines of entity-to-module mappings that should inform module selection.

Files to Modify

backend/app/services/investigations/orchestrator_v3_llm.py - Core fix
backend/app/services/investigations/entity_tools.py - Already exists, needs to be called
backend/app/services/investigations/entity_module_map.py - Should be utilized

Test Plan

After fix:

Run investigation on known domain (e.g., "example.com")
Verify organization is created in database
Verify persons are created from employee_scraper results
Verify relationships are created between entities
Verify findings are created with proper evidence citations

## 🔴 CRITICAL: Agentic Flow is Completely Broken ### Summary The "active" orchestrator (`orchestrator_v3_llm.py`) is **missing all entity creation tools** and has **fundamentally weak prompts**. The LLM can only call modules but cannot create structured entities (organizations, people, findings, relationships). This makes the entire agentic flow non-functional. ### Root Cause Analysis #### Problem 1: Missing Tools The v3_llm orchestrator ONLY provides the `run_module` tool: ```python # orchestrator_v3_llm.py line 182 tools = [build_run_module_tool(allowed_modules=allowed_modules)] ``` But the ORIGINAL orchestrator.py provided **ALL tools**: ```python # orchestrator.py lines 492-496 tools = ( [build_run_module_tool(allowed_modules=allowed_modules)] + entity_tools_list # create_or_update_org, create_or_update_person, etc. + get_all_evidence_tools() # get_evidence_artifact_chunk ) ``` **Missing tools:** - `create_or_update_org` - Create organizations - `create_or_update_person` - Create persons - `create_lead` - Create investigation leads - `create_finding` - Create evidence-backed findings - `create_relationship_edge` - Create entity relationships - `create_evidence_snippet` - Create citeable evidence snippets - `get_evidence_artifact_chunk` - Retrieve evidence for analysis #### Problem 2: Weak System Prompts The v3_llm prompt (lines 455-473) is missing the detailed workflow guidance from the original orchestrator.py. #### Problem 3: No Anti-Hallucination Safeguards The v3_llm has no guidance on: - Only creating entities from actual evidence - Proper citation of evidence IDs - Distinguishing real data from metadata ### Impact 1. **LLM can gather data but cannot create entities** - It runs 50 modules but all raw data is lost 2. **No structured intelligence output** - No organizations, people, relationships in database 3. **Investigations are useless** - Even successful "50 module calls" produce no actionable intel 4. **Broken feedback loop** - Cannot use discovered entities to inform next module selection ### Required Fix #### Step 1: Add All Tools Back In `orchestrator_v3_llm.py`, change tools list to include entity and evidence tools. #### Step 2: Restore Strong System Prompts Restore the detailed prompts from `orchestrator.py` (lines 537-623). #### Step 3: Wire Up Entity Tool Execution The v3_llm orchestrator needs handlers for all entity tools. #### Step 4: Use entity_module_map The `entity_module_map.py` has 1000+ lines of entity-to-module mappings that should inform module selection. ### Files to Modify 1. `backend/app/services/investigations/orchestrator_v3_llm.py` - Core fix 2. `backend/app/services/investigations/entity_tools.py` - Already exists, needs to be called 3. `backend/app/services/investigations/entity_module_map.py` - Should be utilized ### Test Plan After fix: 1. Run investigation on known domain (e.g., "example.com") 2. Verify organization is created in database 3. Verify persons are created from employee_scraper results 4. Verify relationships are created between entities 5. Verify findings are created with proper evidence citations