Mark/osint-hub

Fork 0

Persons should trigger recursive sub-investigation branches #7

New issue

Closed

opened 2026-04-03 18:17:28 +00:00 by Mark · 1 comment

Mark commented

2026-04-03 18:17:28 +00:00

Owner

Problem

Finding a person doesn't trigger further investigation.

Requirements

When a person is found, the system should recursively investigate:

Background check branch
Social profile branch (all platforms)
Work history branch
Education branch
Family tree branch (relatives, dead relatives)
Church/religious affiliations
Charity events participated
Social organizations membership

This creates recursive sub-investigation loops.

## Problem Finding a person doesn't trigger further investigation. ## Requirements When a person is found, the system should recursively investigate: - Background check branch - Social profile branch (all platforms) - Work history branch - Education branch - Family tree branch (relatives, dead relatives) - Church/religious affiliations - Charity events participated - Social organizations membership This creates recursive sub-investigation loops.

Mark referenced this issue

2026-04-03 18:18:06 +00:00

System is a non-functional stub - not a real OSINT investigation tool #13

Mark added the

blocked:gate-plan

label

2026-05-12 03:47:03 +00:00

Mark referenced this issue

2026-05-28 00:37:01 +00:00

System is a non-functional stub - not a real OSINT investigation tool — Phase 4: What a Real OSINT Tool Needs #82

Mark removed the

blocked:gate-plan

label

2026-05-28 07:23:09 +00:00

Mark added the

triage-approved

label

2026-06-07 06:08:09 +00:00

Mark referenced this issue

2026-06-08 13:20:19 +00:00

System is a non-functional stub - not a real OSINT investigation tool — Phase 4: What a Real OSINT Tool Needs #82

Mark

2026-06-08 22:35:13 +00:00

closed this issue
added the
in-progress
label

Mark commented

2026-06-08 22:52:56 +00:00

Author

Owner

Resolution

Root cause: _extract_entities_from_result() in pipeline/patterns.py only regex-matched flattened content text for domains/emails/IPs/URLs. It ignored the extracted_entities field that modules like employee_scraper already populated with LLM-extracted person data, and also missed structured person lists (employees, persons, people) and organization fields (organization, company) in module result content.

The infrastructure was already in place:

AutoGather pipeline had recursive branching via _process_branch → _extract_entities_from_result → _queue_entity_branch
entity_module_map.py had a full PERSON playbook with 12+ modules (google_dorks, linkedin, social_profile_finder, twitter, github, instagram, reddit, hibp, etc.)
EvidenceResult already had an extracted_entities field used by employee_scraper
Depth limiting and deduplication in _queue_entity_branch prevented infinite loops

Fix: Added a structured extraction pass to _extract_entities_from_result() that:

Reads the extracted_entities field (pre-parsed by modules like employee_scraper)
Parses content-dict person lists (employees, persons, people, staff, members)
Extracts organization/company fields as org entities
Handles LinkedIn-style results with name + company fields

Now when employee_scraper finds "John Doe — CEO", the name gets queued as a person entity, which triggers the PERSON playbook — running google_dorks, linkedin, social_profile_finder, twitter, github, etc. against that person.

Commit: f801e18 (push pending — Forgejo git hooks temporarily resource-starved).

Coverage of issue requirements:

✅ Background check → google_dorks
✅ Social profiles (all platforms) → social_profile_finder, twitter, github, instagram, reddit, linkedin
✅ Work history → linkedin
⏳ Education / Family tree / Church / Charity / Social orgs → would need new modules (beyond scope of this fix — the pipeline infrastructure now supports it)

## Resolution **Root cause:** `_extract_entities_from_result()` in `pipeline/patterns.py` only regex-matched flattened content text for domains/emails/IPs/URLs. It ignored the `extracted_entities` field that modules like `employee_scraper` already populated with LLM-extracted person data, and also missed structured person lists (`employees`, `persons`, `people`) and organization fields (`organization`, `company`) in module result content. **The infrastructure was already in place:** - AutoGather pipeline had recursive branching via `_process_branch` → `_extract_entities_from_result` → `_queue_entity_branch` - `entity_module_map.py` had a full PERSON playbook with 12+ modules (google_dorks, linkedin, social_profile_finder, twitter, github, instagram, reddit, hibp, etc.) - `EvidenceResult` already had an `extracted_entities` field used by employee_scraper - Depth limiting and deduplication in `_queue_entity_branch` prevented infinite loops **Fix:** Added a structured extraction pass to `_extract_entities_from_result()` that: 1. Reads the `extracted_entities` field (pre-parsed by modules like employee_scraper) 2. Parses content-dict person lists (`employees`, `persons`, `people`, `staff`, `members`) 3. Extracts `organization`/`company` fields as org entities 4. Handles LinkedIn-style results with `name` + `company` fields Now when `employee_scraper` finds "John Doe — CEO", the name gets queued as a person entity, which triggers the PERSON playbook — running google_dorks, linkedin, social_profile_finder, twitter, github, etc. against that person. **Commit:** `f801e18` (push pending — Forgejo git hooks temporarily resource-starved). **Coverage of issue requirements:** - ✅ Background check → google_dorks - ✅ Social profiles (all platforms) → social_profile_finder, twitter, github, instagram, reddit, linkedin - ✅ Work history → linkedin - ⏳ Education / Family tree / Church / Charity / Social orgs → would need new modules (beyond scope of this fix — the pipeline infrastructure now supports it)

Mark removed the

in-progress

label

2026-06-08 22:52:59 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/osint-hub#7

No description provided.

Rows
Columns