Mark/osint-hub
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

No module blocking/availability tracking - blocked modules not configurable — Phase 1: Problem #76

New issue
Closed
opened 2026-05-28 00:31:50 +00:00 by Mark · 3 comments
Mark commented 2026-05-28 00:31:50 +00:00
Owner
Copy link

Phase 1: Problem

There's no system for:

  1. Tracking which modules are blocked (require API keys that aren't configured)
  2. Tracking which modules are available (have required configuration)
  3. Configuring module availability per investigation
  4. Dynamic module selection based on availability

Currently orchestrator hardcodes:

disallowed = set()  # Allow all modules including playbooks
allowed_modules = [m for m in sorted(ModuleRegistry._modules.keys()) if m not in disallowed]

But many modules require API keys that may not be configured:

  • virustotal requires VIRUSTOTAL_API_KEY
  • shodan requires SHODAN_API_KEY
  • etc.

Decomposed from osint-hub#19

## Phase 1: Problem There's no system for: 1. Tracking which modules are blocked (require API keys that aren't configured) 2. Tracking which modules are available (have required configuration) 3. Configuring module availability per investigation 4. Dynamic module selection based on availability Currently orchestrator hardcodes: ```python disallowed = set() # Allow all modules including playbooks allowed_modules = [m for m in sorted(ModuleRegistry._modules.keys()) if m not in disallowed] ``` But many modules require API keys that may not be configured: - virustotal requires VIRUSTOTAL_API_KEY - shodan requires SHODAN_API_KEY - etc. --- _Decomposed from [osint-hub#19](https://forgejo.tail018ac4.ts.net/Mark/osint-hub/issues/19)_
Mark commented 2026-05-28 04:42:57 +00:00
Author
Owner
Copy link

⏸ Needs Decision

Blocking factors:

  • Didn't meet auto-approve criteria — scope=medium, risk=low, spec=aligned

Scope: medium Risk: low Spec: aligned Type: feature
Summary: Migrate main orchestrators (orchestrator.py, orchestrator_enhanced.py) from hardcoded disallowed = set() to dynamic availability-based module filtering with per-investigation blocking configuration.
Suggestion: Reuse existing module_availability.py in main orchestrators to filter by API key availability, add blocked_modules field to InvestigationRun/Case DB model, expose API endpoints for configuring blocked modules per investigation, and integrate into orchestrator.py and orchestrator_enhanced.py. orchestrator_v2.py already uses is_module_available() successfully — replicate that pattern.
Rationale: The infrastructure already exists: module_availability.py has is_module_available(), filter_available_modules(), and get_module_availability_info() with 128 lines of tests in test_module_availability.py. orchestrator_v2.py already uses it. The gap is that the two main orchestrators (orchestrator.py, orchestrator_enhanced.py) still use disallowed = set() which is a no-op — they run modules that will fail at runtime because API keys are missing. Per-investigation blocking requires a DB schema change and new API endpoints, but the availability-checking core is done. Medium scope because it touches ~8-12 files (2x orchestrators, models, router, tests) but low risk since the pattern is proven in v2.

Action: Review each blocking factor above. Remove blocked:needs-decision label to route back to triage.

## ⏸ Needs Decision **Blocking factors:** - **Didn't meet auto-approve criteria** — scope=medium, risk=low, spec=aligned --- **Scope:** medium **Risk:** low **Spec:** aligned **Type:** feature **Summary:** Migrate main orchestrators (orchestrator.py, orchestrator_enhanced.py) from hardcoded `disallowed = set()` to dynamic availability-based module filtering with per-investigation blocking configuration. **Suggestion:** Reuse existing `module_availability.py` in main orchestrators to filter by API key availability, add `blocked_modules` field to InvestigationRun/Case DB model, expose API endpoints for configuring blocked modules per investigation, and integrate into orchestrator.py and orchestrator_enhanced.py. `orchestrator_v2.py` already uses `is_module_available()` successfully — replicate that pattern. **Rationale:** The infrastructure already exists: `module_availability.py` has `is_module_available()`, `filter_available_modules()`, and `get_module_availability_info()` with 128 lines of tests in `test_module_availability.py`. `orchestrator_v2.py` already uses it. The gap is that the two main orchestrators (orchestrator.py, orchestrator_enhanced.py) still use `disallowed = set()` which is a no-op — they run modules that will fail at runtime because API keys are missing. Per-investigation blocking requires a DB schema change and new API endpoints, but the availability-checking core is done. Medium scope because it touches ~8-12 files (2x orchestrators, models, router, tests) but low risk since the pattern is proven in v2. **Action:** Review each blocking factor above. Remove `blocked:needs-decision` label to route back to triage.
Mark commented 2026-05-29 18:05:09 +00:00
Author
Owner
Copy link

GSD PLAN: response missing GATE: line.

GSD PLAN: response missing GATE: line.
Mark closed this issue 2026-06-08 16:40:18 +00:00
Mark commented 2026-06-08 16:50:05 +00:00
Author
Owner
Copy link

✅ Phase 1 Resolved — Per-Investigation Module Blocking Implemented

Changes made

  1. Model (backend/app/models.py):

    • Added blocked_modules: str field to InvestigationRun (JSON list of blocked module names)

  2. Availability layer (backend/app/services/investigations/module_availability.py):

    • Added get_run_blocked_modules(session, run_id) — combines global ModuleBlock DB blocks with per-run blocked_modules

  3. Orchestrators updated:

    • orchestrator.py — uses get_run_blocked_modules(session, run.id) instead of get_blocked_modules(session)
    • orchestrator_v3_llm.py — same migration

  4. API endpoint (backend/app/routers/investigations.py):

    • PATCH /investigations/{run_id}/blocked-modules — accepts {"blocked_modules": ["shodan", "virustotal"]}, pass empty list to clear

  5. Tests — 5 new tests for get_run_blocked_modules covering global-only, combined, empty, None run_id, and empty string edge cases. All 24 tests pass.

## ✅ Phase 1 Resolved — Per-Investigation Module Blocking Implemented ### Changes made 1. **Model** (`backend/app/models.py`): - Added `blocked_modules: str` field to `InvestigationRun` (JSON list of blocked module names) 2. **Availability layer** (`backend/app/services/investigations/module_availability.py`): - Added `get_run_blocked_modules(session, run_id)` — combines global `ModuleBlock` DB blocks with per-run `blocked_modules` 3. **Orchestrators updated**: - `orchestrator.py` — uses `get_run_blocked_modules(session, run.id)` instead of `get_blocked_modules(session)` - `orchestrator_v3_llm.py` — same migration 4. **API endpoint** (`backend/app/routers/investigations.py`): - `PATCH /investigations/{run_id}/blocked-modules` — accepts `{"blocked_modules": ["shodan", "virustotal"]}`, pass empty list to clear 5. **Tests** — 5 new tests for `get_run_blocked_modules` covering global-only, combined, empty, None run_id, and empty string edge cases. All 24 tests pass.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
agentic-osint

Agentic OSINT features

  
audit

   
blocked:commit-ready-failed

   
blocked:gate-fetch

   
blocked:gate-fetch

   
blocked:gate-lint

   
blocked:gate-plan

   
blocked:gate-research

   
blocked:needs-decision

   
blocked:needs-human

   
blocked:no-changes

   
blocked:preflight

Repo has dirty lint; kilo-gsd preflight refuses to dispatch until lint-cleanup lands

  
blocked:verify-failed

   
bug

Something is not working

  
chore

Config/cleanup item

  
deferred

Deferred — will revisit later

  
enhancement

Enhancement request

  
feature-request

   
in-progress

   
lint-baseline

   
lint-cleanup

kilo-gsd preflight lint gate: repo has dirty ruff state, cleanup required before fixer dispatch

  
needs-decompose

Issue needs decompose processing

  
needs-triage

   
performance

   
quickfix

   
security/medium

Medium severity security issue

  
triage-approved

Approved by triage

No labels
agentic-osint
audit
blocked:commit-ready-failed
blocked:gate-fetch
blocked:gate-fetch
blocked:gate-lint
blocked:gate-plan
blocked:gate-research
blocked:needs-decision
blocked:needs-human
blocked:no-changes
blocked:preflight
blocked:verify-failed
bug
chore
deferred
enhancement
feature-request
in-progress
lint-baseline
lint-cleanup
needs-decompose
needs-triage
performance
quickfix
security/medium
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/osint-hub#76
No description provided.