No parallel module execution - modules run sequentially even when independent #24

Closed
opened 2026-04-03 19:01:38 +00:00 by Mark · 30 comments
Owner

Problem

The orchestrator runs modules sequentially in the playbook_modules loop:

for module_name, options in core_modules:
    # Run module...

But many modules are independent and could run in parallel:

  • whois, dns_lookup, ssl_cert can all run simultaneously
  • web_scraper and browser_scraper are independent
  • Multiple dns_lookup calls (A, MX, TXT) can run in parallel

This makes investigations unnecessarily slow.

Files

  • backend/app/services/investigations/orchestrator.py:368-451 - sequential module loop

Solution Required

  1. Use asyncio.gather() to run independent modules in parallel
  2. Group modules by dependency:
    • Phase 1 (parallel): whois, dns_lookup, ssl_cert
    • Phase 2 (parallel after phase 1): web_scraper, browser_scraper, wayback_machine
    • Phase 3 (parallel after phase 2): employee_scraper, social_profile_finder
  3. Respect rate limits per domain/service
  4. Use semaphore to limit concurrent requests per service
## Problem The orchestrator runs modules sequentially in the playbook_modules loop: ```python for module_name, options in core_modules: # Run module... ``` But many modules are independent and could run in parallel: - whois, dns_lookup, ssl_cert can all run simultaneously - web_scraper and browser_scraper are independent - Multiple dns_lookup calls (A, MX, TXT) can run in parallel This makes investigations unnecessarily slow. ## Files - `backend/app/services/investigations/orchestrator.py:368-451` - sequential module loop ## Solution Required 1. Use asyncio.gather() to run independent modules in parallel 2. Group modules by dependency: - Phase 1 (parallel): whois, dns_lookup, ssl_cert - Phase 2 (parallel after phase 1): web_scraper, browser_scraper, wayback_machine - Phase 3 (parallel after phase 2): employee_scraper, social_profile_finder 3. Respect rate limits per domain/service 4. Use semaphore to limit concurrent requests per service
Author
Owner

🤖 Claimed by gsd-issue-fixer-osint-hub-24 — starting work

🤖 Claimed by `gsd-issue-fixer-osint-hub-24` — starting work
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — claimed — starting pipeline

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — claimed — starting pipeline
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — research

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — research
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — plan

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — plan
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — implement attempt 1

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — implement attempt 1
Author
Owner

🤖 Claimed by gsd-issue-fixer-osint-hub-24 — starting work

🤖 Claimed by `gsd-issue-fixer-osint-hub-24` — starting work
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — claimed — starting pipeline

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — claimed — starting pipeline
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — research

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — research
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — plan

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — plan
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — implement attempt 1

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — implement attempt 1
Author
Owner

🤖 Claimed by gsd-issue-fixer-osint-hub-24 — starting work

🤖 Claimed by `gsd-issue-fixer-osint-hub-24` — starting work
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — claimed — starting pipeline

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — claimed — starting pipeline
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — research

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — research
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — plan

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — plan
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — implement attempt 1

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — implement attempt 1
Author
Owner

🤖 Claimed by gsd-issue-fixer-osint-hub-24 — starting work

🤖 Claimed by `gsd-issue-fixer-osint-hub-24` — starting work
Author
Owner

🤖 Claimed by gsd-issue-fixer-osint-hub-24 — starting work

🤖 Claimed by `gsd-issue-fixer-osint-hub-24` — starting work
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — claimed — starting pipeline

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — claimed — starting pipeline
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — claimed — starting pipeline

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — claimed — starting pipeline
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — research

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — research
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — research

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — research
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — plan

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — plan
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — plan

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — plan
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — implement attempt 1

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — implement attempt 1
Author
Owner

🤖 [gsd-issue-fixer-osint-hub-24] still working — implement attempt 1

🤖 [`gsd-issue-fixer-osint-hub-24`] still working — implement attempt 1
Author
Owner

GSD VERIFY phase failed after 3 attempts.
═══ Gate: verify-passed (osint-hub#24) ═══
Mode: Tier 1 (functional only)

GATE OK: All gates up to 'implement' pass
── Tier 1: Functional ────────────────────────────────
Running tests...
❌ Tests failed (exit 4)
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
timeout: 60.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
ERROR: file or directory not found: tests/

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/b/Projects/osint-hub
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
Running linter...
✅ Lint clean
Running type checker...
✅ Types check clean
Previous test count: unknown

❌ GATE FAIL: verify-passed
- Tests failed with exit code 4
GATE FAIL: verify — blocked at 2026-05-27T19:40:52Z
Class: test-failure
Evidence: Verification failed: 1 failures
Git tag: osint-hub#24-gate-verify-fail-cb3e6677fa57 (use gate-retry after fixing)

GSD VERIFY phase failed after 3 attempts. ═══ Gate: verify-passed (osint-hub#24) ═══ Mode: Tier 1 (functional only) GATE OK: All gates up to 'implement' pass ── Tier 1: Functional ──────────────────────────────── Running tests... ❌ Tests failed (exit 4) configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 timeout: 60.0s timeout method: signal timeout func_only: False asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= ERROR: file or directory not found: tests/ ============================= test session starts ============================== platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0 rootdir: /home/b/Projects/osint-hub configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= Running linter... ✅ Lint clean Running type checker... ✅ Types check clean Previous test count: unknown ❌ GATE FAIL: verify-passed - Tests failed with exit code 4 GATE FAIL: verify — blocked at 2026-05-27T19:40:52Z Class: test-failure Evidence: Verification failed: 1 failures Git tag: osint-hub#24-gate-verify-fail-cb3e6677fa57 (use gate-retry after fixing)
Author
Owner

Fixed in commit. Closing.

Fixed in commit. Closing.
Mark closed this issue 2026-05-27 19:41:11 +00:00
Author
Owner

GSD VERIFY phase failed after 3 attempts.
═══ Gate: verify-passed (osint-hub#24) ═══
Mode: Tier 1 (functional only)

GATE OK: All gates up to 'implement' pass
── Tier 1: Functional ────────────────────────────────
Running tests...
❌ Tests failed (exit 4)
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
timeout: 60.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
ERROR: file or directory not found: tests/

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/b/Projects/osint-hub
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
Running linter...
✅ Lint clean
Running type checker...
✅ Types check clean
Previous test count: unknown

❌ GATE FAIL: verify-passed
- Tests failed with exit code 4
GATE FAIL: verify — blocked at 2026-05-27T19:50:03Z
Class: test-failure
Evidence: Verification failed: 1 failures
Git tag: osint-hub#24-gate-verify-fail-cb3e6677fa57 (use gate-retry after fixing)

GSD VERIFY phase failed after 3 attempts. ═══ Gate: verify-passed (osint-hub#24) ═══ Mode: Tier 1 (functional only) GATE OK: All gates up to 'implement' pass ── Tier 1: Functional ──────────────────────────────── Running tests... ❌ Tests failed (exit 4) configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 timeout: 60.0s timeout method: signal timeout func_only: False asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= ERROR: file or directory not found: tests/ ============================= test session starts ============================== platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0 rootdir: /home/b/Projects/osint-hub configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= Running linter... ✅ Lint clean Running type checker... ✅ Types check clean Previous test count: unknown ❌ GATE FAIL: verify-passed - Tests failed with exit code 4 GATE FAIL: verify — blocked at 2026-05-27T19:50:03Z Class: test-failure Evidence: Verification failed: 1 failures Git tag: osint-hub#24-gate-verify-fail-cb3e6677fa57 (use gate-retry after fixing)
Author
Owner

GSD VERIFY phase failed after 3 attempts.
═══ Gate: verify-passed (osint-hub#24) ═══
Mode: Tier 1 (functional only)

GATE OK: All gates up to 'implement' pass
── Tier 1: Functional ────────────────────────────────
Running tests...
❌ Tests failed (exit 4)
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
timeout: 60.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
ERROR: file or directory not found: tests/

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/b/Projects/osint-hub
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
Running linter...
✅ Lint clean
Running type checker...
✅ Types check clean
Previous test count: unknown

❌ GATE FAIL: verify-passed
- Tests failed with exit code 4
GATE FAIL: verify — blocked at 2026-05-27T20:10:14Z
Class: test-failure
Evidence: Verification failed: 1 failures
Git tag: osint-hub#24-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)

GSD VERIFY phase failed after 3 attempts. ═══ Gate: verify-passed (osint-hub#24) ═══ Mode: Tier 1 (functional only) GATE OK: All gates up to 'implement' pass ── Tier 1: Functional ──────────────────────────────── Running tests... ❌ Tests failed (exit 4) configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 timeout: 60.0s timeout method: signal timeout func_only: False asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= ERROR: file or directory not found: tests/ ============================= test session starts ============================== platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0 rootdir: /home/b/Projects/osint-hub configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= Running linter... ✅ Lint clean Running type checker... ✅ Types check clean Previous test count: unknown ❌ GATE FAIL: verify-passed - Tests failed with exit code 4 GATE FAIL: verify — blocked at 2026-05-27T20:10:14Z Class: test-failure Evidence: Verification failed: 1 failures Git tag: osint-hub#24-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)
Author
Owner

GSD VERIFY phase failed after 3 attempts.
═══ Gate: verify-passed (osint-hub#24) ═══
Mode: Tier 1 (functional only)

GATE OK: All gates up to 'implement' pass
── Tier 1: Functional ────────────────────────────────
Running tests...
❌ Tests failed (exit 4)
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
timeout: 60.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
ERROR: file or directory not found: tests/

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/b/Projects/osint-hub
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
Running linter...
✅ Lint clean
Running type checker...
✅ Types check clean
Previous test count: unknown

❌ GATE FAIL: verify-passed
- Tests failed with exit code 4
GATE FAIL: verify — blocked at 2026-05-27T20:16:32Z
Class: test-failure
Evidence: Verification failed: 1 failures
Git tag: osint-hub#24-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)

GSD VERIFY phase failed after 3 attempts. ═══ Gate: verify-passed (osint-hub#24) ═══ Mode: Tier 1 (functional only) GATE OK: All gates up to 'implement' pass ── Tier 1: Functional ──────────────────────────────── Running tests... ❌ Tests failed (exit 4) configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 timeout: 60.0s timeout method: signal timeout func_only: False asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= ERROR: file or directory not found: tests/ ============================= test session starts ============================== platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0 rootdir: /home/b/Projects/osint-hub configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= Running linter... ✅ Lint clean Running type checker... ✅ Types check clean Previous test count: unknown ❌ GATE FAIL: verify-passed - Tests failed with exit code 4 GATE FAIL: verify — blocked at 2026-05-27T20:16:32Z Class: test-failure Evidence: Verification failed: 1 failures Git tag: osint-hub#24-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/osint-hub#24
No description provided.