Implement distributed task queue for parallel investigation execution #11

New issue

Closed

opened 2026-04-03 18:17:51 +00:00 by Mark · 0 comments

Mark commented

2026-04-03 18:17:51 +00:00

Owner

Problem

Current investigation runs in a single linear process. Should be distributed/async for scale.

Requirements

Distributed Execution

Celery task queue for module execution
Multiple workers for parallel execution
Redis as message broker
Async module results

Investigation Queue

Priority queue for investigations
Queue management UI
Retry failed investigations
Dead letter queue for failed tasks

Scheduling

Scheduled investigations (recurring)
Cron-like scheduling for monitoring
Batch investigation processing

## Problem Current investigation runs in a single linear process. Should be distributed/async for scale. ## Requirements ### Distributed Execution - Celery task queue for module execution - Multiple workers for parallel execution - Redis as message broker - Async module results ### Investigation Queue - Priority queue for investigations - Queue management UI - Retry failed investigations - Dead letter queue for failed tasks ### Scheduling - Scheduled investigations (recurring) - Cron-like scheduling for monitoring - Batch investigation processing