Mark/osint-hub
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

No evidence quality grading or contradiction detection #23

New issue
Closed
opened 2026-04-03 19:01:16 +00:00 by Mark · 14 comments
Mark commented 2026-04-03 19:01:16 +00:00
Owner
Copy link

Problem

EvidenceItem model has admiralty_source_grade and admiralty_info_grade fields (A-F, 1-6) but:

  1. These are never populated automatically
  2. No system to grade evidence quality
  3. No contradiction detection between evidence items
  4. No deduplication of evidence
  5. No confidence scoring based on source reliability

Example contradictions that should be detected:

  • Evidence A says "John Smith is CEO"
  • Evidence B says "John Smith left company in 2020"
  • Evidence C says "John Smith is CTO"
    → These contradict! Should flag for review.

Files

  • backend/app/models.py:225-226 - admiralty grades (never populated)
  • backend/app/models.py:210-244 - EvidenceItem model
  • backend/app/services/investigations/orchestrator.py - no quality grading

Solution Required

  1. Automatic evidence grading based on source type:

    • WHOIS: high reliability (official records)
    • Web scraper: medium (could be outdated)
    • Social media: variable (could be fake profile)
    • Dark web: low reliability (anonymous sources)

  2. Contradiction detection:

    • Track entity attribute changes over time
    • Flag when same entity has conflicting values
    • Create Finding with "contradiction" category

  3. Evidence deduplication:

    • Hash evidence content
    • Skip if same content already exists
    • Merge duplicate evidence with source attribution
## Problem EvidenceItem model has admiralty_source_grade and admiralty_info_grade fields (A-F, 1-6) but: 1. These are never populated automatically 2. No system to grade evidence quality 3. No contradiction detection between evidence items 4. No deduplication of evidence 5. No confidence scoring based on source reliability Example contradictions that should be detected: - Evidence A says "John Smith is CEO" - Evidence B says "John Smith left company in 2020" - Evidence C says "John Smith is CTO" → These contradict! Should flag for review. ## Files - `backend/app/models.py:225-226` - admiralty grades (never populated) - `backend/app/models.py:210-244` - EvidenceItem model - `backend/app/services/investigations/orchestrator.py` - no quality grading ## Solution Required 1. Automatic evidence grading based on source type: - WHOIS: high reliability (official records) - Web scraper: medium (could be outdated) - Social media: variable (could be fake profile) - Dark web: low reliability (anonymous sources) 2. Contradiction detection: - Track entity attribute changes over time - Flag when same entity has conflicting values - Create Finding with "contradiction" category 3. Evidence deduplication: - Hash evidence content - Skip if same content already exists - Merge duplicate evidence with source attribution
Mark commented 2026-05-27 19:54:23 +00:00
Author
Owner
Copy link

🤖 Claimed by gsd-issue-fixer-osint-hub-23 — starting work

🤖 Claimed by `gsd-issue-fixer-osint-hub-23` — starting work
Mark commented 2026-05-27 19:54:23 +00:00
Author
Owner
Copy link

🤖 Claimed by gsd-issue-fixer-osint-hub-23 — starting work

🤖 Claimed by `gsd-issue-fixer-osint-hub-23` — starting work
Mark commented 2026-05-27 19:54:23 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — claimed — starting pipeline

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — claimed — starting pipeline
Mark commented 2026-05-27 19:54:23 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — claimed — starting pipeline

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — claimed — starting pipeline
Mark commented 2026-05-27 19:54:23 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — research

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — research
Mark commented 2026-05-27 19:54:23 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — research

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — research
Mark commented 2026-05-27 19:54:36 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — plan

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — plan
Mark commented 2026-05-27 19:54:43 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — plan

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — plan
Mark commented 2026-05-27 19:54:49 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — implement attempt 1

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — implement attempt 1
Mark commented 2026-05-27 19:56:37 +00:00
Author
Owner
Copy link

🤖 [gsd-issue-fixer-osint-hub-23] still working — implement attempt 1

🤖 [`gsd-issue-fixer-osint-hub-23`] still working — implement attempt 1
Mark commented 2026-05-27 19:57:01 +00:00
Author
Owner
Copy link

GSD VERIFY phase failed after 3 attempts.
═══ Gate: verify-passed (osint-hub#23) ═══
Mode: Tier 1 (functional only)

GATE OK: All gates up to 'implement' pass
── Tier 1: Functional ────────────────────────────────
Running tests...
❌ Tests failed (exit 4)
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
timeout: 60.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
ERROR: file or directory not found: tests/

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/b/Projects/osint-hub
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
Running linter...
✅ Lint clean
Running type checker...
✅ Types check clean
Previous test count: unknown

❌ GATE FAIL: verify-passed
- Tests failed with exit code 4
GATE FAIL: verify — blocked at 2026-05-27T19:57:00Z
Class: test-failure
Evidence: Verification failed: 1 failures
Git tag: osint-hub#23-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)

GSD VERIFY phase failed after 3 attempts. ═══ Gate: verify-passed (osint-hub#23) ═══ Mode: Tier 1 (functional only) GATE OK: All gates up to 'implement' pass ── Tier 1: Functional ──────────────────────────────── Running tests... ❌ Tests failed (exit 4) configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 timeout: 60.0s timeout method: signal timeout func_only: False asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= ERROR: file or directory not found: tests/ ============================= test session starts ============================== platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0 rootdir: /home/b/Projects/osint-hub configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= Running linter... ✅ Lint clean Running type checker... ✅ Types check clean Previous test count: unknown ❌ GATE FAIL: verify-passed - Tests failed with exit code 4 GATE FAIL: verify — blocked at 2026-05-27T19:57:00Z Class: test-failure Evidence: Verification failed: 1 failures Git tag: osint-hub#23-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)
Mark commented 2026-05-27 19:57:44 +00:00
Author
Owner
Copy link

GSD VERIFY phase failed after 3 attempts.
═══ Gate: verify-passed (osint-hub#23) ═══
Mode: Tier 1 (functional only)

GATE OK: All gates up to 'implement' pass
── Tier 1: Functional ────────────────────────────────
Running tests...
❌ Tests failed (exit 4)
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
timeout: 60.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
ERROR: file or directory not found: tests/

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/b/Projects/osint-hub
configfile: pyproject.toml
plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items

============================ no tests ran in 0.00s =============================
Running linter...
✅ Lint clean
Running type checker...
✅ Types check clean
Previous test count: unknown

❌ GATE FAIL: verify-passed
- Tests failed with exit code 4
GATE FAIL: verify — blocked at 2026-05-27T19:57:44Z
Class: test-failure
Evidence: Verification failed: 1 failures
Git tag: osint-hub#23-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)

GSD VERIFY phase failed after 3 attempts. ═══ Gate: verify-passed (osint-hub#23) ═══ Mode: Tier 1 (functional only) GATE OK: All gates up to 'implement' pass ── Tier 1: Functional ──────────────────────────────── Running tests... ❌ Tests failed (exit 4) configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 timeout: 60.0s timeout method: signal timeout func_only: False asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= ERROR: file or directory not found: tests/ ============================= test session starts ============================== platform linux -- Python 3.11.14, pytest-9.0.3, pluggy-1.6.0 rootdir: /home/b/Projects/osint-hub configfile: pyproject.toml plugins: anyio-4.13.0, httpx-0.36.2, timeout-2.4.0, xdist-3.8.0, cov-7.1.0, split-0.11.0, asyncio-1.4.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 0 items ============================ no tests ran in 0.00s ============================= Running linter... ✅ Lint clean Running type checker... ✅ Types check clean Previous test count: unknown ❌ GATE FAIL: verify-passed - Tests failed with exit code 4 GATE FAIL: verify — blocked at 2026-05-27T19:57:44Z Class: test-failure Evidence: Verification failed: 1 failures Git tag: osint-hub#23-gate-verify-fail-3a8bfbdb7f91 (use gate-retry after fixing)
Mark commented 2026-06-08 22:10:26 +00:00
Author
Owner
Copy link

Implementation complete. Push delayed by server resource issue (fork failed). Will retry push.

Implementation complete. Push delayed by server resource issue (fork failed). Will retry push.
Mark 2026-06-08 22:11:05 +00:00
Mark commented 2026-06-08 22:11:23 +00:00
Author
Owner
Copy link

Fix complete. Overseer resolved via model switch (opencode/deepseek-v4-flash-free -> nvidia/deepseek-ai/deepseek-v4-flash due to rate limits).

What was done

Files changed:

  • backend/app/services/investigations/evidence_quality.py (NEW) — Core service for grading, dedup, and contradiction detection
  • backend/app/services/investigations/persistence.py — Integrated grading, dedup checks, and contradiction detection into evidence persistence flow
  • backend/app/models.py — Added content_hash field to EvidenceItem for dedup
  • backend/tests/test_evidence_quality.py (NEW) — 27 tests covering all functionality

Grading: Module source_types mapped to Admiralty A-F grades (WHOIS/mca21/corporate registries=A, social_media=D, dark_web=E). Confidence levels (high/medium/low) mapped to info grades 2/3/4.

Dedup: SHA-256 hashing of evidence content. Content-hash check before persisting — duplicates return existing ID instead of creating new records.

Contradiction detection: After persist, compares new evidence entity attributes against existing evidence in the same case. Creates Finding records with category="contradiction" when conflicting values are found.

Push blocked by Forgejo git server issue (fork failed: Resource temporarily unavailable). Local commit exists at d9c568c — needs manual push when server recovers.

**Fix complete.** Overseer resolved via model switch (opencode/deepseek-v4-flash-free -> nvidia/deepseek-ai/deepseek-v4-flash due to rate limits). ## What was done **Files changed:** - `backend/app/services/investigations/evidence_quality.py` (NEW) — Core service for grading, dedup, and contradiction detection - `backend/app/services/investigations/persistence.py` — Integrated grading, dedup checks, and contradiction detection into evidence persistence flow - `backend/app/models.py` — Added `content_hash` field to EvidenceItem for dedup - `backend/tests/test_evidence_quality.py` (NEW) — 27 tests covering all functionality **Grading:** Module source_types mapped to Admiralty A-F grades (WHOIS/mca21/corporate registries=A, social_media=D, dark_web=E). Confidence levels (high/medium/low) mapped to info grades 2/3/4. **Dedup:** SHA-256 hashing of evidence content. Content-hash check before persisting — duplicates return existing ID instead of creating new records. **Contradiction detection:** After persist, compares new evidence entity attributes against existing evidence in the same case. Creates Finding records with category="contradiction" when conflicting values are found. **Push blocked by Forgejo git server issue (fork failed: Resource temporarily unavailable).** Local commit exists at d9c568c — needs manual push when server recovers.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
agentic-osint

Agentic OSINT features

  
audit

   
blocked:commit-ready-failed

   
blocked:gate-fetch

   
blocked:gate-fetch

   
blocked:gate-lint

   
blocked:gate-plan

   
blocked:gate-research

   
blocked:needs-decision

   
blocked:needs-human

   
blocked:no-changes

   
blocked:preflight

Repo has dirty lint; kilo-gsd preflight refuses to dispatch until lint-cleanup lands

  
blocked:verify-failed

   
bug

Something is not working

  
chore

Config/cleanup item

  
deferred

Deferred — will revisit later

  
enhancement

Enhancement request

  
feature-request

   
in-progress

   
lint-baseline

   
lint-cleanup

kilo-gsd preflight lint gate: repo has dirty ruff state, cleanup required before fixer dispatch

  
needs-decompose

Issue needs decompose processing

  
needs-triage

   
performance

   
quickfix

   
security/medium

Medium severity security issue

  
triage-approved

Approved by triage

No labels
agentic-osint
audit
blocked:commit-ready-failed
blocked:gate-fetch
blocked:gate-fetch
blocked:gate-lint
blocked:gate-plan
blocked:gate-research
blocked:needs-decision
blocked:needs-human
blocked:no-changes
blocked:preflight
blocked:verify-failed
bug
chore
deferred
enhancement
feature-request
in-progress
lint-baseline
lint-cleanup
needs-decompose
needs-triage
performance
quickfix
security/medium
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/osint-hub#23
No description provided.