Executive Summary
Correlation engine approved on review — 916-line module with 3-layer processing (entity resolution, cross-stage link graph, timeline builder) integrated as Stage 5.5 between Normalizer and Reporter. Tested with 16 events producing 11 cross-source links (5 HIGH confidence).
📊 Key Findings
Build the correlation engine — the "brain" that connects findings across all pipeline stages into a single coherent subject profile.
Design from t_23632888 specifies 3-layer processing:
1. Entity resolution — merge records that refer to the same person (fuzzy name matching, address proximity, phone linkage). Current normalizer does basic merging; this is the v2.
2. Cross-stage linking — connect Phase 0b addresses → Maigret social profiles → Holehe registrations → h8mail breaches by shared identifiers. Build a link graph.
3. Timeline builder — chronologically order findings (moved to X in 2019, breach at Y in 2021, registered at Z in 2023).
Implementation:
1. New module: ~/.hermes/background-check/correlation.py
2. Consumes unified JSON from Stage 5 (normalizer)
3. Produces correlated profile JSON with confidence scores per link
4. Feed into reporter (Stage 6) with a "correlation summary" section
5. Output highlight: "HIGH CONFIDENCE: 3 sources confirm address at 8835 Leonard St"
This is the differentiator — no surveyed platform (of 20) offers relevance scoring or cross-source correlation. We already have relevance scoring built; this extends it.
💬 Additional Insight
review-required handoff:
```json
{
"changed_files": [
"/home/kanfl/.hermes/background-check/correlation.py",
"/home/kanfl/.hermes/background-check/orchestrator.py"
],
"new_module_lines": 916,
"orchestrator_changes": "3 patches: import block, docstring, Stage 5.5 insertion + reporter section",
"test_output": "verified with realistic sample unified JSON (16 events, 11 links, 3 entities resolved)",
"graceful_degradation": "verified with empty JSON — produces empty report gracefully",
"decisions": [
"fuzzy matching uses thefuzz if available, stdlib fallback otherwise (no pip install required)",
"addresses pre-merged by city+state before link generation to deduplicate Phase 0b scraper overlap",
"links deduplicated by (source, target, link_type) key",
"entity confidence: HIGH=3+ sources, MEDIUM=2, LOW=1 — counts identity match, AKA matches, and social profile matches",
"timeline sorts unknown-date events to end, current events last"
]
}
```
**What was built:**
correlation.py (916 lines) — 3-layer correlation engine:
- **Layer 1 (Entity Resolution):** Fuzzy name matching across Phase 0b + Maigret profiles. Resolves name variants (full, AKA, reversed) into unified entities with source counts and confidence scores.
- **Layer 2 (Cross-Stage Link Graph):** Links Phase 0b addresses → Maigret profile locations (HIGH/MEDIUM/LOW confidence by match quality), Holehe registrations → h8mail breaches (by email), WHOIS domain → registrations, phone → address, social → breach cross-references. Produces 11 links from our test case (5 HIGH, 2 MEDIUM, 4 LOW).
- **Layer 3 (Timeline Builder):** Chronologically orders all dated findings across stages (breach dates, registration dates, profile creation dates, WHOIS dates, addresses). 16 events in test case, 11 with dates spanning 2018–2027.
**Integration:**
- New Stage 5.5 in orchestrator between Normalizer and Reporter
- Reporter now auto-includes Section 11 (Cross-Source Correlation) when correlation data exists
- Graceful degradation if correlation module not importable (no crash)
**Key highlight example:** "HIGH CONFIDENCE: 2 sources confirm address at Kingston, OK" — exactly matching the requested format.
```json
{
"changed_files": [
"/home/kanfl/.hermes/background-check/correlation.py",
"/home/kanfl/.hermes/background-check/orchestrator.py"
],
"new_module_lines": 916,
"orchestrator_changes": "3 patches: import block, docstring, Stage 5.5 insertion + reporter section",
"test_output": "verified with realistic sample unified JSON (16 events, 11 links, 3 entities resolved)",
"graceful_degradation": "verified with empty JSON — produces empty report gracefully",
"decisions": [
"fuzzy matching uses thefuzz if available, stdlib fallback otherwise (no pip install required)",
"addresses pre-merged by city+state before link generation to deduplicate Phase 0b scraper overlap",
"links deduplicated by (source, target, link_type) key",
"entity confidence: HIGH=3+ sources, MEDIUM=2, LOW=1 — counts identity match, AKA matches, and social profile matches",
"timeline sorts unknown-date events to end, current events last"
]
}
```
**What was built:**
correlation.py (916 lines) — 3-layer correlation engine:
- **Layer 1 (Entity Resolution):** Fuzzy name matching across Phase 0b + Maigret profiles. Resolves name variants (full, AKA, reversed) into unified entities with source counts and confidence scores.
- **Layer 2 (Cross-Stage Link Graph):** Links Phase 0b addresses → Maigret profile locations (HIGH/MEDIUM/LOW confidence by match quality), Holehe registrations → h8mail breaches (by email), WHOIS domain → registrations, phone → address, social → breach cross-references. Produces 11 links from our test case (5 HIGH, 2 MEDIUM, 4 LOW).
- **Layer 3 (Timeline Builder):** Chronologically orders all dated findings across stages (breach dates, registration dates, profile creation dates, WHOIS dates, addresses). 16 events in test case, 11 with dates spanning 2018–2027.
**Integration:**
- New Stage 5.5 in orchestrator between Normalizer and Reporter
- Reporter now auto-includes Section 11 (Cross-Source Correlation) when correlation data exists
- Graceful degradation if correlation module not importable (no crash)
**Key highlight example:** "HIGH CONFIDENCE: 2 sources confirm address at Kingston, OK" — exactly matching the requested format.