Processing pipeline

The vulnmgmt engine takes a raw software inventory (typically an Intune / Defender CSV export) and drives it through five stages to produce a scored, prioritized list of vulnerabilities. This page walks each stage, names the classes involved, and summarizes the PostgreSQL schema the results land in.

See also: CLI commands for how to invoke each stage, and the CLI & engine overview for how the engine relates to the webapp.

Flow at a glance

Raw inventory CSV
   │  inventory/  (ApplicationInventory, Application)
   ▼
Normalized applications        name • version • vendor • asset_count • criticality
   │  mapping/  (CVEMapper → CPEClient / CVEResearcher)
   ▼
CPE match + CVE candidates     CVEMatch objects (CVE id, CVSS, severity, KEV flag)
   │  intelligence/  (ThreatIntelligenceAggregator)
   ▼
Enriched CVEs                  EnrichedCVE (EPSS, OTX, MITRE, VirusTotal, Shodan…)
   │  scoring/  (RiskScoringEngine, Priority)
   ▼
Scored / prioritized output    ScoredCVE (0–100 risk score, priority, recommendation)
   │  database/  +  output/
   ▼
PostgreSQL findings  •  Excel / DOCX reports

Two lineages

The codebase actually contains two parallel application pipelines that grew up at different times:

• Modern pipeline — ApplicationInventory → CVEMapper → ThreatIntelligenceAggregator → RiskScoringEngine. This is what vulnmgmt vuln / vulnmgmt intune drive and what the webapp reuses. This page documents it as the primary flow.
• Legacy pandas path — core.processor.VulnerabilityProcessor + analysis.risk_scoring.RiskAnalyzer, used by the older vulnmgmt process, analyze, and versions commands to build Excel workbooks directly from a DataFrame.

They use different Application types and different Priority enums — see the caveats below.

Stage 1 — Inventory ingestion (inventory/)

vulnmgmt.inventory.app_discovery normalizes raw software rows into a clean list of unique applications.

Application (a dataclass) is the unit of inventory:

Field	Notes
name, version	Required
vendor, product	Optional; used for CPE generation
asset_count	Number of devices with this app (default 1)
criticality	low / medium / high / critical (default medium)
source	Where the row came from (intune, cli, manual, …)
cpe	Auto-generated in __post_init__ when vendor+product+version exist

When vendor, product, and version are all present, the constructor synthesizes a CPE 2.3 string: cpe:2.3:a:{vendor}:{product}:{version}:*:*:*:*:*:*:* (lowercased, spaces → underscores).

ApplicationInventory manages a collection of Applications and knows how to load them from several sources:

Method	Source
import_from_intune_csv(path)	Intune AppInventory export — dedupes on (name, version, vendor), sums device counts, and builds a device_software_map
import_from_csv(path)	Generic CSV (name,version,vendor,product,asset_count,criticality)
import_from_json(path)	Reads data["applications"]
discover_from_rapid7(...) / discover_from_defender(...)	Pull inventory straight from a platform API

Supporting operations include deduplicate() (merges duplicate keys and adds asset counts), filter_by_criticality(), get_summary(), and save_to_database() which persists rows as software records linked to assets.

Device count drives an automatic criticality rating in the CLI importers: >=100 → critical, >=50 → high, >=10 → medium, else low.

Vendor normalization (core/vendor_mapping.py)

Raw publisher strings from Intune are messy (CN=…, O=Foxit Software Inc., GUID publishers, Microsoft Corporation vs Skype Software Sarl). The core.vendor_mapping module cleans them up with:

• extract_vendor() — parses CN= / O= code-signing certificate strings.
• normalize_vendor() — collapses ~65 VENDOR_ALIASES to canonical names.
• normalize_application_name() — strips Microsoft Store prefixes (Microsoft., AD2F1837., …).
• VENDOR_SECURITY_URLS — ~55 canonical vendor → security-bulletin URLs (surfaced via vulnmgmt vendors --list).

Stage 2 — CPE / product mapping (mapping/)

vulnmgmt.mapping.cve_mapper.CVEMapper turns each normalized Application into a set of candidate CVEs, producing CVEMatch dataclasses (cve_id, cvss_score, severity, description, published_date, is_exploited, asset_count, source).

map_application(app) uses two strategies:

1. CPE-resolved lookup (preferred). _resolve_cpe_names() matches the app name against a CPE_MAP table (~40 regex → vendor:product entries, e.g. google chrome → google:chrome, screenconnect → connectwise:screenconnect, java → oracle:jdk). If resolved, it queries NVD with a virtualMatchString so NVD does the version-range matching server-side.
2. Keyword fallback. If no CPE mapping exists, it does a keyword NVD search and filters client-side with _version_matches(), which evaluates NVD cpeMatch ranges (versionStartIncluding/EndExcluding, wildcards, exact) using internal version-comparison helpers.

Every match set is then cross-checked against the CISA KEV catalog; CVEs found there get is_exploited = True.

map_applications_batch(apps) runs the whole inventory with a progress bar. get_summary() reports unique/exploited/critical/high counts and average CVSS.

The lower-level CPE dictionary itself is exposed through vulnmgmt.integrations.cpe.CPEClient (see vulnmgmt vuln cpe search|match|lookup), which searches the NVD CPE dictionary and scores application → CPE matches by confidence.

Stage 3 — CVE enrichment (intelligence/)

vulnmgmt.intelligence.aggregator.ThreatIntelligenceAggregator takes the CVEMatch list and enriches each one into an EnrichedCVE, pulling from multiple threat-intelligence sources in parallel (ThreadPoolExecutor, max_workers=5).

Source groups:

Group	Sources	Adds
CVE-keyed	OTX, MITRE ATT&CK, CVE AWG, NVD	pulse counts, threat actors, malware families, MITRE techniques/tactics, CWE, detailed CVSS
EPSS	FIRST.org EPSS	epss_score, epss_percentile (exploitation probability)
IOC-keyed	VirusTotal, Shodan, GreyNoise, AbuseIPDB, URLhaus	marked not_applicable for CVE-id enrichment, used when an IOC is available

The aggregator is constructed with an api_keys dict and an optional enabled_sources list (default ['otx', 'cve_awg', 'nvd']). enrich_batch() returns the enriched list; get_summary() and get_high_risk_cves() provide roll-ups.

Stage 4 — Risk scoring (scoring/)

vulnmgmt.scoring.risk_engine.RiskScoringEngine converts each EnrichedCVE into a ScoredCVE with a 0–100 risk_score, a Priority, a recommendation, and a remediation_urgency.

Scoring is a weighted sum of six factors:

Factor	Weight	Basis
CVSS	0.20	(cvss / 10) × 100
EPSS	0.15	epss_score × 100
Exploitation	0.25	100 if KEV-listed, 75 if GreyNoise scanner activity, 50 if OTX pulses > 10
Threat actors	0.15	min(#actors × 25, 100)
Malware	0.15	min(#families × 20, 100) + VirusTotal / URLhaus bonuses
Exposure	0.10	asset-count tiers (>100→100, >50→75, >10→50, >1→25) + Shodan bonus

The Priority enum here has five values — CRITICAL, HIGH, MEDIUM, LOW, INFO. A CVE is forced to CRITICAL when it is actively exploited and CVSS ≥ 9.0; otherwise thresholds apply (≥80 CRITICAL, ≥60 HIGH, ≥40 MEDIUM, ≥20 LOW, else INFO). Remediation urgency ranges from “IMMEDIATE (0–24 hours)” for CRITICAL down to “90 days” / “As needed”.

score_batch() scores and sorts descending; get_top_risks(n) and get_summary() (priority breakdown, average score, immediate-action count) feed the CLI output and reports.

Stage 5 — Output & persistence

Scored results are emitted two ways:

• Reports via output/ — vulnmgmt vuln export builds a six-sheet Excel workbook (Executive Summary, All Vulnerabilities, Risk Scoring, Threat Intelligence, Enrichment Details, Action Items) or a branded DOCX report.
• Database via database/ — findings persist to PostgreSQL for the multi-tenant workflow (state tracking, SLA, remediation history).

Database schema (database/schema.sql)

The engine is PostgreSQL-backed (declares PostgreSQL 14+, uses the uuid-ossp extension, INET / MACADDR types, JSONB, TEXT[] arrays, and CHECK constraints). Every table uses a UUID primary key defaulting to uuid_generate_v4(). SQLAlchemy models in database/models.py mirror these tables (Client, Asset, Software, Vulnerability, Finding, RemediationHistory, DiscoveryJob, Report).

Table	Purpose	Key columns
clients	Multi-tenant company records	name, identifier (unique), cw_company_id / cw_board_id, service_tier (assessment/copilot/managed)
assets	Servers, workstations, devices	client_id → clients, hostname, ip_address (INET), mac_address (MACADDR), asset_type, os_type, criticality, status, last_seen
software	Software discovered on assets	asset_id → assets, name, vendor, version, cpe, cpe_confidence, install_path
vulnerabilities	CVE database with enrichment	cve_id (unique, regex-checked), cvss_v2/v3_score, severity, epss_score/epss_percentile, is_exploited, kev_due_date, mitre_techniques[], threat_actors[], malware_families[], references (JSONB), enrichment_sources[]
findings	Central link of asset × software × vulnerability	FKs to clients/assets/software/vulnerabilities, state, risk_score, priority, assigned_to, cw_ticket_id, lifecycle timestamps, sla_due_date, sla_breached. Unique on (asset_id, software_id, vuln_id)
remediation_history	Audit trail of finding state changes	finding_id → findings, old_state, new_state, changed_by, changed_at
discovery_jobs	Scan / discovery job tracking	client_id, job_type (defender/intune/rapid7/manual), status, counts of assets/software/findings, timing
reports	Generated report tracking / delivery	client_id, report_type, format, file_path, findings_count, delivered

A finding moves through states such as discovered → assessed → assigned → in_progress → verified → closed (plus accepted_risk, deferred, exception, reopened). A trigger (update_updated_at_column) auto-maintains updated_at on the core tables.

Reporting views (database/views.sql)

The schema ships ~12 dashboard/reporting views, including:

• vw_client_summary, vw_asset_risk_ranking, vw_top_vulnerabilities — executive roll-ups.
• vw_findings_by_severity, vw_sla_status, vw_asset_inventory — operational.
• vw_epss_distribution, vw_mitre_techniques, vw_kev_tracking — threat intelligence.
• vw_software_inventory, vw_remediation_progress, vw_assignment_workload — software & remediation.

schema.sql additionally defines v_active_findings and v_sla_summary.

Connection (database/session.py)

The engine reads DATABASE_URL (get_database_url()); when unset it falls back to a local SQLite file for dev convenience — but the Postgres-specific types and views mean PostgreSQL is required for real use. The Postgres engine is configured with pool_size=10, max_overflow=20, and pool_pre_ping=True. init_db(force=…) creates (or drops and recreates) all tables; vulnmgmt db init wraps it (see CLI commands).

Caveats & gotchas

• Two Application types. The dataclass in inventory.app_discovery (used by CVEMapper) is not the same as the pydantic core.models.Application (used by VulnerabilityProcessor). They carry different fields.
• Three Priority enums. core.models.Priority has four values; scoring.risk_engine.Priority adds INFO (five values). They are defined independently.
• Schema ↔ model drift. database/models.py adds a few columns not in schema.sql (e.g. Vulnerability.cpe_configurations, Finding.rapid7_vuln_id / port / protocol) and relaxes some nullability for Rapid7 asset-level findings. Treat the SQLAlchemy models as the runtime source of truth.