Data model

Lockstep VM stores all webapp data in a single MongoDB database (vulnmgmt), accessed asynchronously via motor. There is no fixed schema — documents are written directly by the integration syncs and CSV pipeline. This page documents the shapes those code paths actually produce, so you can query and reason about the data.

Two conventions run through everything:

• Everything is scoped by tenant_id. Every document carries a tenant_id string, and every query filters on it. Uniqueness keys are compound and always lead with tenant_id. See Multi-tenancy.
• The source tag. Rows in the shared collections (inventory_items, csv_*) carry a source string identifying which integration or CSV provider produced them: intune, sentinelone, rapid7, connectwise_automate, or a CSV provider key (e.g. autoelevate, badge). This lets one collection hold data from every source while keeping provenance and enabling per-source purges.

inventory_items — the shared device × app join

The single source of truth for installed software. One document per (device, application, version), written by every source (Intune, SentinelOne, Rapid7, ConnectWise Automate, and CSV “software”-target providers). This is what CVE-to-device matching and the software rollup read.

Upsert key: {tenant_id, source, device_id, application_name, version}.

Field	Type	Notes
tenant_id	string	Owning tenant.
source	string	intune / sentinelone / rapid7 / connectwise_automate / CSV provider key.
device_id	string	Source device identifier. For SentinelOne rows this is the matched Intune UUID when available (so CVE→device links line up), else the S1 agent id.
device_name	string	Hostname as reported by the source.
application_name	string	Software name (raw, pre-normalization).
version	string	Version string (may be empty).
vendor	string	Publisher / vendor (naming varies by source).
os_platform	string	OS family (e.g. Windows, macOS).
os_version	string	OS version, when the source provides it.
user_name	string	Assigned/last user display name (enriched from Intune where matched).
user_email	string	Assigned user UPN/email (enriched from Intune where matched).
synced_at	datetime	Timestamp of the sync that wrote the row.
agent_id	string	SentinelOne rows only — the S1 agent id (used to prune stale S1 rows).
intune_device_id	string	SentinelOne rows only — matched Intune device id.
upload_id	string	CSV rows only — the csv_uploads job that produced the row.

Each sync prunes its own stale rows: it deletes rows for its source whose device id is no longer live (e.g. source: "intune", device_id: {$nin: live_ids}).

applications — the per-tenant software rollup

The Software-inventory page reads this, not inventory_items. One document per software group (name) per tenant, derived entirely from inventory_items by app/services/inventory_rollup.py (rebuild_applications). It is rebuilt after every sync and CSV import.

Upsert key: {tenant_id, normalized_name}.

Field	Type	Notes
tenant_id	string	Owning tenant.
name / normalized_name / product	string	Software name after normalization (see below).
vendor / normalized_vendor	string	Dominant vendor — the one seen on the most distinct devices.
version	string	Newest version observed.
versions	string[]	All observed versions, sorted numerically-aware.
device_count	int	Count of distinct devices running the software.
os_platforms	string[]	Deduped OS platforms (case-insensitive, Title-case preferred).
platform	string | null	First platform, for convenience.
publisher	string	Publisher, when carried by the source.
sources	string[]	Distinct source tags the software was seen from.
version_tracking	object	See below.
created_at / updated_at / rollup_at	datetime	rollup_at marks the run; older docs are pruned.

version_tracking shape:

{
  "most_common_version": "1.41.2",
  "newest_version": "1.41.2",
  "oldest_version": "1.32.0",
  "version_counts": [
    { "version": "1.32.0", "count": 3 },
    { "version": "1.41.2", "count": 12 }
  ]
}

version_counts is a list (not a version-keyed object) so arbitrary version strings can never produce an invalid MongoDB field name.

How the rollup is derived

rebuild_applications(tenant_id, db) runs an aggregation over inventory_items for the tenant, then post-processes in Python:

1. Group by application_name. A $group collects, per app name, the set of (version, device_name) pairs, (vendor, device_name) pairs, the set of os_platform values, and the set of source values.
2. App-name normalization. Raw-name groups are merged when they share a normalized name. _normalize_app_name strips a trailing per-install instance-id/hash or GUID in parentheses — e.g. ScreenConnect Client (51a7836ecd7be670) → ScreenConnect Client — so every per-install variant collapses into one group. Meaningful parentheticals like (x64), (64-bit), (en-us) are deliberately kept (they aren’t hex ids). Values are also cleaned of NULL bytes and control characters that MongoDB rejects or that pollute version strings.
3. Distinct device counts. device_count is the number of distinct device_name values across all versions of the group; per-version device counts feed version_tracking.
4. Dominant vendor / most-common version. Both are chosen by “installed on the most distinct devices.”
5. Platform dedupe. _dedupe_platforms folds windows/Windows together, preferring the capitalized display form.
6. Prune. Rollup documents whose rollup_at doesn’t match the current run (no longer backed by any inventory row) are deleted.

Writes are chunked into bulk upserts of 1000.

NetBIOS device-name handling. Device grouping in the rollup keys on device_name as reported. Cross-source device reconciliation happens at query/lookup time elsewhere, because sources disagree on hostname form: ConnectWise Automate and some CSV exports (e.g. AutoElevate) report the 15-character NetBIOS hostname, while Intune/SentinelOne carry the full name. Device-lookup endpoints therefore match on both the exact lowercased name and its 15-char NetBIOS truncation (name[:15]) — see devices.py, csv_sources.py’s device-by-name, and ConnectWise Automate’s device match — so a full-name asset and its NetBIOS twin resolve to the same device.

Per-integration config collections

One document per tenant per integration. The tenant_id: "default" document is seeded from env vars at startup; other tenants inherit it via the _resolve_config adopt-on-read pattern (details). All share a common lifecycle shape: enabled, sync_status, last_sync, sync_error, count fields, updated_at.

Collection	Credentials fields	Count fields
sentinelone_config	api_url, api_token	agent_count, threat_count
intune_config	azure_tenant_id, azure_client_id, auth_method, cert_thumbprint, cert_private_key (or client_secret)	device_count, app_count
entra_config	same Azure app + cert as Intune (azure_tenant_id, azure_client_id, cert_thumbprint, cert_private_key)	user_count, group_count
rapid7_config	api_key, region, organization_id	asset_count, app_count, critical_count, severe_count
connectwise_config	api_url, company_id, public_key, private_key, client_id	—
connectwise_automate_config	api_url, username, password, client_id	device_count, app_count

Endpoints never return raw secrets; a _safe_config projection replaces tokens/keys with booleans like has_token / has_cert.

Per-source device / asset collections

Each API integration keeps a raw, full-fidelity copy of its devices (beyond the software rows it contributes to inventory_items). All are keyed by {tenant_id, <source id>} and carry synced_at; stale rows are pruned each sync.

intune_devices

Upsert key {tenant_id, intune_device_id}. Stores the client’s device dict verbatim plus tenant/sync fields.

Field	Notes
intune_device_id	Graph device id (from id).
device_name	deviceName.
os / os_version	operatingSystem / osVersion.
compliance_state	complianceState.
user_display_name / user_upn	userDisplayName / userPrincipalName.
manufacturer / model / serial_number	Hardware fields.
synced_at	Sync timestamp.

(A companion intune_apps collection holds the app catalog, keyed by {tenant_id, intune_app_id}.)

sentinelone_devices

Upsert key {tenant_id, agent_id}. Rich agent record from the S1 API.

Field group	Fields
Identity	agent_id, uuid, computer_name, domain, serial_number, machine_type, model_name
OS	os_type, os_version, os_revision, os_arch
Agent	agent_version, is_active, is_infected, is_decommissioned, is_up_to_date, network_status, scan_status, last_active_date
Org	site_name, site_id, account_name, group_name, group_id, tags
Security	threat_count, apps_vulnerability_status
Cross-ref	intune_device_id (matched by lowercased computer name), last_logged_in_user

A sibling sentinelone_threats collection (key {tenant_id, threat_id}) holds detections, and sentinelone_app_risks holds app-risk/CVE data surfaced during sync.

rapid7_assets

Upsert key {tenant_id, asset_id}.

Field	Notes
asset_id, hostname, ip_address, mac_address	Identity / network.
operating_system, os_version, os_family	OS.
risk_score, vulnerabilities_count, critical_vulnerabilities, severe_vulnerabilities	Risk.
last_scan_date, tags	Scan metadata.
intune_device_id, s1_agent_id, user_email, user_name	Cross-referenced from Intune/S1 by hostname (exact + short/FQDN-stripped form).
synced_at	Sync timestamp.

connectwise_automate_devices

Upsert key {tenant_id, device_id} (the Automate ComputerID).

Field	Notes
device_id, device_name, friendly_name	Identity (ComputerName / FriendlyName).
operating_system, os_version	OS.
domain_name, type, ip_address (LocalIPAddress)	Network / classification.
client_name	Owning client from the nested Client.Name.
last_contact, agent_version, last_user	Agent + last logged-in user.
synced_at	Sync timestamp.

Entra ID collections

Populated by the Entra sync (entra.py), which also resolves group memberships.

entra_users

Upsert key {tenant_id, id} (Graph object id).

Field	Notes
id, upn	Object id and userPrincipalName.
aliases	SMTP aliases parsed from proxyAddresses — used to link badge/HR CSV rows whose email is an alias, not the UPN.
display_name, mail, job_title, department, office_location, mobile_phone	Profile.
sam_account	onPremisesSamAccountName.
account_enabled	Enabled flag.
groups	Names of groups the user belongs to (resolved from membership lookups).
synced_at	Sync timestamp.

entra_groups

Upsert key {tenant_id, id}.

Field	Notes
id, display_name, description, mail	Group identity.
is_m365	true when Unified is in groupTypes.
is_security	From securityEnabled.
visibility	Group visibility.
synced_at	Sync timestamp.

CSV pipeline collections

A generic CSV/Excel framework (csv_sources.py + the provider registry) drives every CSV-fed source. A provider’s target decides which collection its parsed rows land in:

Provider target	Destination collection	Behaviour
software	inventory_items (with source = <provider key>)	Upserted, then rebuild_applications runs so the software shows up in the rollup.
device	csv_devices	Device-enrichment. Replaced per source each upload; keyed by lowercased hostname to link to an existing asset (never creates duplicates).
user	csv_users	User-enrichment. Replaced per source; keyed by lowercased email to link to an Entra user.
records	csv_records	Generic reference/tagging rows; replaced per source each upload.

csv_uploads

One document per upload job — the audit trail and status for the background parse.

Field	Notes
tenant_id, provider_key	Owner + CSV source key.
filename, file_size	Sanitized original filename and byte size.
status	processing → processed / failed.
row_count, imported_count	Rows parsed vs. records written.
detected_shape, warnings, error	Parser diagnostics.
created_at, processed_at	Job timestamps.

csv_devices

Key {tenant_id, source, device_key} where device_key = device_name.lower().

Field	Notes
source, device_key, device_name	Provenance + hostname (matched exact and NetBIOS-15 forms on lookup).
attributes	Provider-specific key/values.
upload_id, synced_at	Origin upload + timestamp.

csv_users

Key {tenant_id, source, email} (lowercased email).

Field	Notes
source, email, display_name, site	Identity.
attributes, flags	Provider-specific details / booleans.
upload_id, synced_at	Origin upload + timestamp.

csv_records

Generic records-target rows: the parsed record fields (**r) plus tenant_id, source, upload_id, synced_at, and a raw field. Replaced wholesale for a source on each upload.

See also

• Architecture — how the services read and write these collections.
• API reference — the routes that expose this data.