Backend & services

The backend is a FastAPI application (webapp/backend/app) backed by MongoDB (via Motor, the async driver). It exposes the REST API the frontend calls, runs every integration sync, and owns the reconciliation logic that turns raw per-source rows into the unified Assets / Software / Users views.

Application structure

webapp/backend/app/
├── main.py            FastAPI app: lifespan, CORS, rate limit, config seeding
├── core/              cross-cutting infrastructure
│   ├── config.py      pydantic-settings Settings (env-driven)
│   ├── database.py    Motor client + get_database() dependency
│   ├── security.py    auth: Better Auth cookie → JWT bearer fallback
│   ├── better_auth.py session-cookie validation against the frontend
│   ├── rate_limit.py  slowapi limiter
│   └── uploads.py     validate_upload() — extension + libmagic MIME sniffing
├── api/v1/            routers (endpoints/*.py) mounted under /api/v1
├── models/            pydantic request/response models
└── services/          the reusable, router-independent service layer
    ├── inventory_rollup.py           applications rollup rebuild
    ├── query_builder.py              structured query → Mongo filter
    ├── csv_providers/                CSV parser framework (base + providers)
    ├── intune_client.py              Microsoft Graph (Intune) client
    ├── entra_client.py               Microsoft Graph (Entra ID) client
    ├── connectwise_automate_client.py  ConnectWise Automate RMM client
    ├── dashboard_service.py / metrics/ / widget_data_service.py

App lifecycle (`main.py`)

The lifespan context manager connects to MongoDB on startup and then runs a set of config-seeding helpers — _seed_intune_config, _seed_sentinelone_config, _seed_connectwise_config, _seed_rapid7_config, _seed_connectwise_automate_config and _seed_entra_config. Each one checks the relevant *_config collection for the default tenant and, if empty, populates it from environment variables so a freshly deployed instance is ready to sync without manual configuration. Existing config is never overwritten (adopt-on-read).

Intune and Entra share the same Azure app registration: _seed_entra_config reuses INTUNE_TENANT_ID / INTUNE_CLIENT_ID / INTUNE_CERT_THUMBPRINT and the same PEM key, extracting the private-key block with _extract_private_key().

Other app-level wiring:

Concern	Detail
CORS	Origins from `CORS_ORIGINS` (comma-separated); `allow_credentials=True` so Better Auth cookies work; methods/headers tightened to what the frontend uses.
Rate limiting	`slowapi` limiter on `app.state`, default from `RATE_LIMIT_PER_MINUTE`, per-route overrides via `@limiter.limit`.
Health	`GET /health` returns `{status, environment, database}`; `GET /` returns version + docs link.
Errors	A global exception handler returns a JSON 500 `{detail, type, path}` instead of leaking a traceback.
OpenAPI	Swagger at `/docs`, ReDoc at `/redoc`, schema at `/openapi.json`.

Authentication (`core/security.py`)

get_current_user resolves identity from two sources, in order:

Better Auth session cookie (vulnmgmt.session_token) — the primary path for browser traffic, validated via the frontend’s /api/auth/get-session.
Bearer JWT in the Authorization header — the legacy path kept alive so the Better Auth cutover doesn’t log everyone out at once.

Both normalize to {user_id, email, tenant_id, role}. get_current_admin_user layers a role == "admin" check on top. Every data query is scoped by current_user["tenant_id"].

The reusable service layer

Everything under app/services/ is router-independent — it takes a config dict or the database handle and raises on error rather than returning HTTP responses. This lets the same code run from an API endpoint, a background task, or a future durable worker (see Durable workflows).

The shared source of truth is the inventory_items collection: one row per (device, application, version, source). Every source — CSV upload, SentinelOne, Intune, Rapid7, ConnectWise Automate — writes into it. The unified views are derived from it.

Software rollup — `inventory_rollup.rebuild_applications`

The Software-inventory page does not read inventory_items directly; it reads applications, a per-tenant rollup grouped by software with a distinct-device count, version tracking, and source attribution. Historically only the CSV path built that rollup, so software from API integrations never appeared. rebuild_applications(tenant_id, db) rebuilds it straight from inventory_items, so it is correct regardless of source and after every sync.

Stage 1 — group by application name

An aggregation pipeline groups inventory rows for the tenant by application_name (ignoring null/empty names). It deliberately groups by name only, not (name, vendor): a single app often carries several vendor spellings (e.g. SentinelOne, Inc. vs Sentinel Labs, Inc.), and the by-application device view also matches on name. For each name-group it collects, as bounded $addToSet sets:

Field	Purpose
`pairs` = `{v: version, d: device_name}`	derive distinct device count and per-version device counts
`vendor_pairs` = `{ven: vendor, d: device}`	pick the dominant vendor
`os_platforms`	observed OS labels
`sources`	which integrations reported the app
`publisher`	`$max` publisher string

Stage 2 — instance-id name normalization

Some agents report a per-install instance id or hash in the app name, so the “same” app appears as thousands of unique names. _normalize_app_name strips a trailing hex/GUID parenthetical and the raw-name groups are merged by their normalized name, unioning their device/version/source sets.

_INSTANCE_SUFFIX = re.compile(
    r"\s*\((?:[0-9a-fA-F]{8,}|[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-...)\)\s*$"
)

Raw name	Normalized
`ScreenConnect Client (51a7836ecd7be670)`	`ScreenConnect Client`
`Some App (3f2504e0-4f89-41d3-9a0c-0305e82c3301)`	`Some App`
`7-Zip 22.01 (x64)`	unchanged — `x64` is not a hex id
`Notepad++ (en-us)`	unchanged — meaningful parenthetical kept

The regex only matches an 8+-char hex run or a full GUID, so meaningful parentheticals like (x64), (64-bit), (en-us) are preserved.

Stage 3 — per-group aggregation

For each merged group:

Dominant vendor = the vendor installed on the most distinct devices (max over vendor_devices), so the display vendor is the most common spelling.
Distinct device count = number of distinct device_name across all versions.
Version tracking — from per-version device sets:

Field Meaning
most_common_version version on the most devices
newest_version / oldest_version by _version_key ordering
version_counts a list of {version, count} (never a dict keyed by version)

_version_key splits on . - _ + and orders numeric chunks numerically (1.32.0 < 1.41.2 < 10.2.1.8267844), falling back to lexical for non-numeric chunks. version_counts is a list, not a version-keyed dict, so an arbitrary version string can never produce an invalid MongoDB field name.
Platforms — _dedupe_platforms does a case-insensitive dedupe preferring a Title-case display form (windows/Windows → Windows).
Sources — sorted set of the reporting integrations.

Field	Meaning
`most_common_version`	version on the most devices
`newest_version` / `oldest_version`	by `_version_key` ordering
`version_counts`	a list of `{version, count}` (never a dict keyed by version)

_clean() strips NULL bytes and control characters (some agent exports embed them) that MongoDB rejects in field names or that pollute values.

Stage 4 — upsert + stale cleanup

Rows are upserted in batches of 1000 keyed on (tenant_id, normalized_name). A run_ts timestamp is written to every rebuilt doc via rollup_at; afterwards any applications doc for the tenant whose rollup_at != run_ts is deleted — so software no longer backed by any inventory row is removed. This is what makes a re-sync self-cleaning.

Device-identity reconciliation

A single physical machine is reported under different names by different sources: Intune and SentinelOne use the full DNS name (LTG-62DTTJ4.corp.example.com), while ConnectWise Automate (and Windows NetBIOS generally) truncates to 15 characters (LTG-62DTTJ4). Naïvely they look like two assets.

The Assets aggregation (devices.search_devices, POST /devices/search) reconciles them by grouping on a NetBIOS key: the uppercased, 15-char-truncated hostname.

{"$addFields": {"_nbkey": {"$toUpper": {"$substrCP": [
    {"$ifNull": ["$device_name", ""]}, 0, 15]}}}}
{"$group": {"_id": "$_nbkey", "_names": {"$addToSet": "$device_name"}, ...}}

So LTG-62DTTJ4.corp.example.com and its 15-char NetBIOS twin LTG-62DTTJ4 merge into one asset. The canonical display name is then chosen as the longest observed name (the full DNS form) via a $reduce.

After grouping, the pipeline $lookups per-source detail into nested subdocuments (matched case-insensitively on the short/lowercased name):

Lookup	Collection	Match
`intune`	`intune_devices`	`lower(device_name) == name`
`s1`	`sentinelone_devices`	`lower(computer_name) == name`
`r7`	`rapid7_assets`	`lower(hostname) == name` or short-host of an FQDN hostname

The Intune device UUID (when present) is preferred as device_id for asset-detail deep links, and a rapid7 chip is added to sources when an r7 asset matched. The simpler GET /devices list does a lighter version of the same idea: it groups by exact device_name but still matches CSV device-enrichment sources against both the exact and 15-char NetBIOS-truncated hostname.

User linking — Entra aliases (`entra.py`)

Users come from Microsoft Entra ID (Graph /users), and are enriched with badge-access and HR-roster rows that arrive as CSV (csv_users). The problem: a badge/HR export often keys a person by an email alias, not their Entra UPN.

The Entra client selects proxyAddresses on every user. During sync, each user’s SMTP aliases are extracted from proxyAddresses (the smtp:-prefixed entries) and stored as aliases:

for pa in (u.get("proxyAddresses") or []):
    if pa.lower().startswith("smtp:"):
        aliases.append(pa.split(":", 1)[1].strip().lower())

When enriching a user, the match set is the union of the UPN, primary mail, and every alias, and csv_users rows are joined on any of them:

email_set  = {upn.lower(), mail.lower()} | {a.lower() for a in aliases}
async for cu in db.csv_users.find({"tenant_id": tid, "email": {"$in": emails}}):
    ...

So a badge row addressed to an alias still links to the right person. Enrichment flags (e.g. badge_admin) are folded onto the user record.

Structured query builder — `query_builder.py`

The Assets and Software query builders send a nested Group/Rule payload; the backend translates it into a MongoDB filter. The key safety property: callers may only reference allow-listed fields — never arbitrary Mongo paths.

Payload shape

{
  "combinator": "and" | "or" | "none" | "not_all",
  "rules": [
    { "kind": "rule",  "field": "vendor", "operator": "contains", "value": "microsoft" },
    { "kind": "group", "combinator": "or", "rules": [ ... ] }
  ]
}

Modeled by QueryRule / QueryGroup (recursive) and wrapped by SearchRequest (query, free-text search, skip, limit, sort).

FieldSpec allow-lists

A FieldSpec maps a logical field name to a db_field + a type (string | number | boolean | date). build_mongo_filter(group, allowlist) resolves every rule’s field through the allow-list; an unknown field is silently skipped and an unknown operator drops the rule, so no arbitrary path or operator can ever reach Mongo.

Allow-list	Notable fields
`APPLICATION_FIELDS`	`name`, `vendor`, `product`, `version`, `device_count`, `site`, plus intune-app enrichment: `publisher`, `platform`, `size_bytes`, `category`, `install_status`
`DEVICE_FIELDS`	shared inventory fields (`device_name`, `os_platform`, `user_email`, `source`, …) plus source-prefixed joins: `intune.` (compliance, owner, serial…), `s1.` (agent, infected, threat_count…), `r7.*` (risk_score, vulnerabilities_count, tags…)

The intune. / s1. / r7. prefixed device fields reference data joined by the $lookup stages above, so the UI can filter on source-specific attributes without those attributes living on the shared inventory row. In search_devices these post-lookup fields are applied after the $group, while bare fields are pushed into the $match before it.

Operator mapping

Operator(s)	Mongo translation
`contains` / `not_contains`	`$regex` (escaped), case-insensitive, negated with `$not`
`equals` / `not_equals`	anchored `^…$` regex for strings; exact `$eq`/`$ne` for non-strings
`starts_with` / `ends_with` (+ `not_`)	anchored regex `^v` / `v$`
`gt` `gte` `lt` `lte`	`$gt` … `$lte`
`is_empty` / `is_not_empty`	presence check across `$exists` / `null` / `""`
`in` / `not_in`	`$in` / `$nin` (comma-splits a string value)

Values are run through _coerce for the field’s type, and all string values are re.escaped before going into a regex, so user input can’t inject regex or operators.

Combinators map to $and / $or / $nor (none) / NOT(AND) (not_all). An empty group returns {}, which the caller safely $ands with the tenant filter.

Integration clients

Microsoft Graph — Intune & Entra (MSAL cert auth)

IntuneClient and EntraClient both authenticate with MSAL client-credentials against https://login.microsoftonline.com/{tenant}, using either a client secret or a certificate ({thumbprint, private_key} PEM). In production Lockstep uses the certificate path (auth_method: "certificate"). Tokens are acquired for the .default scope and cached (acquire_token_silent → acquire_token_for_client).

Client	Endpoints	Required Graph permissions
`IntuneClient`	`/deviceManagement/managedDevices`, `/deviceManagement/detectedApps`, per-device `detectedApps` (beta), `$batch`	`DeviceManagementManagedDevices.Read.All`, `DeviceManagementApps.Read.All`
`EntraClient`	`/users`, `/groups`, `/groups/{id}/members`	`User.Read.All`, `Group.Read.All` (or `GroupMember.Read.All`)

Both follow @odata.nextLink pagination and raise typed *AuthError (401/403, with permission hints) / *APIError. The Intune client additionally supports Graph JSON $batch (get_device_apps_batch) — up to 20 sub-requests per call, with per-sub-request 429 handling (Retry-After with exponential backoff), pagination re-batching, and a 30-round ceiling — to fetch per-device app inventories at scale.

ConnectWise Automate (integrator token auth)

ConnectWiseAutomateClient uses Automate’s integrator token flow, not MSAL:

POST /cwa/api/v1/apitoken  {UserName, Password}  (+ optional clientId header)
   → {AccessToken, ExpirationDate}
Subsequent calls: Authorization: Bearer <AccessToken>

Tokens are cached until 60s before ExpirationDate (_ensure_token), and a 401 on a data call triggers a single re-login + retry in case the token was revoked. It raises on error (never sys.exit) — critical when embedded in a long-running web service. Scoping is server-side via the Automate condition filter: get_computers_for_client(client_id) issues GET /Computers?condition=Client.Id=<id>. See ConnectWise Automate scoping & durable workflows for why client scoping matters and how stale data crept in.

Upload validation — `uploads.validate_upload`

Every file upload must pass validate_upload(file, kind, max_size=None) before it is persisted. It enforces three layers and returns the raw bytes (so callers don’t re-read the stream), raising UploadRejected (HTTP 400) on any failure:

Extension — the filename must end with an extension whitelisted for the kind.
Size — non-empty and under MAX_UPLOAD_SIZE (default 100 MB) or a per-call max_size.
Content (libmagic) — python-magic sniffs the real MIME from the first 8 KB and it must match a prefix allowed for the kind. This stops a malware.exe being uploaded as malware.csv.

`kind`	Extensions	Accepted MIME prefixes
`csv`	`.csv`	`text/`, `application/csv`, `application/vnd.ms-excel`
`excel`	`.xlsx`, `.xls`	`application/vnd.openxmlformats…`, `application/vnd.ms-excel`, `application/zip` (xlsx is a zip)
`pem`	`.pem`, `.key`	`text/plain`, `application/x-pem-file`, `application/octet-stream`
`json`	`.json`	`application/json`, `text/plain`

If libmagic is unavailable at runtime (missing on the host OS) content sniffing degrades to a warning-and-continue; extension + size checks still run. libmagic1 is installed in the backend image (see the Dockerfile). sanitize_filename() additionally strips path separators and control characters from user-supplied names.

CSV provider framework

app/services/csv_providers/ mirrors the API-integration pattern (client → config → sync → data → rollup) for sources that have no API and are fed by CSV/Excel export. The “client” is a Python parser and the “sync” is an upload + parse job. Each provider subclasses CsvProvider, declares a descriptor (key, name, category, target), and implements parse(rows, tenant_id):

`target`	Written to	Effect
`software`	`inventory_items` (`source=<key>`)	rolls up into Software inventory alongside the API sources
`records` / `device` / `user`	`csv_records` / `csv_devices` / `csv_users`	reference/enrichment data (badge access, HR rosters)

load() auto-detects CSV vs XLSX by magic bytes (PK\x03\x04 → xlsx via openpyxl) and honours header_signals to skip a preamble/rules block before the real header row (e.g. a PTC badge export). Column matching is loose (case/space/punctuation-insensitive) via pick, pick_contains, and has_columns, and normalize_platform() maps arbitrary OS labels onto windows / macos / linux. See CSV provider framework for the full guide.