# CBOMkit 2.3.0 on pyca/cryptography — comparison run

Prepared 12 October 2026 by CRYPTAGION. This is an independent comparison; it is not
endorsed by PQCA, the CBOMkit maintainers or the pyca project.

## What was run

| | |
|---|---|
| Target | https://github.com/pyca/cryptography at commit `d45b7930f93c9f139ab09cfb91990e3250bc989b` (the same commit as our [pyca evidence bundle](../../pyca-cryptography/2026-10-10/RUNBOOK.md)) |
| Tool | CBOMkit 2.3.0 (git tag `2.3.0`, commit `feace38c0e9b4ea196d8d026bf20476ed0a80a08`), image `ghcr.io/cbomkit/cbomkit@sha256:20d37926b979d7c901d130fe8a4178c3f87601df4a69e66e366710f767f6b007` |
| Database | `postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea` |
| Host | macOS 26.5.2, arm64, Docker 28.3.0 |
| Mode | CBOMkit service API (the documented quickstart path), default settings, no build of the target |

The CBOMkit service clones a git URL and checks out a branch or tag; it cannot be pointed at a
bare commit. To pin the commit, a local bare mirror of pyca/cryptography was given a branch
`cbomkit-pin` at `d45b7930…` and mounted read-only into the backend container, then scanned as
`file:///repos/cryptography` with branch `cbomkit-pin`. Both CBOMs record `commit: d45b793`.

`compose.yaml` is the locked-down deployment used (API on 127.0.0.1 only, Postgres not
published, no frontend, no OPA). `scan.py` starts a scan over CBOMkit's websocket API and saves
the progress messages and the CBOM. The scan was run twice.

## Files

| File | Content |
|---|---|
| `cbomkit-run-1.cbom.json`, `cbomkit-run-2.cbom.json` | CBOMs returned by CBOMkit (CycloneDX 1.6) |
| `cbomkit-run-1.progress.json`, `cbomkit-run-2.progress.json` | Progress messages (detections, module index, file and line counts) |
| `compose.yaml`, `scan.py` | How CBOMkit was deployed and called |
| `analyse.py`, `comparison.json` | Comparison against the CRYPTAGION CBOM at the same commit |

`analyse.py <workdir> <repo-root>` expects the CycloneDX schemas from our evidence bundle in
`<workdir>/evidence/schemas`, the two CBOMkit outputs in `<workdir>/run/out1` and
`<workdir>/run/out2`, and reads the CRYPTAGION CBOM from
`samples/pyca-cryptography/2026-10-10/cbom.json`.

## Results

- **Schema:** both CBOMkit CBOMs validate against the official CycloneDX 1.6 schema with zero
  errors (same schema files and validator settings as our bundle).
- **Repeatability:** the two runs are identical once random identifiers (bom-refs, generated
  key names) are normalised.
- **Scope:** CBOMkit indexed 131 files — 124 Python files in the root module, 2 in `vectors/`,
  1 Java file and 4 Go files under `docs/development/custom-vectors/`. The 124 matches the
  number of tracked Python files outside `tests/` and `vectors/`: the 117 files under `tests/`
  were not scanned. The CRYPTAGION run covered all 243 tracked Python files and tagged each
  finding by scope; it did not scan the Java or Go files.
- **Findings:** CBOMkit returned 17 components (9 algorithms, 8 related keys) with 25
  occurrences: 8 in library source (4 lines in `ssh.py` and `dsa.py`) and 17 in documentation
  vector scripts (Python, Java, Go). CRYPTAGION returned 1,804 detections: 46 in library source
  (11 files), 1,746 in tests and 12 in documentation scripts.
- **Overlap:** one source line is reported by both tools
  (`docs/development/custom-vectors/rsa-oaep-sha2/generate_rsa_oaep_sha2.py:25`).

## Limits

- One repository, one commit, one host. pyca/cryptography is a cryptographic library, not an
  application that calls one; tools that resolve calls into crypto APIs have less to work with
  in the library's own source. Results on an application repository will differ.
- CBOMkit was run as a service with default settings and without building the target, as its
  README warns this reduces completeness for Java. The sonar-cryptography plugin or
  CBOMkit-action with a build may give different results.
- "Detections" and "occurrences" are not vulnerabilities and the two tools count different
  things (CBOMkit: resolved API calls and related keys; CRYPTAGION: imports, constructors,
  lookup tables and capability checks). Counts are not a quality score.
- Both tools labelled the DSA key generation in pyca as Diffie-Hellman in at least one place
  (CBOMkit: `FFDH` at `src/cryptography/hazmat/primitives/asymmetric/dsa.py:185`; CRYPTAGION:
  a test call, noted in our bundle's `validation.json`).