Checklist · Buyers

Evaluating a cryptographic discovery or CBOM tool: 24 questions to ask every vendor

Published 11 October 2026 · 9 min read · By Ali Korsi Every cryptographic discovery tool finds RSA somewhere. These questions separate the tools whose inventory will stand up in front of an auditor, an architect and a migration budget from the ones that only produce a chart.

Lire en français

The EU post-quantum roadmap expects organisations to start with a cryptographic inventory by the end of 2026, and DORA, NIS2 and the Cyber Resilience Act already expect cryptography to be governed. The market has responded with a wave of tools that promise to find your cryptography and export a CBOM. Their datasheets look alike. Their output does not.

This checklist is vendor-neutral and meant to be printed and taken into vendor meetings. Each question comes with what a good answer looks like, so you can tell a demonstrated capability from a roadmap slide. It applies to us as much as to anyone else.

Before anything else: ask for a CBOM of a public repository

Before you give any vendor access to your code, ask each one for the same thing: a CBOM of a public repository, pinned to a commit, with the scan scope, the validation result and the stated limits. Better still, ask them to run it live on a public repository you choose. It costs the vendor little if the tool works, and it lets you check findings against source code anyone can read.

What that looks like: our pyca/cryptography CBOM example publishes the CycloneDX 1.6 CBOM, the validation record (schema result, file counts, scope split and a written list of limits) and the runbook for one pinned commit. Re-running the scan itself requires access to our scanner; the outputs can be checked by anyone against the public source.

1. Coverage and detection

Which languages, file types and surfaces do you discover, and which do you not?
Cryptography lives in source code, certificates, live endpoints, key stores, binaries and container images. A tool that covers one surface well and implies it covers the rest will leave gaps you only find during migration.

Good answer: A written coverage matrix by language, certificate format and surface, with each line marked available, planned or not supported. “We cover everything” is not an answer.
How does detection work, and what does it miss?
AST analysis, pattern rules and handshake inspection each have different blind spots: algorithms chosen at runtime, cryptography inside dependencies, compiled code.

Good answer: The detection method per language or surface, the known blind spots, and the detector named on each finding so you can judge it.
Do you separate production code from tests, examples and vendored code?
In a cryptographic library, test vectors can outnumber production calls by an order of magnitude. Without scope tags, the inventory measures your test suite.

Good answer: Every finding tagged by scope (production, test, example, vendor), with counts reported separately and filters to exclude non-production code from scoring and gating.

2. Evidence quality

Can every finding be traced to where it was found?
An auditor or an engineer will ask “where?”. An aggregate count cannot be verified or fixed.

Good answer: File and line (or certificate, or endpoint) on every finding, plus the repository revision that was scanned.
What do you record beyond the algorithm name?
“RSA” is not enough to plan a migration. Key size, curve, mode, padding and use case decide what replaces it and how urgent it is.

Good answer: Primitive, key size or curve, mode and padding where they are detectable, and an explicit “unknown” where they are not, rather than a guess.
Is a scan repeatable?
If two runs on the same code disagree, neither result is evidence.

Good answer: Scanner version and target revision recorded with each run, and the same revision scanned twice giving the same normalised findings.

3. CBOM standard compliance

Is the export a CycloneDX 1.6 CBOM, validated against the official schema?
A proprietary JSON “inspired by” CycloneDX locks your inventory to the vendor and may not load in your GRC or SBOM tooling.

Good answer: CycloneDX 1.6 cryptographic-asset components, validated against the published CycloneDX schema, with a validation result you can rerun yourself.
Are inventory facts kept separate from the vendor’s interpretation?
Risk scores and recommendations are opinions; algorithm, location and parameters are facts. Mixing them makes the CBOM less portable and harder to audit.

Good answer: Facts in standard CBOM fields; scores, reasoning and recommendations in separate, attributable outputs.
What is not in the CBOM?
Every CBOM has limits: references in code are not deployed keys, and few tools capture a complete dependency graph.

Good answer: A written list of limits that ships with the CBOM, not discovered by your auditor.

4. Deployment and data handling

Where does the scan run, and does source code ever leave your environment?
You are handing the tool a map of your most sensitive security controls. Where that map is built matters as much as what it contains.

Good answer: On-premises or in your own CI runners, a documented data flow, and no source code in the exported artefacts.
Can it run fully offline, and which features need a network connection?
“Air-gapped” claims often hide a licence check, a cloud dashboard or an AI feature that calls out.

Good answer: A list of every feature that makes an external call, each with an offline fallback, and a straight answer on whether offline operation has been independently tested.
What happens to your inventory if you stop using the tool?
An inventory you can only read inside the vendor’s platform is a dependency, not an asset.

Good answer: Open-format outputs that stay usable without the vendor, a reversibility clause, and a security posture document your procurement team can file.

5. CI/CD integration

Can it run in the pipeline and fail a build on a policy you define?
Inventory without a gate is a leaking bucket: new quantum-vulnerable cryptography arrives while you remove the old.

Good answer: A CLI with meaningful exit codes, a configurable severity threshold and a tolerated count, so you can start in report-only mode and tighten over time.
Do results land where developers already work?
A separate dashboard nobody opens does not change code.

Good answer: SARIF output for code scanning, with file, line, severity and a recommended replacement on each alert.
Can you see cryptographic drift between releases?
The question after the first inventory is always “what changed?”.

Good answer: Inventory snapshots per release and a diff that lists added and removed cryptography, exportable for your pipeline or GRC tool.

6. Risk scoring and prioritisation

Is risk scored per asset, with the reasoning shown?
A single organisation-wide rating cannot tell you where to start, and a black-box score cannot be defended.

Good answer: A per-asset score with written reasoning that names the factors: algorithm weakness, exposure, data lifetime, scope.
Does the score account for how long your data must stay confidential?
Harvest-now-decrypt-later risk depends on data lifetime, which the tool cannot guess from code.

Good answer: A configurable sensitivity policy (per path, repository or system) that changes the score, documented and versioned with your code.
Does it go from risk to a migration plan?
A ranked list of findings is not a plan. You need replacements, sequencing and effort.

Good answer: A recommended replacement per asset mapped to FIPS 203/204/205 where relevant, and findings grouped into migration waves.

7. EU regulatory mapping

Which obligations are findings mapped to, and at what level of detail?
“DORA-ready” on a datasheet tells your compliance team nothing.

Good answer: Mapping to named provisions, for example DORA Article 9 and Delegated Regulation 2024/1774, NIS2 Article 21(2)(h) and Implementing Regulation 2024/2690, and the EU Cyber Resilience Act, plus the EU post-quantum roadmap milestones.
Does the vendor avoid claiming that the tool makes you compliant?
Compliance is a judgement about your organisation, not a property of software. Over-claiming is a red flag for the rest of the pitch.

Good answer: Wording such as “supports compliance activities”, and evidence an auditor can read: dated inventory, method, limits.
Can your board and auditors use the report without the vendor’s platform?
Supervisors and boards read documents, not dashboards.

Good answer: A dated report in a standard format, in the language your board and supervisor work in, that stands on its own.

8. Commercial and support

Is the first engagement fixed in scope and price?
Open-ended discovery projects are hard to approve and harder to close.

Good answer: Published or fixed pricing for a first, bounded engagement, with a clear deliverable, before any platform commitment.
Who runs the tool, and who supports you?
Interpreting a cryptographic inventory takes expertise; so does fixing a false positive in a gate that blocks releases.

Good answer: Named people who scope and run the engagement, a support commitment for the platform tier, and a published vulnerability disclosure policy.
Will the vendor run the tool in front of you, on public code?
A slide deck proves nothing, and you should not have to hand over code to find out whether a tool works.

Good answer: A live run on a public repository that resembles your stack, ending with a CBOM you keep and can inspect.

How to score the answers

Score each answer per vendor, and only count what you have seen, not what you were told.

Score	Meaning
0	No answer, or an answer that avoids the question.
1	Claimed in writing (datasheet, proposal, contract), not yet shown.
2	Demonstrated: you saw it on real code or hold the artefact (CBOM, report, SARIF file).

Weight the sections to your situation. A bank under DORA will weight evidence, deployment and regulatory mapping; a software vendor preparing for the CRA may weight CI/CD and the CBOM standard. Treat a 0 on questions 4, 7 or 10 as a blocker: without traceable findings, a valid CBOM and control over where your code goes, the rest does not matter.

How CRYPTAGION answers these questions

We wrote this checklist to be used on us too. In short: CRYPTAGION discovers cryptography in Python (AST), JavaScript/TypeScript, Java, Go and C/C++ source, X.509 certificates (PEM, DER, CRT, CER) and live TLS endpoints. Every finding carries its file and line or endpoint and a scope tag (production, test, example, vendor). It exports a schema-validated CycloneDX 1.6 CBOM and keeps risk scores, with their reasoning, in separate outputs. Scores take data sensitivity and HNDL exposure from a per-path .cryptagion.yaml policy. It runs on-premises or air-gapped, and in CI with a policy gate and SARIF output. Container image and binary analysis are on our roadmap, not available today. The deployment and data flow page and the security posture document cover the data-handling questions; pricing is public.

Sources

Put us through the checklist

In a 30-minute demo we run CRYPTAGION live on a public repository that matches your stack, and you leave with the CBOM. No access to your code needed.

Request a free demo →

← All resources · Home