GitLab candidate pipeline

Scope: IHL-CTR-0001 lab candidates; executor: dedicated GitLab Runner on a separately managed VM; registry: project GitLab Container Registry with approved production promotion to indianhilllabs.azurecr.io; release status: signed private-lab publication with a separately approved, workload-federated ACR boundary.

The pipeline turns a reviewed source revision into an immutable runtime and reference API candidate at the typed container-level pipeline input, mapped to CONTAINER_LEVEL for the jobs, then evaluates those same target digests through additive Secure → Hardened → Validated qualification stages. The selectable levels are secure, hardened, and validated; the default is hardened. It uses the short-lived GitLab job credentials already provided to each job. No long-lived registry password is stored in the repository.

Stage flow

Stage	Job	Result
Validate	validate:source	Checks digest pins, non-root runtime invariants, rootless build settings, pinned helper images, and build-context exclusions.
Validate	validate:documentation	Builds the public documentation with MkDocs strict mode.
Validate	validate:azure-federation	Uses a short-lived, branch-bound GitLab OIDC token to verify the expected Azure subscription and registry through a read-only call. It does not import content or bypass production approval.
Build	build:images	Uses rootless BuildKit to build the pipeline-scoped external OpenSCAP scanner, the Secure runtime, its selected Hardened or Validated descendant, and the reference API; pushes the candidates and records immutable ancestry and scanner identity.
Test	test:reference-api	Starts the exact reference API candidate and verifies health, identity, UID 1654, Hardened diagnostic lockdown, and the Validated FIPS approved-mode probe when applicable.
Evidence	evidence:sbom	Produces native Syft, SPDX JSON, and raw CycloneDX JSON SBOMs for both digest-bound images.
Evidence	evidence:gitlab-cyclonedx	Preserves the raw SBOMs and adds GitLab's required CycloneDX ingestion metadata to separate report files.
Scan	scan:vulnerabilities	Scans the native Syft catalogs with Grype and retains the complete JSON reports.
Scan	container_scanning:runtime	Runs GitLab Container Scanning against the exact runtime digest and uploads GitLab's native vulnerability report.
Scan	container_scanning:reference-api	Runs GitLab Container Scanning against the exact reference API digest and uploads GitLab's native vulnerability report.
Scan	scan:container-compliance	Reads the controlled container catalog, creates digest-bound application/assessment plans, unpacks each image without executing it, and runs every registered OpenSCAP STIG/CIS profile with XCCDF, ARF, HTML, and normalized JSON output.
Secure	secure:qualification	Summarizes findings, intersects CVEs with CISA KEV, validates immutable build/SBOM/runtime identities, and records the Secure decision.
Hardened	hardened:qualification	Requires Secure qualification, a distinct Hardened/Validated image, observed .NET diagnostic lockdown, and accepted tailored CIS Level 1 results for both exact image filesystems.
Validated	validated:qualification	Requires Hardened qualification, accepted tailored STIG results for both exact image filesystems, and verified exact OpenSSL module/base identities, provider digest, approved-mode enforcement, integrity/self-tests, and application tests.
Report	export:assurance-evidence	Validates cross-report identity, uploads GitLab Requirement results, and creates a SHA-256-manifested evidence handoff for the exact source job.
Report	trigger:assurance-dashboard	Starts the separate private dashboard pipeline with exact upstream lineage. Dashboard presentation is not a container-promotion control.
Publish	publish:secure-lab, publish:hardened-lab, publish:validated-lab	Protected manual gates apply the tier-specific lab tag to exact independently qualified digests and bind the unchanged manifest SHA-256 and export job.
Sign	sign:*	Uses GitLab OIDC and Sigstore keyless signing for both exact image digests after protected publication.
Verify	verify:*	Enforces the exact GitLab main pipeline certificate identity, GitLab issuer, required manifest/tier/revision annotations, and transparency-backed Cosign result.
Retain	retain:*	Fails closed on the original manifest hash, builds a deterministic release-evidence archive, and uploads it to a pipeline-and-manifest-specific GitLab Generic Package Registry version.
Promote	promote:*‑acr	After a protected production approval, uses GitLab-to-Azure workload federation to import only verified digests into ACR and locks the target tags.
Deploy	deploy:lab, deploy:validated-lab	Manually deploys the promoted digest through the restricted SSH identity. Both paths serialize changes to the single private lab; Validated adds live FIPS approved-mode verification.

Build, test, and evidence consumers pass image identities in repository@sha256:... form. Tags are convenient pointers; they are not used as the identity between stages. variant-lineage.json binds the selected target images to the Secure and, when applicable, Hardened parent digests. Candidate tags contain candidate-<level>-<commit>; the tag does not replace the recorded digest.

GitLab security views

The two container_scanning:* jobs use GitLab's supported Container Scanning analyzer and upload gl-container-scanning-report.json as artifacts:reports:container_scanning. GitLab merges the job reports for the pipeline Security tab. Findings from a successful default-branch pipeline are ingested into Secure > Vulnerability report at the project level. Merge request pipelines also provide security comparison data when the target branch has a completed baseline report.

The runtime and reference API use separate scan jobs so both image digests have an explicit report and job result. Supported operating-system and language-package vulnerabilities, unfixed findings, every severity, and end-of-life operating-system findings are enabled.

GitLab's analyzer bundles its vulnerability database and disables database downloads at runtime. The analyzer therefore uses GitLab's supported :8 major-version reference instead of a digest, and the dedicated runner pulls it on every job. This is a narrow exception to the pipeline's helper-image digest rule so each assessment receives GitLab's current v8 analyzer and advisory data. The analyzer identity recorded in each report and job log is part of the assessment evidence; it does not affect the built container digest.

GitLab's native report is the source for GitLab vulnerability workflow, status, and triage. The retained Grype reports and vulnerability-summary.json remain the source for the current automated promotion decision. Differences between the two scanners require review; one report is not evidence that the other is incorrect.

Secure qualification boundary

The Secure qualification is intentionally conservative. It requires immutable BuildKit identities, structurally valid SPDX inventories, successful reference API identity and UID 1654 observations, no High or Critical matches for either image, and no reported CVE intersection with the current CISA KEV catalog. GitLab's two native scanning jobs are also blocking dependencies of this qualification.

A reported scanner match is not automatically an applicable vulnerability. When the policy blocks promotion, the correct next action is applicability review and supporting evidence. This initial pipeline does not implement exceptions or VEX overrides, so it cannot convert that review into an automated approval. Changing the gate requires a reviewed policy implementation, not editing generated evidence.

Hardened qualification boundary

Before Hardened qualification, scan:container-compliance applies the catalog at compliance/container-compliance/catalog.json. The current catalog binds both produced images to the Ubuntu 24.04 Chiseled STIG and CIS Level 1 profiles. It records the exact tier Dockerfile that applied the controls, its SHA-256, the immutable target digest, and the immutable scanner digest. Skopeo and Umoci expose the merged image filesystem to oscap-chroot; scanner software is never added to the product image.

Every catalogued data stream is installed into the scanner from its SHA-512-pinned release archive and recorded in openscap/content-manifest.json. Raw OpenSCAP profiles remain report-only for visibility. The tailored CIS profile is enforced for Hardened and Validated candidates; the tailored STIG profile is enforced only for Validated candidates. Both are report-only for lower tiers. Missing evidence, scanner execution errors, mutable image identities, mixed artifact identities, or a tailored result that violates its reviewed matrix block the applicable tier. See Repeatable Container Compliance.

The Hardened job still produces the reusable common Container Platform and complete Ubuntu V1R6 applicability reports for visibility and later Validated evaluation. Those engineering overlays do not define the Hardened claim. The Hardened qualification gate is the tailored CIS Level 1 container-filesystem assessment for both exact images, plus product-specific .NET diagnostic lockdown.

Lab promotion requires a valid evidence package in addition to the vulnerability-policy decision. This is a completeness and consistency gate, not a claim that a STIG was passed. The common profile covers 20 selected controls; the rest of the 188-rule benchmark requires platform, host, registry, identity, audit, or organizational assessment.

The same job applies IHL-STIG-UBUNTU-24.04-CONTAINER-1 to both images. It classifies all 194 Ubuntu 24.04 V1R6 rules across image, runtime, host, organization, and not-applicable boundaries. Before generating the report, CI evaluates the Syft inventory for the expected Ubuntu userspace, prohibited packages, interactive and package-manager binaries, regular-file SHA-256 coverage, unsafe modes, and non-system owners. Required checks can downgrade a classification for an individual digest; they cannot silently leave a stale not-applicable or supported result in place.

Hardened qualification requires Secure qualification, a separately constructed image that inherits the exact Secure runtime digest, observed .NET diagnostic lockdown with no diagnostic IPC socket, accepted enforced tailored CIS results for both artifacts, and indexed private-lab runtime controls. It is a CIS-aligned engineering assessment, not CIS certification. The STIG reports continue to expose inherited, partially supported, planned-gap, and not-applicable statuses without changing the Hardened decision. See Container and Ubuntu STIG Evidence for the control status definitions, applicability boundaries, and current gaps.

Validated qualification boundary

Validated qualification inherits Hardened and then requires complete legacy STIG applicability evidence plus accepted enforced tailored STIG results for both exact image filesystems. This is a bounded container-image assessment, not full host or system STIG compliance. Validated also requires fips-validation.json with the exact candidate digests and source revision. The controlled initial target is the OpenSSL FIPS Provider 3.1.2 under CMVP certificate 4985. The report must also bind the provider SHA-256 and show approved mode, module integrity tests, startup self-tests, and application cryptographic tests all passed.

Dockerfile.validated builds the official OpenSSL 3.1.2 source archive after checking the OpenSSL-published SHA-256 and retains only its unchanged validated fips.so provider. It separately builds the checksum-pinned OpenSSL 3.5.7 LTS base library with no-fips, preventing that base build from substituting an unvalidated provider. The supported base generates the module integrity configuration with fipsinstall -pedantic -self_test_onload, activates only the FIPS and base providers, and enforces default_properties = fips=yes.

The reference application then independently verifies the supported base-library version and source identity, validated module source identity and digest, provider activation, on-demand self-test, approved SHA-256 fetch, blocked MD5 fetch, and .NET SHA-256/AES-GCM operations. test:reference-api emits those results as fips-validation.json only for the Validated candidate. Missing or mismatched evidence produces not-qualified rather than a FIPS claim. See Container Assurance Levels.

Actionable assurance view

The report stage validates that all three assurance decisions, every raw and tailored CIS/STIG assessment, common and Ubuntu STIG reports, automated Ubuntu image assessments, vulnerability-policy summary, runtime observation, and variant lineage all refer to the current source revision, constructed level, exact tier mode, and exact image digests. It rejects a higher qualified tier when its predecessor did not qualify or the corresponding image variant was not constructed, and publishes separate Secure, CIS, STIG, and FIPS claim states before uploading binary results for all 20 GitLab Requirements, and creates a SHA-256-manifested artifact bundle. A multi-project trigger passes the exact source pipeline and export-job identity to the separate private dashboard project.

That project owns normalization, detailed category views, GitLab Pages, confidential issue draft actions, and downloadable reports. Keeping presentation separate prevents dashboard dependencies from entering the container build and promotion trust boundary. See the Container Assurance Dashboard for the handoff contract and assessment boundary.

Protected publication

On a successful default-branch pipeline, an authorized operator may run only the publication job matching the independently constructed candidate: publish:secure-lab, publish:hardened-lab, or publish:validated-lab. The jobs recheck the tier decision, the exact image digests, the original assurance-manifest SHA-256, and the export-job identity before applying 10.0.11-<tier>-lab.1 and resolving the tag back to its digest. secure-lab, lab, and validated-lab must be protected environments.

Promotion is a blocking manual gate. After automated policy passes, GitLab holds the pipeline at the applicable publication job until an authorized operator confirms promotion. Only a successful promotion makes the separate deploy:lab manual action available.

The lab tag is mutable and is not a production release. Consumers that need repeatable verification must record and use the digest written to the pipeline evidence.

On a successful default-branch Validated pipeline, publish:validated-lab is a separate blocking manual gate associated with the validated-lab environment. It requires all three additive qualifications and binds the exact runtime/reference digests, FIPS module SHA-256, assurance-manifest SHA-256, source revision, and evidence export job in validated-lab-promotion.txt before applying 10.0.11-validated-lab.1. Configure validated-lab as a protected environment and add a deployment approval rule before treating this gate as authorization. The tag is private-lab distribution, not a production release.

Keyless signing and verification policy

Publication automatically unlocks a tier-specific signing job. Cosign receives a short-lived GitLab OIDC token with audience sigstore and signs the runtime and reference API by immutable digest. No long-lived signing key is stored. The following verification job requires the exact certificate identity https://gitlab.com/indianhilllabs/indian-hill-secure-containers//.gitlab-ci.yml@refs/heads/main, issuer https://gitlab.com, and the expected tier, source revision, and manifest-hash annotations. The machine-readable policy is compliance/supply-chain/cosign-keyless-policy.json.

The signature proves that the approved GitLab workflow signed a specific digest. It does not prove vulnerability absence, complete STIG compliance, or that the entire container is FIPS validated.

Durable release evidence

After signature verification, the retention job independently checks the byte hash of the existing assurance-manifest.json, the publication record, both verification outputs, exact digests, source revision, tier, and export job. It then creates a deterministic archive and uploads the archive, checksum, and metadata to the GitLab Generic Package Registry under container-assurance-evidence/<tier>-<pipeline-id>-<manifest-sha256>. This operationally durable package is separate from expiring job artifacts. GitLab project owners can still delete a generic package; regulatory WORM retention requires a separately approved storage control.

Production ACR approval boundary

Each tier has a manual promote:<tier>-acr job associated with the protected production/<tier> environment. The job requires the signed verification record and durable package result, authenticates to Azure through the branch-bound GitLab OIDC federated identity, imports only the exact digests, signs both ACR digest identities with the protected GitLab keyless identity, verifies the destination signatures and OCI referrers, and disables writes and deletion on both the release tags and manifests. Destination signing evidence is uploaded to a separate pipeline-and-manifest-specific Generic Package Registry version. The Entra identity has Container Registry Data Importer and Data Reader plus Container Registry Repository Writer only at the IndianHillLabs registry scope. GitLab requires one Maintainer approval and does not allow the pipeline triggerer to self-approve. At least two eligible project members are therefore required to execute this production path.

Private lab deployment

The protected manual deploy:lab job consumes the exact reference API digest and the promotion record rather than deploying the mutable lab tag. GitLab's job-scoped registry credential is streamed to a forced-command SSH account and is never written to the repository or retained on the host. The host deployment wrapper validates the repository and digest, uses a temporary root-only Docker client configuration, starts the Hardened Compose application, and verifies health, UID 1654, a read-only root filesystem, writable temporary storage, dropped capabilities, no-new-privileges, resource limits, the Hardened assurance label, disabled .NET diagnostics, and absence of a diagnostic IPC socket.

The API is published only on host loopback at 127.0.0.1:8080. Administrators reach it through a separate restricted local SSH tunnel; no application port is allowed through the host firewall.

The protected manual deploy:validated-lab job consumes only the Validated promotion record and sends the exact image and module digests to the same forced-command account. The host repeats every Hardened runtime check, verifies the live OpenSSL FIPS Provider 3.1.2 identity, CMVP certificate 4985, supported OpenSSL 3.5.7 identity, the expected module hash, approved-mode enforcement, integrity/self-tests, approved SHA-256 and AES-GCM operations, blocked MD5 resolution, and the image crypto labels. Both deployment jobs use resource_group: lab, preventing concurrent changes to the single Compose application.

Evidence retained by CI

Candidate jobs retain the following under pipeline-evidence/ for 30 days. Publication, signature, verification, and release-package artifacts are retained for one year, and the release bundle is also uploaded to the Generic Package Registry:

• BuildKit metadata, exact runtime/reference image identities, and Secure/Hardened ancestry;
• reference API functional and UID observations;
• native Syft, SPDX JSON, and CycloneDX JSON SBOMs;
• GitLab-native container vulnerability reports for both images;
• Syft and Grype version output and complete Grype JSON reports;
• CISA KEV source, catalog metadata, payload hash, match list, and policy decision;
• per-digest common STIG gap reports with selected control status, evidence paths, and evidence-file hashes;
• per-digest Ubuntu userspace assessments and complete 194-rule Ubuntu V1R6 applicability reports;
• additive Secure, Hardened, and Validated qualification decisions with the controlled assurance-policy hash and explicit claim limitations;
• for a Validated candidate, the exact OpenSSL module and supported base-library source identities plus approved-mode fips-validation.json runtime evidence;
• the GitLab Requirement report and SHA-256-manifested dashboard evidence handoff; and
• a tier-specific promotion record when a manual publication job is run;
• Cosign version, signing identity, and per-image signature verification results; and
• deterministic release archive, checksum, durable package URL, and manifest binding.

The Generic Package Registry is durable operational retention, not immutable regulatory archive storage. A formal retention schedule and WORM-capable archive remain organizational controls.

The GitLab report variants identify gitlab:meta:schema_version=1, the applicable repository-relative Dockerfile as the dependency input and remediation source, and the candidate image repository and tag. These properties allow GitLab to ingest third-party CycloneDX files without changing the raw Syft output. Syft's CycloneDX output is explicitly pinned to specification 1.6 because GitLab does not yet ingest 1.7. The Dockerfile path is provenance context for an image scan; it does not claim that every discovered package came from a language lockfile.

Current limitations

• The automated service test verifies application behavior and image UID, but GitLab service syntax does not assert the manual lab's read-only root filesystem, dropped capabilities, or no-new-privileges controls.
• BuildKit-produced minimal provenance is attached to candidates. Published digests now have an approved GitLab keyless signing identity and verification policy; a separate signed SLSA provenance attestation is not yet produced.
• The pipeline targets linux/amd64 only.
• Candidate tags older than 30 days are selected by the weekly registry cleanup policy; the ten most recent tags per image and tier/version/signature tags are preserved.
• A live vulnerability feed can change a later assessment of an unchanged digest.
• GitLab's mutable v8 analyzer can change between assessments; the runner's mandatory pull policy keeps its bundled advisory data current, while retained reports identify the scanner that produced each result.
• The Container Platform common profile is a selected pre-assessment overlay. The Ubuntu overlay classifies the complete V1R6 rule catalog, but it remains an engineering applicability assessment and does not replace official checks, assessor-approved not-applicable decisions, or inherited host, platform, and organizational evidence.
• Certificate 4985 identifies the validated module and tested operational environments; it does not certify this container or its Ubuntu 24.04 deployment. Production use still requires Cryptographic Officer review of the exact environment and vendor guidance.
• Tier lab tags remain mutable discovery labels. The signature, verification, release package, ACR import, and consumers all use immutable digests. ACR release tags and manifests are locked after destination signature verification.

The remaining assurance increments are enforcing this signature policy in any future runtime platform outside the GitLab-to-ACR promotion boundary, adding WORM evidence retention where required, and completing separate host/platform control overlays.