UBI10 base-image migration for Facekom container fleet

FKITDEV-8252 — UBI10 base-image migration for Facekom container fleet

Phase A.6.1 complete (2026-05-18, later in day)

All 5 UBI10 base images probe-build green on UBI10 + libkrun (8 GiB / 6 CPUs): portal_css 658 MB, vuer_css 805 MB, vuer_oss 2.44 GB, janus 313 MB, vuer_cv 5.93 GB. Two new commits on feature/FKITDEV-8252-ubi10 (f38c8e2 probe-buildable existing bases, b984006 add UBI10 vuer_cv base) — branch is now 17 commits ahead of origin/main, not pushed. Author identity: andras.lederer@alpiq.com (work). The earlier “per-key rpmkeys” approach (Q1 decision below) did not survive UBI10 reality — see Key lessons learned (Phase A.6.1) for why we ended up on update-crypto-policies --set LEGACY after all.

One-line summary

Migrate the entire Facekom container fleet from UBI8/UBI9 to Red Hat UBI 10 ahead of RHEL 9 EOL. Three repos in series: vuer_build (Phase A.6.1 done; A.6.2 next), vuer-release (62 Dockerfiles), vuer_docker (Bence’s PR #203 pending, currently conflicting). vuer_cv is now in-scope — a fresh UBI10 base Dockerfile was added today and probe-builds green at 5.93 GB.

Ticket / scope

Property	Value
JIRA	FKITDEV-8252
Owner	Bence László
Severity	P2 (proactive, ahead of RHEL 9 EOL)
Driver	RHEL 9 EOL deadline; ahead-of-time fleet posture
Repos in scope	vuer_build, vuer-release (62 Dockerfiles), vuer_docker
vuer_cv	in-scope — new UBI10 base Dockerfile added & probe-builds green (5.93 GB)
Branch	feature/FKITDEV-8252-ubi10 in /Users/levander/coding/facekom/vuer_build
Branch state	17 commits ahead of origin/main, not pushed (was 15 at note creation)
Author identity used	andras.lederer@alpiq.com (work)
Build host	macOS Apple Silicon, podman machine on libkrun, 8 GiB RAM minimum, 6 CPUs
Ticket source	/Users/levander/Downloads/FKITDEV-8252-Ubi10.md

Phase plan

Phase	Repo / scope	Status	Notes
A.6.1	vuer_build — 4 existing base/* Dockerfiles + new base/vuer_cv	DONE	All 5 probe-build green (see size table above)
A.6.2	vuer_build — remaining 3 base/* Dockerfiles (likely common/*)	open	Need to identify which 3 — see
B	vuer-release (62 Dockerfiles)	not started	After Phase A fully closes
C	vuer_docker / Bence’s PR #203	not started	PR #203 currently conflicting; reconcile after A+B

Phase A.6.1 — results (2026-05-18)

Five-for-five

Every base image in vuer_build/base/ now probe-builds clean on UBI10 + libkrun (Apple Silicon emulated linux/amd64). Iteration counts below are how many probe-build rounds it took to land each one — useful as a complexity indicator for Phase B planning.

Image	Final size	Iterations to green	Notable
base/portal_css	658 MB	3	Simplest; no microdnf update before key import — masked the rpmkeys-flag-disappearance bug for a while
base/vuer_css	805 MB	3	Slim portal-sister parallel to portal_css
base/vuer_oss	2.44 GB	5	Heaviest package surface in the existing four
base/janus	313 MB	7	Multi-stage; exposed the rpmkeys-flag-disappearance bug, the missing gzip, the GitHub archive v-prefix gotcha
base/vuer_cv	5.93 GB	7	Brand-new UBI10 base; ML deps inflate size; git-lfs + libmicrohttpd (EPEL10) + ENV_VERSION=8 matching requiredEnvVersion

New commits on the branch

Commit	Subject
f38c8e2	fix(FKITDEV-8252): make existing UBI10 base images probe-buildable
b984006	feat(FKITDEV-8252): add UBI10 base image for vuer_cv

Not pushed. Branch is 17 commits ahead of origin/main.

Key lessons learned (Phase A.6.1)

Capture these for the team — they will hit every other Dockerfile in Phases B and C

Seven probe-build iterations on base/janus and base/vuer_cv surfaced ten reusable lessons. The first three together explain why the original “per-key rpmkeys” decision (Q1) had to be revised.

1. rpmkeys --import --allow-sha1-signatures is unreliable on UBI10

The --allow-sha1-signatures flag exists in the UBI10 base image’s rpmkeys but disappears after a microdnf -y update — because the package update pulls in a newer rpm-libs from the UBI10 stream that has the flag stripped.

This is why portal_css passed in iter 4 but janus stage 1 failed simultaneously: portal_css does not microdnf update before importing keys, so its rpmkeys still had the flag. janus stage 1 does microdnf update first, then tries rpmkeys --import --allow-sha1-signatures against an rpmkeys that no longer accepts that flag.

2. update-crypto-policies --set DEFAULT:SHA1 doesn’t work on UBI10

The sub-policy module pattern (DEFAULT:SHA1, which would additively re-enable SHA1 only on top of DEFAULT) requires the SHA1.pmod sub-policy file. That file doesn’t ship in crypto-policies-scripts-20250905-2.gitc7eb7b2.el10_1.1.noarch.

Available modules in that package: LEGACY, FUTURE, FIPS, DEFAULT, and the NO-* removal modules. No additive SHA1 module exists on UBI10. The clean per-policy-knob path is closed.

3. LEGACY is the correct mechanism on UBI10

update-crypto-policies --set LEGACY is the documented Red Hat escape hatch for accepting SHA1-signed GPG keys on UBI10 — and the narrower alternatives (per-key flag in §1, additive sub-policy in §2) don’t exist in practice. Yes, LEGACY’s surface area is wider (it also re-enables SHA1 for TLS/SSH/kernel module signatures inside the build sandbox), but at base-image-build time that exposure is bounded to the build stage. Used in 7 stages across portal_css, vuer_css, vuer_oss, janus (both stages), vuer_cv.

This supersedes Q1 decision below (per-key rpmkeys). Treat that decision as "tried and reverted." Anything in Phases B/C that still says "use per-key rpmkeys" should be flipped to LEGACY.

4. Order matters when installing crypto-policies-scripts

If you COPY yum/CentOS-Base.repo before installing crypto-policies-scripts, microdnf’s solver will pick the CentOS Stream 10 SHA1-signed version of crypto-policies-scripts over UBI10’s SHA2-signed one — and then fail to verify it (chicken-and-egg: you need crypto-policies-scripts to run before you can accept SHA1, and you can’t install the SHA1-signed one without first relaxing crypto policy).

Fix: install crypto-policies-scripts from the UBI10 BaseOS stream first, then COPY the CentOS Stream 10 repo and run update-crypto-policies --set LEGACY, then import the SHA1 GPG keys.

5. UBI10 minimal does not ship gzip

Must be added to compile-stage microdnf install for any Dockerfile that does tar xzf. Surfaced when janus build couldn’t unpack libnice-0.1.17.tar.gz. Other Phase B Dockerfiles that tar anything need a sanity check.

6. GitHub archive URL strips the v prefix

https://github.com/.../archive/v1.4.1.tar.gz extracts to a directory named janus-gateway-1.4.1/ (no v). Use the parameter expansion cd ${VAR#v} to handle both tag-shaped (v1.4.1) and SHA-shaped (a1b2c3d) values without an if-statement.

7. UBI10 package name differences vs Ubuntu

The ones we hit:

• libopus (Ubuntu) → opus (UBI10); headers in opus-devel
• libmicrohttpd is in EPEL10, not BaseOS — must microdnf -y install epel-release (or equivalent) in stages that need it
• git-lfs is needed to pull LFS-tracked files on git clone; git lfs install --system must run before the clone, not after

8. vuer_cv ENV_VERSION must match config/docker.json requiredEnvVersion

Currently 8. Pass via --build-arg VUERCV_ENV_VERSION=8. If config/docker.json bumps requiredEnvVersion, the build-arg default in the Dockerfile and any wrapper scripts need to bump in lockstep, or runtime startup fails the env-version check.

9. podman machine on macOS Apple Silicon needs 8 GiB RAM minimum

For qemu-emulated linux/amd64 builds. 4 GiB OOMs during the gcc-c++ family installs. 6 CPUs was enough. Other builders on this hardware should be sized the same.

10. Subagent Bash allowlist is more restrictive than main-session shell

Even with settings.local.json adding permissions, subagents may still hit blocks (this is why the prior session believed podman build was sandbox-denied for FKITDEV-8252 work). Critical implication: any build/edit step a subagent can’t do must fall back to the main session, or the subagent’s permissions have to be expanded before re-dispatching. Plan FKITDEV-8252 Phase B work with this in mind.

Files touched / committed 2026-05-18

File	Change	Verified?
base/portal_css/Dockerfile	LEGACY crypto policy + ordering fix + key imports	probe-build green (iter 3, 658 MB)
base/vuer_css/Dockerfile	LEGACY crypto policy + ordering fix + key imports	probe-build green (iter 3, 805 MB)
base/vuer_oss/Dockerfile	LEGACY crypto policy + ordering fix + key imports	probe-build green (iter 5, 2.44 GB)
base/janus/Dockerfile (both stages)	LEGACY crypto policy + ordering fix + gzip + GitHub archive path fix + opus/opus-devel rename	probe-build green (iter 7, 313 MB)
base/vuer_cv/Dockerfile (new)	UBI10 base for vuer_cv with EPEL10, git-lfs system install, ENV_VERSION=8	probe-build green (iter 7, 5.93 GB)

All five rolled into commits f38c8e2 and b984006 on feature/FKITDEV-8252-ubi10.

What’s open after A.6.1

Pickup list for the next session

Five items below. Phase A.6.2 identification is the next concrete chunk of work.

1. Phase A.6.2 — identify the “remaining 3 base/ Dockerfiles”* The prior plan referenced 3 more base Dockerfiles to migrate. vuer_build/base/ itself now contains 5 dirs (the 4 historical + new vuer_cv); the remaining 3 are almost certainly under common/* — best candidates: clamav-base, postgresql-base, rabbitmq-base, turn-base. Do not guess — ls vuer_build/common/ to confirm, identify the right 3, run probe-builds in the same iteration loop as A.6.1.

2. Push the branch (17 commits ahead, not pushed). Pending Bence’s signal that he is ready to review, or until Phase A.6.2 lands so the push is a logical unit.

3. Bence’s response on the Oracle OL10 memo — /Users/levander/coding/facekom/FKITDEV-8252-oracle-ol10-memo.md still awaiting his confirmation on Option 1 (OL9 instantclient RPMs on UBI10 base for kh / bb partner Dockerfiles) and escalation to the partner-contract owners.

4. Phase B — vuer-release 62 Dockerfiles. Will benefit from the LEGACY pattern + ordering fix + the 10 lessons captured above. Not started.

5. Phase C — vuer_docker / Bence’s PR #203 reconcile. PR #203 currently conflicting. Not started.

Quality concerns flagged (for follow-up tickets, not blockers)

These didn't block A.6.1 but deserve their own follow-up

Drop into a separate Jira / a follow-up note rather than letting them silently rot.

• | debug_log pipe masks microdnf install failures. We saw libmicrohttpd silently miss in iter 6 and only caught it via a cascading groupadd error two layers later. The pipe is swallowing non-zero exit. Worth either dropping the pipe, using set -o pipefail, or routing debug_log differently so failures surface at the layer that caused them.
• vuer_cv at 5.93 GB is heavy. Multi-stage split (drop build-time deps: git-lfs, gcc-c++, python3-devel) could roughly halve it. Out of scope for A.6.1 (functional first), strong candidate for a follow-up size-reduction ticket.
• microdnf -y remove --allowerasing git git-lfs openssh-clients cleanup in vuer_cv may cascade through git-core dependencies unintentionally. Image is the final stage so runtime behavior is fine, but worth a sanity audit to confirm we’re not removing more than intended.

Decisions made 2026-05-18

Q1 — Crypto policy: revised from per-key rpmkeys → LEGACY

Original decision did not survive contact with UBI10 reality

Captured here as a historical record so the next person doesn’t try per-key again and burn the same iterations we did.

Original decision (morning): switch from update-crypto-policies --set LEGACY to per-key rpmkeys --import --allow-sha1-signatures to narrow the surface area of SHA1 acceptance.

Why it didn’t work (afternoon discoveries — see Key lessons learned (Phase A.6.1) §1-3):

• The --allow-sha1-signatures flag disappears after microdnf -y update (newer rpm-libs strips it).
• The additive sub-policy DEFAULT:SHA1 mechanism doesn’t exist on UBI10 (no SHA1.pmod).
• The narrower alternatives don’t exist in practice.

Current decision: use update-crypto-policies --set LEGACY in all build stages that need to import SHA1-signed GPG keys. Build-time scope only; runtime crypto policy of the final image is unaffected unless an intermediate stage carries it forward. Applied in 7 stages across portal_css, vuer_css, vuer_oss, janus (×2), vuer_cv.

Affected keys (signers needing SHA1 acceptance): EPEL-10, RPM-Fusion, CentOS Stream 10 (centosofficial), Oracle Linux 10 (ol-10), nginx (nginxofficial).

Q2 — Oracle Instant Client OL10 unavailable: stay on OL9 RPMs (unchanged)

Decision: use OL9 instantclient RPMs on UBI10 base for kh and bb partner Dockerfiles (Option 1, pragmatic ship-it path).

Rationale: verified 2026-05-18 that https://yum.oracle.com/repo/OracleLinux/OL10/ returns 404 across oracle/instantclient/x86_64/, the root OL10 path, and the alternate OracleLinux10/ naming. Oracle Linux 10 itself is GA per Oracle’s release notes, but the public yum mirrors at yum.oracle.com have not been populated for OL10 as of today. OL9 binaries are forward-compatible with UBI10’s glibc 2.39 (OL9 was built against glibc 2.34); swap to .el10 is a trivial single-URL change once Oracle ships them.

Memo for Bence: /Users/levander/coding/facekom/FKITDEV-8252-oracle-ol10-memo.md — needs Bence to confirm Option 1 and escalate to whoever owns the kh / bb partner contracts so they are aware their integration containers will be running OL9-vintage binaries on UBI10 until Oracle catches up.

Alternatives considered, not chosen:

• Option 2 — keep kh/bb on UBI9 (fleet fragmentation)
• Option 3 — build/extract from source (Oracle licensing prohibits; fragile)
• Option 4 — switch to thin-mode oracledb Python driver (only viable if container code is Python; would need ~1h grep to confirm)

Q3 — vuer_cv scope: in-scope (resolved)

Decision: vuer_cv is in scope for FKITDEV-8252. A new UBI10 base Dockerfile was authored and probe-builds green at 5.93 GB (iter 7). The 2-hour spike originally planned to decide in-scope vs split is resolved by execution.

Caveats:

• Image size is heavy (5.93 GB) — flagged as a follow-up size-reduction concern, not a blocker.
• Build-time deps (git-lfs, gcc-c++, python3-devel) are removed in final layer via microdnf remove --allowerasing — that removal cascade is worth a sanity audit (see [[#quality-concerns-flagged-for-follow-up-tickets-not-blockers|Quality concerns flagged (for follow-up tickets, not blockers)]]).
• ENV_VERSION must track config/docker.json requiredEnvVersion (currently 8) — see lesson §8.

Pre-existing migration context (still relevant)

Carry-forward from 2026-05-13 to 2026-05-15 prior agent

These are issues / fixes already discovered and applied in earlier commits on the branch. Listed so the next session does not re-discover them.

Already-fixed in committed history

Commit	Issue	Fix
dc28071	UBI10 microdnf is dnf5-based and hangs on prompts if -y missing	Add -y to every microdnf invocation
(in current diff)	CS10 package renames	pcre-devel → pcre2-devel; zlib-devel → zlib-ng-compat-devel
474d9af	redis package removed from UBI10 streams	switched to valkey with compat symlinks for tooling expecting redis-server/redis-cli
56d4f5e	coturn .el8 pin no longer satisfiable	dropped .el8 pin, took plain EPEL10 4.10.0-1.el10_3
01eca7c	rabbitmq packagecloud /el/10/ is empty	fallback to /el/9/ with 3.13.7 .el8.noarch
4fac31a	groupadd / useradd missing in UBI10 minimal	install shadow-utils explicitly
4cc85d7	Hardcoded x86_64 in OL10 repo defs	replaced with $basearch for multi-arch readiness
f38c8e2 (today)	Existing 4 base Dockerfiles not probe-buildable on UBI10	LEGACY crypto policy + crypto-policies-scripts ordering + key imports + multi-stage fixes for janus (gzip, opus rename, archive path)
b984006 (today)	vuer_cv had no UBI10 base	new base/vuer_cv/Dockerfile with EPEL10 + git-lfs system install + ENV_VERSION=8

Additional package renames to verify in Phase B

These were on the radar at session start and none have been hit in A.6.1 — they’re flagged for Phase B (vuer-release) probe-builds:

• ffmpeg-devel
• libogg-devel
• libconfig-devel
• gtk-doc
• jansson-devel
• pkgconf
• gengetopt
• libnice (hit in A.6.1 via janus — built from source libnice-0.1.17.tar.gz, no rename needed at package level)
• libsrtp2

Other carry-forward gotchas

• NVIDIA cuda-rhel10.repo availability — was an open question at session start; not hit in A.6.1 because the new base/vuer_cv Dockerfile didn’t need CUDA-from-NVIDIA-repo at base layer. Will re-surface in Phase B if any vuer-release image pulls CUDA directly.
• PR #203 (Bence’s WIP on vuer_docker) is currently CONFLICTING — will be reconciled in Phase C after Phase A/B settle.

Cross-references

• Memo for Bence (Oracle OL10): /Users/levander/coding/facekom/FKITDEV-8252-oracle-ol10-memo.md
• Ticket source: /Users/levander/Downloads/FKITDEV-8252-Ubi10.md
• Auto-memory entry: [FKITDEV-8252 UBI10](/Users/levander/coding/facekom/FKITDEV-8252-oracle-ol10-memo.md) — “Oracle OL10 missing; per-key SHA1; vuer_cv in-scope (spike 8/12)” — note the per-key SHA1 reference is superseded by Q1 revision; vuer_cv in-scope is confirmed by A.6.1 execution
• Bence’s vuer_docker PR: PR #203 — currently conflicting
• Branch: feature/FKITDEV-8252-ubi10 in /Users/levander/coding/facekom/vuer_build, 17 commits ahead of origin/main, not pushed

Related

• FaceKom
• vuer_docker
• vuer_oss
• vuer_css
• vuer_cv
• portal_css
• infrastructure
• release-process