Embedded CI/CD & DevOps, Built for Regulated Hardware

You don't need another generic "DevOps consultancy." You need a build and release pipeline that survives a decade of scale, three timezones of developers, and an auditor asking hard questions about traceability. That's what we build.

Executive Summary

For over a decade, we've designed and operated the CI/CD and release infrastructure behind a large device manufacturer's embedded product line — a self-hosted system handling 100+ developers across 3 timezones, tens to hundreds of Yocto builds a day with 15-minute feedback, and 5 parallel release branches spanning development through in-market production. Every stage is gated by automated dependency analysis, static analysis, and external system testing; every release is signed through an HSM-backed, on-demand signing process; and the entire operation runs on-premises to meet FDA, IEC 62304, and data governance requirements, at 99% uptime with no major incidents. It's proof that regulatory rigor and engineering speed don't have to be a trade-off.

> Self-hosted by design. Every core service — build orchestration, source control integration, artifact storage — runs on infrastructure inside your own lab network, not a third-party cloud. It still integrates cleanly with external, cloud-hosted tools where that makes sense, but nothing sensitive ever has to leave your governance boundary to get there.

Contents

• Proven at Scale, Not Just in Theory
• Deep Integration, Not Just Off-the-Shelf Tooling
• Architecture Philosophy
• Yocto at Real-World Scale
• Smarter Gating, Not Just More Gating
• Continuous Build Cadence & System Testing
• Managing the Full Release Lifecycle
• Built for Regulated Environments
• Production Firmware Signing
• Beyond the Build: Full Lifecycle Coverage
• A Complete Evidence Package, Every Release
• Why This Matters to You

Proven at Scale, Not Just in Theory

• 10 years of continuous production use. The Open Build Service deployment we designed and operate has been in active, uninterrupted service for a decade — not a pilot, not a proof of concept, a system your teams depend on every day.
• 100+ active developers, 3 timezones. Built to handle real-world concurrency and follow-the-sun development, not a single-team demo.

This isn't a greenfield toy setup. It's infrastructure that has already absorbed a decade of organizational change, tooling churn, and scale — and kept working. 99% uptime, with no major incidents, across that entire history.

Deep Integration, Not Just Off-the-Shelf Tooling

• Custom SCM connectors. Purpose-built integration first with Gerrit, then migrated to GitLab — connecting your source review workflow directly into the build system rather than bolting on generic webhooks.
• Multi-architecture build support. Bare-metal ARM, embedded Linux on ARM, virtual machine targets, and Windows builds driven through WSL — all handled from the same CI backbone, and all using the same declarative build spec format, not a separate one-off pipeline per platform.
• Multiple compiler toolchains, integrated and orchestrated across CI stages — not a single hardcoded toolchain assumption.
• Artifactory integration. Build outputs flow into Artifactory for versioned, traceable artifact storage — no ad-hoc file shares or "which build is this again" guesswork.
• Hybrid connectivity. Local-first, self-hosted infrastructure that still integrates cleanly with external, cloud-hosted tools and services where needed — you get the reach of cloud tooling without moving sensitive builds or source off-premises.

Architecture Philosophy

The system isn't just a collection of integrations bolted together — it's built on a small set of deliberate architectural principles that keep it maintainable at scale:

• Git-first build specs. Build specifications live in git, alongside the code they build — not in some separate configuration system or a UI-driven pipeline editor. Every build definition is versioned, reviewed, and branched the same way your source code is, so what built a given release is always reconstructable from history, not tribal knowledge.
• Verified reproducible builds. Every release ships with a container image that is verified to rebuild byte-identical binaries from the same source. This isn't "should be reproducible in theory" — it's actively checked, so you have a concrete, verifiable answer to "can we prove this exact binary came from this exact source" years after the fact, not just a promise.
• The CI environment, in developers' hands. The same build environment that runs in CI is packaged as a cross-platform SDK — a Docker container on Linux and macOS, and a WSL distro on Windows — so developers can reproduce a CI build or failure locally instead of guessing at environment drift. Reproducibility isn't just a release-time property; it's available to every developer, every day.
• Yocto SDKs, properly supported. Cross-compilation SDKs for target architectures are generated and maintained as a first-class part of the offering. This sounds basic, but in practice many vendors either overlook Yocto SDK usage entirely or are outright reluctant to support it — leaving developers to fight cross-compilation setup on their own. It isn't offered here as a novelty; it's offered because too many teams don't get it at all.

All of this reflects the same underlying principle: durability and scale come from treating infrastructure like software — versioned, reviewable, and built to be understood years later by people who weren't in the room when it was designed.

Yocto at Real-World Scale

Embedded Linux builds run on Yocto, tuned for speed and volume:

• 4 target images across 3 machines, built for every incoming change — and results delivered in 15 minutes, not hours.
• Tens to hundreds of Yocto builds handled every day, without the pipeline buckling under load.

That kind of turnaround at that volume isn't accidental — it comes from build architecture (sstate cache strategy, dependency graph tuning, parallelization) purpose-built for Yocto's quirks at scale, not a generic CI template pointed at a bitbake command.

Smarter Gating, Not Just More Gating

• Automatic reverse-dependency gating, built on Open Build Service's native handling of BuildRequires declared in the build spec — when a change lands, everything that declares a dependency on it is automatically identified and rebuilt/retested, not just the component that changed. This is a capability generic pipeline tools (GitLab CI, GitHub Actions) don't provide out of the box; getting equivalent behavior there means heavily custom, hand-maintained pipeline logic.
• Cross-project gating across repositories, coordinated through a topic model — Gerrit-native, and reimplemented in the custom backend for GitLab, so the same cross-repo, single-logical-change gating works regardless of which SCM a given team is on.
• Unit testing and coverage reporting wired into the pipeline as a first-class citizen, not an afterthought bolted on post-release.
• Coverity static analysis, scoped and gated at merge time — not just run as a full-branch audit. Most Coverity deployments run full scans against a release branch as a late-stage audit checkpoint, which means defects surface right when the code is supposed to be shipping, not being refactored — turning "clean up findings" into a scramble against the release date. This pipeline instead runs partial, incremental scans against each incoming change and rejects any newly introduced defects before merge. Static analysis becomes part of the definition of done, not a bill that comes due at release time.

Continuous Build Cadence & System Testing

Validation isn't a once-a-day checkbox — it runs continuously, at every level:

• Rolling master build, automatically rebuilding every affected component after every batch of merged commits — not just on a timer.
• Nightly builds and snapshots, giving a consistent, dependable baseline on top of the rolling cadence.
• External system testing infrastructure triggered at every stage — for every individual change, for the rolling master build, and for nightly snapshots alike. Nothing ships or merges on unit tests and static analysis alone; it's exercised at the system level too.

Managing the Full Release Lifecycle

Device manufacturers don't run one branch — they run several, at different stages of maturity, simultaneously. The process is built for exactly that: 5 active branches currently maintained in parallel, spanning development, verification, and in-market/production stages, each following the same rigorous gating, build, and test process appropriate to its place in the lifecycle.

Built for Regulated Environments

CI and release workflows designed around FDA and IEC 62304 (medical device software lifecycle) requirements from day one — because for device manufacturers, "it built successfully" isn't the bar. Traceability, auditability, and repeatability are the bar, and the pipeline is designed to clear it. All of it running on fully self-hosted, on-premises infrastructure — no third-party cloud CI, no external SaaS build runners — built to satisfy strict data governance requirements rather than work around them.

Production Firmware Signing

Release integrity doesn't stop at "the build succeeded." The pipeline handles production signing of firmware images as an integrated release step — not a manual, out-of-band process run by whoever remembers to do it.

• HSM-backed secret storage. Signing secrets live in a hardware security module. Build machines never have access to the private key material — there's no path from a compromised build job to a compromised signing key.
• On-demand signing via DigiCert, invoked only within an explicitly opened signing time window — signing capability isn't standing open, it's deliberately gated.

The result: consistent, auditable release provenance, with key handling designed around "the build system should never be able to sign anything on its own" rather than trusting process alone.

Beyond the Build: Full Lifecycle Coverage

A pipeline that only gets code to a release tag isn't finished — devices ship, run in the field for years, and need to be updated, audited, and supported long after the initial release. The offering covers that full lifecycle, not just the build:

• OTA update artifact generation. Update packages are produced as an integrated part of the release pipeline, not a separate manual process bolted on afterward — so what ships to devices in the field traces back to the same build and gating process as everything else.
• Open-source license compliance scanning. Every release is scanned for FOSS license obligations across the full layer set, catching copyleft and licensing risk before it ships rather than discovering it after a product is already in the field.
• Hardware-in-the-loop (HIL) testing, through a custom-built solution integrated into the gating pipeline — validating changes against real target hardware, not just emulation or virtual machines, before they merge.
• Disaster recovery and backup strategy for the self-hosted infrastructure itself — because a build system that's been trusted with 10 years of continuous operation earns that trust partly by having a real answer to "what happens if something fails," not just an uptime number. The entire infrastructure is deployed and maintained through Ansible, so recovery isn't a manual, tribal-knowledge process — it's rebuildable from version-controlled configuration, the same way the build specs themselves are.
• Engineering and release metrics dashboards, giving both engineering and business stakeholders visibility into build health, release cadence, and pipeline performance over time — not just a black box that either works or doesn't.
• CVE monitoring, ready to integrate. Dependency and component vulnerability scanning against shipped software is currently handled by a separate internal team, but it's a natural, low-friction addition to wire directly into this pipeline as an additional gating or reporting stage.

A Complete Evidence Package, Every Release

When an auditor asks "prove it," the answer shouldn't be a scramble to reconstruct what happened months ago. Every release automatically bundles the full evidence trail as a build artifact:

• Release metadata
• SOUP (Software of Unknown Provenance) lists — the IEC 62304 term for tracking third-party and off-the-shelf software components in a medical device
• Complete build logs
• Unit test run results
• Coverage reports
• Static analysis reports
• Accompanying documentation

This isn't paperwork assembled after the fact for an audit — it's generated as a direct output of the pipeline itself, tied to the exact build that produced it. Combined with verified reproducible builds and HSM-backed signing, every release comes with both the artifact and the proof of how it was built, tested, and secured — in one place, every time.

One SCMP, standardized across the full product range. Rather than maintaining a patchwork of per-product Software Configuration Management Plans, the process has been standardized to the point where a single SCMP covers the entire range of products the team handles. That's a level of process maturity that's hard to reach — most organizations end up with product-specific variations that drift apart over time — and it means consistent, predictable compliance behavior regardless of which product line a given release comes from.

Why This Matters to You

Interested in what this could look like for your build infrastructure? Let's talk about your current pipeline and where the gaps are.