I have around 72 different apps, and two environments, staging and production. Generally, the idea is to deploy to staging, test it out, and then promote to production.

In practice, there was no tooling that made this mandatory, so it was often skipped. Over time, environments drifted, and staging was soon forgotten as a testing ground.

I set about to change this by creating a connector that would act as a bridge between the two meshes, allowing production to ask staging to build, test, and then ship any component.

One of the main challenges with this was that checking if everything is identically configured on both environments meant needing to know exactly what should have been, which led me to first develop a way to deploy new applications using this method. After all, if it knows how to deploy a new app successfully, why wouldn't it have that capability?

Once deployed, updates can be instigated from production by clicking "Pull from staging" which results in a fully auditable log as the following events take place:

proc-mesh is a special component which runs unprivileged, with no network, and one unix socket. I decided not to write any programs which continuously run as root, instead using doas for carefully-audited privilege escalation when needed, invoked by proc-mesh. The other components have NoNewPrivileges=yes, so even if they wanted to use doas, they would be unable. It was necessary to use a layered approach to security where gaining privileges loses the ability to access the network.

```
permit nopass xfproc as root cmd /usr/local/sbin/xf-restart-plug
permit nopass xfproc as root cmd /usr/local/sbin/xf-trustbase-verify

Install a staged artifact: . The bridge owns the

allowlist and derives both the artifact path and the destination itself.

permit nopass xfproc as root cmd /usr/local/sbin/xf-install-binary
```
Shown in the log:

And fully auditable at every stage:

Now that it knows how to check the setup of existing apps and create new apps, it becomes easy to perform bulk tasks like bringing the environments back in sync. Each app is staged first, tested, then shipped into production.

What's next?

You may have spotted one issue: plug-mesh cannot deploy itself, and still uses my old Gitea runner deployment path. My original fix for its own deployment was to connect a new plug-mesh to the isolated mesh-to-mesh NATS bridge (which is different from the main bridge, which connects all components within one mesh) and perform the tests on it there using the existing plug-mesh component, before having the new one kill the old one and seamlessly transition to the main NATS network. In theory, this would work, but in practice I found that I was not often amending this component, so did not feel the need to enable it to deploy itself.

If I do implement this, it will likely be a side effect of a smarter deployment path with more rigorous testing which stages a new component without stopping the old one, requiring an isolated NATS. Currently, we accept that staging can break, and this is the whole point. If staging an application fails, well, at least it wasn't on production!