Use zigbuild to unify the build process of binary files and docker #1093

vickunwu · 2025-01-15T16:11:27Z

This PR does the following things:

Split the build matrix from 1 to 3 [linux,macos,windows]
Create a multi-stage Dockerfile for build binary files as well as the publishing docker image
Improve the cache mechanism with sccache and docker build cache (Speed up the build process)
Add 3 comment out targets [arm-unknown-linux-gnueabihf,armv7-unknown-linux-musleabihf,arm-unknown-linux-musleabihf] (Not tested, but successfully compiled)
Update foundationdb lib to latest stable version and add an aarch64-apple-darwin variant
Add an alpine flavor docker image for musl binary files (tagged with a suffix of -alpine)
Reduce the size of generated binary files (Guess it's because the use of zig)
Update all actions to the latest stable version

CLAassistant · 2025-01-15T16:11:35Z

All committers have signed the CLA.

mdecimus · 2025-01-15T18:01:05Z

I haven't tested it but it looks great, thanks @vickunwu ! If it's not too much of an ask, would be willing to add SLSA provenance to the builds?

@33KK since you created the current build.yml I'm tagging you in case you have time to review this PR. Thanks.

vickunwu · 2025-01-16T15:41:53Z

@mdecimus While I was working on the SLSA provenance, I just found that when specifying a single $TARGETPLATFORM in docker bake file, it will preserve only one arch under the same tag in Container Registry. Clearly this PR needs some refinements, I am converting this into a draft.

Considering about the workaround, do you think it's ok that we just copy the artifacts from previous build jobs to the runtime container when building the image in a seperate job? If so, should we add attestations for the artifacts, or just generate provenances for the compressed release files?

vickunwu · 2025-01-17T15:01:08Z

@mdecimus While I was working on the SLSA provenance, I just found that when specifying a single $TARGETPLATFORM in docker bake file, it will preserve only one arch under the same tag in Container Registry. Clearly this PR needs some refinements, I am converting this into a draft.

Considering about the workaround, do you think it's ok that we just copy the artifacts from previous build jobs to the runtime container when building the image in a seperate job? If so, should we add attestations for the artifacts, or just generate provenances for the compressed release files?

Never mind. I find a way to build multi-arch image across different runner.
Working on refactoring the CI process.

mdecimus · 2025-01-17T18:08:49Z

Never mind. I find a way to build multi-arch image across different runner.

Great, thanks! Please let me know once it is ready and I'll test it out.

Just two comments and one question:

We are currently specifying the feature list when building the mail server with the default databases, this is to avoid accidental changes in Cargo.toml that break the build process. The current CI job is doing it as follows:

 build -p mail-server --no-default-features --features "sqlite postgres mysql rocks elastic s3 redis azure enterprise"
 build -p stalwart-cli

Yesterday Github finally released arm64 runners as a public preview. Not sure if they are mature enough to be used by the CI job yet though.
Many people would like to see FreeBSD support. Are you aware of any CI jobs that will allow us to build FreeBSD targets?

Thanks again.

vickunwu · 2025-01-18T14:40:47Z

We are currently specifying the feature list when building the mail server with the default databases, this is to avoid accidental changes in Cargo.toml that break the build process.

Changes ready for review now.

Yesterday Github finally released arm64 runners as a public preview. Not sure if they are mature enough to be used by the CI job yet though.

As for arm64 runner, right now it seems unnecessary to delegate the corresponding tasks with it. I believe we can add another attribute(e.g. host_os) to the linux build matrix once foundationdb releases a stable version of arm64 debian pkg.

Many people would like to see FreeBSD support. Are you aware of any CI jobs that will allow us to build FreeBSD targets?

Sorry, I don't have much clue about it.

33KK

Well, this isn't using zigbuild to unify anything, but docker-bake, which I have zero clue about. And replaces cross with cargo-zigbuild, which is a good idea if it works properly. Don't really like the amount of complexity and separate tools and build systems here, but I guess there's not that much you can do about it.

It's crazy how unnecessarily complex and fragile are CI and build systems, and all the container related tools are. (warning, rant)

Yaml became the programming language for all the CI systems, which has zero static analysis, and really sucks to write since it's not an actual programming language but a JSON clone bodged to function as a """programming language""". It pretty much requires pushing your changes to a repo a hundred times and waiting for the CI to fail a hundred to see if it works or not. It pretty much requires you to use shell scripts or some other scripting language to get anything done, which also means you have to deal with two different similar variable syntaxes ${VAR} for shells and ${{expr}} for github actions.

Docker only ever "fixed" the dependency hell from the POV of whoever has to deploy the built containers, it's still a disaster in terms of actually building them, it's still a mix of package managers, random file downloads, awkward copying of files from other images or using base images. Plus the build tools around it is a disaster.

Also everything relies on POSIX compatible shells which, let's be real, are not really portable, extremely fragile, error prone and painful to get anything done with, or just simply terrible.

All of this just doesn't integrate well together and is extremely fragile and very hard to debug. Also the amount of parties to trust sure doesn't give me much hope that any of this is at all resistant to supply chain attacks. The "attest" thing barely gives any reassurance considering that all it ensures is that the image was built on github actions (if it actually even does that and you can't just fake it with some amount of effort and using a ssh server action) but does nothing about all the external tools and dependencies.

33KK · 2025-01-19T00:29:22Z

.github/workflows/ci.yml

+  SCCACHE_GHA_ENABLED: true
+  RUSTC_WRAPPER: sccache


Probably shouldn't be enabled on releases

Can you please be more specific about the reason?

More or less a "just in case" thing, I imagine there's a chance it could introduce miscompilations, release builds aren't terribly frequent and don't take that much longer anyway

33KK · 2025-01-19T00:31:53Z

.github/workflows/ci.yml

+      - name: Inject cache into docker
+        uses: reproducible-containers/buildkit-cache-dance@v3.1.2
+        with:
+          cache-map: |
+            {
+              "var-cache-apt": "/var/cache/apt",
+              "var-lib-apt": "/var/lib/apt",
+              "usr-local-cargo-registry": "/usr/local/cargo/registry",
+              "usr-local-cargo-git": "/usr/local/cargo/git"
+            }
+          skip-extraction: ${{ steps.cargo-cache.outputs.cache-hit }} && ${{ steps.apt-cache.outputs.cache-hit }}


Not sure how exactly reproducibility goes together with injecting cache only ever accessible from the CI

Haha is that a judgement from the action name?

Yeah, not really about the PR I guess, it's just kinda funny to me how that github org hosts an action that very much seems to do the opposite of reproducibility

33KK · 2025-01-19T00:38:37Z

.github/workflows/ci.yml

+            type=ref,event=tag
+            type=ref,event=branch,prefix=branch-
+            type=edge,branch=main
+            type=semver,pattern=v{{major}}.{{minor}}


Not sure how exactly this works, but it should end up creating tags like latest and vMAJOR.MINOR.PATCH

you can think the 2nd and 3rd rule are for manual build.

I mean to me it seems like it creates "edge" instead of "latest" but idk, docker build tools are really confusing

33KK · 2025-01-19T00:53:08Z

Dockerfile.build

+RUN ZIG_VERSION=$(curl --retry 5 -sL "https://api.github.com/repos/ziglang/zig/releases/latest" | jq -r '.tag_name') && \
+    [ ! -z "$ZIG_VERSION" ] && \
+    curl --retry 5 -Ls "https://ziglang.org/download/${ZIG_VERSION}/zig-linux-$(uname -m)-${ZIG_VERSION}.tar.xz" | tar -J -x -C /usr/local && \
+    ln -s "/usr/local/zig-linux-$(uname -m)-${ZIG_VERSION}/zig" /usr/local/bin/zig
+# Install cargo-binstall
+RUN curl --retry 5 -L --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/cargo-bins/cargo-binstall/main/install-from-binstall-release.sh | bash
+# Install FoundationDB
+# TODO According to https://github.com/apple/foundationdb/issues/11448#issuecomment-2417766293
+# Once FoundationDB v7.3.53 gets released, we should be able to build the aarch64-unknown-linux-gnu target.
+# The last command is for future build use, so if you are building on a native arm64 device, please use docker qemu.
+RUN curl --retry 5 -Lso /usr/lib/libfdb_c.so "$(curl --retry 5 -Ls 'https://api.github.com/repos/apple/foundationdb/releases' | jq --arg arch "$(uname -m)" -r '.[] | select(.prerelease == false) | .assets[] | select(.name | test("libfdb_c." + $arch + ".so")) | .browser_download_url' | head -n1)"
+# Install cargo-chef & sccache & cargo-zigbuild
+RUN cargo binstall --no-confirm cargo-chef sccache cargo-zigbuild


Don't particularly like all the curling action, but I guess that's just a Docker moment. The alternative is to use cargo-zigbuild image as the base and copy the cargo-chef binary from it's image, and sccache from apt 💀

cargo zigbuild image imports unnecessary macos sdk and all supported rust targets. most importantly, it may not track the latest stable rust version.

That makes sense, I guess there's not much you can do here

33KK · 2025-01-19T00:59:40Z

Dockerfile.build

+    --mount=type=secret,id=ACTIONS_CACHE_URL,env=ACTIONS_CACHE_URL \
+    --mount=type=secret,id=ACTIONS_RUNTIME_TOKEN,env=ACTIONS_RUNTIME_TOKEN \


No idea what is this about, don't see anything that [I'd expect to] use it.

This is the required env vars for sccache to use Github actio cache.

Makes sense

33KK · 2025-01-19T01:01:03Z

Dockerfile.build

+# *****************
+# Binary stage
+# *****************
+FROM scratch AS binaries
+COPY --from=builder /app/artifact /


Why not just COPY from builder in the images below?

Because when the devs want to get the binaries without building a whole image, the docker build stops here.

Is this how the docker-bake thing ends up generating binary artifacts? I guess that makes sense

vickunwu · 2025-01-19T17:13:29Z

It pretty much requires pushing your changes to a repo a hundred times and waiting for the CI to fail a hundred to see if it works or not.

Couldn't agree more.

Docker only ever "fixed" the dependency hell from the POV of whoever has to deploy the built containers, it's still a disaster in terms of actually building them,

I think there's still some upside about it. Personall I use it to build stuff because it sort of prevents the cache "pollution".

vickunwu force-pushed the main branch 14 times, most recently from d8c9e20 to 342bbc0 Compare January 16, 2025 13:27

vickunwu marked this pull request as draft January 16, 2025 15:42

vickunwu force-pushed the main branch 10 times, most recently from f05376b to 552fa81 Compare January 17, 2025 14:07

vickunwu force-pushed the main branch 2 times, most recently from ea64780 to 5421dad Compare January 18, 2025 13:26

CI refactor & Add SLSA provenance for release and docker image

37c25d6

vickunwu force-pushed the main branch from 5421dad to 37c25d6 Compare January 18, 2025 13:32

vickunwu marked this pull request as ready for review January 18, 2025 14:26

33KK reviewed Jan 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use zigbuild to unify the build process of binary files and docker #1093

Use zigbuild to unify the build process of binary files and docker #1093

vickunwu commented Jan 15, 2025

CLAassistant commented Jan 15, 2025 •

edited

Loading

mdecimus commented Jan 15, 2025

vickunwu commented Jan 16, 2025

vickunwu commented Jan 17, 2025

mdecimus commented Jan 17, 2025

vickunwu commented Jan 18, 2025

33KK left a comment

33KK Jan 19, 2025

vickunwu Jan 19, 2025

33KK Jan 20, 2025 •

edited

Loading

33KK Jan 19, 2025

vickunwu Jan 19, 2025

33KK Jan 20, 2025 •

edited

Loading

33KK Jan 19, 2025

vickunwu Jan 19, 2025

33KK Jan 20, 2025

33KK Jan 19, 2025

vickunwu Jan 19, 2025

33KK Jan 20, 2025

33KK Jan 19, 2025

vickunwu Jan 19, 2025

33KK Jan 20, 2025

33KK Jan 19, 2025

vickunwu Jan 19, 2025

33KK Jan 20, 2025

vickunwu commented Jan 19, 2025

		--mount=type=secret,id=ACTIONS_CACHE_URL,env=ACTIONS_CACHE_URL \
		--mount=type=secret,id=ACTIONS_RUNTIME_TOKEN,env=ACTIONS_RUNTIME_TOKEN \

Use zigbuild to unify the build process of binary files and docker #1093

Are you sure you want to change the base?

Use zigbuild to unify the build process of binary files and docker #1093

Conversation

vickunwu commented Jan 15, 2025

CLAassistant commented Jan 15, 2025 • edited Loading

mdecimus commented Jan 15, 2025

vickunwu commented Jan 16, 2025

vickunwu commented Jan 17, 2025

mdecimus commented Jan 17, 2025

vickunwu commented Jan 18, 2025

33KK left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

33KK Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

33KK Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vickunwu commented Jan 19, 2025

CLAassistant commented Jan 15, 2025 •

edited

Loading

33KK Jan 20, 2025 •

edited

Loading

33KK Jan 20, 2025 •

edited

Loading