nixosTests.acme is very flaky

[vcunat]

This is a longer-term issue. I'm annoyed from having to restart the test all the time to unblock channels.

Examples (unstable, 24.11, both Linuxes):
https://hydra.nixos.org/build/285602524#tabs-buildsteps
https://hydra.nixos.org/build/285602583#tabs-buildsteps

.meta.maintainers: @aanderse, @arianvp, @emilazy, @flokli, @m1cr0man

[m1cr0man]

Current plan to tackle this:

• Review history and ongoing build failures for patterns (it tends to be something different each time)
• Investigate the dependency chains in acme systemd components for potential race conditions during activation
  • One thing I only recently learned about config activation is that systemd jobs start one service at a time rather than a bunch together. This challenges some assumptions I had about service startup ordering. I need to review each service used in an acme renewal to ensure it depends on the right services in the right order.
• Restructure the test suite by breaking it into many smaller integration tests for particular feature sets
• Get nixos/acme: Refactor setup process #355087 merged to reduce the number of services and thus the dependency chain
• Consider adding a target unit for correct ordering of the web server required for HTTP-01 renewal. There are some vague assumptions made currently and this could be causing issues.

I'm open to any other suggestions too. If anyone is eager to help, the first and second tasks would be great to get eyes on.

cc @NixOS/acme