05-Specifications / 05.04.Speed-Up-CICD.spec

05.04.Speed Up CICD.spec

05.04. Speed Up CICD.spec

This document specifies the requirements for improving the CI/CD performance for both the API and WUI projects by implementing parallel execution, caching, and workflow restructuring.

1. Problem Statement

The current CI/CD workflows (ci.yaml in both API and WUI) are highly sequential and redundant: * Redundant Cloning: Every job re-clones all repositories in the monorepo structure. * Redundant Building: Deployment jobs often re-build Docker images that were already built and tested in previous steps. * Sequential Testing: Tests run one after another within a single large job. * Heavyweight Setup: Starting the entire Docker compose environment for simple unit tests or type checks adds significant overhead.

2. Proposed Optimization Strategy

2.1 Parallel Execution (Matrix & Independent Jobs)

We will split the monolithic build-and-test jobs into discrete, parallelizable jobs:

• API Pipeline:
  • lint-and-typecheck: Parallel job for ruff and mypy.
  • unit-tests: Parallel job for fast unit tests (no Docker needed if possible, or use a lightweight container).
  • integration-tests: Parallel job for tests requiring Redis/Tesla Mock.
• WUI Pipeline:
  • typecheck: Parallel job for vue-tsc.
  • lint: Parallel job for eslint.
  • unit-tests: Parallel job for Vitest/Jest.
  • e2e-tests: Parallel job for Puppeteer (using the existing language matrix).

2.2 Build Once, Reuse Everywhere

• Artifact Registry: The build job should produce a versioned Docker image (tagged with sha-${{ github.sha }}) and push it to the GCP Artifact Registry early in the pipeline.
• Pull vs. Build: Subsequent test and deploy jobs should pull this pre-built image instead of running make do-setup-api-no-cache or similar commands.

2.3 Selective CI (Path Filtering)

Implement dorny/paths-filter to detect which sub-projects changed: * If only bnc-cpt-wui/ changed, skip the full API test suite. * If only bnc-cpt-api/ changed, skip the WUI E2E tests. * Always run full suite on merges to master.

2.4 Advanced Caching

• Docker Layers: Use type=gha cache backend for Docker builds.
• Package Managers:
  • Cache ~/.cache/pip and ~/.cache/pypoetry.
  • Cache ~/.npm and node_modules.
• Workspace Sharing: Use actions/upload-artifact to share the initial git clone/setup across jobs within a single run, reducing 3-5 minutes of cloning per job.

3. Implementation Plan

3.1 Step 1: Workflow Restructuring (utl doc/md/)

Update the utl project with helper scripts for CI: * do_ci_upload_workspace: Script to bundle and upload the cloned repos. * do_ci_download_workspace: Script to restore the workspace.

3.2 Step 2: API Workflow Update

1. Introduce a setup job that clones and uploads the workspace.
2. Introduce a build job that pushes the candidate image to Artifact Registry.
3. Split tests into parallel jobs that depend on build.
4. Update deploy to use the image from build.

3.3 Step 3: WUI Workflow Update

1. Apply similar setup and build separation.
2. Parallelize type-check, lint, and e2e-tests.
3. Optimize the E2E matrix so workers share a single pre-built WUI container.

4. Expected Outcomes

• Time Reduction: Aim for a 50% reduction in total "wall clock" time (e.g., from 20 minutes to < 10 minutes).
• Resource Efficiency: Reduced CPU minutes by avoiding redundant builds and clones.
• Developer Experience: Faster feedback loops for PRs.

5. Constraints

• Coverage: Must maintain 100% of current test coverage.
• Environment Consistency: Parallel workers must use identical Docker images to ensure "test-once-deploy-anywhere" integrity.
• Secrets: All parallel jobs must have access to necessary GCP/GitHub secrets.