05-Specifications / 05.04.v02.Speed-Up-CICD.spec

05.04.v02.Speed Up CICD.spec

05.04. v02.Speed Up CICD.spec

This document provides an improved iteration of the CI/CD optimization strategy, focusing on specific technical fixes identified during the analysis of the current ci.yaml files in bnc-cpt-api and bnc-cpt-wui.

1. Analysis Findings (Current Infficiencies)

Bottleneck	API Workflow	WUI Workflow
Redundant Cloning	Deploy job re-clones all repos.	Deploy job re-clones all repos.
Redundant Building	N/A (Builds once in monolithic job)	Deploy job runs make do-setup-wui-no-cache.
Docker I/O	N/A	Uses docker save/load (very slow artifact transfer).
Package Management	poetry install happens inside container.	npm install runs in every test job.
Caching	No Docker layer caching; uses -no-cache.	No Docker layer caching; uses -no-cache.
Path Filtering	None. Runs on every push.	None. Runs on every push.

2. Refined Strategy (Iteration v02)

2.1 The "Gold Image" Pattern (Registry-Centric)

Instead of docker save/load or re-building in deploy: 1. Job: build: Build the Docker image once using docker/build-push-action with cache-from: type=gha and cache-to: type=gha. 2. Push: Push to GCP Artifact Registry with two tags: sha-${{ github.sha }} and latest-branch-${{ github.ref_name }}. 3. Pull: All subsequent jobs (test-unit, test-ui, deploy) pull this specific SHA tag. This guarantees that exactly what was tested is what is deployed, without the overhead of artifact zipping/unzipping.

2.2 Smarter Workspace Sharing

Instead of actions/upload-artifact for the entire 500MB+ workspace: 1. Job: setup: Clone once. 2. Path Filtering: Determine which sub-projects changed. 3. Sparse Artifacts: Only upload the specific project directory needed (e.g., bnc-cpt-api/ for API tests).

2.3 Parallel Testing Architecture

• API:
  • job: test-unit: Run pytest tests/unit.
  • job: test-integration: Run pytest tests/integration (starts Redis/Tesla Mock).
  • These jobs run in parallel after the build job completes.
• WUI:
  • job: test-unit: Vitest.
  • job: lint-typecheck: ESlint + vue-tsc.
  • job: test-ui-matrix: Parallel Puppeteer workers by language.

2.4 Pre-baked Dependencies

Stop running npm install and poetry install in test jobs. 1. The build job must ensure node_modules and .venv are fully populated inside the Docker image. 2. The Dockerfile should be optimized with multi-stage builds to keep the final test/deploy image lean but complete.

3. Specific Technical Fixes

3.1 utl/bin/ci-helper.sh

Create a suite of CI helper functions in the utl project: * do_ci_get_image_tag: Generates the unique SHA-based tag. * do_ci_wait_for_registry: Ensures the image is available before pulling.

3.2 API Workflow Fixes

• Remove --no-cache from make targets in CI.
• Implement docker/login-action for GCP Artifact Registry.
• Split build-and-test into build -> test (parallel).

3.3 WUI Workflow Fixes

• Replace docker save/load with docker pull from Artifact Registry.
• Remove npm install from test-support and test-unit jobs; rely on the image.
• Update deploy job to use the pre-built image for the build step.

4. Path Filtering Logic

Implement dorny/paths-filter in a pre-flight job:

filters:
  api:
    - 'bnc-cpt-api/**'
    - 'bnc-cpt-utl/**'
    - 'bnc-cpt-cnf/**'
  wui:
    - 'bnc-cpt-wui/**'
    - 'bnc-cpt-utl/**'
    - 'bnc-cpt-cnf/**'

5. Performance Targets (v02)

• API Build+Test: < 5 minutes (down from ~12m).
• WUI Build+Test: < 7 minutes (down from ~18m).
• Deployment: < 3 minutes (down from ~8m).
• Total "Cold" Run: < 10 minutes.

6. Implementation Checklist

1. [ ] Update bnc-cpt-utl with CI helper scripts.
2. [ ] Configure GCP Artifact Registry for all environments.
3. [ ] Refactor bnc-cpt-api ci.yaml to use the Registry.
4. [ ] Refactor bnc-cpt-wui ci.yaml to remove docker save/load.
5. [ ] Add path filtering to both.
6. [ ] Enable GitHub Actions Cache for Docker.