03-Development / 03.07.UTL-Development

03.07.UTL Development

03.07. UTL Development

Overview

The bnc-cpt-utl module is the orchestration layer for the Car Pulse Tracker project. It provides Docker container management, Make targets for Terraform operations, and shell actions for deployment and version management.

Quick Start

cd /opt/bnc/bnc-cpt/bnc-cpt-utl
export UID=$(id -u) GID=$(id -g)

# Start API container
make do-setup-api

# Start WUI container
make do-setup-wui

# Start Terraform runner
make do-setup-tf-runner

# Start all infra containers
make do-setup-bnc-cpt-all

Version Management

Bumping Versions

The do_version_bump shell action manages semantic version tags for any project.

# Bump WUI version (default target)
./run -a do_version_bump

# Bump API version
TARGET_PROJ=/opt/bnc/bnc-cpt/bnc-cpt-api ./run -a do_version_bump

# Bump types
BUMP=patch ./run -a do_version_bump    # v0.2.3 -> v0.2.4 (default)
BUMP=minor ./run -a do_version_bump    # v0.2.3 -> v0.3.0
BUMP=major ./run -a do_version_bump    # v0.2.3 -> v1.0.0

# Push the tag
git -C /opt/bnc/bnc-cpt/bnc-cpt-wui push origin <tag>

Increment Logic

Patch increments 0-9, then rolls to minor. Minor 0-9, then rolls to major:

v1.0.0 -> v1.0.1 -> ... -> v1.0.9 -> v1.1.0 -> ... -> v1.9.9 -> v2.0.0

How Versions Reach Deployments

WUI (Vue frontend): - gcp-deploy-wui.func.sh reads latest git tag from bnc-cpt-wui - Passes VITE_APP_VERSION env var to Docker build - Vite plugin injects <meta name="version" content="v0.2.3" /> into HTML - Check: curl -s https://dev.carpulsetracker.com/index.html | grep version

API (FastAPI backend): - cd.yaml reads latest git tag from bnc-cpt-api - Passes APP_VERSION env var to Cloud Run - Visible at /, /api, /health endpoints - Check: curl -s https://dev.api.carpulsetracker.com/ | jq .version

WUI Deployment

# Deploy to specific environment
ENV=dev ./run -a do_gcp_deploy_wui

# With explicit version override
APP_VERSION=v1.0.0 ENV=dev ./run -a do_gcp_deploy_wui

Flow: Read git tag -> Build Vue app in container -> gsutil rsync to GCS -> CDN invalidation -> Health check

Terraform Operations

All terraform commands run inside the con-bnc-cpt-tf-runner container.

# Generate config (YAML -> JSON -> tfvars)
make do-generate-config-for-step ENV=dev STEP=029-create-gcp-secrets

# Provision
make do-provision ENV=dev STEP=029-create-gcp-secrets

# Deprovision
make do-deprovision ENV=dev STEP=029-create-gcp-secrets

# Plan only
make do-tf-plan ENV=dev STEP=029-create-gcp-secrets

# State operations
make do-tf-state-list ENV=dev STEP=029-create-gcp-secrets
make do-tf-state-show ENV=dev STEP=029-create-gcp-secrets TARGET="resource.name"

# Import existing resource
make do-tf-import ENV=all STEP=120-github-general-secrets \
  TARGET='github_actions_variable.my_var[0]' ID='repo-name:VAR_NAME'

# Apply single resource
make do-tf-apply-target ENV=dev STEP=029-create-gcp-secrets TARGET="resource.name"

# Destroy single resource
make do-tf-destroy-target ENV=dev STEP=029-create-gcp-secrets TARGET="resource.name"

Provision all environments

for env in inf dev tst prd; do
  make do-generate-config-for-step ENV=$env STEP=029-create-gcp-secrets
  make do-provision ENV=$env STEP=029-create-gcp-secrets
done

Docker Containers

Container Purpose Port Compose File
con-bnc-cpt-api FastAPI backend 8100 docker-compose-app.yaml
con-bnc-cpt-wui Vue frontend (Vite dev server) 3333 docker-compose-app.yaml
con-bnc-cpt-the-bot Puppeteer UI testing (Chromium) - docker-compose-app.yaml
con-bnc-cpt-gateway Nginx reverse proxy 80 docker-compose-app.yaml
con-bnc-cpt-redis Redis (session store and cache) 6379 docker-compose-app.yaml
con-bnc-cpt-tf-runner Terraform execution - docker-compose-infra.yaml
con-bnc-cpt-tpl-gen Template generator - docker-compose-infra.yaml
con-bnc-cpt-conf-validator Config validation - docker-compose-infra.yaml
# Check running containers
docker ps | grep bnc-cpt

# Prune Docker cache
make do-prune-docker-system

Shell Actions Reference

Deployment

Action Usage Description
do_gcp_deploy_wui ENV=dev ./run -a do_gcp_deploy_wui Build + deploy WUI to GCS
do_gcp_deploy_api_full ENV=dev ./run -a do_gcp_deploy_api_full Build + push + deploy API
do_gcp_build_and_push_api_image ENV=dev ./run -a do_gcp_build_and_push_api_image Build + push API image
do_gcp_deploy_cloud_run ENV=dev ./run -a do_gcp_deploy_cloud_run Deploy existing image

Version Management

Action Usage Description
do_version_bump ./run -a do_version_bump Bump WUI version tag
do_version_bump TARGET_PROJ=.../bnc-cpt-api ./run -a do_version_bump Bump API version tag

Secrets Management

Action Usage Description
do_gcp_fetch_secrets ENV=dev ./run -a do_gcp_fetch_secrets Fetch all GCP secrets as env vars
do_gcp_fetch_secrets ENV=dev SECRETS_FILTER=test ./run -a do_gcp_fetch_secrets Fetch test credentials only

Utility

Action Usage Description
do_push_all_app_repos ./run -a do_push_all_app_repos Push all project repos
do_pull_all_app_repos ./run -a do_pull_all_app_repos Pull all project repos
do_zip_all_projects ./run -a do_zip_all_projects Archive all projects

Finding Actions

./run --help                        # List all available actions
SRCH=gcp ./run -a do_help_with      # Search by keyword
SRCH=deploy ./run -a do_help_with   # Search deploy actions

Tab-Completion for Shell Actions

Tab-completion lets you discover actions by typing ./run -a do_<TAB> instead of scrolling through --help output.

One-time setup (installs completion + schedules cache refresh):

cd /opt/bnc/bnc-cpt/bnc-cpt-utl
./run -a do_setup_shell_completion
source ~/.bashrc

Usage:

./run -a do_<TAB>            # List all actions in the current repo
./run -a do_gcp<TAB>         # Narrow to gcp-related actions
./run -a do_version<TAB>     # Complete to do_version_bump

The completion works in any repo that has a run symlink — it scans src/bash/run/*.func.sh and lib/bash/funcs/*.func.sh relative to the current directory.

Cache: Action lists are cached in /tmp/.run_completions_* per directory. The cache auto-invalidates when any *.func.sh file is modified. A cron job also pre-warms the cache at 10:00, 15:00, and 19:00 daily.

Manual source (without installing):

source /opt/bnc/bnc-cpt/bnc-cpt-utl/lib/bash/completions/run.completion.bash

Make Targets

Testing

make test-api             # Run all API tests
make test-api-cov         # With coverage report
make test-api-payment     # Payment tests only
make test-api-debug       # Verbose output
make test-api-shell       # Shell into test container

Configuration

make do-generate-env-json ENV=dev                              # YAML -> JSON
make do-generate-config-for-step ENV=dev STEP=030-gcp-cloud-run # JSON -> tfvars

Help

make help                 # List all make targets with descriptions

Directory Structure

bnc-cpt-utl/
├── Makefile                      # Main orchestrator
├── run -> src/bash/run/run.sh    # Shell action dispatcher
├── lib/
│   ├── bash/funcs/               # Shared bash utilities
│   └── make/                     # Make includes
└── src/
    ├── docker/
    │   ├── cpt-api/              # API Dockerfile
    │   ├── cpt-wui/              # WUI Dockerfile
    │   ├── tf-runner/            # Terraform Dockerfile
    │   ├── tpl-gen/              # Template generator Dockerfile
    │   ├── docker-compose-api.yaml
    │   ├── docker-compose-wui.yaml
    │   ├── docker-compose-infra.yaml
    │   ├── .env                  # Active Docker env vars
    │   └── .env.cin              # CI env vars template
    ├── bash/
    │   ├── run/                      # Shell action files
    │   │   ├── version-bump.func.sh
    │   │   ├── gcp-deploy-wui.func.sh
    │   │   ├── gcp-deploy-api-full.func.sh
    │   │   ├── gcp-fetch-secrets.func.sh  # Fetch GCP secrets as env vars
    │   │   └── ...
    │   └── scripts/
    │       ├── docker-init-cpt-api.sh     # API container init
    │       ├── docker-init-the-bot.sh     # Test container init
    │       ├── gcp-fetch-secrets.sh       # Standalone eval-able secrets fetcher
    │       └── ...
    └── make/                      # Function-specific makefiles
        ├── tf-tasks.func.mk       # Terraform targets
        ├── setup-api.func.mk      # API container targets
        ├── setup-wui.func.mk      # WUI container targets
        └── ...

GCP Secret Manager

Shell Actions

Action Usage Description
do_gcp_fetch_secrets ENV=dev ./run -a do_gcp_fetch_secrets Fetch all secrets, export as env vars
do_gcp_fetch_secrets ENV=dev SECRETS_FILTER=test ./run -a do_gcp_fetch_secrets Fetch test credentials only

Standalone Script (eval-able)

# On host, before starting containers:
eval "$(ENV=dev bash src/bash/scripts/gcp-fetch-secrets.sh)"
make do-setup-app   # containers inherit secrets from host env

# Test creds only:
eval "$(ENV=dev SECRETS_FILTER=test bash src/bash/scripts/gcp-fetch-secrets.sh)"

Environment Variable Mapping

Secret IDs are auto-converted: strip bnc-cpt- prefix, - to _, uppercase.

Test credential env vars passed through docker-compose to the-bot container: - STRIPTE_TEST_ACCOUNT_CARD_NUMBER - STRIPTE_TEST_ACCOUNT_CARD_VALIDITY_MONTH - STRIPTE_TEST_ACCOUNT_CARD_VALIDITY_YEAR - STRIPTE_TEST_ACCOUNT_CARD_CRC_CODE - PAYPAL_TEST_USER_USERNAME - PAYPAL_TEST_USER_PASSWORD

Security Model

Observability & Monitoring

Structured Logging

All logs are JSON-formatted via app/core/logging_config.py, compatible with Google Cloud Logging. Context fields are automatically attached:

Field Source Purpose
request_id X-Request-ID header or auto-generated UUID Trace correlation
session_id Tesla OAuth / order session User flow tracking
payment_id Payment intent ID Payment flow tracking
vin_suffix Last 4 chars of VIN Vehicle identification (privacy-safe)
from app.core.logging_config import set_log_context
set_log_context(session_id="abc123", payment_id="pi_xyz")

Distributed Tracing (OpenTelemetry)

Tracing is configured in app/core/tracing.py. Controlled by OTEL_ENABLED setting (default: False for local dev).

Auto-instrumented: - FastAPI incoming requests (spans for each endpoint) - httpx outgoing HTTP calls (Tesla API, Stripe, PayPal) - Redis operations

Local development: Set OTEL_ENABLED=true in .env to enable console trace output.

Cloud Run: Set OTEL_ENABLED=true and GCP_PROJECT=bnc-cpt-{env} as env vars. Traces export to Google Cloud Trace.

Dependencies (in requirements.txt): - opentelemetry-api, opentelemetry-sdk - opentelemetry-exporter-gcp-trace - opentelemetry-instrumentation-fastapi - opentelemetry-instrumentation-httpx - opentelemetry-instrumentation-redis

Cloud Monitoring (Terraform Step 140)

Provisioned via bnc-cpt-inf/src/terraform/140-gcp-cloud-monitor/:

Log-based Metrics: - tesla-api-errors — Tesla API failures (401, 403, 429, 5xx) - payment-errors — Payment processing failures - pdf-errors — PDF generation/rendering failures - server-errors-5xx — All HTTP 5xx responses

Alert Policies: | Alert | Threshold | Window | Severity | |-------|-----------|--------|----------| | Tesla API Error Rate | >10 errors | 5 min | WARNING | | Payment Failure Rate | >5 errors | 15 min | CRITICAL | | Server 5xx Error Rate | >10 errors | 5 min | WARNING | | PDF Generation Failures | >5 errors | 15 min | WARNING |

Health Dashboard (10 panels): - API request rate, latency p95/p99 - Tesla/payment/PDF error rates - CPU/memory utilization, instance count, startup latency

Provision monitoring:

cd /opt/bnc/bnc-cpt/bnc-cpt-utl
for env in dev tst prd; do
  make do-generate-config-for-step ENV=$env STEP=140-gcp-cloud-monitor
  make do-provision ENV=$env STEP=140-gcp-cloud-monitor
done

CI/CD Integration

The CI/CD workflows in bnc-cpt-api and bnc-cpt-wui repos use: - .env.cin for CI environment variables - Make targets for container setup - Shell actions for deployment - MOUNT_WORK_DIR override for GitHub runner paths

When modifying CI-related code, ensure the .env.cin file has all required variables for both API and WUI containers.

Post-Deploy Testing

Both API and WUI pipelines include a post-deploy-test job that runs after deployment:

API (cd.yaml): Curl-based smoke tests against the deployed API URL: - GET /health - health check - GET /docs - Swagger UI accessible - POST /api/v1/payment/create-intent - payment endpoint responsive

WUI (ci.yaml): The-bot Puppeteer suite against the deployed frontend: - Builds the-bot container in CI - Fetches test credentials from GCP Secret Manager - Runs full Puppeteer test suite against the deployed environment - Uploads Mochawesome HTML report as artifact

Both jobs: - Skip prd environment (production is never auto-tested) - Run against inf, dev, tst environments - Fetch test credentials from GCP Secret Manager (masked in logs) - Produce GitHub Actions step summaries with results