01.11.Tracing Feature

01.11. Tracing Feature

1. Overview

This feature implements a non-global, session-based "switch" that enables full request/response capture for specific Tesla Fleet API calls. It supports high-fidelity diagnostics for designated users while maintaining the project's core "no-save" privacy mandate for the general public.

Core Principles

No-Save by Default: Standard users never have their Tesla data persisted.
Selective Activation: Tracing is only active if a specific TRACE_SECRET is provided.
Ephemeral Storage: All trace data and temporary PDF reports are subject to a strict cleanup policy (1-hour TTL in Redis, 1-day minimum in GCS).
Invisible to Users: The feature is triggered via hidden headers or URL parameters.

2. Usage Guide

A. Browser (WUI) — For End Users / QA

Append ?trace_secret=<value> to any page URL. The secret is captured into sessionStorage, stripped from the address bar, and automatically attached to every subsequent API call for the duration of the browser tab.

https://dev.carpulsetracker.com/?trace_secret=0024PlaPla__

From that point on every API request made by the WUI includes the X-Trace-Secret and X-Session-ID headers. The backend buffers all log records for each request and uploads them to GCS as a JSON bundle.

Cleanup happens automatically: - After a PDF is downloaded, the WUI calls DELETE /tesla/reports/{task_id}. - When the tab closes or the user navigates away, the WUI calls DELETE /tesla/traces/{session_id} (best-effort via keepalive). - GCS lifecycle deletes anything left after 24 hours.

B. curl / CLI — For Developers

1. Health check with tracing (observe structured logs on the API container):

curl -s \
  -H "X-Trace-Secret: 0024PlaPla__" \
  -H "X-Session-ID: demo-session-001" \
  https://dev.carpulsetracker.com/api/v1/tesla/health

2. Any API call — just add the two headers:

curl -s \
  -H "Content-Type: application/json" \
  -H "X-Trace-Secret: 0024PlaPla__" \
  -H "X-Session-ID: my-debug-session" \
  https://dev.carpulsetracker.com/api/v1/pricing/catalog

3. Cleanup — delete trace data for a session:

curl -s -X DELETE \
  -H "X-Trace-Secret: 0024PlaPla__" \
  https://dev.carpulsetracker.com/api/v1/tesla/traces/my-debug-session

4. Cleanup — delete a specific PDF report:

curl -s -X DELETE \
  -H "X-Trace-Secret: 0024PlaPla__" \
  https://dev.carpulsetracker.com/api/v1/tesla/reports/<report_id>

5. Local dev (Docker):

# Health check with tracing against the local API container
curl -s \
  -H "X-Trace-Secret: 0024PlaPla__" \
  -H "X-Session-ID: local-test" \
  http://localhost:8100/api/v1/tesla/health

# Watch the API container logs for trace output
docker logs -f con-bnc-cpt-api 2>&1 | jq 'select(.trace_on == true)'

C. What Happens Under the Hood

Middleware (main.py:request_context_middleware) reads X-Trace-Secret.
If the value matches the TRACE_SECRET env var, it sets trace_on=True in the request context and initialises an in-memory log buffer.
TraceHandler (a Python logging.Handler) captures every log record emitted during the request into that buffer.
Logs also flow to stdout as normal (picked up by Cloud Logging).
After the response is sent, the middleware uploads the buffer as a JSON file to GCS: traces/{session_id}/{timestamp}_{request_id}_{endpoint}.json.
The buffer is cleared and trace_on is reset to False.

D. Viewing Trace Data

In GCS (Cloud Console or gsutil):

# List traces for a session
gsutil ls gs://bnc-cpt-dev-reports/traces/my-debug-session/

# Download and inspect a trace bundle
gsutil cat gs://bnc-cpt-dev-reports/traces/my-debug-session/20260317T194200Z_a1b2c3d4_api_v1_tesla_health.json | jq .

In Cloud Logging (deployed environments):

resource.type="cloud_run_revision"
jsonPayload.trace_on=true

3. Technical Analysis

A. Backend Architecture (`bnc-cpt-api`)

Request Context: Uses ContextVar to propagate a trace_on flag throughout the request lifecycle.
TraceHandler: A logging.Handler subclass buffers all log records into a per-request list when trace_on is active.
Storage: GCS (REPORTS_BUCKET) under the traces/ prefix — each request produces a JSON log bundle.
Cleanup Policy:
- PDF tasks in Redis: configurable TTL (TASK_STATUS_TTL_SECONDS).
- GCS Lifecycle: 1-day "Delete" rule for the entire bucket.
- Active Deletion: Two endpoints for WUI-triggered cleanup:
- DELETE /tesla/reports/{report_id} — deletes a single PDF blob after download.
- DELETE /tesla/traces/{session_id} — deletes all trace blobs for a session.

B. Frontend Integration (`bnc-cpt-wui`)

Activation: Hidden URL parameter ?trace_secret=... captured into sessionStorage on first load. The parameter is stripped from the URL to prevent leaking in bookmarks/shares.
Propagation: services/api.ts buildHeaders() automatically attaches X-Trace-Secret and X-Session-ID to all outgoing calls when a trace secret is present.
Cleanup Triggers:
ReportDashboard.vue calls DELETE /tesla/reports/{task_id} after a successful PDF download (awaited, error shown if fails).
beforeunload event and onBeforeUnmount lifecycle hook call DELETE /tesla/traces/{session_id} with keepalive: true for reliable delivery.

C. Logging & Observability: The `do_log TRACE` Pattern

The Python backend implements a specialized tracing logger aligned with project-wide bash conventions.

Unified Trace Function (do_log):
- do_log(message: str, data: Optional[dict] = None) maps the first token (e.g., TRACE, INFO, ERROR) to standard Python log levels.
- If trace_on is enabled, or if the level is explicitly TRACE, payload data is included in the structured JSON log output.
Automatic Log Routing via TraceHandler:
- TraceHandler in app/core/logging_config.py checks the _trace_on ContextVar.
- When active, all log records (DEBUG and above) are buffered into a per-request list via _trace_buffer ContextVar.
- On request completion, the middleware uploads the buffer to GCS as a JSON bundle and also lets records flow through stdout (Cloud Logging) as normal.

4. Implementation Details

Files Modified

Repository	File	Change
`bnc-cpt-api`	`app/core/logging_config.py`	Added `TraceHandler`, `_trace_buffer` ContextVar, `begin_trace_buffer()`, `flush_trace_buffer()`
`bnc-cpt-api`	`app/core/gcs.py`	Added `upload_trace_to_gcs()`, `delete_gcs_blob()`, `delete_gcs_prefix()`
`bnc-cpt-api`	`app/config.py` + `.tpl`	Added `TRACE_SECRET: str = ""` setting
`bnc-cpt-api`	`app/main.py`	Enhanced `request_context_middleware` to validate `X-Trace-Secret`, manage trace buffer lifecycle, upload trace bundle on request completion
`bnc-cpt-api`	`app/api/v1/routers/tesla.py`	Added `DELETE /tesla/reports/{report_id}` and `DELETE /tesla/traces/{session_id}` endpoints
`bnc-cpt-wui`	`src/services/api.ts`	Added `initTraceSecret()`, `getTraceHeaders()`, `buildHeaders()`, `deleteReport()`, `deleteTraces()`
`bnc-cpt-wui`	`src/components/ReportDashboard.vue`	Added post-download report cleanup and `beforeunload`/`onBeforeUnmount` trace cleanup
`bnc-cpt-inf`	`src/terraform/016-gcp-reports-bucket/04.step-vars.tf`	Changed `lifecycle_age_days` default from 7 to 1
`bnc-cpt-cnf`	`{env}.env.yaml` (all envs)	Added `bnc-cpt-trace-secret` to step 029 secrets list

GCS Storage Pattern

Reports: gs://{REPORTS_BUCKET}/reports/vehicle-report-{report_id}.pdf
Traces: gs://{REPORTS_BUCKET}/traces/{session_id}/{timestamp}_{request_id}_{endpoint}.json

Security & Authentication

A global TRACE_SECRET is stored in GCP Secret Manager (bnc-cpt-trace-secret, provisioned in step 029).
The request_context_middleware validates the X-Trace-Secret header against the TRACE_SECRET env var.
If valid, set_log_context(trace_on=True) is called and the trace buffer is initialized.

API Endpoints

Endpoint	Method	Purpose
`/api/v1/tesla/reports/{report_id}`	DELETE	Delete a PDF report blob from GCS after download
`/api/v1/tesla/traces/{session_id}`	DELETE	Delete all trace blobs for a session from GCS

5. Data Privacy Compliance

Manual Cleanup: Two separate DELETE endpoints for reports and traces.
Automatic Cleanup: GCS lifecycle ensures no abandoned reports or traces persist beyond 24 hours.
Audit Logs: Every traced request logs an INFO event: Selective tracing enabled for session {session_id}.

6. Provisioning Status

Infrastructure provisioned across all environments (2026-03-17):

Step	inf	dev	tst	prd
016 (reports bucket lifecycle=1d)	N/A	OK	OK	OK
029 (trace-secret container)	OK	OK	OK	OK
Secret value (version 1)	OK	OK	OK	OK

Rotating the Secret

To change the trace secret value in an environment:

gcloud auth activate-service-account --key-file=$HOME/.gcp/.bnc/key-bnc-cpt-{env}.json
echo -n "NEW_SECRET_VALUE" | \
  gcloud secrets versions add bnc-cpt-trace-secret --data-file=- --project=bnc-cpt-{env}

Then redeploy the API container so it picks up the new value.