Series 1 — Part 5 of 8

Explainability traces are not an afterthought in the behavioral AI platform — they are first-class objects generated at scoring time, stored in the database, and surfaced through a dedicated API. This article shows how to build them and why post-hoc explanations fail.

The Problem with Post-Hoc Explanations

LIME and SHAP work by perturbing an input and watching how outputs change. That tells you about the model's local decision boundary — not about what actually happened when the specific score was computed. In a multi-engine system, that distinction matters enormously: two engines may have produced conflicting outputs that were resolved by a third. Post-hoc analysis on the final number misses the conflict entirely.

Causal traces — built at scoring time — capture the conflict.

The Trace Schema

// Generated inside each engine's score() function
function buildTrace(engineId, signals, output) {
  return {
    engineId,
    timestamp:   new Date().toISOString(),
    inputFields: Object.keys(signals),
    evidenceChain: signals.map(([field, value]) => ({
      field,
      value,
      weight:       getFieldWeight(engineId, field),
      contribution: value * getFieldWeight(engineId, field),
      interpretation: interpretField(engineId, field, value),
    })),
    score:         output.score,
    confidence:    output.confidence,
    reducers:      output.reducers,
    recommendation: buildRecommendation(output.score, engineId),
  };
}

Storing Traces for Audit

-- bc_explainability_traces
CREATE TABLE bc_explainability_traces (
  trace_id    CHAR(36)     PRIMARY KEY,  -- UUID
  engine_id   VARCHAR(80)  NOT NULL,
  context_id  VARCHAR(80)  NOT NULL,     -- e.g. work_log_id or lead_id
  score       DECIMAL(5,4) NOT NULL,
  confidence  DECIMAL(5,4) NOT NULL,
  trace_json  JSON         NOT NULL,     -- the full evidenceChain
  created_at  DATETIME     NOT NULL DEFAULT CURRENT_TIMESTAMP,
  INDEX idx_trace_context (context_id),
  INDEX idx_trace_engine  (engine_id)
);

From Technical Trace to Legal Evidence

A trace stored at created_at with an immutable trace_id can be produced in discovery. The evidenceChain shows exactly which fields drove the score on a specific date. See S1-ADV5 — Explainability as Legal Evidence for chain-of-custody requirements.