Skip to main content

Retraining & Continuous Learning

AI Products are living systems.
Their performance can degrade over time due to data drift, concept drift, or changing environments.
To remain trustworthy and valuable, AI Products must support retraining and continuous learning as first-class lifecycle requirements.

Why This Matters​

  • Performance → Prevents accuracy degradation in dynamic environments.
  • Fairness → Ensures bias does not accumulate undetected.
  • Compliance → Some regulations (e.g., EU AI Act) require demonstrable monitoring and retraining.
  • Sustainability → Prolongs the useful lifecycle of the AI Product.

Retraining Modes​

AI Products must declare supported retraining approaches:

  1. Periodic Retraining

    • Scheduled updates using new or augmented data.
    • Example: monthly retraining on updated transaction datasets.

  2. Triggered Retraining

    • Initiated when drift or performance degradation crosses thresholds.
    • Example: retrain if accuracy drops below 90% or fairness metrics degrade.

  3. Continuous / Online Learning

    • Model updates incrementally as new data arrives.
    • Example: reinforcement learning from user feedback.

  4. Human-in-the-Loop Retraining

    • Retraining guided by human oversight.
    • Example: curating edge cases or correcting misclassifications.

Required Declarations​

Every AI Product must declare:

  • Retraining strategy → periodic, triggered, online, human-in-the-loop.
  • Retraining frequency or thresholds → time-based or metric-based triggers.
  • Data sources → where training data originates (must align with governance rules).
  • Governance approval → who authorizes retraining and deployment.
  • Versioning → how retrained versions are labeled and communicated (see Versioning & Lifecycle).

Risks & Safeguards​

  • Data Contamination → Ensure retraining data does not leak private or sensitive information.
  • Bias Amplification → Monitor for cumulative bias after repeated retraining.
  • Overfitting → Prevent degradation of generalization ability.
  • Model Regression → Require regression testing to confirm retrained models do not reduce quality.

Example​

Fraud Detection Product

  • Retraining Strategy: Triggered + periodic.
  • Thresholds: Retrain if precision falls below 95% or false positives exceed 5%.
  • Frequency: Monthly refresh on new transaction data.
  • Data Sources: Anonymized transaction logs, labeled by risk analysts.
  • Governance: Retraining approved by compliance officer.
  • Versioning: Semantic versioning with retrain date encoded.

Summary​

  • Retraining is not optional — it is a core lifecycle responsibility of an AI Product.
  • Must declare strategy, frequency, data sources, governance, and versioning.
  • Continuous learning must include safeguards against bias, overfitting, and regression.

Principle: An AI Product that cannot evolve through retraining is destined to drift into obsolescence.