Skip to main content

Execution, Serving & Scaling

AIPDS defines how the AI Product executes, how its intelligence is made available, and how its runtime responds to demand and operational constraints.

The historical term inference serving remains relevant to model-backed AI Products, but it is not universal enough for symbolic reasoners, planners, agents, or embodied closed-loop AI.


Execution Modes​

AIPDS should align execution declarations with the AIPROD MO — Interaction / Execution Mode vocabulary.

Batch​

Processes bounded workloads or collections on demand or on a schedule.

Real-Time / Event-Driven​

Responds to events with bounded latency, for example fraud detection or anomaly detection.

Synchronous API​

Returns an intelligence effect within a request/response interaction.

Interactive​

Maintains a user or system interaction context across turns or sessions.

Embedded​

Executes inside an application, device, robot, vehicle, or other host runtime.

Continuous​

Runs continuously against changing state or observations.

Agentic / Goal-Driven​

Pursues goals through multi-step reasoning, planning, tool use, or action.

An AI Product may support multiple execution modes.


Execution Characteristics​

The deployment should declare applicable characteristics such as:

  • latency and deadline profile;
  • throughput / work rate;
  • concurrency and session limits;
  • statefulness and context retention;
  • resource consumption;
  • warm-up / cold-start behavior;
  • timeout and retry semantics;
  • degraded-mode / fallback behavior;
  • tool/action invocation limits;
  • control-loop frequency and jitter for physical AI;
  • human approval latency where AR = ActWithApproval;
  • service-level or safety-level objectives.

Not every characteristic applies to every AI Product.


Scaling Strategies​

Scaling may include:

  • horizontal replication;
  • vertical resource expansion;
  • elastic autoscaling;
  • partitioned or sharded execution;
  • session/agent pool scaling;
  • edge fleet scaling;
  • fixed-capacity embedded execution;
  • hybrid cloud/edge distribution.

For safety-critical or closed-loop physical AI, fixed bounded execution may be preferable to elastic scaling. The specification therefore requires explicit behavior rather than assuming cloud autoscaling.


Model Inference as a Specialization​

Where a learned model participates directly in execution, the AIPDS may additionally declare:

  • batch/online/streaming inference;
  • model server characteristics;
  • accelerator requirements;
  • token/image/frame throughput;
  • model warm-up and caching behavior;
  • fallback model strategy.

These are conditional model-serving details, not universal AIPDS requirements.


Principle: Every AI Product must declare its execution behavior; only model-backed Products need model-specific inference declarations.