Architecture

Internals of the ai-agents execution engine. The engine uses Kubernetes CRDs as the sole state store — no Redis, no BullMQ, no database.

For the top-level system view (context, building blocks, deployment, RAD conformance) see the Software Architecture Document. This page is the engine-internals deep-dive.

Deployable Processes

One TypeScript codebase (npm workspace) produces two server entrypoints, a Next.js frontend, and a set of executor images:

The operator controller is conditional on the API process via ENABLE_OPERATOR (default true). Webhook pods run the same API image in MODE=webhook: when a WebhookSource CR is created, the API server auto-provisions a Deployment + Service + Ingress for it (src/k8s/webhook-sources.ts), each cleaned up via a finalizer on CR deletion. There are no separate processor/webhook images or charts — everything ships from the single charts/ai-agents-main chart.

GitHub event ──► Webhook pod ──► InferenceRequest CR
POST /api/jobs/submit ─────────►        │
POST /api/events/ingest ───────►        ▼
            ┌──────────────────────────────────────────┐
            │  API server (operator reconcile loop)     │
            │  watches IR CRs, creates k8s Jobs         │
            └──────────────────────────────────────────┘
                                         │
                                         ▼
                              ephemeral k8s Job
                              (executor container)
                                         │ inference + MCP tool callbacks
                                         ▼
                              Inference Proxy ──► Anthropic / OpenAI / GX10 vLLM / Ollama

CRDs (group: labrats.work, version: v1alpha1)

Prompts are not stored inline in IR specs (CRD size limit ≈1 MiB). Each IR holds a promptRef pointing to a per-job ConfigMap ({irName}-prompt).

InferenceRequest (full shape, src/k8s/inference-request-types.ts)

spec: {
  source: string                    // "github", "manual", "dashboard", "github-actions", ...
  type: string                      // "implement-issue", "code-review", "document-extraction", ...
  agentRole?: string                // matches AiAgent.metadata.name or trigger
  priority?: number                 // default 2; lower = higher priority
  promptRef: { configMapName, key } // ConfigMap holding the prompt
  workspaceSetup?: {
    gitRepo, gitRef, gitDepth, githubToken
  }
  workspaceFiles?:   [{ name, data }]      // base64-encoded
  workspaceUploads?: [{ id, name }]        // RWX uploads PVC references
  config?: { model, effort, fast, provider, timeoutMs, maxTurns, executorImage, reactEnabled }
  callbackUrl?: string
  pipeline?: { type: "pdf-sections" }
  chain?: {
    chainId, chainDepth, parentRequestName, chainPath[]
    nextAgentTrigger, nextAgentCondition, nextFailurePolicy
  }
  retryPolicy?: { maxAttempts, backoffMs }
  metadata?: Record<string, unknown>
  errors?: [{ context, message, stack, timestamp }]
}

status: {
  phase: "Pending"|"Running"|"Succeeded"|"Failed"|"Cancelled"
  attempts, accountName, k8sJobName, executorPodName
  startedAt, completedAt
  result: { exitCode, timedOut, budgetExceeded, budgetReason, usage{...}, provider, model, effort, fast }
  failedReason, logs[], conditions[]
}

Operator (src/operator/)

Controller (controller.ts)

• Informer-driven plus a 5-second poll fallback (DEFAULT_POLL_INTERVAL_MS = 5_000).
• Default concurrency is 5; override with WORKER_CONCURRENCY.
• Tracks in-flight reconciles in an internal Set of IR names (processing).
• Pending IRs are sorted by priority ascending, then by creationTimestamp.
• Running IRs are reconciled before Pending IRs so status polling stays fresh.
• New events (informer add/update) are debounced via setImmediate into the next reconcile pass.
• A slow retention GC sweep (gc.ts, default every 10 min via IR_GC_INTERVAL_MINUTES) hard-deletes terminal IRs past their per-phase TTL — IR_TTL_SUCCEEDED (24h), IR_TTL_FAILED (168h), IR_TTL_CANCELLED (24h); 0 disables a phase. (spec 0016)

Reconciler (reconciler.ts)

On terminal status the reconciler also calls maybeChainNextAgent() to optionally enqueue a child IR.

Prompt Builder (prompt-builder.ts)

Resolution at job start:

config.model    : IR.spec.config.model
                  → AiAgent.spec.defaultModel
                  → AiConfig.spec.providers[provider].model
                  → AiConfig.spec.model
                  → CLI default (no flag passed)

config.effort   : IR.spec.config.effort
                  → AiAgent.spec.defaultEffort
                  → AiConfig.spec.providers[provider].effort
                  → AiConfig.spec.effort
                  → "medium"

config.provider : IR.spec.config.provider
                  → AiAgent.spec.provider
                  → "claude"

Instruction injection (instructionWatcher.resolveForAgent()):

1. Collect all global AiInstructions sorted by priority ascending.
2. Append local instructions listed in the agent's spec.instructions[].
3. Prepend each as an <instruction name="…" scope="…">…</instruction> XML block to the prompt.

Pipeline jobs (currently only pdf-sections) skip instruction injection.

Chain (chain.ts)

After job completion, maybeChainNextAgent():

1. Returns immediately if spec.chain.nextAgentTrigger is unset.
2. Loop guard — refuses if nextAgentTrigger already appears in chainPath.
3. Depth guard — MAX_CHAIN_DEPTH = 10.
4. Condition — evaluates nextAgentCondition (always | on-success | on-failure) against exitCode === 0.
5. Failure policy — stop | skip | notify (only blocks chaining when policy says so).
6. Captures up to 4,000 chars of parent stdout, wraps it in <chain-context>…</chain-context>, and prepends it to the parent prompt to form the child prompt.
7. Creates a child IR with chainDepth + 1, extended chainPath, and the next agent's own next config (resolved from its eventTriggers).

See Agent Chains for configuration details.

Recursive sub-agent delegation (spec 0013). Separate from event chains, the bundled MCP server's delegate_subtask tool spawns a focused sub-agent — a child IR with its own tools and a small turn budget — capped at depth 3 with a cycle guard via the same spec.chain lineage. To stay deadlock-free the parent parks while waiting: POST /api/jobs/:id/park releases its AiModel slot and unpark re-acquires before it resumes, so leaf sub-agents free slots and the tree drains bottom-up (status.parked tracks it).

Job Executor (src/k8s/job-executor.ts)

Each IR transition to Running creates one ephemeral k8s batch/v1 Job.

Mounts on the executor pod:

Defaults:

• Image: per-provider executor image (default ghcr.io/labrats-work/apps.ai-agents/executor-claude:<app-version>, or the EXECUTOR_IMAGE env). Resolved per provider from AiExecutor CRs / AiConfig.spec.executors, and overridable per-job via the IR's spec.config.executorImage.
• Resources: requests cpu: 100m, memory: 512Mi; limits cpu: 2, memory: 4Gi (uniform default for all providers). Requests are intentionally a low floor — executors are I/O-bound on remote inference; the limits carry burst headroom (#1261). Override per provider via the AiExecutor CR's spec.resources.
• activeDeadlineSeconds = ceil(timeoutMs / 1000) + 60 (default timeoutMs = 3 300 000 ms / 55 min).
• Labels: app.kubernetes.io/managed-by=ai-agents, app.kubernetes.io/component=executor, ai-agents.labrats.work/job-id=<truncated jobId>.
• ServiceAccount: from EXECUTOR_SERVICE_ACCOUNT env. Must allow PATCHing the credentials Secret (used by the executor entrypoint to persist OAuth refresh on exit).

The Job is cleaned up by the executor on completion via JSON patch back to the IR status.

Executor Container (executor/)

One image per CLI provider, built from executor/Dockerfile.{claude,codex,qwen-code} (Node Alpine base, CLIs pre-installed). All run as user executor (uid 1001). The qwen-code image bundles Qwen Code (a Gemini-CLI fork) pointed at an OpenAI-compatible endpoint — the in-cluster Inference Proxy, which fronts the GX10 vLLM backend.

vllm and ollama are not executor images: they are HTTP inference backends. When an AiModel has a spec.providers[].url, the reconciler routes inference via HttpInferenceClient (src/k8s/http-inference-client.ts) to that URL instead of creating a Job (falling back to a Job if the endpoint is unhealthy). mock is a test provider.

The entrypoint.sh still contains legacy gemma / bonsai / qwen (llama-cli + GGUF) branches, but no image is built for them in the current matrix — they are not part of the supported provider set above.

Entrypoint flow (executor/entrypoint.sh):

1. Raise file descriptor limit (claude-code spawns many fsnotify watchers).
2. Credentials:
   • PROVIDER=claude → copy /credentials/token into $HOME/.claude/.credentials.json (mode 0600). On EXIT, PATCH the Secret to persist refreshed tokens.
   • PROVIDER=codex → read /credentials/token into CODEX_API_KEY.
   • PROVIDER=qwen-code → render ~/.qwen/settings.json from the template, pointing at OPENAI_BASE_URL (the proxy); the caller's key is presented as the API key.
3. Git clone: read repo URL from /git-config/repo-url (or GIT_REPO_URL env); shallow clone with depth from GIT_DEPTH (default 1).
4. Workspace uploads: if WORKSPACE_UPLOADS env is set (JSON [{id,name}]), copy /uploads/{id}/{name} into the workspace.
5. MCP servers & skills: translate the MCP_SERVERS env into ~/.claude.json / ~/.qwen/settings.json, and install any mounted /skills/<name> into the CLI's skills dir for auto-discovery.
6. Prompt: read from /prompt/prompt.txt (or PROMPT_TEXT env).
7. CLI invocation (wrapped in timeout ${TIMEOUT_SEC}s):
   • claude — claude --print --verbose --dangerously-skip-permissions --output-format stream-json --no-session-persistence [--model M] [--effort E] [--fast] [--max-turns N]
   • codex — codex exec --json --full-auto --skip-git-repo-check [--model M] [--config model_reasoning_effort=E]
   • qwen-code — the Qwen Code CLI against the configured OpenAI-compatible endpoint.
   • mock — log metadata, exit 0 (used in tests).
8. Exit code is propagated; 124 means timeout.

Model Slots & Credentials (src/k8s/models.ts)

There is no AiAccount pool — it was removed in spec 0003; credential and concurrency tracking moved onto AiModel. A watcher-driven informer keeps a local cache of AiModel CRs.

• Status states (AiModelCR.status.state): idle, busy, unavailable, error. spec.paused: true keeps a model out of rotation.
• Concurrency: spec.maxConcurrency caps simultaneous jobs per model (0 = unlimited); status.activeJobs / status.activeJobIds[] track in-flight IRs.
• Acquisition (acquire()) uses an optimistic JSON Patch with a test op on status.activeJobs to avoid races; on conflict the next candidate is tried.
• Selection filters out paused / error / unavailable models and those at max concurrency, optionally filters by model name, then sorts by load ratio ascending, then least-recently-used. A provider URL is chosen from spec.providers[] (skipping unavailable ones).
• Crash recovery: on startup, models whose activeJobIds reference IRs no longer Running are reconciled back to idle.

Credentials

• Caller identity: every IR carries spec.apiKeyRef (required) → executor AI_AGENTS_API_KEY for the MCP callback. Sourced from the request's Bearer token (submit/events/ingest), WebhookSource.spec.apiKeyRef (webhook events), AiAgent.apiKeyRef (triggers), or inherited (chains). No long-lived executor service credential. (spec 0007)
• Backend token: optional AiModel.spec.credentialsRef → /credentials/token, consumed only by the claude executor. qwen-code/codex use the caller's key, so discovered models need none.
• GitHub: short-lived App installation tokens minted per webhook event; webhooks HMAC-verified with the App webhookSecret. No PAT.

Webhook Ingest

GitHub webhooks land at POST /api/webhooks/github on the webhook pod (or the API pod for the unified deployment).

src/sources/github/index.ts flow:

1. Look up the WebhookSource CR by spec.type === "github".
2. Read webhookSecret, appId, privateKey, installationId from the credentials Secret referenced by spec.credentialsRef.
3. Verify HMAC-SHA-256 (x-hub-signature-256 header vs raw body).
4. Normalize x-github-event + payload.action into a canonical event name.
5. Public-repo filter: if payload.repository.private === false, deny unless allowPublicRepos: true or repository.full_name is in allowedPublicRepos. Returns 200 with matched: 0 on deny (not 4xx).
6. Match agents via agentWatcher.findByEvent(source, event) plus mention parsing (@ai-{role}, @agents-assemble).
7. Create one InferenceRequest per matched agent via irClient.create(...).

Workspace and Uploads

The shared RWX uploads PVC is mounted read-write by the API pod and read-only by executor Jobs. Files referenced via workspaceUploads are copied from /uploads/{id}/{name} into the workspace before the CLI runs. The PVC name comes from UPLOADS_PVC_NAME.

Per-job ConfigMaps and git Secrets are deleted after Job termination via owner references and TTL.

Metrics and Logs

• GET /metrics exposes Prometheus metrics (registry from src/metrics.ts). The dashboard timeseries page reads from PROMETHEUS_URL.
• GET /api/jobs/:id/logs is a Server-Sent Events stream that:
  1. Streams executor pod logs live (follow: true) when an executorPodName is recorded.
  2. Falls back to tailing the persisted log file in the uploads PVC (used by pipeline jobs).
  3. Closes with {done:true,status:…} once the IR reaches a terminal phase.

Config Defaults Resolution

Job submission spec.config
        │
        ▼ (per-field fallback)
AiAgent.spec.defaultModel / defaultEffort / provider
        │
        ▼
AiConfig.spec.providers[provider].model / effort
        │
        ▼
AiConfig.spec.model / effort
        │
        ▼
Hard defaults: provider="claude", effort="medium"

AiConfig.spec.executorImage and executorResources apply uniformly to all jobs unless an IR explicitly overrides them.