Tenancy

How ai-agents tells callers apart, what each may use, and how their usage is counted and paced. Each section is self-contained and links the spec or ADR that holds the reasoning.

flowchart LR
  W["Workload pod<br/>(projected SA token)"] -->|/auth/workload-token| API
  K["API key / user / webhook / schedule"] --> API
  API["API server<br/>resolves caller"] -->|IR + caller annotations| OP["Operator<br/>maxJobs, priority"]
  OP -->|per-job token| PX
  W -->|workload token, /v1| PX["Proxy<br/>maxInFlight queue"]
  PX --> M["ai_proxy_*{tenant, call}"]
  OP --> J["ai_ir_*{tenant}"]

What a tenant is

A tenant is the low-cardinality label every job and /v1 call is attributed to. Every entry point resolves a caller (kind, id, tenant) once, in src/auth/caller.ts:

• Per job: the entry point that creates the InferenceRequest stamps the caller on it as ai-agents.labrats.work/caller-kind, caller and tenant. Chain and delegation children copy them. A client cannot set them, and they are honoured only on IRs in the operator namespace.
• Per /v1 call: the proxy resolves the caller from the bearer. A per-job token resolves to its IR's recorded caller, so calls an executor makes count against whoever submitted the job. A call with no resolvable caller is unattributed.
• The tenant is never an email or an IR name; user identity stays in id.
• People need grant:ai-models-active in their user space (runtime.k8s.bor.gitops spec 0016 Phase 8 d7): every user has a space, not every user may use ai-agents. Dashboard and proxy sign-in, Authelia Bearer tokens and CLI tokens all make the same cached SubjectAccessReview (use aimodels in user-<id> as oidc:<id>) and are refused without it. The ai-agents and ai-agents-chat Authelia clients are one_factor with no group policy, so this check is what restricts access. Series recorded before this change keep the old shared tenant user.
• A dashboard login re-keys the user's jobs recorded under their verified email (before sessions carried the ID) to the ID, keeping the email in ai-agents.labrats.work/caller-email (spec 0077).

Why: spec 0074.

Identity and credentials

Workload token exchange. The pod mounts a projected ServiceAccount token with audience ai-agents-inference and posts it as the Bearer to /auth/workload-token on the API server, with a body naming what it wants: model and/or agentRole (a string or list), optional ttlSeconds (default 3600, clamped 60 s to 24 h). The server runs a TokenReview, derives the namespace from the verified system:serviceaccount:<ns>:<sa> (the body cannot name an identity), and SAR-checks each requested name for verb use in that namespace. It returns a short-lived HS256 token (username: ns:<namespace>) whose models and agentRoles claims hold only the allowed names; nothing allowed is a 403. models is always present, and empty when only agent roles were requested or granted: a caller that needs both submits and direct calls names both in one exchange or exchanges twice. Revoking the RoleBinding stops new exchanges at once.

GitHub Actions exchange. A trusted workflow requests a GitHub OIDC ID token (permissions: id-token: write) with the environment's API URL as audience and posts it as the Bearer to /auth/github-token, with the same body as /auth/workload-token. The server verifies it against GitHub's JWKS and matches it against the trust list, a gitops ConfigMap (chart value githubOidc.trustConfigMap, key trust.json). A match needs the org's repository_owner_id, the entry's repository_id, a ref the entry lists (default refs/heads/main), and a job_workflow_ref naming one of the entry's workflow files. pull_request and pull_request_target events are always refused. The entry names a ServiceAccount, and the server then authorizes and mints exactly as /auth/workload-token does for it, capped at one hour. With no trust list configured the endpoint is not mounted. Why: spec 0079-ci-oidc-tokens.

Executor mints use the same pod-identity pattern with their own audiences (ai-agents-git-mint, ai-agents-mcp-mint). They are bound to an executor pod running an IR, and the scope is derived from that IR's agent, never from the request. A tenant workload cannot use them.

API keys are for machines without pod identity; people use a CLI token (lab ai login) instead. Annotate the key Secret with labrats.work/tenant: <namespace> to attribute the key to that namespace (and put it under that namespace's AiTenant).

Why: spec 0069 (workload token), spec 0070 (model use), spec 0071 (role use), spec 0067 (keyless jobs), spec 0058 and ADR 0006 (git mint), ADR 0007 (MCP mint), ADR 0005 (the reviewer's GitHub App identity).

Tenant resources

What a tenant may define. A namespace may author its own namespaced CRs (AiAgent, AiInstruction, AiTool, AiSkill, AiMcpServer, WebhookSource) under ordinary namespace RBAC; there is no enrollment step. The app repo that owns the namespace owns these CRs. The operator watches them cluster-wide, keyed by namespace/name.

What the platform owns.

• AiModel, AiExecutor and the central AiMcpServers: the shared catalog, referenced by name.

• ClusterAiInstruction: the instruction floor every agent in every namespace inherits; a namespaced instruction cannot mask it.

• The org-wide responder agents (reviewer, triage) in the operator namespace. Webhook events route only to operator-namespace agents, so a tenant agent's eventTriggers are ignored.

• Grants, all in the tenant namespace and bound to the workload ServiceAccount: use on aimodels/<name>, aiagents/<role> and aimcpservers/<server> (apiGroup labrats.work). A tenant MCP server also needs the platform half: the central AiMcpServer must list the namespace in its ai-agents.labrats.work/tenant-namespaces annotation.

• Per-tenant MCP servers (runtime.k8s.bor.gitops spec 0017): a central AiMcpServer annotated mcp.labrats.work/per-tenant: "true" stands for one instance in each user space, with {namespace} in its spec.url. A tenant run whose tenant (the IR's ai-agents.labrats.work/tenant) is user-<id> gets the URL resolved to user-<id> and a minted token with aud <token-audience>.user-<id>; a user-space tenant counts as opted in, so no tenant-namespaces entry is needed. Other tenants, central runs and workload tokens are refused, as is a user-space agent attributed to a different user. The RBAC use aimcpservers/<server> grant is still required.

• Person-bound MCP servers (runtime.k8s.bor.gitops spec 0018): a central AiMcpServer annotated mcp.labrats.work/person-bound: "true" holds personal data. Every token minted for it has aud <token-audience>.user-<id> and the server's catalog scope, and a mint with no person is a 403, never a plain token. The person is:

  Caller	Endpoint	The person
  A tenant run	/internal/mcp-token	the run's recorded tenant, when it is user-<id> (as for per-tenant servers)
  A workload in user-<id>	/auth/workload-mcp-token	the caller's namespace
  An app elsewhere, with X-On-Behalf-Of	/auth/workload-mcp-token	the header's user, after the delegate check

  Both sides are checked at every mint. The person's space grants the server: use aimcpservers/<server> in user-<id> for user-<id>:agent (runs and on-behalf apps) or for the pod's own ServiceAccount (a workload in the space). The actor is bound to it: the run's agent lists it, and an app acting on the person's behalf still holds <service>-delegate in user-<id> (cached 60 s, so a revoke takes effect within a minute, even mid-run). A run's caller must be the person, their space's schedule or workload, or an on-behalf app; the service an app named at submit is recorded on the IR (ai-agents.labrats.work/on-behalf-service) for this check. API-key and webhook callers never act for a person. The usual two-sided grant still applies: the caller's own use grant and the central server's tenant-namespaces opt-in for the agent's or workload's namespace. A server that is also per-tenant keeps the per-tenant rules (no opt-in listing, refused for workloads) plus these checks. Servers without the annotation keep plain audiences.

• AiTenant: cluster-scoped and optional, one per tenant, keyed by spec.namespace (the tenant). Absent, the tenant runs under cluster defaults; it is never a participation gate. It carries priority and concurrency (concurrency and priority), and the admission fields quotas.maxAgents, allowedModels, allowedExecutors and allowedMcpServers.

Admission rules (src/k8s/admission.ts, served at /admission/validate): only the platform identity may write AiTenant or ClusterAiInstruction; in a namespace with an AiTenant, an AiAgent may reference only the allowed models, executors and MCP servers, and may not exceed quotas.maxAgents. These rules are enforced only when the chart value admission.enabled is true (default false); otherwise only cluster RBAC guards AiTenant.

Admission webhook (charts/ai-agents-main, admission.* values; needs cert-manager). The kube-apiserver calls webhooks over HTTPS only, so the API pod gets a second listener (src/admission-tls.ts) that serves only /admission/validate, on admission.port (default 8443, Service port 443). It starts when ADMISSION_TLS_CERT_FILE and ADMISSION_TLS_KEY_FILE are set, which the chart does, and re-reads the files every 5 minutes so a renewed cert is served without a restart. When enabled the chart renders:

• a self-signed Issuer, a CA Certificate and a CA Issuer, then the serving Certificate <fullname>-admission-serving for <fullname>.<namespace>.svc, stored in the Secret <fullname>-admission-tls and mounted at /etc/admission-tls;
• the ValidatingWebhookConfiguration <fullname>-<namespace>-tenant-boundaries (cluster-scoped, so the name carries the namespace and dev and prod do not collide). cert-manager's CA injector fills its caBundle from the serving certificate (cert-manager.io/inject-ca-from). It matches aiagents CREATE/UPDATE in every namespace except kube-system, and aitenants and clusteraiinstructions CREATE/UPDATE/DELETE. failurePolicy (default Ignore, so a webhook outage does not block writes) and timeoutSeconds (default 5) come from values;
• a NetworkPolicy ingress rule on the admission port. The kube-apiserver calls from the host network, which no pod or namespace selector matches, so with admission.apiserverCIDRs empty (the default) the port is open to any source. That port serves only /admission/validate, which has no side effects. Setting apiserverCIDRs restricts it to those ipBlocks; on Cilium an ipBlock matches node addresses only with policy-cidr-match-mode: nodes.

failurePolicy: Ignore only covers webhook errors: a deny is still a deny. Whoever applies AiTenant and ClusterAiInstruction objects must therefore be in the platform identity, or its writes are refused. The defaults are this release's ServiceAccount and the group system:masters; admission.platformIdentity.serviceAccounts, .users and .groups replace them (ADMISSION_PLATFORM_*), so a GitOps controller that syncs these objects is listed there together with the release ServiceAccount. Each enabled release validates every matching request with its own platform identity, so all of them must allow the same writers.

Why: spec 0062.

Acting on behalf of a user

An app may make a /v1 call or submit a job for a user (runtime.k8s.bor.gitops spec 0016 d30, d31). It sends its own workload token plus two headers:

• Check. A SubjectAccessReview: may the app's ServiceAccount use grants.labrats.work/services/<service>-delegate in user-<id>? The user-space chart binds that (ClusterRole grant:<service>-delegate) only while the user holds grant:<service>, so the user's grants decide which apps act for them. Cached 60 s; fails closed.
• Refusals. A half or malformed pair is 400. A caller that is not an app's workload token (API key, user, per-job token, a token without a ServiceAccount) is 403, and so is a missing grant. The headers are never ignored.
• Attribution. The call or job is the user's: tenant user-<id>, so it counts against the user's AiTenant limits and spend. The caller id stays the app's <ns>/<sa> for audit.
• Jobs. The role resolves in user-<id> first, then the app's own namespace, then central. A user-space agent is authorized by the delegate grant (the token's agentRoles claim covers only the app's namespace), and its MCP servers are those the user's grants give their own agent identity (user-<id>:agent), plus the central server's tenant opt-in for user-<id>. submitter-namespace stays the app's namespace, so the app reads its job back as usual.
• MCP tokens. An app may also send the headers to /auth/workload-mcp-token. The delegate check runs on every mint, and a person-bound server's token carries the user's space in its audience (person-bound MCP servers). The headers are verified for any server, but only person-bound servers change the audience.

Job submission

A tenant workload submits with its workload token, no API key:

1. Exchange with {"agentRole": "<role>"}. RBAC use on aiagents/<role> decides.
2. POST /api/jobs/submit with the token as Bearer. body.agentRole must be in the token's agentRoles claim (403 otherwise). A workload submit may set only the public fields, may not use a reserved source (delegation, chain, schedule, github, dashboard), pipeline, config.executorImage or env.
3. Read back with the same token: GET /api/jobs/:id and /status return only IRs this namespace submitted; any other IR is a 404. Uploads (POST /api/workspace-uploads) are readable only by the uploading namespace (or an API key) and referenceable only by its submits.

Which agent runs. The role resolves first in the caller's namespace (a tenant agent), then in the operator namespace (a central agent). A tenant agent with a central agent's name is refused with 409. API-key callers see central agents only.

Where it runs. Every job runs centrally: the IR is created in the operator namespace and the operator runs it there. The IR records agent-namespace, submitter-namespace, submitter-sa and granted-mcp-servers (all ai-agents.labrats.work/). The operator reconciles only IRs in its own namespace; an IR created in a tenant namespace is not run.

What a tenant agent's run gets. Only the MCP servers it lists that the submitting ServiceAccount may use, that the central entry opts its namespace into, and that are mintable. It gets no GitHub token (workspaceSetup.gitRepo is refused), no skills, no static MCP secrets, no model credential (so no claude provider), no executor ServiceAccount token, and runs keyless. A central agent's run keeps its normal credentials.

Why: spec 0071, spec 0062 "Tenant agents".

Direct model calls

A tenant calls the model directly through the proxy's OpenAI-compatible /v1 (/chat/completions, /responses) with its workload token or an API key as Bearer. Who may call which model depends on the credential:

A workload token is recognised by typ: workload or an ns:<namespace> username. Grant use on the alias aimodels/active rather than a concrete AiModel name, so a model swap does not break the grant.

Every proxy call is labelled call="direct" or call="job":

• direct: no irId on the token. Counted, priced and paced per tenant in the proxy.
• job: a per-job token. Labelled with the job caller's tenant; its tokens and cost belong to the job and are counted once, in the ai_ir_* metrics, when the job finishes.

Streaming calls always ask the backend for usage; the proxy strips the extra chunk when the client did not ask for it, so clients see no difference.

Why: spec 0074 §3.

Metering and cost

Metrics (all with a tenant label):

Cost basis. A cost the executor reports (Claude's CLI) is kept (usage.costBasis: provider-reported). Everything else is priced notionally per MTok from the AiConfig spec.pricing table (notional-per-mtok); no matching rate means costUsd: 0 and no basis. The lookup within a provider is AiModel name, then wire model id, then model family, then the provider default, so one AiModel can carry its own price. Direct calls are priced under provider qwen-code, the same key a job's proxy calls use. Cached prompt tokens are priced once at the cache-read rate; cache writes use cacheWritePerMTok, defaulting to the input rate. Prices are configuration in platform.shared, not code.

Where to read it.

• Grafana "AI Agents — Tenant Costs" dashboard (ai-agents-tenant-costs): cost per namespace tenant over time and for the range, jobs versus direct calls, cost by model, tokens, and the waiting panels. It shows namespace tenants only; key:<name>, github, user and unattributed callers are left out.
• Grafana "AI Agents" dashboard, "Usage by tenant" row: the same usage for every tenant, key tenants included.
• Both dashboards are shipped by the release in metrics.grafanaDashboard.namespace (the prod namespace); their $namespace variable picks the environment and $tenant filters tenants.
• GET /api/stats/tenants?range=24h|7d|30d and the "Usage by tenant" table on /stats: jobs and direct calls per tenant from Prometheus, so it outlives IR garbage collection.
• GET /api/jobs?tenant=<t>&callerKind=<k>: jobs by tenant and caller kind.

The counters are per pod, and a tenant's series starts on its first call, so a plain increase() drops that first sample and undercounts tenants with little traffic. The dashboards (Helm helper ai-agents-main.irIncrease) and /api/stats/tenants (podCounterIncrease) add back the value of a series that started inside the range; use the same correction in any new query.

Why: spec 0074 §4–6, spec 0022.

Concurrency and priority

All tenants share the active model. A tenant's AiTenant sets priority (high, normal, low), concurrency.maxInFlight (direct /v1 calls at once, enforced by the proxy's priority wait queue, 429 on timeout) and concurrency.maxJobs (Running jobs at once, enforced at dispatch). Priority orders the proxy queue and shifts the tenant's jobs in WSJF ordering. Tenants without a CR get the chart's tenantLimits defaults (unlimited, normal). Edits apply without a restart.

The operator guide, with the queue algorithm, multi-replica behaviour, examples and metrics, is tenant-concurrency.md.

Why: spec 0075.

Onboarding a tenant

For a workload in a new namespace. Names in angle brackets are placeholders.

1. ServiceAccount and token. In the app's manifests (runtime.k8s.bor.gitops apps/<app>/): a ServiceAccount with automountServiceAccountToken: false, and on the Deployment or CronJob a projected volume serviceAccountToken: { audience: ai-agents-inference, expirationSeconds: 3600, path: token } mounted read-only.
2. App config env. AI_AGENTS_URL=http://ai-agents-main.<ai-agents-ns>.svc.cluster.local (exchange and submit), AI_AGENTS_CHAT_URL=http://ai-agents-main-proxy.<ai-agents-ns>.svc.cluster.local/v1 (direct calls), AI_AGENTS_TOKEN_FILE=/var/run/secrets/ai-agents/token. The kubelet rotates the file, so read it at each exchange and re-exchange before the minted token expires.
3. Grants. A Role and RoleBinding in the tenant namespace, next to the ServiceAccount: use on aimodels/<model> for direct calls, aiagents/<role> for submits, aimcpservers/<server> for a tenant agent's MCP servers. A grant reused across namespaces can be a ClusterRole in platform.shared ai-agents/k8s/infra/role-use-<role>.yaml, bound per namespace. Audit: kubectl auth can-i use aiagents/<role> --as=system:serviceaccount:<ns>:<sa> -n <ns>.
4. Network. Label the tenant's own Namespace with the grant for each surface it calls: ai-agents.labrats.work/<env>-proxy: "true" (direct calls) and ai-agents.labrats.work/<env>-api: "true" (exchange and submit). The env's values admit namespaces by these labels (proxy.allowedNamespaceSelectors, networkPolicy.allowedNamespaceSelectors); nothing changes in the ai-agents release.
5. Tenant agent (optional). An AiAgent in the tenant namespace, owned by the app's repo, with a name no central agent uses. For each MCP server it lists, the platform adds the namespace to the central AiMcpServer's ai-agents.labrats.work/tenant-namespaces annotation (platform.shared ai-agents/k8s/base/mcp-servers/).
6. Limits (optional). An AiTenant in platform.shared ai-agents/k8s/infra/aitenant-<tenant>.yaml with spec.namespace: <ns>, priority and concurrency.
7. Verify. The exchange response lists the granted models and agentRoles. Usage appears under the namespace as tenant on /stats ("Usage by tenant"), in the Grafana "Usage by tenant" row, and in GET /api/jobs?tenant=<ns>.

A client without pod identity uses an API key Secret instead of steps 1–3, annotated labrats.work/tenant: <ns>.

Glossary

• Tenant: the attribution and policy key of a caller; a namespace for workload tokens.
• Caller: kind, id and tenant, resolved once per request and recorded on each job.
• Workload token: the short-lived token /auth/workload-token mints from a pod's projected ServiceAccount token, scoped by RBAC use grants.
• Per-job token: the token the operator mints for one IR's executor; its calls are call="job".
• Direct call: a /v1 call not made with a per-job token.
• Tenant agent: an AiAgent in a tenant namespace, run centrally without central credentials.
• Central agent: an AiAgent in the operator namespace.
• AiTenant: the optional, platform-owned, cluster-scoped policy for one tenant.
• Grant: a namespace RoleBinding giving a ServiceAccount verb use on an AiModel, AiAgent or AiMcpServer by name.