Tenant concurrency and priority

Every tenant shares the one active GX10 model. A tenant's AiTenant sets how much of it the tenant may use at once and how its waiting work is ordered against other tenants (spec 0075-tenant-concurrency-priority).

The setting

AiTenant is cluster-scoped and platform-owned (spec 0062-namespaced-tenant-crs): a tenant cannot raise its own limit or priority. spec.namespace names the tenant, as resolved in tenancy. Annotate an API key Secret with labrats.work/tenant: <namespace> to put the key under that namespace's AiTenant.

Field	Effect	Default (no CR or field omitted)
`spec.priority`	`high`, `normal` or `low`. Orders the proxy wait queue and shifts the tenant's job priority in WSJF ordering (high −1, low +1).	`tenantLimits.defaultPriority` (`normal`)
`spec.concurrency.maxInFlight`	Direct `/v1` calls in flight at once.	`tenantLimits.defaultMaxInFlight` (unlimited)
`spec.concurrency.maxJobs`	Running jobs at once (a parked job doesn't count).	`tenantLimits.defaultMaxJobs` (unlimited)

tenantLimits.modelMaxInFlight caps direct calls per model across all tenants. All tenantLimits values are chart values (charts/ai-agents-main/values.yaml).

Edits apply without a restart: the proxy and the operator watch AiTenant.

What a caller sees

Direct /v1 call over the limit: the proxy holds the request. When a slot frees, the waiting call with the highest score goes first: priority weight (high 2, normal 1, low 0.5) × (1 + seconds waited / 30), ties to the higher class, then first come first served. After tenantLimits.queueTimeoutMs (60 s) without a slot, the call gets 429 with Retry-After and error.type: rate_limit_exceeded. A client that disconnects while waiting leaves the queue.
Calls made inside a job (per-job token) are not held by the proxy. The job was admitted at dispatch under maxJobs and already holds a model slot.
Job over maxJobs: it stays Pending and is dispatched when one of the tenant's jobs finishes.

Examples

Examples only. CRs live in platform.shared ai-agents/k8s/infra/aitenant-<tenant>.yaml.

# my-diet admin: AI review batches (titles-only, 3 concurrent). Batch work, so
# it yields to interactive tenants when the model is busy.
apiVersion: labrats.work/v1alpha1
kind: AiTenant
metadata:
  name: my-diet-admin
spec:
  namespace: my-diet-admin
  priority: low
  concurrency:
    maxInFlight: 3
---
# my-diet prod: user-facing agents and nightly jobs.
apiVersion: labrats.work/v1alpha1
kind: AiTenant
metadata:
  name: my-diet-prod
spec:
  namespace: my-diet-prod
  priority: normal
  concurrency:
    maxInFlight: 2
    maxJobs: 2

Multiple proxy replicas

Limiter state is in each proxy pod. With proxy.replicaCount replicas, each pod enforces ceil(limit / replicas), so the cluster-wide limit is approximate (it can admit up to replicas × ceil(limit / replicas)), and priority ordering holds within a pod. Limits are exact with one replica.

Metrics

Metric	Meaning
`ai_proxy_queue_depth{tenant}`	Direct calls waiting
`ai_proxy_in_flight{tenant}`	Direct calls admitted
`ai_proxy_queue_wait_seconds{tenant,outcome}`	Wait before admission (`admitted`), 429 (`timeout`) or disconnect (`aborted`)
`ai_proxy_requests_total{status="throttled"}`	Direct calls refused with 429
`ai_ir_tenant_held{tenant}`	Pending jobs held by `maxJobs`

The Grafana "AI Agents" dashboard shows them in the "Usage by tenant" row (Waiting by Tenant, Queue Wait by Tenant).