Skip to main content

Audit trail

The AI App Store can report audit events to the H2O Audit Trail service, recording who did what, where, and when across the App Store API. Events are delivered to the Audit Trail service over gRPC and can be searched by administrators through the Audit Trail API or its Python client.

What is audited​

One audit event is recorded per API request for all user-facing and admin operations on:

  • Apps (import, get, list, update, delete, run, attribute and configuration changes) and app bundles (upload, download)
  • App instances (get, list, update, terminate, suspend/resume, auto-suspend changes, redeploy, log access)
  • Secrets, tags, and aliases (full CRUD plus assign/promote operations)
  • IAM policies and admin user operations (logout, auth status)
  • Federation (admin listing and refresh of federated apps)
  • Credential issuance (CLI and platform token retrieval) and visitor-mode session changes
  • Rejected admin access: an authenticated non-administrator turned away by the admin API guard is recorded as actions/appstore/api/ADMIN_ACCESS_DENIED, with the attempted route as the resource

Admin API operations are recorded with distinct action names (an ADMIN_ prefix, e.g. actions/appstore/apps/ADMIN_DELETE) so that you can distinguish user from administrator activity.

Each event carries the following:

  • the action
  • the affected resource (e.g., //appstore/apps/{id} — App Store resources are not workspace-bound, so resource names carry no workspace segment)
  • the acting principal (users/{id} or services/{id})
  • the request outcome (a gRPC status code — failed and denied requests are audited too)
  • the source IP address
  • the user agent

When an authorization check denies a request and the request fails as a result, the event additionally carries authorization-decision metadata: authz_denied_check names the check that denied it (e.g. CanWriteSecret), and authz_checks summarizes the allow/deny counts of the checks performed by the request. Successful requests carry no such metadata, because many checks shape a response rather than gate it and routinely deny without affecting the outcome.

The following events are deliberately not audited:

  • authentication checks (e.g., CheckAuth)
  • environment configuration reads
  • OIDC login/logout, and the token-to-session exchange at /auth/session (logins are recorded platform-side by the Audit Trail Keycloak collector; the issuance of the token itself is audited)
  • static asset downloads
  • requests rejected before authentication, which have no principal to attribute them to
Source IP is best-effort

source_ip_address is taken from the X-Real-IP / X-Forwarded-For headers and falls back to the peer address. It is intended for operational context, not forensic attribution: a caller that can reach the App Store directly can set those headers itself. The action, resource, principal and outcome on the event are not client-controlled.

Configuration​

Audit trail reporting is disabled by default. When disabled, no audit events are produced — records are built per request but dropped rather than reported.

Prerequisite: service account allow-list​

The App Store authenticates to the Audit Trail service with its Kubernetes service account token. The App Store's service account must be added to the Audit Trail server's serviceAuth.serviceAccounts allow-list; otherwise event delivery is rejected as unauthenticated.

Helm values​

Helm values (under config):

ValueDefaultDescription
auditTrailEnabledfalseReport events to the Audit Trail service; when false, events are dropped.
auditTrailAddress—gRPC address of the Audit Trail service (in-cluster). Required when enabled.
auditTrailSourceImageNameh2oai-appstore-serverContainer image name reported as the event source.
auditTrailMaxBufferSize10000Capacity of the pending event buffer; when full, new events are dropped rather than blocking requests.

Additional server configuration options (TOML [AuditTrail] section / environment):

  • BatchSize (default 50 events per report),
  • BatchTimeoutMillis (default 5000 — how long a partial batch is held),
  • ReportTimeoutMillis (default 1200000 — how long one batch delivery may retry)
  • ServiceAccountTokenPath.

Delivery semantics​

Event delivery is asynchronous and never blocks or fails user requests:

  • Events are buffered in memory and reported in batches. Batching and delivery run on separate goroutines, so a batch retrying against an unavailable service does not stop new events from being collected.
  • Failed deliveries are retried (only the rejected part of a batch is re-sent); each event carries an idempotency key so retries cannot create duplicates. Events the service rejects permanently (validation failures, or a batch rejected as over-sized) are dropped and logged instead of being retried.
  • If the buffer fills (for example while the Audit Trail service is unavailable during a traffic burst), new events are dropped and a warning is logged. The number of batches awaiting delivery is likewise bounded; once that limit is reached whole batches are dropped with a warning rather than growing memory without limit.
  • On shutdown the buffer is flushed on a best-effort basis; events buffered at the moment of a hard crash are lost.

Audit trail delivery is therefore best-effort by design and should be treated as an operational record, not a transactional ledger.

Feedback