Observability UI + Prometheus + Grafana

Start Nexora's built-in observability server:

java -jar nexora-cli/target/nexora.jar observe --port 9464

The server's full HTTP API is described by an OpenAPI 3.1 document at GET /openapi.yaml, browsable with Swagger UI at /docs (served from bundled assets, no external requests). The document's info.version is the API's own version, separate from the Nexora release; the HTTP API page lists what changed in each version and renders the current reference.

It exposes these endpoints with no external dependencies:

Endpoint	What it serves
GET /	Live process UI showing active executions, step timelines, and plan amendments
GET /metrics	Prometheus text format scrape endpoint
GET /openapi.yaml	OpenAPI 3.1 description of every endpoint
GET /docs	Swagger UI for that document
GET /api/process	Raw process snapshot as JSON
POST /api/execute	Trigger an execution remotely; returns 202 with the new executionId
GET /health/live	Liveness probe. Returns 200 while the process is serving HTTP
GET /health/ready	Readiness probe. Returns 200 when persistence, plugins, and the executor are all UP, 503 with a per-check breakdown otherwise
GET /health	Summary: overall status, Nexora version, readiness checks, and capability circuit states (informational)
GET /api/webhook-deliveries/{id}	Audit log of webhook delivery attempts for an execution
GET /api/dead-letters	List dead letter queue entries (paginated via ?page= and ?size=, filterable by ?state=PENDING|RESOLVED|REPLAYED|ALL)
POST /api/dead-letters/{id}/replay	Create a new execution from a dead letter
POST /api/dead-letters/{id}/resolve	Mark a dead letter as resolved
GET /api/executions	Execution history, newest first, cursor-paginated (see Execution History API); filter by status, goal, from, to, limit
GET /api/executions/{id}	One execution with its step records (404 if unknown)
DELETE /api/executions/{id}	Cancel a running execution (200 cancelled, 404 unknown, 409 already finished). See Execution Cancellation
POST /api/executions/{id}/replay	Replay a failed, timed-out, cancelled or compensated execution (202 with the new executionId, 409 otherwise). See Execution Replay
GET /api/schedules	List all cron schedules
POST /api/schedules	Register a new cron schedule
DELETE /api/schedules/{id}	Cancel a cron schedule

Note: The /api/webhook-deliveries, /api/dead-letters, and /api/schedules endpoints were added in v0.2.0. /health/live and /health were added in v0.3.0, when /health/ready changed from capability circuit state to dependency readiness (see Health Checks).

Example execute request with webhook callback:

curl -H "Authorization: Bearer $NEXORA_API_KEY" -X POST http://localhost:9464/api/execute \
  -H "content-type: application/json" \
  -d '{
        "goal": "process order payment notification",
        "context": {"orderId": "ORD-99"},
        "webhookUrl": "https://your-api.com/webhooks/nexora",
        "webhookEvents": ["COMPLETED", "FAILED", "TIMED_OUT"]
      }'

Authentication

Every /api/* route and the live WebSocket require a key, sent as Authorization: Bearer <key>. /metrics, /health/*, the dashboard page, /openapi.yaml and /docs stay public for scrapers and probes.

• Server key (nxk_...): printed once on first start (only its hash is saved), or set with NEXORA_API_KEY for CI and containers. Rotate with --rotate-api-key.
• User key (nxu_...): minted when someone signs in with a username and password. The dashboard gets it as an HttpOnly cookie; nexora login saves it for nexora cancel / nexora replay. Valid 12 hours, logged under the user's name. Manage users with nexora user add/passwd/remove/list.

export NEXORA_API_KEY="$(openssl rand -base64 32)"
nexora observe
curl -H "Authorization: Bearer $NEXORA_API_KEY" http://localhost:9464/api/process

The payment pipeline demo prints a fresh server key in its console on every start and has no user accounts, so paste that key into its sign-in card. To try username and password sign-in, run the login demo.

See Authentication for how each sign-in flow works, user management, the demo, and what 401 and 429 mean.

Health Checks

The three health endpoints need no authentication, so kubelet probes and load balancers can call them directly.

curl http://localhost:9464/health/ready
# 200 {"status":"UP","checks":{"persistence":"UP","migrations":"UP","plugins":"UP","executor":"UP"}}
# 503 {"status":"DOWN","checks":{"persistence":"DOWN","migrations":"DOWN","plugins":"UP","executor":"UP"}}

Check	DOWN when
persistence	The execution store connection fails validation (2s timeout). Reported UP when persistence is disabled.
migrations	A schema migration is recorded as failed in nexora_schema_history, or the database schema is older than this release expects (for example after restoring an old backup). Reported UP when persistence is disabled.
plugins	Any registered plugin is not ACTIVE (never activated, stuck initializing, or failed).
executor	The engine's executor has been shut down.

Capability circuit breaker state does not affect readiness. An OPEN circuit means a downstream dependency is failing, and taking every instance out of rotation for that would turn a partial outage into a full one. Circuit states are reported under capabilities in GET /health.

Kubernetes probe configuration:

livenessProbe:
  httpGet:
    path: /health/live
    port: 9464
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /health/ready
    port: 9464
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2

The nexora_ready gauge mirrors /health/ready (1 = ready, 0 = not ready) and drives the NexoraNotReady alert in observability/prometheus/alerts.yml.

Execution History API

Read-only access to past and running executions, backed by the persistence store (executionStore in nexora.json, or NEXORA_JDBC_URL). Without a store these routes return 503.

Authentication: these endpoints return execution goals and context, so they require the API key. See Authentication.

List executions

curl -H "Authorization: Bearer $NEXORA_API_KEY" "http://localhost:9464/api/executions?status=FAILED,CANCELLED&limit=20"

{
  "data": [
    {
      "executionId": "4f3a1c2d-...",
      "traceId": "9b1e...",
      "goal": "process payment",
      "state": "FAILED",
      "startedAt": "2026-10-01T10:00:00.123Z",
      "completedAt": "2026-10-01T10:00:01.456Z",
      "durationMs": 1333
    }
  ],
  "cursor": "MjAyNi0xMC0wMVQxMDowMDowMC4xMjNafDRmM2ExYzJkLS4uLg",
  "total": 42
}

Parameter	Description
status	One or more states, comma-separated: RUNNING, COMPLETED, FAILED, TIMED_OUT, CANCELLED, COMPENSATING, COMPENSATED
goal	Case-insensitive substring of the goal (max 200 characters)
from	Inclusive lower bound on startedAt, ISO-8601 with an offset (2026-10-01T10:00:00Z). Encode + as %2B: 2026-10-01T15:30:00%2B05:30
to	Exclusive upper bound on startedAt, same format as from
limit	Page size, 1 to 100 (default 20)
cursor	The cursor value from the previous page

Results are ordered by startedAt descending. Pass the returned cursor to get the next page; it is null on the last page. total counts every execution matching the filters, not just this page.

Pagination is keyset-based (the cursor encodes startedAt and executionId), so pages stay consistent while new executions are written. Keep the same filters when following a cursor. Invalid parameters return 400 with an error message.

List rows are summaries. Use the detail route for context and steps.

The observe UI has an Execution History panel built on these routes: filter by status or goal, page with Load more, and click a row to see its steps.

Get one execution

curl -H "Authorization: Bearer $NEXORA_API_KEY" http://localhost:9464/api/executions/4f3a1c2d-...

Returns the execution with its context and steps: state, timings, failure code and message, and output for each step. Unknown ids return 404; if the store cannot be queried the response is 500, never a misleading 404.

output is what the step returned, as a JSON object: a map appears as is, and any other value appears under value. It is {} for steps that returned nothing, have not completed, or failed. Outputs over persistence.stepOutputMaxBytes (64 KB by default) have outputTruncated: true and keep only a preview with the start of the JSON text:

{ "stepId": "charge", "state": "COMPLETED", "output": { "paymentId": "p-1", "status": "CAPTURED" }, "outputTruncated": false }
{ "stepId": "export", "state": "COMPLETED", "output": { "preview": "{\"rows\":[{\"id\":1,..." }, "outputTruncated": true }

The live snapshot (GET /api/process and the WebSocket stream) carries the same output and outputTruncated fields, capped at 4 KB per step because the whole snapshot is resent on every event. The dashboard shows each output under its step; the History panel shows the stored one.

Outputs can contain business data, and anyone with an API key or a dashboard session can read them. Set stepOutputMaxBytes to 0 to record none.

---

Webhook Callbacks

Nexora allows you to register webhook URLs to be notified asynchronously when an execution reaches a terminal state (COMPLETED, FAILED, or TIMED_OUT). This is particularly useful when triggering executions remotely via the API and awaiting their outcome.

To use webhooks securely, configure an HMAC-SHA256 signature secret in your nexora.json or via the NEXORA_WEBHOOK_SECRET environment variable:

{
  "webhookSecret": "your-secure-secret-here",
  "steps": []
}

Nexora will dispatch a JSON payload to your endpoint with the execution outcome. It signs the payload using the configured secret and passes the signature in the nexora-signature HTTP header for validation. Delivery attempts are persisted in the nexora_webhook_deliveries database table and can be queried for auditability via the API.

Cron Schedule API

Requires executionStore to be configured (see CLI Reference). The dashboard includes a live Cron Schedules panel that auto-refreshes every 30 seconds.

# list all schedules
curl -H "Authorization: Bearer $NEXORA_API_KEY" http://localhost:9464/api/schedules

# register a nightly schedule
curl -H "Authorization: Bearer $NEXORA_API_KEY" -X POST http://localhost:9464/api/schedules \
  -H "content-type: application/json" \
  -d '{"cronExpression":"0 0 * * *","goal":"nightly cleanup","missedFirePolicy":"FIRE_ONCE"}'

# cancel a schedule
curl -H "Authorization: Bearer $NEXORA_API_KEY" -X DELETE http://localhost:9464/api/schedules/<id>

See Cron Scheduling for the full reference including missed-fire policies and events.

---

Dead Letter Queue API

Permanently failed executions are captured in the dead letter queue. See Dead Letter Queue for full documentation.

# list PENDING (default)
curl -H "Authorization: Bearer $NEXORA_API_KEY" http://localhost:9464/api/dead-letters

# list all states with pagination
curl -H "Authorization: Bearer $NEXORA_API_KEY" "http://localhost:9464/api/dead-letters?state=ALL&page=0&size=10"

# filter by state
curl -H "Authorization: Bearer $NEXORA_API_KEY" "http://localhost:9464/api/dead-letters?state=RESOLVED&page=0&size=20"

# replay a dead letter (creates a new execution from the original goal + context)
curl -H "Authorization: Bearer $NEXORA_API_KEY" -X POST http://localhost:9464/api/dead-letters/<id>/replay

# resolve with a reason
curl -H "Authorization: Bearer $NEXORA_API_KEY" -X POST http://localhost:9464/api/dead-letters/<id>/resolve \
  -H "content-type: application/json" \
  -d '{"reason":"investigated and closed"}'

# resolve without a reason (reason field is optional)
curl -H "Authorization: Bearer $NEXORA_API_KEY" -X POST http://localhost:9464/api/dead-letters/<id>/resolve

Metrics exposed include

• nexora_plan_started_total, nexora_plan_completed_total, nexora_plan_failed_total: by tenant
• nexora_plan_duration_seconds: histogram by status (completed/failed/timed_out/cancelled) and tenant
• nexora_plan_cancelled_total: cancelled executions by tenant (not counted as failures)
• nexora_step_started_total, nexora_step_completed_total, nexora_step_failed_total: all by capability ID and tenant
• nexora_step_duration_seconds: histogram by capability ID and terminal status
• nexora_plan_amendments_total: by amendment type (ADD_STEP, SKIP_STEP, MODIFY_INPUT)
• nexora_active_executions: current in-flight count by tenant (sum(nexora_active_executions) for the total)

The tenant label is default on a single-tenant deployment, so totals are unchanged; aggregate with sum(...) or sum by (tenant). Step durations are not split by tenant, since every label multiplies the histogram buckets.

• nexora_ready: 1 when /health/ready reports UP, 0 otherwise
• nexora_rate_limit_violations_total: requests refused with 429, by route and client IP (see Rate limiting)

Bring up Prometheus, Grafana, and Alertmanager

cd observability
docker compose up -d

• Prometheus: http://localhost:9090
• Alertmanager: http://localhost:9093
• Grafana: http://localhost:3000 (login: admin / admin)

The Grafana dashboard and alert rules are provisioned automatically. Provisioned assets:

• observability/prometheus/prometheus.yml
• observability/prometheus/alerts.yml
• observability/grafana/dashboards/nexora-overview.json

Attach observability directly to engine

You can also attach observability to an engine in your own application without the HTTP server:

NexoraEngine engine = NexoraEngine.builder()...build();

try (NexoraObservability obs = NexoraObservability.attach(engine)) {
    engine.execute("process order", context).get();
    String metrics = obs.scrapePrometheus();           // Prometheus text format
    ProcessSnapshot snapshot = obs.processSnapshot();  // live execution state
}