Observability

Overview

inPoint.Cortex is instrumented for observability on the three OpenTelemetry signals — logs, traces and metrics. The services and background workers that make up inPoint.Cortex emit this telemetry and export it over the OpenTelemetry Protocol (OTLP). When inPoint.Cortex is launched through its orchestrator the signals are collected by the built-in Aspire dashboard; in production they can be pointed at any OTLP-compatible collector or backend.

OpenTelemetry is already in use across several areas of inPoint.Cortex — the identity server, the web server, the database migration, and the archive jobs — and coverage is being extended to further components (for example core and hybrid store) over time.

This chapter is split in two parts:

• Logging — how log entries are produced, enriched and exported, and how exceptions are recorded.
• OpenTelemetry — the traces/spans each component emits and the attributes they carry, the full catalogue of metrics, and how the OTLP exporter is configured.

All telemetry export is opt-in and driven entirely by configuration: unless an OTLP endpoint is configured, each component still runs and logs locally but exports nothing.

---

Logging

How logs are produced

Every inPoint.Cortex component writes structured log entries. Log output always goes to the component's console/log file, and — when an OTLP endpoint is configured (see OTLP exporter configuration) — is additionally exported to the configured collector (the Aspire dashboard by default).

The default minimum log level and any per-category overrides are controlled from each component's appsettings*.json logging section, so administrators can raise or lower verbosity without code changes.

Contextual enrichment

Logs are enriched with contextual properties so that related entries can be correlated. In particular, every entry written while a background job runs carries the job's identity:

Property	Meaning
Job.Id	The job instance id.
Job.Name	The job short name.

These properties are attached automatically to all entries emitted during a job run, including those from the hosted job itself.

How exceptions are logged

Exceptions are captured as structured data: the exception type, message and full stack trace are recorded alongside the log message rather than flattened into text. The same failure is also recorded on the active trace span (see Exceptions on spans) and, for background jobs, reflected in the job metrics via the result/error tags, so a single failure is visible in logs, traces and metrics.

---

OpenTelemetry

Tracing

Traces follow a request or a job through the system as a tree of spans. Where work crosses a process boundary (for example when a service starts a background worker), the trace context is propagated so that the whole operation appears as one connected distributed trace.

The custom spans are grouped below by the component that produces them. In addition to these, each web-facing component automatically produces standard spans for incoming HTTP requests and outgoing HTTP calls.

Identity server (inPoint.Web.IdentityServer)

Spans covering authentication and token issuance.

Span	Emitted when
identity.login.password	A username/password sign-in is attempted.
identity.login.signin	A user is signed in.
identity.login.cookie_signin	The sign-in cookie is issued.
identity.validate_password	A user's password is validated.
identity.validate_domain_credentials	Domain (directory) credentials are validated.
identity.mfa.enable_verify	A multi-factor enrolment code is verified.
identity.mfa.login_verify	A multi-factor login code is verified.
identity.external_login.callback	An external identity provider callback is handled.
identity.token.get_profile_data	Profile/claims data is assembled for a token.
identity.token.is_active	A token's active state is checked.

Attributes seen on these spans include: identity.user_id, identity.username, identity.login_method, identity.identity_provider, identity.login_provider, identity.login_outcome, identity.password_valid, identity.mfa_code_valid, identity.domain, identity.account, identity.remember_login, identity.machine_name, identity.external_provider, identity.provider_user_id, identity.client_id, identity.is_active, identity.issued_claims_count.

In addition to the custom spans above, the identity server subscribes to the activity sources published by the underlying Open.IdentityServer engine, giving fine-grained visibility into protocol request processing. Each span is named after the class and method performing the operation (for example TokenEndpoint.ProcessAsync or ClientSecretValidator.ValidateAsync). The activity sources are:

Activity source	Covers
Open.IdentityServer	Top-level protocol request processing (token, authorize, etc.).
Open.IdentityServer.Cache	Caching operations (reads, writes, invalidations).
Open.IdentityServer.Services	Internal service operations (token creation, consent, key material).
Open.IdentityServer.Stores	Data store operations (reading/writing grants, clients, resources).
Open.IdentityServer.Validation	Validation operations (token, request and secret validation).

Web server (inPoint.Web.Server)

The web server produces the standard incoming-request and outgoing-call spans. Server spans for the same route are given a readable, per-operation name, and calls to the enterprise back end are tagged with the specific operation being invoked so that otherwise identical requests can be told apart.

Database migration (inPoint.Setup.DbMigration)

Spans covering database provisioning and script execution during setup/upgrade.

Span	Emitted when
database-migration	A migration run starts.
provision-database	The database is provisioned.
create-archive	An archive is created.
provision-hybridstore	The hybrid store is provisioned.
write-ldap-settings	Directory (LDAP) settings are written.
run-database-scripts	The database script phase starts.
execute-script <file>	An individual database script is executed.
pco-import / run-pco-loader	A PCO import / loader run is executed.

Attributes seen on these spans include: migration.schema, migration.db_type, migration.db_version, migration.phase, migration.skipped, db.script.type, db.script.number, db.script.filename, and the PCO loader's pco.loader.config / pco.loader.exit_code.

Archive jobs (inPoint.Jobs)

Jobs run across two processes: the archive service schedules and spawns jobs, and an out-of-process worker executes them. The trace context is propagated to the worker so both sides form a single distributed trace.

Span	Process	Emitted when
provision-jobs	Service	A provisioning run starts (discover + create).
provision-job	Service	Once per job being provisioned.
spawn-worker	Service	A worker process is spawned and connected.
run-job	Service	Once per scheduled run of a remote job.
execute-job	Worker	The worker executes the hosted job.

Attributes seen on these spans include: job.id, job.name, job.class, job.assembly_file, job.provisioned, job.skip_reason, job.remote, job.finished, worker.framework, worker.pipe, and jobs.app_base. Individual jobs may add their own nested spans for the phases of work they perform.

Exceptions on spans

When an operation fails, the failure is recorded on its span: the span is marked with an error status and carries the exception type (exception.type) and message (exception.message). Successful operations are marked with an OK status.

Metrics

Metrics are reported across the components listed above. The table below is the complete catalogue of the custom metrics currently emitted. (Each component also reports standard runtime, incoming-request and outgoing-call metrics automatically.)

Metric	Type	Unit	Tags	Description
inPoint.jobs.execution.duration	Histogram	s	inPoint.jobs.name, inPoint.jobs.remote, inPoint.jobs.result, error.type	Job run duration. Its count is the number of executions (there is no separate execution counter).
inPoint.jobs.execution.active	Up/down counter	{job}	inPoint.jobs.name, inPoint.jobs.remote	Jobs running right now.
inPoint.jobs.restart	Counter	{restart}	inPoint.jobs.name	Worker restarts triggered after a job run failed (lost connection / exited process).
inPoint.audits	Counter	{audit}	inPoint.audit.source, inPoint.audit.action	Audit entries recorded, tagged by audit source and action (see Auditing). Recorded on the producer side as entries are enqueued.

error.type is present only when a run actually threw; it carries the exception's type name.

The identity server additionally subscribes to the Open.IdentityServer meter published by the underlying engine, which reports counters for token issuance and validation activity. These instruments are tagged with values such as result (success / error / internal_error), client, grant_type, endpoint, auth_method and error:

Metric	Type	Description
tokenservice.operation	Counter	General token-service operation outcomes.
tokenservice.active_requests	Counter	Token-service requests in flight.
tokenservice.token_issued	Counter	Tokens issued, tagged by grant type and client.
tokenservice.introspection	Counter	Token introspection calls.
tokenservice.revocation	Counter	Token revocation calls.
tokenservice.client.secret_validation	Counter	Client secret validations.
tokenservice.client.config_validation	Counter	Client configuration validations.
tokenservice.api.secret_validation	Counter	API secret validations.
tokenservice.resourceowner_authentication	Counter	Resource-owner (password) authentications.
tokenservice.device_authentication	Counter	Device authentications.
tokenservice.backchannel_authentication	Counter	Back-channel (CIBA) authentications.
tokenservice.pushed_authorization_request	Counter	Pushed authorization requests (PAR).
tokenservice.dynamic_identityprovider.validation	Counter	Dynamic identity-provider validations.

Tag values

• inPoint.jobs.name — the job short name.
• inPoint.jobs.remote — true when the job ran out-of-process (remote worker), false for an in-process run.
• inPoint.jobs.result — succeeded, failed or cancelled.

Reading the numbers

• There is no separate execution counter: the number of executions is the count of inPoint.jobs.execution.duration. Group it by inPoint.jobs.result for per-result totals.
• inPoint.jobs.execution.active goes up when a run starts and down when it ends; it carries only the job name and remote flag.

---

OTLP exporter configuration

Telemetry export is controlled entirely through environment variables / configuration. The presence of an OTLP endpoint is the switch that turns exporting on: when no endpoint is configured, each component runs normally and logs locally but exports nothing.

Default configuration in inPoint.Cortex (Aspire dashboard)

When inPoint.Cortex is launched through its orchestrator, the Aspire dashboard is started automatically and the required OpenTelemetry environment variables are injected into each component. The relevant variables are:

Variable	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	The OTLP collector endpoint (the Aspire dashboard). Its presence turns exporting on.
OTEL_EXPORTER_OTLP_HEADERS	Any headers required by the collector (for example an API key for the dashboard).
OTEL_EXPORTER_OTLP_PROTOCOL	The OTLP protocol to use.
OTEL_SERVICE_NAME	The logical component name shown in the dashboard.

No manual configuration is required for the default inPoint.Cortex experience — logs, traces and metrics from all instrumented components appear in the Aspire dashboard out of the box.

Configuring a custom OTLP target

To send telemetry to your own collector or observability backend instead of (or in addition to) the Aspire dashboard, set the standard OpenTelemetry environment variables on each component, for example:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.example.com:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_HEADERS=<header-name>=<value>
OTEL_SERVICE_NAME=<component name>

Because export is gated purely on OTEL_EXPORTER_OTLP_ENDPOINT, pointing that variable at a different host is all that is required to redirect the telemetry; the same variables apply to all three signals (logs, traces and metrics).