Sign In

OpenTelemetry Support

This document details Sentry's work in integrating and supporting OpenTelemetry, the open standard for metrics, traces and logs. In particular, it focuses on the integration between Sentry's performance monitoring product and OpenTelemetry's tracing spec.

Background

When Sentry performance monitoring was initially introduced, OpenTelemetry was in early stages. This lead to us adopt a slightly different model from OpenTelemetry, notably we have this concept of transactions that OpenTelemetry does not have. We've described this, and some more historical background, in our performance monitoring research document.

TODO: Add history about OpenTelemetry Sentry Exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/sentryexporter

Approach

TODO: Talk about the approach we are using, based on Matt's hackweek project - https://github.com/getsentry/sentry-ruby/pull/1876

OpenTelemetry Context

TODO: Do we add OpenTelemetry Event Context?

Transaction Protocol

TODO: Talk about generating transactions

Span Protocol

Below describe the transformations between an OpenTelemetry span and a Sentry Span. Related: the interface for a Sentry Span, the Relay spec for a Sentry Span and the spec for an OpenTelemetry span.

This is based on a mapping done as part of work on the OpenTelemetry Sentry Exporter.

OpenTelemetry SpanSentry SpanNotes
trace_idtrace_id
span_idspan_id
parent_span_idparent_span_idIf a span does not have a parent span ID, it is a root span. For a root span:
  • If there is an active Sentry transaction, add it to the transaction
  • If there is no active Sentry transaction, construct a new transaction from that span
    • name,attributes,kinddescriptionThe span description is decided using OpenTelemetry Semantic Conventions. Generally, the OpenTelemetrynamemaps to a Sentrydescription
    name,attributes,kindop
    attributes,kind,statustagsThe OpenTelemetry Span Status message and span kind are set as tags on the Sentry span.
    attributes,statusstatusSee Span Status for more details
    start_time_unix_nanostart_timestamp
    end_time_unix_nanotimestamp
    eventSee Span Events for more details

    Span Status

    In OpenTelemetry, Span Status is an enum of 3 values, while Sentry's Span Status is an enum of 17 values that map to the GRPC status codes. Each of the Sentry Span Status codes also map to HTTP codes. Sentry adopted it's Span Status spec from OpenTelemetry, who used the GRPC status code spec, but later on changed to the current spec it uses today.

    To map from OpenTelemetry Span Status to, you need to rely on both OpenTelemetry Span Status and Span attributes. This approach was adapted from a PR by GH user @anguisa to the OpenTelemetry Sentry Exporter.

    Copied
    // OpenTelemetry span status can be Unset, Ok, Error. HTTP and Grpc codes contained in tags can make it more detailed.

// canonicalCodesHTTPMap maps some HTTP codes to Sentry's span statuses. See possible mapping in https://develop.sentry.dev/sdk/event-payloads/span/
var canonicalCodesHTTPMap = map[string]sentry.SpanStatus{
	"400": sentry.SpanStatusFailedPrecondition, // SpanStatusInvalidArgument, SpanStatusOutOfRange
	"401": sentry.SpanStatusUnauthenticated,
	"403": sentry.SpanStatusPermissionDenied,
	"404": sentry.SpanStatusNotFound,
	"409": sentry.SpanStatusAborted, // SpanStatusAlreadyExists
	"429": sentry.SpanStatusResourceExhausted,
	"499": sentry.SpanStatusCanceled,
	"500": sentry.SpanStatusInternalError, // SpanStatusDataLoss, SpanStatusUnknown
	"501": sentry.SpanStatusUnimplemented,
	"503": sentry.SpanStatusUnavailable,
	"504": sentry.SpanStatusDeadlineExceeded,
}

// canonicalCodesGrpcMap maps some GRPC codes to Sentry's span statuses. See description in grpc documentation.
var canonicalCodesGrpcMap = map[string]sentry.SpanStatus{
	"1":  sentry.SpanStatusCanceled,
	"2":  sentry.SpanStatusUnknown,
	"3":  sentry.SpanStatusInvalidArgument,
	"4":  sentry.SpanStatusDeadlineExceeded,
	"5":  sentry.SpanStatusNotFound,
	"6":  sentry.SpanStatusAlreadyExists,
	"7":  sentry.SpanStatusPermissionDenied,
	"8":  sentry.SpanStatusResourceExhausted,
	"9":  sentry.SpanStatusFailedPrecondition,
	"10": sentry.SpanStatusAborted,
	"11": sentry.SpanStatusOutOfRange,
	"12": sentry.SpanStatusUnimplemented,
	"13": sentry.SpanStatusInternalError,
	"14": sentry.SpanStatusUnavailable,
	"15": sentry.SpanStatusDataLoss,
	"16": sentry.SpanStatusUnauthenticated,
}

code := spanStatus.Code()
if code < 0 || int(code) > 2 {
    return sentry.SpanStatusUnknown, fmt.Sprintf("error code %d", code)
}
httpCode, foundHTTPCode := tags["http.status_code"]
grpcCode, foundGrpcCode := tags["rpc.grpc.status_code"]
var sentryStatus sentry.SpanStatus
switch {
case code == 1 || code == 0:
    sentryStatus = sentry.SpanStatusOK
case foundHTTPCode:
    httpStatus, foundHTTPStatus := canonicalCodesHTTPMap[httpCode]
    switch {
    case foundHTTPStatus:
        sentryStatus = httpStatus
    default:
        sentryStatus = sentry.SpanStatusUnknown
    }
case foundGrpcCode:
    grpcStatus, foundGrpcStatus := canonicalCodesGrpcMap[grpcCode]
    switch {
    case foundGrpcStatus:
        sentryStatus = grpcStatus
    default:
        sentryStatus = sentry.SpanStatusUnknown
    }
default:
    sentryStatus = sentry.SpanStatusUnknown
}
return sentryStatus

    Span Events

    OpenTelemetry, has the concept of Span Events. As per the spec:

    An event is a human-readable message on a span that represents “something happening” during it’s lifetime

    In Sentry, we have two options for how to treat span events. First, we can add them as breadcrumbs to the transaction the span belongs to. Second, we can create an artificial "point-in-time" span (a span with 0 duration), and add it to the span tree. TODO on what approach we take here.

    In the special case that the span event is an exception span, where the name of the span event is exception, we also have the possibility of generating a Sentry error from an exception. In this case, we can create this exception based on the attributes of an event, which include the error message and stacktrace. This exception can also inherit all other attributes of the span event + span as tags on the event.

    In the OpenTelemetry Sentry exporter, we've used this strategy to generate Sentry errors.

    SDK Spec

    You can edit this page on GitHub.