Building a richer waterfall

When you click a request’s trace button and the detail card shows only one span, it’s because nothing in your code opened child spans for that request. The dashboard faithfully renders whatever the OpenTelemetry tracer provider saw — if the middleware was the only thing that Started a span, you get one bar. Here’s the full map of what’s instrumented by default and where to add spans yourself.

What’s traced automatically

Given observability.tracing_enabled: true in config.yaml, these boundaries create spans without you writing a single line:

So /api/v1/users (built-in) and any route produced by gofasta g scaffold <X> already give you a multi-bar waterfall. A brand-new route you wrote by hand in an un-instrumented service won’t, until you add the calls yourself.

Adding spans to existing code

The pattern is three lines, repeated per method:

import "go.opentelemetry.io/otel"
 
const orderServiceTracerName = "yourproject/app/services/order"
 
func (s *OrderService) PlaceOrder(ctx context.Context, input dtos.TPlaceOrderDto) (*dtos.TOrderResponseDto, error) {
    ctx, span := otel.Tracer(orderServiceTracerName).Start(ctx, "OrderService.PlaceOrder")
    defer span.End()
 
    // ... existing body, unchanged.
    // Pass `ctx` (the shadowed one) to downstream calls so their
    // spans nest under this one.
    if err := s.OrderRepo.Insert(ctx, input); err != nil {
        span.RecordError(err) // attach error to the span for the dashboard
        return nil, err
    }
    return &dtos.TOrderResponseDto{...}, nil
}

Three rules to remember:

1. Shadow ctx. The ctx, span := ... assignment returns a new context carrying the span. If you keep using the outer ctx in downstream calls, the children will attach to the parent span of this function, not to this function’s own span — the waterfall won’t nest correctly.
2. defer span.End() immediately. Anything short of this — forgetting to call End, calling it only in the happy path, wrapping in a conditional — leaves unterminated spans that never appear in the trace.
3. Call span.RecordError(err) on every error return. The dashboard marks errored spans red in the waterfall; without this call, a failed call visually looks identical to a successful one.

The tracer name (orderServiceTracerName) should be stable per component — use the module path + package path convention (yourproject/app/services/order) so spans group by subsystem in the “Instrumentation scope” attribute.

Getting SQL queries as waterfall spans

By default, SQL is captured but not turned into OTel spans. This is a deliberate separation:

• devtools.GormPlugin() pushes every query into the in-memory SQL ring with its trace ID attached. That’s what feeds the Recent SQL panel and the Logs tab of the trace card.
• OTel spans for SQL require a separate bridge plugin — devtools.GormPlugin() does not create them.

To get SQL as nested bars under your repository spans, add the GORM OpenTelemetry plugin alongside the devtools plugin. In app/di/providers/core.go, next to the existing db.Use(devtools.GormPlugin()) call:

import "gorm.io/plugin/opentelemetry/tracing"
 
// ... inside the DB provider, after opening the connection:
if err := db.Use(tracing.NewPlugin()); err != nil {
    return nil, err
}
if err := db.Use(devtools.GormPlugin()); err != nil {
    return nil, err
}

Add the dependency:

go get gorm.io/plugin/opentelemetry

After a restart, every query fires its own span; the waterfall shows controller → service → repository → SQL with real duration bars. The two plugins are complementary: the GORM OTel plugin adds the span; the devtools plugin adds the row in the SQL panel with its vars for EXPLAIN. If you only want one, keep devtools — you’ll lose the waterfall bars but still have the SQL panel.

Tracing async tasks and cron jobs

Tasks dispatched through pkg/queue (asynq) and jobs started by pkg/scheduler (robfig/cron) run outside any HTTP request, so there’s no root span to inherit from. Open one yourself at the entry point:

func (h *EmailTaskHandler) Handle(ctx context.Context, task *asynq.Task) error {
    ctx, span := otel.Tracer("yourproject/app/tasks").Start(ctx, "tasks.SendWelcomeEmail")
    defer span.End()
 
    // ... handler body
}

These traces appear in the Traces list alongside HTTP traces. The Recent Requests list, being HTTP-only, doesn’t show them — so the Traces table is how you find them. This is the single reason the Traces list isn’t redundant with Recent Requests.

Verifying you’ve done it right

After adding spans, hit the route or fire the task, open the Traces list, click the new trace’s trace button, and look at the waterfall:

• One bar → only the middleware span fired. Your Start calls either aren’t running (is the code path reached?) or are using a context that didn’t flow through OTel (context.Background() instead of the inherited ctx).
• Multiple bars, but all starting at offset 0 → you’re creating sibling spans instead of children. Double-check you’re shadowing ctx (step 1 above) and passing the shadowed one downstream.
• Multiple bars, properly nested → done. The Logs tab shows only the records emitted under these spans, filtered by trace ID automatically.

Related