Skip to content

Conversation

@fenos
Copy link
Contributor

@fenos fenos commented Dec 31, 2025

What kind of change does this PR introduce?

Feature

What is the current behavior?

Currently metrics are gathered using the prom-client and exposed to a metric endpoint

What is the new behavior?

  • All metrics are now otel metrics and can be pushed to a 3rd party otel backend
  • We still can optionally expose prometheus metrics in the /metrics endpoint using the otel prom exporter (ENABLE_PROMETHEUS_METRICS)
  • All children spans will now include the tenant.ref attribute from a top level span
  • Revamped grafana dashboard using otel metrics

Grafana dashboard

Running completely locally

png (2)

@coveralls
Copy link

coveralls commented Dec 31, 2025

Pull Request Test Coverage Report for Build 21515862680

Details

  • 673 of 1285 (52.37%) changed or added relevant lines in 33 files are covered.
  • 2 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.4%) to 75.548%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/admin-app.ts 5 6 83.33%
src/http/routes/s3/commands/create-multipart-upload.ts 1 2 50.0%
src/internal/monitoring/metrics.ts 194 197 98.48%
src/internal/queue/queue.ts 3 6 50.0%
src/storage/backend/s3/adapter.ts 3 6 50.0%
src/internal/monitoring/logger.ts 9 13 69.23%
src/internal/queue/event.ts 1 5 20.0%
src/internal/database/pool.ts 47 55 85.45%
src/http/plugins/log-request.ts 17 28 60.71%
src/internal/monitoring/system/base-collector.ts 33 45 73.33%
Files with Coverage Reduction New Missed Lines %
src/admin-app.ts 1 89.19%
src/http/routes/s3/commands/upload-part.ts 1 65.68%
Totals Coverage Status
Change from base Build 21449071800: -0.4%
Covered Lines: 25902
Relevant Lines: 34015

💛 - Coveralls

@fenos fenos force-pushed the monitoring/otel-metrics branch from ddcc147 to c9d5fd0 Compare January 7, 2026 13:29
Comment on lines +605 to +617
{
"datasource": {
"type": "prometheus",
"uid": "local_prometheus"
},
"editorMode": "code",
"expr": "sum(increase(storage_api_otel_upload_started_total{region=~\"$region\", instance=~\"$instance\"}[$__range]))",
"hide": false,
"instant": false,
"legendFormat": "__auto",
"range": true,
"refId": "C"
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was supposed to be multipart I guess according to description but instead duplicates query A

hostMetrics.start()

// Register Node.js runtime instrumentations
registerInstrumentations({

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to capture returned function unregister here to pass shutdown below. Otherwise, it might prevent process from exiting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I have double checked but this function is not returning anything.
I guess it registers it internally, and on stop, it will clean it up

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It returns a cleanup function, I believe we need to call it before meter shutdown for clean exit because it will stop timer for metric reader but won't touch the timer inside custom instrument. Another approach could be making timer unblocking as in #819 (comment)

@fenos fenos force-pushed the monitoring/otel-metrics branch 4 times, most recently from c85a64a to 6bd7d5a Compare January 29, 2026 16:29
@fenos fenos force-pushed the monitoring/otel-metrics branch from 6bd7d5a to 2a88ed1 Compare January 30, 2026 12:29
@fenos fenos merged commit f3901ef into master Jan 30, 2026
2 checks passed
@fenos fenos deleted the monitoring/otel-metrics branch January 30, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants