From b3f6138daee64563e6362408218c03bbe673e4ac Mon Sep 17 00:00:00 2001 From: Simon Gurcke Date: Fri, 20 Jun 2025 22:13:09 +1000 Subject: [PATCH 1/7] First draft --- docs/topics/observability.md | 83 ++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 84 insertions(+) create mode 100644 docs/topics/observability.md diff --git a/docs/topics/observability.md b/docs/topics/observability.md new file mode 100644 index 0000000000..91be5a4163 --- /dev/null +++ b/docs/topics/observability.md @@ -0,0 +1,83 @@ +# Observability + +> Measurement is the first step that leads to control and eventually to improvement. If you can't measure something, you can't understand it. If you can't understand it, you can't control it. If you can't control it, you can't improve it. +> +> — H. James Harrington + +REST framework doesn't ship with built-in observability features, but there are many widely-used tools, standards and third party packages to choose from that work well with it. If your needs are simple, you can also implement your own. + +## Custom request logging + +You can implement a custom middleware that logs relevant information about handled API requests and responses using Python's builtin `logging` module. + +```python +import logging +import time + +logger = logging.getLogger('your_app.requests') + +def request_logging_middleware(get_response): + def middleware(request): + start_time = time.time() + response = get_response(request) + duration = time.time() - start_time + logger.info(f'{request.method} {request.path} - {response.status_code} {response.reason_phrase} - {int(duration*1000)}ms') + return response + return middleware +``` + +Then, add the middleware to your Django settings. + +```python +MIDDLEWARE = [ + 'your_app.middleware.request_logging_middleware', + # ... other middleware +] +``` + +## Prometheus + +[Prometheus](https://prometheus.io/) is an open-source monitoring system that collects metrics by scraping HTTP endpoints exposed by applications. It stores the data in a time series database and supports flexible querying and alerting. + +For a REST framework project, Prometheus can be used to track metrics such as request counts, response codes, error rates, and latency. The [django-prometheus](https://pypi.org/project/django-prometheus/) package adds the necessary instrumentation and exposes a `/metrics` endpoint that Prometheus can scrape. You can also add your own application-specific metrics. + +Prometheus can be paired with [Grafana](https://grafana.com/) to visualize metrics with interactive charts and dashboards. + +## OpenTelemetry + +[OpenTelemetry](https://opentelemetry.io/) is an open-source framework for collecting distributed traces and metrics from applications. It provides a vendor-neutral standard for instrumenting code and exporting telemetry data. + +In Django applications, OpenTelemetry can be used to trace requests through views, middleware, and database operations. The [opentelemetry-instrumentation-django](https://pypi.org/project/opentelemetry-instrumentation-django/) package automatically instruments Django and integrates with the wider OpenTelemetry ecosystem. + +The collected data can be exported to and visualized with any tool that supports OpenTelemetry, such as: + +- Jaeger +- Grafana Tempo +- Datadog +- Elastic APM + +## Other third party packages + +### Apitally + +[Apitally](https://apitally.io/) is a simple API monitoring, analytics, and request logging tool with an integration for Django REST framework. It provides intuitive dashboards with metrics and insights into API requests, errors, performance and consumers. + +The [apitally](https://pypi.org/project/apitally/) package integrates with Django applications through middleware, which automatically captures metrics for API requests and responses, and synchronizes with Apitally in the background. See DRF-specific setup guide [here](https://docs.apitally.io/frameworks/django-rest-framework). + +### django-health-check + +[django-health-check](https://pypi.org/project/django-health-check/) provides a set of health check endpoints for Django applications. It allows you to monitor the status of various application components including database connections, cache backends, and custom services. + +The library is helpful for production deployments as it enables load balancers, orchestration systems, and monitoring tools to determine application health. It supports both simple health checks and detailed status reporting for different application components. + +### django-silk + +[django-silk](https://pypi.org/project/django-silk/) is a profiling tool designed specifically for Django applications. It provides detailed insights into request performance, database queries, and custom code blocks or functions through context managers and decorators. + +The package offers a web-based interface for analyzing requests, database queries, and profiling your code. It's particularly useful during development to identify performance bottlenecks before they reach production. + +### Sentry + +[Sentry](https://sentry.io/) is an error monitoring platform that provides real-time error tracking and performance monitoring for applications. It automatically captures exceptions, tracks performance issues, and provides detailed context to help debug problems in production. + +For Django REST framework applications, Sentry can monitor API errors, slow endpoints, and database query performance. The [sentry-sdk](https://pypi.org/project/sentry-sdk/) package provides automatic instrumentation for Django views, middleware, and database operations. diff --git a/mkdocs.yml b/mkdocs.yml index 010aaefe23..aa7afdc9a5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -54,6 +54,7 @@ nav: - 'Settings': 'api-guide/settings.md' - Topics: - 'Documenting your API': 'topics/documenting-your-api.md' + - 'Observability': 'topics/observability.md' - 'Internationalization': 'topics/internationalization.md' - 'AJAX, CSRF & CORS': 'topics/ajax-csrf-cors.md' - 'HTML & Forms': 'topics/html-and-forms.md' From 5288b059512079f7338fecb1f5ed985c9ff0bd73 Mon Sep 17 00:00:00 2001 From: Simon Gurcke Date: Sat, 21 Jun 2025 12:18:14 +1000 Subject: [PATCH 2/7] Tweaks --- docs/topics/observability.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/topics/observability.md b/docs/topics/observability.md index 91be5a4163..ce8e3e8115 100644 --- a/docs/topics/observability.md +++ b/docs/topics/observability.md @@ -4,11 +4,11 @@ > > — H. James Harrington -REST framework doesn't ship with built-in observability features, but there are many widely-used tools, standards and third party packages to choose from that work well with it. If your needs are simple, you can also implement your own. +REST framework doesn’t include built-in observability features, but it works well with popular tools, standards, and third party packages. For simpler needs, you can also implement custom logging or metrics collection using Django middleware. ## Custom request logging -You can implement a custom middleware that logs relevant information about handled API requests and responses using Python's builtin `logging` module. +You can implement a custom middleware that logs relevant information about handled API requests and responses using Python's built-in `logging` module. ```python import logging @@ -39,7 +39,7 @@ MIDDLEWARE = [ [Prometheus](https://prometheus.io/) is an open-source monitoring system that collects metrics by scraping HTTP endpoints exposed by applications. It stores the data in a time series database and supports flexible querying and alerting. -For a REST framework project, Prometheus can be used to track metrics such as request counts, response codes, error rates, and latency. The [django-prometheus](https://pypi.org/project/django-prometheus/) package adds the necessary instrumentation and exposes a `/metrics` endpoint that Prometheus can scrape. You can also add your own application-specific metrics. +For a REST framework project, Prometheus can be used to track metrics such as request counts, error rates, and latency. The [django-prometheus](https://pypi.org/project/django-prometheus/) package adds the necessary instrumentation and exposes a `/metrics` endpoint that Prometheus can scrape. You can also add your own application-specific metrics. Prometheus can be paired with [Grafana](https://grafana.com/) to visualize metrics with interactive charts and dashboards. @@ -49,7 +49,7 @@ Prometheus can be paired with [Grafana](https://grafana.com/) to visualize metri In Django applications, OpenTelemetry can be used to trace requests through views, middleware, and database operations. The [opentelemetry-instrumentation-django](https://pypi.org/project/opentelemetry-instrumentation-django/) package automatically instruments Django and integrates with the wider OpenTelemetry ecosystem. -The collected data can be exported to and visualized with any tool that supports OpenTelemetry, such as: +The collected data can be exported to any OpenTelemetry-compatible backend for visualization and analysis. Examples include: - Jaeger - Grafana Tempo @@ -74,7 +74,7 @@ The library is helpful for production deployments as it enables load balancers, [django-silk](https://pypi.org/project/django-silk/) is a profiling tool designed specifically for Django applications. It provides detailed insights into request performance, database queries, and custom code blocks or functions through context managers and decorators. -The package offers a web-based interface for analyzing requests, database queries, and profiling your code. It's particularly useful during development to identify performance bottlenecks before they reach production. +The package offers a web-based interface for analyzing requests, database queries, and profiling code. It's primarily intended for local development to identify performance bottlenecks before they reach production. ### Sentry From 4e073c1981cf0c981e44a061c003b050628e555d Mon Sep 17 00:00:00 2001 From: Simon Gurcke Date: Sun, 22 Jun 2025 11:37:21 +1000 Subject: [PATCH 3/7] Fix formatting --- docs/topics/observability.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/topics/observability.md b/docs/topics/observability.md index ce8e3e8115..f7824a9447 100644 --- a/docs/topics/observability.md +++ b/docs/topics/observability.md @@ -14,14 +14,16 @@ You can implement a custom middleware that logs relevant information about handl import logging import time -logger = logging.getLogger('your_app.requests') +logger = logging.getLogger("your_app.requests") def request_logging_middleware(get_response): def middleware(request): start_time = time.time() response = get_response(request) duration = time.time() - start_time - logger.info(f'{request.method} {request.path} - {response.status_code} {response.reason_phrase} - {int(duration*1000)}ms') + logger.info( + f"{request.method} {request.path} - {response.status_code} {response.reason_phrase} - {int(duration*1000)}ms" + ) return response return middleware ``` @@ -30,7 +32,7 @@ Then, add the middleware to your Django settings. ```python MIDDLEWARE = [ - 'your_app.middleware.request_logging_middleware', + "your_app.middleware.request_logging_middleware", # ... other middleware ] ``` From 45cfc6698abe5b4362b35ee6d3d99d438a70d3b7 Mon Sep 17 00:00:00 2001 From: Simon Gurcke Date: Sun, 22 Jun 2025 11:39:10 +1000 Subject: [PATCH 4/7] Fix formatting again --- docs/topics/observability.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/topics/observability.md b/docs/topics/observability.md index f7824a9447..54c4d6e029 100644 --- a/docs/topics/observability.md +++ b/docs/topics/observability.md @@ -16,6 +16,7 @@ import time logger = logging.getLogger("your_app.requests") + def request_logging_middleware(get_response): def middleware(request): start_time = time.time() @@ -25,6 +26,7 @@ def request_logging_middleware(get_response): f"{request.method} {request.path} - {response.status_code} {response.reason_phrase} - {int(duration*1000)}ms" ) return response + return middleware ``` From 8ca33c6982b7a74e1e3c7ce227b6beb8e9338836 Mon Sep 17 00:00:00 2001 From: Simon Gurcke Date: Mon, 23 Jun 2025 09:23:33 +1000 Subject: [PATCH 5/7] Move Prometheus section below OpenTelemetry --- docs/topics/observability.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/topics/observability.md b/docs/topics/observability.md index 54c4d6e029..e4c4774162 100644 --- a/docs/topics/observability.md +++ b/docs/topics/observability.md @@ -39,14 +39,6 @@ MIDDLEWARE = [ ] ``` -## Prometheus - -[Prometheus](https://prometheus.io/) is an open-source monitoring system that collects metrics by scraping HTTP endpoints exposed by applications. It stores the data in a time series database and supports flexible querying and alerting. - -For a REST framework project, Prometheus can be used to track metrics such as request counts, error rates, and latency. The [django-prometheus](https://pypi.org/project/django-prometheus/) package adds the necessary instrumentation and exposes a `/metrics` endpoint that Prometheus can scrape. You can also add your own application-specific metrics. - -Prometheus can be paired with [Grafana](https://grafana.com/) to visualize metrics with interactive charts and dashboards. - ## OpenTelemetry [OpenTelemetry](https://opentelemetry.io/) is an open-source framework for collecting distributed traces and metrics from applications. It provides a vendor-neutral standard for instrumenting code and exporting telemetry data. @@ -60,6 +52,14 @@ The collected data can be exported to any OpenTelemetry-compatible backend for v - Datadog - Elastic APM +## Prometheus + +[Prometheus](https://prometheus.io/) is an open-source monitoring system that collects metrics by scraping HTTP endpoints exposed by applications. It stores the data in a time series database and supports flexible querying and alerting. + +For a REST framework project, Prometheus can be used to track metrics such as request counts, error rates, and latency. The [django-prometheus](https://pypi.org/project/django-prometheus/) package adds the necessary instrumentation and exposes a `/metrics` endpoint that Prometheus can scrape. You can also add your own application-specific metrics. + +Prometheus can be paired with [Grafana](https://grafana.com/) to visualize metrics with interactive charts and dashboards. + ## Other third party packages ### Apitally From 04a0871fb184b003a9b2fc52d969a2933d106e70 Mon Sep 17 00:00:00 2001 From: Simon Gurcke Date: Thu, 26 Jun 2025 10:12:06 +1000 Subject: [PATCH 6/7] Move Observability to bottom of Topics menu --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index aa7afdc9a5..ca213794e9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -54,13 +54,13 @@ nav: - 'Settings': 'api-guide/settings.md' - Topics: - 'Documenting your API': 'topics/documenting-your-api.md' - - 'Observability': 'topics/observability.md' - 'Internationalization': 'topics/internationalization.md' - 'AJAX, CSRF & CORS': 'topics/ajax-csrf-cors.md' - 'HTML & Forms': 'topics/html-and-forms.md' - 'Browser Enhancements': 'topics/browser-enhancements.md' - 'The Browsable API': 'topics/browsable-api.md' - 'REST, Hypermedia & HATEOAS': 'topics/rest-hypermedia-hateoas.md' + - 'Observability': 'topics/observability.md' - Community: - 'Tutorials and Resources': 'community/tutorials-and-resources.md' - 'Third Party Packages': 'community/third-party-packages.md' From 6574c6c84c351adbe8cc7a5822401b297de3fd89 Mon Sep 17 00:00:00 2001 From: Simon Gurcke Date: Thu, 26 Jun 2025 10:14:22 +1000 Subject: [PATCH 7/7] Remove django-silk --- docs/topics/observability.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/topics/observability.md b/docs/topics/observability.md index e4c4774162..998dcefcfd 100644 --- a/docs/topics/observability.md +++ b/docs/topics/observability.md @@ -74,12 +74,6 @@ The [apitally](https://pypi.org/project/apitally/) package integrates with Djang The library is helpful for production deployments as it enables load balancers, orchestration systems, and monitoring tools to determine application health. It supports both simple health checks and detailed status reporting for different application components. -### django-silk - -[django-silk](https://pypi.org/project/django-silk/) is a profiling tool designed specifically for Django applications. It provides detailed insights into request performance, database queries, and custom code blocks or functions through context managers and decorators. - -The package offers a web-based interface for analyzing requests, database queries, and profiling code. It's primarily intended for local development to identify performance bottlenecks before they reach production. - ### Sentry [Sentry](https://sentry.io/) is an error monitoring platform that provides real-time error tracking and performance monitoring for applications. It automatically captures exceptions, tracks performance issues, and provides detailed context to help debug problems in production.