As a Lead DevOps Engineer at PolyScale, Benjamin was tasked with designing and implementing an observability platform to provide deep insights into the application's performance, latency, and overall health. The goal was to create a unified platform that would enable the team to monitor, analyze, and optimize the application's behavior in real-time.

Prometheus was deployed in each cluster via Terraform to scrape metrics data from the application's metrics endpoints using a combination of pull-based and push-based ingestion methods. I created custom Prometheus exporters to collect metrics data from the underlying node.

Alert Manager was configured to integrate with Prometheus and define alerting rules based on the application's performance and latency metrics. Slack notification channels were created to notify the team when alerts are triggered.

A Grafana server was configured on an internal cluster, with each cluster's endpoint added as a connection via Terraform. Dashboards were created using Grafana's built in tools to visualize the application's metrics data and its service dependencies.

Improved application performance - The platform has enabled the team to identify performance bottlenecks and optimize the application's behavior in real-time.

Reduced downtime - The platform's alerting capabilities have reduced downtime by detecting issues before they become critical (or customers were impacted).

Increased visibility - The platform has provided the team with a unified view of the application's behavior, enabling them to make data-driven decisions and optimize the application's performance.

Simplified troubleshooting - The platform has simplified troubleshooting by providing a single source of truth for the application's metrics data.

Some key metrics that demonstrate the success of the platform include:

50% reduction in mean time to detect (MTTD): The platform has reduced the time it takes to detect issues from 30 minutes to 15 minutes.

30% reduction in mean time to resolve (MTTR): The platform has reduced the time it takes to resolve issues from 2 hours to 1.5 hours.

25% increase in application availability: The platform has increased the application's availability from 99.5% to 99.9% and resulted in an optimized approach to measuring global availability.