# Site Reliability Engineering Metrics Comprehensive Guide to Understand and Utilize SRE Metrics for Improved Reliability and Performance ## Welcome to the Site Reliability Engineering (SRE) Documentation for PBSA! This documentation serves as a comprehensive guide to implementing SRE practices within PBSA. Whether you are a member of the SRE team, a software engineer, a community member looking for insights on how our infrastructure is maintained, this documentation aims to provide you with the knowledge and resources needed to enhance the reliability, performance, and scalability of our systems and services. In this documentation, you will find a wealth of information specific to maintaining and establishing PBSA's SRE practices, tailored to our unique systems and infrastructure. Here's a glimpse of what you can expect from this documentation: ### Foundational Concepts: Gain a solid understanding of the core SRE concepts that form the basis of our practices at PBSA. Explore topics such as Service Level Objectives (SLOs), Service Level Agreements (SLAs), Error Budgeting, and other SRE practices. ### Incident Management: Learn how we respond to incidents, minimize their impact, and foster a blameless culture of learning and improvement. Discover our incident management process, incident response best practices, and incident postmortem guidelines. ### Monitoring and Alerting: Delve into our monitoring and alerting strategies to proactively detect and address issues. Understand the tools and techniques we employ to ensure reliable performance and real-time visibility into our systems. This documentation is a living resource, continuously evolving alongside our systems and practices. We encourage your active engagement, feedback, and contributions as we strive to build a robust knowledge base specific to PBSA's SRE initiatives. # Service Level Agreements (SLAs) # Log Aggregation This page provides an overview of our Log Aggregation Service Level Agreement (SLA) that outlines the commitments and expectations related to our log aggregation service. Our log aggregation SLA defines the standards and targets for log collection, processing, storage, and accessibility. It serves as a foundation for ensuring the availability, reliability, and performance of the log aggregation service to support the operational needs of our organization. In this section, you will find information on the key components of our log aggregation SLA, including: 1. **Log Collection and Ingestion:** * Supported log formats: Plain Text, JSON, or structured logs. * Ingestion mechanisms: Log forwarding 2. **Processing:** * Processing time: Immediately 3. **Storage and Retention:** * Data retention period: 90 days * Storage capacity: 750GB * Backup Schedule: Weekly 4. **Search and Retrieval:** * Search capabilities: Plain Text, LogQL, Regex 5. **Performance and Scalability:** * Ingestion rate: 4MB/s per log stream * System availability: 99.9% 6. **Monitoring and Alerting:** * Metrics and thresholds: * Storage utilisation, with rocketchat notifications at 80% usage. * Alerting mechanism: Rocketchat/Telegram webhook to monitoring-channel # Covered Services Covered Services/URLs under the SLA This page provides a comprehensive list of all the services and corresponding URLs that fall within the scope of the Log Aggregation Service Level Agreement (SLA). The SLA defines the commitments, expectations, and quality standards for these services, ensuring that they meet the desired level of performance, availability, and reliability. By maintaining a clear and up-to-date record of the services and URLs covered under the SLA, we establish transparency and facilitate effective communication with stakeholders. This list serves as a reference point to identify the specific systems, applications, or endpoints that are subject to the SLA and outlines the agreed-upon service levels for each of them. In this section, you will find an overview of the covered services/URLs:
ServiceURLJob ValueAvailalbility
Grafana LogsGrafanaN/A99.9
Loki IngestorLokiN/A99.9
Production NEXGrafanaproduction-nex-output99.9
Production NEXGrafanaproduction-nex-errors99.9
Alphanet NEXGrafanaalphanet-nex-output99.9
Alphanet NEXGrafanaproduction-nex-output99.9
Alphanet BlockchainGrafanaalphanet-blockchain-logs99.9
Alphanet FaucetGrafanaalphanet-faucet-logs99.9
Devnet NEXGrafanadevnet-nex-output99.9
Devnet NEXGrafanadevnet-nex-errors99.9
Devnet BlockchainGrafanadevnet-blockchain-logs99.9
Devnet FaucetGrafanadevnet-faucet99.9
By referring to this list, stakeholders can gain a clear understanding of the services and URLs covered by this SLA and the associated service levels that have been committed to. This promotes transparency, aligns expectations, and ensures that the agreed-upon standards for performance and reliability are consistently met. Please note that this list may be subject to updates and changes as new services or URLs are added or modified. It is essential to review and communicate any updates to stakeholders to ensure everyone has the most up-to-date information on the covered services/URLs under the SLA. # Monitoring Logs Of A New System This page shows you how to scrap logs and push them to our ingestor The provided script automates the deployment and configuration of the Promtail agent, which is used for log aggregation and forwarding to a Loki deployment. Here's a brief summary of how to use the script: 1. Review, download and run the *promtail* agent installer script: ``` wget https://gitlab.com/PBSA/infrastructure/devops-misc-issues/-/raw/main/scripts/loki/promtail-install.sh ``` 2. Add execute permissions and run the script: ``` sudo chmod +x promtail-install.sh && ./promtail-install.sh ``` 3. System Update: The script begins with a system update, running: `apt update` and `apt dist-upgrade` to ensure the system is up to date. 4. Promtail Installation: The script then downloads the Promtail binary from the official Grafana Loki repository, extracts it, and moves it to `/usr/local/bin/`. It also verifies the version of Promtail. 5. Configuration: The script creates the necessary directories for Promtail's configuration and data files (`/etc/promtail` and `/data/promtail`). It then adds a default configuration file (`config.yaml`) for Promtail, which you need to edit with your specific settings. 6. Systemd Service: The script sets up a systemd service file for Promtail (`/etc/systemd/system/promtail.service`) to enable automatic start and management of the Promtail agent as a service. 7. Configuration Reminder: The script reminds you to edit the `/etc/promtail/config.yaml` file with your specific configuration details before starting the Promtail agent. 8. Starting and Checking Service: The script provides instructions for starting the Promtail agent using `sudo systemctl start promtail.service` and checking the service's status with `systemctl status promtail.service`. Remember to review and update the configuration files with your desired settings to ensure Promtail is properly configured for your environment. # Service Level Objectives (SLOs) Setting the Standard: Empowering Performance Excellence with Service Level Objectives (SLOs) ## Welcome to the Service Level Objective (SLO) Documentation! This documentation serves as a comprehensive guide to understanding and implementing SLOs to drive performance excellence within our organization. SLOs are powerful tools that allow us to set specific targets and measure the performance, reliability, and quality of our services. By defining clear objectives, we can align our efforts and continually strive for service excellence. SLOs provide a framework to establish measurable goals and track our progress in meeting them. By setting realistic yet ambitious targets, we can hold ourselves accountable and continually improve the performance and availability of our systems. In this documentation, you will find information about SLOs, including their importance, benefits, implementation strategies, and best practices. We will explore the key components of SLOs, such as defining appropriate metrics, setting meaningful targets, and establishing thresholds for acceptable performance. Additionally, we will discuss monitoring techniques, alerting mechanisms, and data analysis approaches to effectively measure and manage our SLOs. Here's an overview of what you can expect from this documentation: ### Understanding SLOs: Gain a clear understanding of what SLOs are, their purpose, and the benefits they bring to our organization. Explore why they are essential for driving performance excellence and enhancing customer satisfaction. {% content-ref url="/pages/LgvA4NwxNyIwK2MO6Dx9" %} [Understanding Our SLOs](/introduction/service-level-objectives-slos/understanding-our-slos) {% endcontent-ref %} ### Defining SLO Metrics: Learn how to identify and select appropriate metrics that accurately reflect the performance and reliability of our services. Discover best practices for choosing relevant indicators and establishing meaningful targets for each metric. ### Monitoring and Alerting for SLOs: Dive into monitoring techniques and alerting mechanisms to measure our services against their defined SLOs. Explore how to implement effective monitoring systems that provide real-time insights and proactive notifications when performance deviations occur. ### Analyzing SLO Data: Explore methods for analyzing SLO data to gain actionable insights and drive continuous improvement. Learn how to interpret trends, identify patterns, and make data-driven decisions to enhance service performance. ### SLO Management and Iteration: Discover strategies for managing and iterating on our SLOs. Explore approaches for updating targets, adjusting thresholds, and incorporating customer feedback to ensure our objectives remain relevant and aligned with evolving business needs. By embracing the power of SLOs, we can drive a culture of performance excellence, establish clear expectations, and continuously improve the reliability and quality of our services. Let's embark on this SLO journey together and unlock new levels of service excellence! # Understanding Our SLOs