Observability & ChatOps
End-to-end multi-node telemetry: metric scraping, centralized Loki log pipelines, and Telegram ChatOps incident triage.
# Architectural Topology
Strict separation of application workloads and telemetry infrastructure. All metrics and logs transit through an encrypted VPN tunnel.
- • Nginx Web Container: Astro Landing + /api/crash
- • nginx-prometheus-exporter: Scrapes /stub_status (:9113)
- • node-exporter: CPU, RAM, Disk I/O (:9100)
- • Promtail: Shipping access logs in JSON
- • Tinyproxy: Egress gateway for Telegram API (:8888)
- • Prometheus 2.52: TSDB, 3-day retention (:9090)
- • Loki 3.0: High-performance LogQL engine (:3100)
- • Alertmanager: Deduplication & routing pipeline (:9093)
- • Grafana + Renderer: Headless Chromium for chart snapshots
- • ChatOps Bot: Operational triage & incident control
# Key Reliability Indicators (SLO & Telemetry)
Workload availability and infrastructure tracking structured by RED and USE design principles.
Availability & HTTP Traffic
Continuous request rate and error surge detection preventing production service degradation.
Host Footprint (Amsterdam)
CPU saturation and RAM consumption. Alert triggers when CPU remains > 80% for > 30s.
Log Stream Analysis & Scanners
Real client parsing through X-Forwarded-For alongside live tracking of automated web vulnerability scans.
# Live Demo: Incident Lifecycle
Interactive workflow of the @yankovich_alerting_bot ChatOps assistant in action
# Engineering Highlights
WireGuard Mesh Isolation
Telemetry exporters and Grafana bind strictly to the private 10.100.0.0/24 interface, inaccessible from public networks.
Telegram API Tunneling
Outbound alert webhooks are forwarded via Tinyproxy in Amsterdam, ensuring resilient delivery regardless of network filtering.
Headless Chromium Tuning
Strict concurrency queues in grafana-image-renderer prevent memory spikes on lightweight 2 vCPU monitoring instances.