O11yStack.ai
Matrix Portal Grafana GitHub
Stack Active • neoson-o11y.duckdns.org • 29/29 Services Healthy

Unified Observability &
Autonomous AI SRE Platform

Zero-overhead telemetry correlation with VictoriaMetrics, VictoriaLogs, VictoriaTraces, and Grafana β€” augmented by Mezmo AURA for autonomous multi-agent root-cause incident investigation.

Launch Grafana Portal Explore AURA SRE AI LiteLLM AI Gateway Langfuse LLM Traces
29/29
Running Containers
100%
Telemetry Coverage
1 Year
Data Retention
216
AURA MCP Tools
TLS 1.3
Let's Encrypt Active
Autonomous Root-Cause Analysis

Mezmo AURA AI SRE Agent

An autonomous multi-agent reasoning engine that ingests alerts, interrogates metrics, logs, and distributed traces, and isolates the precise root cause in seconds.

Multi-Agent Architecture

How AURA Investigates Incidents

Rather than relying on a single prompt, AURA deploys an Orchestrator Coordinator that delegates investigation workflows to specialized domain sub-agents using 216 live Model Context Protocol (MCP) tools:

1
Anomaly & Metric Anomaly Detection

Queries VictoriaMetrics via PromQL to pinpoint p99 latency spikes, CPU throttling, or memory saturation.

2
Distributed Trace & Span Drilldown

Interrogates VictoriaTraces to traverse parent/child spans and identify bottleneck microservices or slow DB queries.

3
LogsQL Clustering & Error Signature Mining

Filters VictoriaLogs with high-speed regex to locate stack traces, panic strings, and HTTP 5xx anomalies.

4
Root Cause Synthesis & Mitigation

Synthesizes full cross-domain telemetry into an executive RCA report with exact commit/configuration fix recommendations.

⚑ Native Grafana App ⏱ Live & Recorded Elapsed Time πŸ“‹ 1-Click Investigation Copy πŸ” 216 Live MCP Tools
aura-sre-investigator • live reasoning trace CONNECTED

sre@o11ystack:~$ curl -X POST https://neoson-o11y.duckdns.org/aura/v1/chat/completions \

  -H "Authorization: Basic <credentials>" \

  -d '{"model":"aura-sre-model", "messages":[{"role":"user","content":"Investigate 504 Gateway Timeouts in checkout-service"}]}'


[AURA Orchestrator] Incident investigation initialized. Plan depth: 3 turns.

[Worker: metrics-analyst] PromQL: histogram_quantile(0.99, sum(rate(http_duration_bucket{service="checkout"}[5m])) by (le))

→ p99 latency spiked from 42ms to 4,820ms at 05:32:10 UTC.


[Worker: trace-analyst] Trace query: service=checkout status=error minDuration=4s

→ Isolated span db_query: SELECT * FROM payment_ledger FOR UPDATE in postgres-backend taking 4.6s.


[Worker: log-analyst] LogsQL: service:checkout AND level:error AND "connection pool exhausted"

→ 412 log occurrences: "FATAL: remaining connection slots reserved for non-replication superuser".


ROOT CAUSE DETERMINED:

Checkout service connection pool exhaustion triggered by unindexed row-locking query on payment ledger during batch reconciliation.

• Recommended fix: Add index on payment_ledger(status, created_at) and increase max_connections from 100 to 300 in postgresql.conf.

Enterprise Open Source Software

The Complete Observability & AI Stack

18 integrated, best-of-breed open-source containers running in harmony with mutual networks and optimized storage pipelines.

Basic Auth

VictoriaMetrics

Time-Series Metrics TSDB

Ultra-fast, cost-effective Prometheus-compatible TSDB. Consumes up to 10x less RAM than vanilla Prometheus, with 1-year persistent data retention.

  • Port: :8428 (Internal) • /vmetrics/ (Public)
  • Full PromQL & MetricsQL support
  • Built-in VMUI graphical metrics explorer
  • Zero-cost high cardinality metric ingestion
vmetrics/vmui Launch VMUI →
Basic Auth

VictoriaLogs

High-Throughput Log Engine

Next-generation logging solution designed for structured and unstructured logs. Offers up to 30x compression and revolutionary LogsQL syntax.

  • Port: :9428 (Internal) • /vlogs/ (Public)
  • Powerful LogsQL pipeline queries
  • Integrated VictoriaLogs Web UI
  • 1-year log retention on host SSD
vlogs/select/vmui Launch Logs →
Basic Auth

VictoriaTraces

Distributed Tracing Backend

Scalable, high-volume distributed tracing storage natively receiving OTLP gRPC traces from OpenTelemetry collectors and eBPF probes.

  • Port: :10428 (HTTP) • :4317 (OTLP gRPC)
  • Compatible with Jaeger & Tempo visualization
  • Trace spans correlated with metrics and logs
  • Built-in VMUI trace waterfall explorer
vtraces/select/vmui Launch Traces →
Grafana Login

Grafana 11+

Unified Observability Portal

The premier visualization dashboard suite pre-configured with automated datasources for VictoriaMetrics, VictoriaLogs, and VictoriaTraces.

  • Port: :3000 (Internal) • /grafana/ (Public)
  • Backed by dedicated MySQL 8.0 for persistence
  • Pre-provisioned dashboards for Node, Processes & VM
  • Integrated Grafana MCP Service Account token
Default: admin Open Grafana →
Grafana App Basic Auth

Mezmo AURA

Autonomous SRE AI Agent

Multi-agent incident investigator using LiteLLM MCP hub to execute 216 live tools across metrics, logs, traces, and Grafana alert states. Available natively in Grafana or via standalone console.

  • Native Grafana App: /grafana/a/mezmo-aura-app/console
  • Live & recorded elapsed investigation time (MTTD)
  • 1-Click investigation copy & Grafana trace reasoning
  • Port: :8080 (Internal) • /aura/ (Web Console)
Grafana / Basic Auth
Grafana App Basic Auth

OpenSRE

Open Source AI SRE Agent

Autonomous incident response and root-cause analysis agent connecting to VictoriaLogs, VictoriaMetrics, Grafana, and LiteLLM model routing (sharing aura-sre-model).

  • Native Grafana App: /grafana/a/opensre-app/console
  • Shared LiteLLM Model Hub & Redis prompt cache
  • Port: :8000 (Internal) • /opensre/v1/ (API)
  • Alert Webhook Intake: /opensre/alerts
Grafana / Basic Auth
Master Key

LiteLLM Gateway

AI Proxy & MCP Hub

High-performance universal LLM proxy routing 100+ providers (OpenRouter, Gemini, Groq, Ollama) and hosting authenticated MCP tool connections.

  • Port: :4000 (Internal) • /litellm/ (Public)
  • Interactive Web UI Playground with model testing
  • Backed by PostgreSQL 16 & Redis 7 cache
  • Native Model Context Protocol (MCP) streamable hub
litellm/ui LiteLLM UI →
Master Key

Langfuse

LLM & MCP Call Tracing

Self-hosted LLM observability: every LiteLLM completion and MCP tool call captured as an OpenTelemetry trace with tokens, cost, latency and full prompt / response payloads.

  • Port: :3000 (Internal) • /langfuse/ (Public)
  • ClickHouse OLAP + S3 blob store + Postgres & Redis state
  • Ingests LiteLLM OTLP spans via the OpenTelemetry Collector
  • Observability data queryable from Grafana (ClickHouse)
langfuse Langfuse UI →
eBPF Core

Grafana Beyla

Zero-Code eBPF Auto-Instrumentation

Kernel-level eBPF probes inspecting HTTP/HTTPS, gRPC, and SQL traffic with zero code modifications, sending RED metrics and traces to OTEL Collector.

  • Kernel eBPF: Ingests all system ports (1-65535)
  • Zero-overhead HTTP/gRPC span generation
  • Auto-detects backend latency and error codes
  • Exports directly to OTLP pipeline
Kernel Layer Auto Active
Pipeline

OTEL Collector

Telemetry Routing Engine

Vendor-agnostic proxy receiving metrics, traces, and host logs, processing batch buffers, and fanning out telemetry to VictoriaMetrics and VictoriaTraces.

  • Port: :4317 (gRPC) • :4318 (HTTP) • :55679 (zPages)
  • Prometheus scrape jobs for Node & Process exporters
  • Host filelog receiver streaming /var/log to VLogs
  • Batching, memory limiter, and retry queueing
Internal OTLP Active
System Metrics

Exporters (Node & Proc)

Host Hardware & Thread Telemetry

Granular hardware metric collection including CPU cores, memory pages, disk I/O, network bandwidth, and individual per-process resource tracking.

  • Node Exporter (:9100) • Process Exporter (:9256)
  • Host filesystem /proc and /sys inspection
  • Tracks Docker daemon, databases, and LLM workers
  • Scraped every 15s into VictoriaMetrics
Kernel Metrics Active
Port 80/443

Nginx Gateway

SSL Termination & Routing

High-performance reverse proxy terminating HTTPS with HTTP/2, enforcing Basic Auth over sensitive backends, and proxying subpaths seamlessly.

  • HTTP → HTTPS automated 301 redirection
  • Secured with Let's Encrypt TLS 1.3 certificates
  • HTTP Basic Auth (.htpasswd) on Victoria suite & AURA
  • Subpath rewriting for unified single-port access
Port 443 Home →
Auto-Sync

DuckDNS & Let's Encrypt

Dynamic DNS & ACME SSL

Automated dynamic IP synchronization every 5 minutes with automated DNS-01 Let's Encrypt challenge for trusted zero-warning TLS certificates.

  • Domain: neoson-o11y.duckdns.org
  • Host Public IP: 213.136.84.111
  • Automated ACME DNS-01 verification script
  • Continuous IP drift detection and updates
DNS-01 Active Healthy
Persistence

Databases & Cache

MySQL 8, Postgres 16, Redis 7

Dedicated database backends isolating application state: MySQL for Grafana dashboards, PostgreSQL for LiteLLM keys & audit logs, Redis for caching.

  • MySQL 8.0: Grafana relational schema & alerts
  • PostgreSQL 16: LiteLLM user tokens & spend logs
  • Redis 7: In-memory LLM response cache & rate limiting
  • Isolated docker volumes for zero-loss restarts
Docker Volumes Protected
Zero-Latency Pipeline

End-to-End Data & AI Flow

How requests traverse from host processes into kernel eBPF probes, the OpenTelemetry Collector, Victoria storage, and Mezmo AURA reasoning.

1. Telemetry Sources
Grafana Beyla
eBPF auto-instrumentation of kernel syscalls, HTTP/gRPC latencies, and SQL queries.
Node Exporter
CPU load, memory RAM usage, network interfaces, disk I/O performance.
Process Exporter
Thread states, user/kernel CPU, per-process memory consumption.
Host System Logs
Linux systemd journal, /var/log directory, docker container stdout/stderr.
2. Ingestion & OTEL
OTEL Collector Contrib
OTLP gRPC (:4317) & HTTP (:4318) endpoints buffering incoming streams.
Batch & Memory Processors
De-duplicates spans, enforces rate limits, and batches telemetry for efficient write operations.
Exporters Dispatch
Routes Prometheus metrics to VMetrics, OTLP traces to VTraces, and logs to VLogs.
3. Victoria Trio Storage
VictoriaMetrics (:8428)
TSDB storing 1-year telemetry with ultra-compressed block format.
VictoriaLogs (:9428)
LogsQL database handling millions of log entries/sec with sub-second queries.
VictoriaTraces (:10428)
OpenTelemetry trace repository storing end-to-end distributed span graphs.
4. AI & Visualization
Grafana 11 Portal
Pre-configured dashboards for real-time human SRE exploration.
LiteLLM MCP Hub
Exposes authenticated streamable tools to LLM reasoning models.
Mezmo AURA SRE Agent
Orchestrates multi-agent incident triage and automates root-cause discovery.
Developer & SRE Quickstart

Query Telemetry & Trigger AI SRE

Standard HTTP REST and curl snippets for querying VictoriaMetrics, VictoriaLogs, and triggering AURA RCA investigations.

Trigger AURA SRE Incident Investigation 
curl -s -X POST https://neoson-o11y.duckdns.org/aura/v1/chat/completions \
  -u "admin:<BASIC_AUTH_PASSWORD>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "aura-sre-model",
    "messages": [
      {
        "role": "user",
        "content": "Diagnose elevated 5xx error rate and identify offending microservice."
      }
    ],
    "stream": false
  }'
Query VictoriaLogs with LogsQL 
curl -s -u "admin:<BASIC_AUTH_PASSWORD>" \
  "https://neoson-o11y.duckdns.org/vlogs/select/logsql/query" \
  -d 'query=_time:5m AND level:error'
Query VictoriaMetrics with PromQL 
curl -s -u "admin:<BASIC_AUTH_PASSWORD>" \
  "https://neoson-o11y.duckdns.org/vmetrics/api/v1/query" \
  -d 'query=sum(rate(node_cpu_seconds_total{mode="idle"}[5m]))'
LiteLLM Unified Multi-Model Gateway 
curl -s -X POST https://neoson-o11y.duckdns.org/litellm/v1/chat/completions \
  -H "Authorization: Bearer <LITELLM_MASTER_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openrouter-llama-70b",
    "messages": [{"role":"user","content":"Summarize recent system health."}]
  }'