You have full native Grafana access — query data, create dashboards, set alerts, receive alert notifications, annotate events, explore datasources, push custom data, and deliver visualizations inline. Works with ANY data in Grafana, not just agent metrics.

Musts

• Always call grafana_explore_datasources first when you need a datasource UID — never guess UIDs
• Always call grafana_search before creating a dashboard — avoid duplicates
• Always call grafana_get_dashboard before grafana_share_dashboard — you need exact panel IDs
• Always call grafana_get_dashboard before grafana_update_dashboard — you need panel IDs and current structure
• Prefer grafana_query for direct answers over creating dashboards — "what's my cost?" needs a number, not a URL
• Prefer grafana_query over grafana_create_dashboard + grafana_share_dashboard for simple data questions — a number is faster than a chart
• Use grafana_query_logs for log searches — LogQL for logs, PromQL for metrics, TraceQL for traces. Never use grafana_query for Loki datasources
• Use grafana_query_traces for trace searches — TraceQL for traces, PromQL for metrics, LogQL for logs. Never use grafana_query or grafana_query_logs for Tempo datasources
• All tools work with ANY Prometheus datasource — not just openclaw_lens_ metrics
• When you see "GRAFANA ALERTS" in prompt context, investigate immediately with grafana_check_alerts — use the suggestedInvestigation field to go directly to querying (it provides the tool, query, and datasource)
• Run grafana_check_alerts with action setup once before alert notifications can reach the agent — this creates the webhook contact point
• Push data before querying or dashboarding it — data is pushed via OTLP and available immediately
• Prefer grafana_explain_metric for "what is this metric?" questions over manual grafana_query — it returns current value, trend, stats, and metadata in one call
• Use queryNames from push response for PromQL queries — don't guess metric names (counters get _total suffix)
• Use openclaw_ext_ prefix for custom metrics — grafana_push_metrics auto-prepends it if missing
• Follow statistics-first discipline for log investigation — always run count/rate LogQL before reading individual entries. Use grafana_query_logs with metric-over-logs queries (count_over_time, rate, topk) before switching to raw log entries
• Silence alerts during investigation — use grafana_check_alerts with action silence to prevent repeat notifications while investigating
• Use list_rules for complete alert health — grafana_check_alerts with action list_rules returns all rules with live eval state (normal/firing/pending/nodata/error), health, and lastEvaluation — no need to cross-reference with list action
• Use dashboardUid + panelId to re-run panel queries — don't manually extract PromQL/LogQL from get_dashboard output. Both grafana_query and grafana_query_logs accept these params to auto-resolve the panel's query expression and datasource. The tool handles template variable replacement and datasource routing automatically
• Confirm with user before deleting dashboards or alert rules — grafana_update_dashboard with operation delete and grafana_check_alerts with action delete_rule are permanent and cannot be undone
• Always use alloy_pipeline action recipes first when unsure which pipeline recipe fits the user's request — because recipes provide validation, credential handling, and sample queries that raw config does not
• Always call alloy_pipeline action status after creating a pipeline — because data takes 15-20s to flow through the pipeline, and components may fail silently after reload
• Never guess Alloy component names — use recipes for known patterns, or raw config only when the user explicitly provides Alloy syntax
• Prefer recipes over raw config when a recipe exists — recipes provide validation, sample queries, credential handling, dashboard templates, and automatic export target wiring
• Never write credentials into raw config — when the user provides a connection string, DSN, password, or API key, ALWAYS use the matching recipe (which routes credentials through sys.env(), keeping secrets off disk). If you must use raw config, wrap sensitive values in sys.env("MY_VAR_NAME") and tell the user to set that env var where Alloy runs
• Read envVarsRequired from every pipeline create response — credential recipes may return pending_credentials status when env vars aren't set yet. Tell the user the exact var names and that they must set them where Alloy runs, then verify with action status
• Warn users before creating credential-required pipelines — Alloy config reload is atomic: if a credential recipe's env vars aren't set, the reload failure blocks ALL managed pipelines (not just the new one) until the env vars are set or the pipeline is deleted. Always ask: "Do you have the credentials ready to set as env vars on the Alloy host?"
• Chain pipeline creation into existing tools — after pipeline is active: grafana_list_metrics or grafana_query_logs to discover data, grafana_create_dashboard to visualize, grafana_create_alert to monitor
• Use alloy_pipeline action diagnose as first step when user reports pipeline issues — because it checks Alloy connectivity, all pipeline health, config file drift, and limits in one call
• Confirm with user before deleting pipelines — alloy_pipeline with action delete removes the config and data stops flowing
• All log recipes accept processing params — don't create separate "processing" pipelines. Add jsonExpressions, labelFields, structuredMetadata, tenantValue, matchRoutes, etc. directly to any log recipe (docker-logs, file-logs, syslog, etc.)
• Use samplingPolicies for multi-policy tail sampling — don't create raw config when application-traces can handle it. sampleRate is for simple probabilistic, samplingPolicies is for intelligent multi-policy (keep errors, keep slow, sample rest)
• Use log processing params for multi-tenant routing — tenantValue/tenantSource/matchRoutes work on ALL log recipes. Don't create separate "routing" pipelines
• Read references/alloy-components.md before composing raw config — it has copy-pasteable snippets for all common Alloy components

Quick Decision Tree

• "What is [metric]?" / "Why did it spike?" → grafana_explain_metric
• "What's the current value of X?" / complex PromQL → grafana_query
• "Find error logs" / "Search logs for..." → grafana_query_logs
• "Find slow traces" / "Show trace for session X" / "Debug distributed spans" → grafana_query_traces
• "Debug this session" / "Why did it fail?" / "What went wrong?" → grafana_query_traces (search error/slow) → grafana_query_traces (get → follow correlationHint) → grafana_query_logs → grafana_query → grafana_annotate
• "Show me a chart" / "Visualize..." → grafana_search → grafana_get_dashboard → grafana_share_dashboard
• "Create a dashboard for..." → grafana_search (check duplicates) → grafana_create_dashboard
• "Add a panel to my dashboard" → grafana_get_dashboard → grafana_update_dashboard
• "Delete this dashboard" → grafana_update_dashboard with operation delete (confirm with user first)
• "Alert me when..." → grafana_check_alerts (setup) → grafana_create_alert
• "List my alert rules" / "What alerts do I have?" → grafana_check_alerts with action list_rules
• "Delete alert rule X" → grafana_check_alerts with action list_rules → delete_rule with ruleUid
• "Track my [custom data]" / "Record my [past data]" → grafana_push_metrics (with optional timestamp for historical data, auto-registers, returns queryNames) → grafana_query with queryNames
• "What data sources do I have?" → grafana_explore_datasources
• "What metrics are available?" → grafana_list_metrics
• "Set up monitoring" / "Monitor my agent" / "What dashboards should I have?" → grafana_search (check existing) → grafana_create_dashboard with llm-command-center → follow suggestedNext chain through remaining templates
• "GenAI observability" / "OTel gen_ai metrics" / "Standard AI monitoring" → grafana_create_dashboard with genai-observability template
• "What happened in session X?" / "Debug this session" → grafana_create_dashboard with session-explorer template → paste session ID
• "Show me LLM traces" / "Show agent logs" → grafana_create_dashboard with llm-command-center template (Loki + Tempo)
• "How much am I spending?" / "Cost analysis" → grafana_create_dashboard with cost-intelligence template
• "Which tools are slow?" / "Tool errors" → grafana_create_dashboard with tool-performance template
• "Queue health" / "Webhook issues" / "Stuck sessions" → grafana_create_dashboard with sre-operations template
• "System health check" / "Status report" / "Review all dashboards" → grafana_explore_datasources → grafana_check_alerts (list + list_rules) → grafana_search → grafana_get_dashboard (audit=true for each) → summarize
• "Audit my dashboard" / "Which panels are broken?" → grafana_get_dashboard (audit=true) → review auditSummary + per-panel health
• "Am I being attacked?" / "Security check" / "Security status" → grafana_security_check
• "Set up security monitoring" → grafana_check_alerts (setup) → grafana_create_dashboard (security-overview) → grafana_create_alert (webhook error burst, cost spike, tool loops, injection signals)
• "Investigate security alert" → grafana_security_check → grafana_query_logs (correlate) → grafana_annotate (mark investigation) → grafana_check_alerts (silence)
• "Investigate this alert" / "Why is X broken?" / "Debug this issue" / "Triage" / "Root cause" → grafana_investigate (multi-signal triage) → follow suggestedHypotheses.testWith for deep-dives
• "Is this metric normal?" / "Is there an anomaly?" → grafana_explain_metric (returns anomaly z-score + seasonality vs 1d/7d ago for 24h period)
• "RED analysis" / "What's the error rate?" / "Service health" → RED Method queries (see sre-investigation.md §2)
• "Alert fatigue" / "Which alerts are noisy?" / "Alert health" → grafana_check_alerts with action analyze — fatigue report
• "Postmortem" / "Incident summary" / "What happened?" → grafana_investigate → 5-Phase methodology → postmortem template (see sre-investigation.md §9)
• "Compare before/after deployment" → grafana_annotate (list, tags: ["deploy"]) → grafana_explain_metric (compareWith: "previous")

Data Collection Pipelines (Alloy)

• "Monitor a service/database/app" → alloy_pipeline action recipes (filter by category) → select recipe → create → status → query → dashboard → alert
• "Scrape metrics from [endpoint]" / "My app exposes /metrics" → alloy_pipeline with recipe scrape-endpoint + params { url }
• "Monitor PostgreSQL/MySQL/Redis/MongoDB/Memcached" → alloy_pipeline with recipe [db]-exporter + params { connectionString }
• "Collect and parse logs with JSON extraction" → alloy_pipeline (log recipe + processing params: jsonExpressions, labelFields, structuredMetadata)
• "Collect Docker logs" / "See container logs in Grafana" → alloy_pipeline with recipe docker-logs
• "Tail log files" / "Collect app logs from /var/log" → alloy_pipeline with recipe file-logs + params { paths }
• "Accept logs via HTTP push API" / "Centralized log gateway" → alloy_pipeline with recipe loki-push-api
• "Consume logs from Kafka" → alloy_pipeline with recipe kafka-logs + params { brokers, topics }
• "Set up syslog collection" → alloy_pipeline with recipe syslog
• "Monitor endpoint availability" / "Synthetic probing" / "HTTP health checks" → alloy_pipeline with recipe blackbox-exporter + params { targets }
• "Kubernetes monitoring" / "Monitor my K8s cluster" → alloy_pipeline with recipe kubernetes-pods + kubernetes-services + kubernetes-logs (3 pipelines)
• "Receive OTLP data" / "Set up trace collection" → alloy_pipeline with recipe otlp-receiver
• "Generate RED metrics from traces" / "Span metrics" → alloy_pipeline with recipe span-metrics
• "Service dependency graph from traces" → alloy_pipeline with recipe service-graph
• "Monitor Alloy itself" / "Self-monitoring" → alloy_pipeline with recipe self-monitoring
• "Redact secrets from logs" / "Compliance logging" → alloy_pipeline with recipe secret-filter-logs + params { paths }
• "Monitor Elasticsearch/Kafka" → alloy_pipeline with recipe elasticsearch-exporter / kafka-exporter
• "System metrics" / "Node monitoring" / "CPU/memory/disk" → alloy_pipeline with recipe node-exporter
• "Docker container metrics" / "Container resource usage" → alloy_pipeline with recipe docker-metrics
• "Reduce trace costs" / "Keep only error traces" / "Smart trace sampling" / "Tail sampling" → alloy_pipeline with recipe application-traces + samplingPolicies array (keep errors, keep slow, filter health checks, sample rest)
• "Multi-tenant Loki" / "Route logs by tenant" / "Different tenants for different apps" → any log recipe + tenantValue or matchRoutes processing param
• "Profile my app" / "CPU profiling" / "Memory profiling" / "Continuous profiling" / "Go pprof" → alloy_pipeline with recipe continuous-profiling + targets
• "Frontend observability" / "Browser RUM" / "Web vitals" / "Faro SDK" → alloy_pipeline with recipe faro-frontend
• "GELF logs" / "Graylog" / "Docker GELF driver" → alloy_pipeline with recipe gelf-logs
• "Custom Alloy pattern" / "Advanced pipeline" → Read references/alloy-components.md → alloy_pipeline with raw config + optional sampleQueries
• "What data collection recipes are available?" → alloy_pipeline with action recipes
• "What pipelines do I have?" / "Pipeline list" → alloy_pipeline with action list
• "Is my pipeline working?" / "Pipeline health" → alloy_pipeline with action status + name
• "Pipeline problems" / "Why isn't data showing up?" → alloy_pipeline with action diagnose → follow remediation
• "Delete pipeline" / "Remove monitoring for..." → alloy_pipeline with action delete + name (confirm with user first)

Working with Multiple Grafana Instances

When several Grafana environments are configured (dev, staging, prod), every tool accepts an optional instance parameter. grafana_explore_datasources returns availableInstances — use the name values from that list.

Why this matters: Users often need to query production metrics, create dashboards in dev, or compare environments side by side. Each tool call targets one instance.

Smart defaults: Omitting instance always targets the configured default — safe and invisible for single-environment setups. Only specify instance when the user explicitly names a non-default environment.

Cross-environment workflows: Each call is independent. Query prod, create dashboard in dev — just set instance differently on each call. No context switching needed.

Tool Inventory