AIOps & SecOps Case Study using local ai and n8n - Part 2

Part 2: Mini Self-Healing Infrastructure & Dynamic AIOps Remediation

1. Executive Summary & The Shift to Active Remediation

In Part 1, we established a foundational Zero-Cloud AIOps & Security Digest Engine, focusing on telemetry collection, log aggregation, and local AI-driven analysis to maintain complete data privacy without recurring cloud API costs. However, passive visibility only solves half the problem. Systems administrators still face the operational burden of manually executing routine fixes for recurring infrastructure anomalies.

This installment explores Dynamic AIOps Remediation—bridging the gap between automated insight and system-level execution. By embedding an orchestration engine within our local architecture, telemetry alerts no longer just trigger notifications; they trigger intelligent, context-aware recovery workflows.

2. Architectural Enhancements & Component Stack

To execute automated remediation safely within a self-hosted environment, the architecture integrates a tightly coupled local stack:

• Telemetry & Event Source (Zabbix): Monitors resource health, container states, and service metrics, dispatching rich JSON payloads via webhooks the moment an anomaly or threshold breach occurs.
• Orchestration Engine (n8n on Proxmox LXC): Hosted locally inside a dedicated Linux Container with explicit systemd network bindings (0.0.0.0) and static local routing ([Local IP of n8n]:5678) to ensure reliable internal and relay-based communication.
• Cognitive Processing (Local LLM via Ollama): Evaluates unstructured alert messages locally. Instead of relying on brittle regex scripts or hardcoded conditionals, the local model parses the operational context and dynamically structures a valid shell command.
• Execution Layer: Interfaces directly with host environments or container APIs (e.g., executing Proxmox container management commands like pct start) to resolve faults autonomously.

3. Implementation Workflow: From Alert to Execution

The dynamic remediation pipeline translates raw monitoring data into corrective system changes through a streamlined execution flow:

1. Webhook Ingestion: When a service crashes or a container stops unexpectedly, Zabbix fires a structured POST request containing event metadata and error logs to the n8n webhook endpoint.
2. Cognitive Root-Cause Parsing: The payload is passed to the local LLM node. The model analyzes the error signature and outputs a deterministic JSON object containing the exact remediation command required (e.g., {"actionCommand": "pct start 108"}).
3. Command Validation & Execution: The orchestration workflow extracts the payload dynamically (leveraging explicit node references to avoid execution errors) and passes it to the command execution module, completing the closed-loop recovery cycle.

4. Engineering Trade-offs & Operational Risks

While automating infrastructure remediation drastically lowers Mean Time to Resolution (MTTR), introducing autonomous execution introduces architectural risks that must be managed:

• The Hallucination Vector: Small-parameter local models can occasionally misinterpret ambiguous stack traces or log outputs, risking the generation of inaccurate or suboptimal commands.
• Unbounded Autonomy: In a pure zero-touch model, an erroneous AI interpretation translates immediately into a direct system command, highlighting the critical need for strict input validation and command whitelisting.
• Practical Safety Scope (The "Start/Stop" Rule): Given the natural apprehension around raw local AI execution, initial testing must be strictly sandboxed. Restrict the AI's operational scope exclusively to non-destructive lifecycle states—such as executing simple start or stop commands for a non-critical LXC or VM (e.g., pct start 108)—completely avoiding configuration-altering or data-destructive commands during the initial rollout phase.

Check the complete workflow: Visit GitHub Repository →

System in Action: End-to-End-Execution:

here is the entire workflow

here is the self healing in action

Looking Ahead: Introducing Governance

While dynamic remediation streamlines routine infrastructure management, executing raw AI output directly on production systems requires a robust safety net. Future iterations introduce asynchronous governance gates—such as Human-in-the-Loop (HITL) approval loops—to ensure absolute administrative control over automated infrastructure changes.