Aiden POC — Accuracy, Memory, SLOs & Privacy
Aiden POC — Accuracy, Memory, SLOs & Privacy
Plain-language answers for customer POC stakeholders who ask: How accurate is root cause? How long do we keep memory? How do SLOs and error budgets work? What about PII and privacy?
ELI5: We measure Aiden on your past incidents with a shared scorecard, you control how long memory sticks around, SLO burn can wake investigations and weekly digests, and personal data is scrubbed before the model sees it—while your SE can walk security through the formal egress and trust packs under NDA.
Suggested POC agenda
| Session | Focus | Outcome |
|---|---|---|
| 1 | Scorecard kickoff | Agree golden incidents, rubric (correctness / completeness / actionability), and pass thresholds |
| 2 | Live or replayed investigations | Run Aiden on those incidents; capture timings and human scores |
| 3 | Memory & retention walkthrough | Show Memory Explorer, curated shared incident memory, and write down keep-for-N-days settings in the handoff |
| 4 | SLO / error-budget path | Trigger or review a burn-rate investigation; show weekly digest + remaining-budget chat skill |
| 5 | Privacy & controls | Confirm PII scrubbing, optional deny/allow overlays, policy / vault / approve-before-change posture |
Related SE material: Incident triage POC limits, category taxonomy, SE playbook.
1. Root cause accuracy
What we claim (and what we do not)
We do not publish a single company-wide “accuracy %” that applies to every customer and every incident type. Environments, tooling, and golden narratives differ too much for one number to be honest.
What we do in a POC is measure Aiden on your golden incidents with a shared scorecard:
| Dimension | Plain meaning |
|---|---|
| Correctness | Did the root cause match (or reasonably match) the agreed golden narrative? |
| Completeness | Was enough evidence and context present to trust the answer? |
| Actionability | Could an on-call engineer act on the write-up without guessing? |
You set pass thresholds with your SE (a common correctness bar in internal evals is “pass if correctness > 0.4,” but the POC should use your bar).
Illustrative internal live eval (not a customer case study)
These numbers come from an internal live replay against a StackGen demo alert set. They are not a named U.S. (or any) customer case study, and they are not a contractual accuracy guarantee.
One scored internal pass looked roughly like this:
| Metric | Approx. result |
|---|---|
| Incidents scored | ~4 |
| Average human correctness | ~0.84 |
| Pass rate at correctness > 0.4 | ~94% (16/17) |
| Finished within ~2 minutes | ~73% |
| RCA present in the run | 100% |
Treat these as illustrations of method and ballpark, not a guarantee for your estate. Detailed category tables and caveats live in the taxonomy notes for SEs.
Honest caveats
- Quality improves when Kubernetes, GitHub (change correlation), and curated incident memory are wired and healthy.
- Out of the box, Aiden is often weaker on network, database-internals, and “unknown” incidents that need labels or extra integrations—see POC limits.
- The POC replaces these illustrations with your numbers on your golden set. That report is what procurement and engineering should use.
2. Memory retention & TTL
How long Aiden keeps remembered incident lessons is operator-controlled—not a hidden product default that quietly deletes (or keeps) everything forever without your say.
| Control | Behavior |
|---|---|
| Automatic retention sweeper | Disabled by default. Nothing is auto-purged until you turn retention on. |
| When enabled | Sweeps on a schedule; default interval is every 24 hours. |
| Day-to-day lesson retention (episodic) | Configurable in days (a common example is 30 days). |
| Rejected / unused evidence | Often kept for fewer days (a common example is 7 days). |
| Zero days for a category | Means keep forever for that category (no age-based delete for it). |
Shared incident memory (the curated “similar past incidents” store) is meant to be deliberate: seed what you trust, review what agents write, and browse or delete entries in Memory Explorer. There is no silent “promote everything to permanent wisdom” switch in the default POC path—see memory layers in the limits doc.
POC handoff: write down the exact day values (lesson memory / rejected / expired, and whether automatic cleanup is on) in the customer runbook so security and ops share one source of truth.
3. SLOs & error budget
Aiden can sit next to the SLO tools you already use—it does not replace Grafana, Datadog, or New Relic as the system of record.
| Capability | What customers see |
|---|---|
| Burn-rate / error-budget alerts | When Grafana, Datadog, or New Relic (or similar) say the service is burning its reliability budget too fast, Aiden can start an investigation automatically—so “we’re burning budget” becomes analysis, not only a page. |
| Weekly SLO health digest | A scheduled health review can summarize remaining error budget and whether the written SLO definitions still match what is live in monitoring (aios-agent-slo-health). |
| Chat skill for remaining budget / change risk | Operators can ask how much budget is left and get a plain recommendation such as healthy / caution / freeze / exhausted (wording may vary slightly by skill version). |
| SLO definition bootstrap & drift PRs | Discover or reconcile standard OpenSLO definition files via GitHub PRs so Git stays the source of truth for what “good” means. |
| Prioritization | What gets investigated first is driven by alert routing, schedules, and policies—not an opaque AI that silently reorders your whole ticket queue. |
Situation → module mapping: Use-case catalog (SLO health rows).
4. Data privacy & PII
| Control | Plain meaning |
|---|---|
| Personal-data scrubbing before the AI model | On by default. Likely personal data in chat and tool content is redacted before it is sent to the model. Best-effort—not a promise that every novel format is caught without extra rules. |
| Stream / export redaction | Chat streams and exports follow the same redaction posture so operators are not surprised by raw personal data in the UI trail. |
| Per-agent hide / keep rules | Tune one agent: e.g. always hide employee badge IDs (EMP-######) while keeping Grafana dashboard IDs needed for Explore links. Demo path: incident-triage PII overlays. |
| Policy firewall | Written policy rules can allow, deny, or require human approval for sensitive tools—see Writing Rego policies. |
| Vault secrets | Credentials stay in vault-backed secret stores; agents receive resolved access without baking secrets into prompts. |
| Approve-before-change | Destructive or write paths can pause for a human before they run. |
| Audit & deployment modes | Actions are auditable; SaaS, private SaaS, and self-hosted postures change where compute and models run—ask your SE which mode matches the deal. |
For the deeper diligence questions—are vault secrets in telemetry? can a vendor store PII from agent traces? how do we scrub?—see the self-contained Telemetry, PII & vault secrets FAQ.
For the security team (under NDA)
Ask your StackGen SE for the formal packs (not published as public marketing pages):
- Data egress matrix — what leaves the tenant boundary, to where, and under which controls
- Security & Trust overview — product security posture in buyer language
- SOC 2 / ISO evidence — shared under NDA as appropriate
Enterprise rollout context (webhooks, skills sync, LLM egress milestones): Enterprise deployment profile.
Closing
A strong POC ends with customer-owned numbers, documented retention days, a clear SLO alert → investigation → digest story, and a security checklist your InfoSec lead can sign off on with the SE packs above—not with a single marketing accuracy percentage.