Telemetry, PII & vault secrets FAQ

Security and diligence often ask the same three questions about Aiden and the SRE agent: Do vault secrets show up in telemetry? If chat or tool output has PII or pasted secrets, does a vendor keep them? How do we delete or scrub that data?

This page is the public, self-contained answer for SEs and customers. The formal channel-by-channel egress matrix stays an NDA pack (ask your SE). Broader POC privacy and retention: Aiden POC — Accuracy, Memory, SLOs & Privacy.

ELI5: Integration passwords live in a locked vault drawer, not in the scoreboard. The scoreboard (metrics) mostly counts events. The diary (agent traces) can quote what people and tools said. We try to black out personal data before the diary is written, but that blackout is a careful filter, not a magic wipe. Cleaning a diary that already left the building means cleaning it at the diary vendor.

Quick answers

Question	Answer
Are vault secrets stored in telemetry?	By design, no. Secret material lives in StackGen Vault (encrypted at rest). Aiden stores refs, resolves secrets at runtime, and logs name + length—not plaintext values.
Can we be sure vendor telemetry never has PII or secrets?	No absolute product guarantee for every sink. Service metrics are low risk. Langfuse receives structure, usage, and by default full observation/trace I/O (watch/debug need it). Opt in via Helm `config.langfuse.sendInfo` (`full` \| `omit` \| `pii_redact`). Recover stream content from local `execution_events` and session messages (not the audit table). LLM providers still see model prompts (subject to boundary redaction).
How do we delete / scrub vendor telemetry?	Not via an Aiden or SRE “delete my vendor traces” API. Use the sink’s own UI, API, or retention (e.g. Langfuse project/trace delete; Observe / metrics retention). Local memory and app purges do not scrub already-exported vendor data.

What “telemetry” means here

Operators often lump three different signals under “telemetry.” Risk differs by signal.

Signal	Typical sink	What it usually contains	Leaves the trust boundary?
Service metrics	Prometheus scrape → StackGen Observe (when wired)	Counts, outcome codes, agent names—not alert bodies or chat text	Only if scrape / remote-write is configured
Agent / LLM / tool traces	Langfuse (when `LANGFUSE_*` keys + host are set)	Tree, model/tool names, tokens/cost, timing, and I/O by default. Helm `config.langfuse.sendInfo` selects how much I/O to send. Full stream also in Aiden `execution_events` / sessions — not the audit table.	Yes when Langfuse is configured (bodies full unless omit/pii_redact)
Infra OTLP spans	Optional collector / Tempo	Infrastructure spans—not the rich agent tree	Only if an OTLP endpoint is set
Structured logs	stdout (shipping is operator choice)	Operational logs; vault resolve logs name + value length	Only if the operator ships logs
LLM provider calls	OpenAI / Anthropic / Azure / self-hosted	Same content as the (redacted or unredacted) model prompt	Yes on public-model deployments; can stay in-network with a self-hosted model

SRE Copilot app metrics are operational counters and error codes (sync failures, investigation starts). They are not designed to carry alert bodies or chat transcripts. Rich investigation traces for SRE workflows still run through Aiden when the agent executes.

flowchart LR
  Run[Aiden_or_SRE_agent_run]
  Run -->|"prompts_and_tool_IO"| LLM[LLM_provider]
  Run -->|"rich_spans_if_configured"| LF[Langfuse]
  Run -->|"counters"| Prom[Prometheus_9091]
  Prom -->|"if_scraped"| Obs[StackGen_Observe]
  Run -.->|"only_if_OTLP_set"| Tempo[Infra_OTLP]
  Vault[StackGen_Vault] -->|"resolve_at_runtime"| Run

Langfuse I/O dial (Helm `sendInfo`)

Watch DAG and execution debug ZIP enrich node input/output from Langfuse today. That is why the default still sends full observation/trace I/O. Diligence can reduce what the vendor stores without turning Langfuse off.

Operator dial (Guild chart ≥ 0.7.2): config.langfuse.sendInfo

Helm `sendInfo`	Env on guild pod (`LANGFUSE_OBSERVATION_BODIES`)	What Langfuse gets for I/O	Watch / debug ZIP I/O
unset / `full` (default)	`off`	Full prompts and tool payloads	Real content
`omit`	`omit`	`[omitted]` placeholder	Shows `[omitted]` (re-reads Langfuse)
`pii_redact`	`pii_redact`	Strings run through global `pii:` scanner → `[HIDDEN:…]`	Still readable; secrets blacked out
invalid / typo	treated as `off`	Full I/O	Real content

Genie uses off omit pii_redact internally. Helm uses full instead of off so the shipping label matches intent (full → Genie off). Structure, tokens, model names, tool names, and timing always export in every mode.

config:
  langfuse:
    enabled: true
    sendInfo: "pii_redact" # full | omit | pii_redact
  # Optional: extra deny/allow patterns (also used by LLM-boundary redaction)
  pii:
    customRegexes:
      - name: employee_id
        pattern: '\bEMP-[0-9]{6}\b'
    safeRegexes:
      - name: grafana_uid
        pattern: '^[A-Za-z0-9]{8,16}$'

Ops notes:

Change takes effect after a guild pod restart (env injection).
Prefer pii_redact for diligence windows before hard omit.
Empty config.pii lists omit the pii: block so Genie built-in defaults stay. If you set safeRegexes, that list replaces Genie’s default allowlist (Guild still appends tool-name / kebab / sha256 safe patterns at startup).
If guild.env already sets LANGFUSE_OBSERVATION_BODIES, that value wins and the chart does not emit a duplicate.
Local/dev: set LANGFUSE_OBSERVATION_BODIES in .env or guild.env. Legacy LANGFUSE_OMIT_OBSERVATION_BODIES=true still maps to omit when the new env is unset.
Historical Langfuse rows are unchanged; only new exports follow the dial.

FAQ

Are vault secrets stored in telemetry?

No—by design.

Credentials and integration secrets live in StackGen Vault. The secret payload is stored encrypted at rest (AES-GCM keyed per secret row).
Aiden holds secret references, not raw secret material in its application database.
At runtime Aiden resolves refs from Vault into the process that needs them (and, for remote runners, uses an encrypted sync path when refs are attached).
Vault-related logs are expected to include the secret name and value length, not the plaintext value.
Prometheus-style metrics are built as counters and stable reason codes—not as dumps of secret values.

So the common claim is correct: secrets are not meant to be written into telemetry; they live in Vault, encrypted.

If the traffic includes PII or secrets, will a vendor store them?

Split vault secrets from everything else.

Kind of data	Vendor storage risk
Vault-managed credentials	Not designed into metrics or traces as secret payloads
Chat text, alert paste, tool results, investigation context	LLM provider: yes (after best-effort redaction). Langfuse: structure/usage yes; I/O full by default; opt-in `omit` or `pii_redact` via Helm/env. Full copy stays in Aiden `execution_events` / sessions.
Pasted API keys, tokens, emails, badge IDs inside that content	May survive if redaction misses them or is skipped

Mitigations (best-effort, not absolute):

LLM-boundary redaction — likely personal data and secret-shaped strings in user messages and tool results are replaced (e.g. [HIDDEN:…]) before the model call. Heuristic scanner; novel formats can slip through.
Export / stream redaction — chat streams and exports follow per-agent policy so operators are not surprised by raw personal data in the UI trail.
Optional egress policy — Rego obligations can require further redaction on tool paths.
Per-agent allow / deny overlays — e.g. keep Grafana rule UIDs visible while hiding employee badge patterns. SRE investigators typically use narrow allow rules, not “redaction disabled.”
Config gates — empty Langfuse keys → no Langfuse export; unset OTLP endpoint → no infra OTLP push; metrics leave only if scraped; Helm config.langfuse.sendInfo defaults to full (→ Genie LANGFUSE_OBSERVATION_BODIES=off). For diligence, prefer pii_redact before hard omit. With pii_redact, operators can add deny/allow regexes via Helm config.pii.customRegexes / safeRegexes.
Local recover path — watch replay and forensics use execution_events and session messages. Watch DAG / debug zip also re-read Langfuse I/O today, so omit blanks those surfaces until a local rehydrate path exists. The audit table is for lifecycle/security events, not a transcript store.

Honest caveats:

Redaction can be skipped on some unattended paths, or gated off by middleware / agent settings.
System prompts and some stream chunk shapes are not fully covered the same way as user/tool content at the model boundary.
pii_redact is heuristic; novel secret shapes can slip through.
You cannot claim vendor telemetry is guaranteed free of PII or accidentally pasted secrets.

What about the SRE agent specifically?

SRE app service metrics → low PII risk (counts and codes).
Investigation runs → agent prompts and tool results still go through Aiden’s LLM + optional Langfuse path, with the same redaction posture and overlays.
Skill guidance says not to put credentials into long-lived memories; platform redaction is re-applied on memory capture when the originating agent’s policy says so.

How do we delete or scrub data?

Where the data is	How to scrub
Langfuse (exported agent traces)	New exports keep I/O by default; opt-in omit or pii_redact via Helm `config.langfuse.sendInfo` (restart guild). Custom PII patterns via `config.pii`. Delete old traces via Langfuse UI/API; set retention. Not an Aiden API.
Observe / metrics / Tempo	Retention TTL and ops procedures for that cluster. Not an Aiden API.
LLM provider	Subject to the provider’s retention and DPA.
Local vector / incident memory	Operator retention sweeper, Memory Explorer delete, bulk memory delete APIs.
SRE app DB (alerts tied to an integration)	Integration data purge paths in the SRE app.
Local `execution_events` / sessions	Primary recover path for chat and tool stream content. Internal cleanup by trace id exists for AG-UI events; it is not a public privacy “delete everything everywhere” API and does not erase Langfuse/Observe.
Audit table	Lifecycle/security events only — not a transcript recover path.

Bottom line: Langfuse is the structure/usage diary with full I/O by default. Diligence can switch to pii_redact or hard omit via Helm/env. Full content for recovery is local (execution_events / sessions). Scrubbing anything already shipped to a vendor is still a vendor/ops action.

Diligence phrasing (copy-friendly)

Integration credentials and vault secrets are stored encrypted in StackGen Vault and are not designed to be written into metrics or traces. Langfuse receives agent structure, tokens, timing, and by default observation/trace I/O (needed for watch/debug). Body handling is selected via Helm config.langfuse.sendInfo (full omit pii_redact; chart maps full → Genie env LANGFUSE_OBSERVATION_BODIES=off). Full conversation and tool stream content also lives in Aiden execution_events and session messages — not the audit table. LLM providers still receive model prompts subject to best-effort boundary redaction. Scrubbing already-exported vendor data is done in the vendor (or via local memory / integration purge APIs), not via an automated cross-vendor scrub.

Resource	Use when
Aiden POC — Accuracy, Memory, SLOs & Privacy	POC stakeholders: scorecard, memory TTL, SLOs, high-level PII controls
Enterprise deployment profile	Rollout milestones including LLM egress posture
Writing Rego policies	Policy firewall for sensitive tools
SE NDA packs (data egress matrix, Security & Trust, SOC 2 / ISO)	Formal channel-by-channel guarantees—ask your SE

Closing

Treat Vault secrets and agent conversation content as different stories. Secrets stay in Vault. Conversation and tool I/O can leave when the LLM and optional trace sinks are configured. Metrics are the quiet scoreboard; traces are the diary. Plan retention and scrubbing for each diary you actually enable.