Clean up SD-WAN alarm rendering + explain overlay vs physical

The wanAlarms check was pasting raw Prisma `info` JSON blobs (nested
`vpn_reasons` arrays with element/site/vpnlink ids) into the chat
message field. On a store with 20 NETWORK_ANYNETLINK_DOWN flaps this
produced a wall of unreadable stringified JSON where the actual
signal ("SD-WAN overlay tunnels are flapping") was lost.

Introduces a shared alarmSemantics module that:
  - Buckets each code into overlay / physical / device / other so
    the check + renderer stay consistent
  - Humanizes codes (NETWORK_ANYNETLINK_DOWN → "SD-WAN overlay
    tunnel down") with a Title-Cased fallback for unknown codes
  - Rolls up (code + severity) tuples so 20 identical alarms show as
    a single line with ×20 and a "just now / Nm / Nh / Nd" age

Rewrites wanAlarms.run() to use those helpers + cross-reference the
site's physical link state so operators aren't left wondering why 20
alarms fired while every metric shows green: overlay flaps get a
"physical WAN paths are all up per Link State" clarifier, and
physical alarms point back at the Link State check. The label loses
its hardcoded "(last 1h)" suffix since the alarm window is now
dynamic (defaults to 24h to match the WAN window).

The follow-up renderer used by /phonestatus imports the same helpers
so the two surfaces cannot drift.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
jmcqueen 2026-07-09 10:19:15 -04:00
parent 44ab6559ac
commit 21fa8f1436
7 changed files with 718 additions and 68 deletions

View file

@ -0,0 +1,234 @@
// src/services/enrichment/alarmSemantics.js
//
// Shared alarm-code semantics for Prisma SD-WAN events. Two things
// live here so both the check runner (services/voiceDiag/checks/wan
// /wanAlarms.js) and the follow-up renderer (services/renderers/
// wanDiagnosticsRenderer.js) present the same story:
//
// 1. categorizeAlarm(code) — buckets a raw alarm code into one of:
// - 'overlay' SD-WAN VPN tunnels between peer elements. These
// DO NOT correspond to the physical WAN interfaces
// shown in the Link State / LQM checks; a store
// can have all physical circuits green while the
// overlay flaps repeatedly.
// - 'physical' A specific WAN circuit / interface is affected.
// Should generally line up with what Link State
// reports.
// - 'device' Element / ION appliance itself (heartbeat, boot).
// - 'other' Everything else (LAN, DHCP, config, etc.).
//
// 2. humanizeAlarmCode(code) — one-line English label so the check
// message and the recent-samples list don't dump
// raw Prisma constants at the operator.
//
// 3. rollupAlarms(samples) — groups near-identical alarm samples
// by (code + severity) so 20 successive
// NETWORK_ANYNETLINK_DOWN flaps show as a single
// line with "×20" instead of a 20-line wall.
// Sorted critical → major → minor, then by count.
//
// The category / humanizer tables err on the side of "unknown code
// falls through to a sensible default" — Prisma introduces new codes
// with tenant updates and we do NOT want a new code to blow up the
// diagnostic.
/**
* Alarm category operator meaning.
*
* overlay SD-WAN VPN link between peer elements; will NOT
* appear as a physical link-down in the Link State
* check. Common when a hub site or a peer is temporarily
* unreachable.
* physical An actual WAN circuit / interface is down or
* degraded. Should correlate with Link State / LQM.
* device Element itself (ION appliance) missed heartbeats,
* rebooted, or entered a bad state.
* other LAN / DHCP / configuration / anything else.
*/
const CATEGORY_TABLE = {
// ── overlay (VPN tunnel / SD-WAN mesh) ──────────────────────────
NETWORK_ANYNETLINK_DOWN: 'overlay',
NETWORK_VPNLINK_DOWN: 'overlay',
NETWORK_VPNLINK_FLAP: 'overlay',
NETWORK_VPNLINK_UNREACHABLE: 'overlay',
NETWORK_DIRECTPRIVATE_DOWN: 'overlay',
NETWORK_STANDBY_LINK_DOWN: 'overlay',
NETWORK_SITE_UNREACHABLE: 'overlay',
// ── physical (WAN circuit / interface) ──────────────────────────
NETWORK_INTERNET_DOWN: 'physical',
NETWORK_PUBLICWAN_UNREACHABLE: 'physical',
NETWORK_PRIVATEWAN_UNREACHABLE: 'physical',
NETWORK_DIRECTINTERNET_UNREACHABLE: 'physical',
DEVICE_INTERFACE_STATE_CHANGED: 'physical',
DEVICE_INTERFACE_DOWN: 'physical',
DEVICE_INTERFACE_FLAP: 'physical',
NETWORK_LTE_LINK_DOWN: 'physical',
// ── device (element / ION itself) ───────────────────────────────
DEVICE_HB_MISSED: 'device',
DEVICE_UNREACHABLE: 'device',
DEVICE_REBOOT: 'device',
DEVICE_HIGH_CPU: 'device',
DEVICE_HIGH_MEMORY: 'device',
DEVICE_DISK_UTILIZATION_HIGH: 'device',
DEVICE_SOFTWARE_UPGRADE_FAILED: 'device',
DEVICE_HA_STATE_CHANGE: 'device',
// ── other (LAN / DHCP / config) ─────────────────────────────────
DHCP_POOL_UTILIZATION_HIGH: 'other',
DHCP_FAILURE: 'other',
CONFIG_SYNC_FAILED: 'other',
LAN_INTERFACE_DOWN: 'other',
BGP_PEER_DOWN: 'other',
};
/**
* Human-readable one-liner per alarm code. Kept short operators
* scan the message field, they don't need a paragraph. Falls back to
* a Title-Cased version of the raw code when we don't know it.
*/
const HUMANIZE_TABLE = {
NETWORK_ANYNETLINK_DOWN: 'SD-WAN overlay tunnel down',
NETWORK_VPNLINK_DOWN: 'VPN link to peer site down',
NETWORK_VPNLINK_FLAP: 'VPN link flapping',
NETWORK_VPNLINK_UNREACHABLE: 'VPN link unreachable',
NETWORK_DIRECTPRIVATE_DOWN: 'Direct private link down',
NETWORK_STANDBY_LINK_DOWN: 'Standby link down',
NETWORK_SITE_UNREACHABLE: 'Peer site unreachable',
NETWORK_INTERNET_DOWN: 'Internet WAN circuit down',
NETWORK_PUBLICWAN_UNREACHABLE: 'Public WAN unreachable',
NETWORK_PRIVATEWAN_UNREACHABLE: 'Private WAN unreachable',
NETWORK_DIRECTINTERNET_UNREACHABLE: 'Direct internet unreachable',
NETWORK_LTE_LINK_DOWN: 'LTE link down',
DEVICE_INTERFACE_STATE_CHANGED: 'Physical interface state change',
DEVICE_INTERFACE_DOWN: 'Physical interface down',
DEVICE_INTERFACE_FLAP: 'Physical interface flap',
DEVICE_HB_MISSED: 'ION heartbeat missed',
DEVICE_UNREACHABLE: 'ION unreachable',
DEVICE_REBOOT: 'ION reboot',
DEVICE_HIGH_CPU: 'ION high CPU',
DEVICE_HIGH_MEMORY: 'ION high memory',
DEVICE_DISK_UTILIZATION_HIGH: 'ION disk full',
DEVICE_SOFTWARE_UPGRADE_FAILED: 'Software upgrade failed',
DEVICE_HA_STATE_CHANGE: 'HA state change',
DHCP_POOL_UTILIZATION_HIGH: 'DHCP pool nearly full',
DHCP_FAILURE: 'DHCP failure',
CONFIG_SYNC_FAILED: 'Configuration sync failed',
LAN_INTERFACE_DOWN: 'LAN interface down',
BGP_PEER_DOWN: 'BGP peer down',
};
/**
* Return one of 'overlay' | 'physical' | 'device' | 'other' for the
* given raw alarm code. Case-insensitive on the code. Unknown codes
* bucket to 'other' Prisma adds new codes over time and we prefer
* a benign fallback to a hard failure.
*/
export function categorizeAlarm(code) {
if (!code) return 'other';
return CATEGORY_TABLE[String(code).toUpperCase()] || 'other';
}
/**
* Return a short human label for the alarm code. Falls back to a
* Title-Cased split of the raw constant when we don't recognise it,
* so `SOMETHING_NEW_DOWN` becomes "Something New Down" instead of
* being dumped raw.
*/
export function humanizeAlarmCode(code) {
if (!code) return 'Unknown alarm';
const key = String(code).toUpperCase();
if (HUMANIZE_TABLE[key]) return HUMANIZE_TABLE[key];
return key
.toLowerCase()
.split('_')
.filter(Boolean)
.map((w) => w.charAt(0).toUpperCase() + w.slice(1))
.join(' ');
}
/**
* Roll alarm samples up by (code + severity). Preserves the newest
* timestamp per rollup and orders results critical major minor,
* then by count descending. This turns a wall of 20 near-identical
* NETWORK_ANYNETLINK_DOWN JSON blobs into one scannable line.
*
* Shape of the returned rollups:
* { code, severity, count, newestTs }
*
* (Kept minimal the caller decorates with humanized label + category
* so we don't force every downstream to import both tables.)
*/
export function rollupAlarms(samples) {
if (!Array.isArray(samples)) return [];
const byKey = new Map();
for (const s of samples) {
if (!s) continue;
const key = `${s.severity}::${s.code}`;
const prev = byKey.get(key);
if (prev) {
prev.count += 1;
if (!prev.newestTs || (s.ts && String(s.ts) > String(prev.newestTs))) {
prev.newestTs = s.ts;
}
} else {
byKey.set(key, {
code: s.code,
severity: s.severity,
count: 1,
newestTs: s.ts,
});
}
}
const sevRank = { critical: 0, major: 1, minor: 2, unknown: 3 };
return [...byKey.values()].sort((a, b) => {
const ra = sevRank[a.severity] ?? 9;
const rb = sevRank[b.severity] ?? 9;
if (ra !== rb) return ra - rb;
return b.count - a.count;
});
}
/**
* Count alarms by category from a set of samples.
* Returns `{ overlay, physical, device, other, total }`.
*/
export function countByCategory(samples) {
const counts = { overlay: 0, physical: 0, device: 0, other: 0, total: 0 };
if (!Array.isArray(samples)) return counts;
for (const s of samples) {
if (!s) continue;
counts.total += 1;
counts[categorizeAlarm(s.code)] += 1;
}
return counts;
}
/**
* Given an alarm timestamp (ISO string), return a coarse "how long
* ago" label. Handles ms drift on both sides — "now", "5m ago",
* "2h ago", "3d ago". Falls back to the raw ts on parse failure.
*/
export function humanizeAge(ts, now = Date.now()) {
if (!ts) return '';
const t = Date.parse(ts);
if (!Number.isFinite(t)) return '';
const deltaSec = Math.max(0, Math.floor((now - t) / 1000));
if (deltaSec < 60) return 'just now';
if (deltaSec < 3600) return `${Math.floor(deltaSec / 60)}m ago`;
if (deltaSec < 86_400) return `${Math.floor(deltaSec / 3600)}h ago`;
return `${Math.floor(deltaSec / 86_400)}d ago`;
}
/**
* Human-friendly minutes label (mirrors the renderer helpers).
* Exported so the alarm check can label windows consistently without
* a separate helper import.
*/
export function humanizeWindow(minutes) {
if (!Number.isFinite(minutes) || minutes <= 0) return `${minutes}m`;
if (minutes >= 1440 && minutes % 1440 === 0) return `${minutes / 1440}d`;
if (minutes >= 60 && minutes % 60 === 0) return `${minutes / 60}h`;
return `${minutes}m`;
}

View file

@ -248,19 +248,45 @@ function renderLinkStateDetails(details) {
} }
/** /**
* Alarm counts {critical, major, minor, recentSamples: [...]}. * Alarm counts + optional category breakdown + rolled-up recent
* samples. Handles both the old shape (bare `critical/major/minor +
* recentSamples` array of raw events) and the new shape emitted by
* wanAlarmsCheck (`byCategory`, rollups with `humanized`, `count`,
* `age`).
*/ */
function renderAlarmsDetails(details) { function renderAlarmsDetails(details) {
const { critical = 0, major = 0, minor = 0, recentSamples = [] } = details; const {
const lines = [` - Counts: 🔴 ${critical} critical · 🟠 ${major} major · 🟡 ${minor} minor`]; critical = 0, major = 0, minor = 0,
byCategory, recentSamples = [],
} = details;
const lines = [
` - Counts: 🔴 ${critical} critical · 🟠 ${major} major · 🟡 ${minor} minor`,
];
if (byCategory && typeof byCategory === 'object') {
const parts = [];
if (byCategory.overlay > 0) parts.push(`${byCategory.overlay} overlay/VPN`);
if (byCategory.physical > 0) parts.push(`${byCategory.physical} physical WAN`);
if (byCategory.device > 0) parts.push(`${byCategory.device} device`);
if (byCategory.other > 0) parts.push(`${byCategory.other} other`);
if (parts.length > 0) lines.push(` - Category: ${parts.join(' · ')}`);
}
if (Array.isArray(recentSamples) && recentSamples.length > 0) { if (Array.isArray(recentSamples) && recentSamples.length > 0) {
lines.push(' - Recent:'); lines.push(' - Recent:');
for (const a of recentSamples) { for (const a of recentSamples) {
const type = a?.type || a?.code || a?.alarm_type || 'unknown'; // Prefer humanized label + code; fall back gracefully for
// pre-humanized (raw event) shapes.
const label = a?.humanized || a?.type || a?.code || a?.alarm_type || 'unknown';
const code = a?.code && a?.humanized ? ` \`${a.code}\`` : '';
const sev = a?.severity ? ` (${a.severity})` : ''; const sev = a?.severity ? ` (${a.severity})` : '';
lines.push(` - ${type}${sev}`); const count = Number.isFinite(a?.count) && a.count > 1 ? ` ×${a.count}` : '';
const age = a?.age ? `${a.age}` : '';
lines.push(` - ${label}${code}${sev}${count}${age}`);
} }
} }
return lines.join('\n'); return lines.join('\n');
} }

View file

@ -16,6 +16,13 @@
// so a scanning operator can locate the bad link visually // so a scanning operator can locate the bad link visually
// without reading numbers. // without reading numbers.
import {
categorizeAlarm,
humanizeAlarmCode,
rollupAlarms as sharedRollupAlarms,
humanizeAge,
} from '../enrichment/alarmSemantics.js';
// Thresholds are read from env at render time so the rendered // Thresholds are read from env at render time so the rendered
// icons stay in sync with the check bucket verdicts. Same defaults // icons stay in sync with the check bucket verdicts. Same defaults
// as services/voiceDiag/checks/wan/_helpers.js — kept in sync by // as services/voiceDiag/checks/wan/_helpers.js — kept in sync by
@ -100,16 +107,20 @@ export function renderWanDiagnosticsMarkdown(data, opts = {}) {
if (critical > 0) parts.push(`${critical} critical`); if (critical > 0) parts.push(`${critical} critical`);
if (major > 0) parts.push(`${major} major`); if (major > 0) parts.push(`${major} major`);
if (minor > 0) parts.push(`${minor} minor`); if (minor > 0) parts.push(`${minor} minor`);
out += `\n🚨 Alarms (last 1h): ${parts.join(', ')}\n`; const alarmWindowLabel = data.window?.alarmMinutes
? humanWindowLabel(data.window.alarmMinutes)
: '1h';
out += `\n🚨 Alarms (last ${alarmWindowLabel}): ${parts.join(', ')}\n`;
const rollups = rollupAlarms(data.alarms.samples || []); const rollups = sharedRollupAlarms(data.alarms.samples || []);
for (const r of rollups.slice(0, 5)) { for (const r of rollups.slice(0, 5)) {
const when = r.newestTs ? new Date(r.newestTs).toLocaleTimeString() : ''; const label = humanizeAlarmCode(r.code);
const category = categorizeAlarm(r.code);
const age = r.newestTs ? humanizeAge(r.newestTs) : '';
const count = r.count > 1 ? ` ×${r.count}` : ''; const count = r.count > 1 ? ` ×${r.count}` : '';
out += out +=
` - ${sevIcon(r.severity)} \`${r.code}\`${count}` + ` - ${sevIcon(r.severity)} **${label}**${count} \`${r.code}\`` +
(when ? ` (most recent ${when})` : '') + ` _(${category}${age ? `, ${age}` : ''})_\n`;
`\n`;
} }
if (rollups.length > 5) { if (rollups.length > 5) {
out += ` - _+${rollups.length - 5} more alarm code${rollups.length - 5 === 1 ? '' : 's'}._\n`; out += ` - _+${rollups.length - 5} more alarm code${rollups.length - 5 === 1 ? '' : 's'}._\n`;
@ -204,40 +215,10 @@ function sevIcon(sev) {
return '⚪'; return '⚪';
} }
/** // rollupAlarms is now shared: services/enrichment/alarmSemantics.js
* Roll alarm samples up by (code + severity). Preserves the newest // (imported as sharedRollupAlarms at the top of this file so both
* timestamp per rollup and orders results critical major minor, // this renderer and the wanAlarms check ship the same rollup shape
* then by count descending. This turns a wall of 20 near-identical // + severity ordering).
* NETWORK_ANYNETLINK_DOWN JSON blobs into one scannable line.
*/
function rollupAlarms(samples) {
const byKey = new Map();
for (const s of samples) {
if (!s) continue;
const key = `${s.severity}::${s.code}`;
const prev = byKey.get(key);
if (prev) {
prev.count += 1;
if (!prev.newestTs || (s.ts && String(s.ts) > String(prev.newestTs))) {
prev.newestTs = s.ts;
}
} else {
byKey.set(key, {
code: s.code,
severity: s.severity,
count: 1,
newestTs: s.ts,
});
}
}
const sevRank = { critical: 0, major: 1, minor: 2, unknown: 3 };
return [...byKey.values()].sort((a, b) => {
const ra = sevRank[a.severity] ?? 9;
const rb = sevRank[b.severity] ?? 9;
if (ra !== rb) return ra - rb;
return b.count - a.count;
});
}
/** /**
* Rank a link on a 0-100 badness scale so the worst path floats to * Rank a link on a 0-100 badness scale so the worst path floats to

View file

@ -7,13 +7,36 @@
// - major → warn // - major → warn
// - minor → info-only (stays 'ok', but details show the count) // - minor → info-only (stays 'ok', but details show the count)
// //
// Window defaults to the composer's 1-hour lookback. An "active // The window matches whatever `collectSdwanForStore` used — same
// value the /voicediag "WAN window" banner shows. An "active
// incident" that Prisma sees is materially more urgent than a // incident" that Prisma sees is materially more urgent than a
// snapshot LQM anomaly, so this deliberately runs even when the // snapshot LQM anomaly, so this deliberately runs even when the
// per-metric checks are all green — a critical alarm from 45 min // per-metric checks are all green — a critical alarm 45 min ago
// ago is still the operator's problem. // is still the operator's problem.
//
// The message field DOES NOT dump raw JSON. Prisma populates each
// alarm's `info` field with a nested object (e.g. `{ vpn_reasons:
// [{ code: 'NETWORK_VPNLINK_DOWN', vpnlink_id: '...' }, …] }`) that
// is useful to a person reading the Prisma UI but noise in a chat
// message. Instead we roll samples up by (code + severity), humanize
// the code, and mention the newest occurrence with an "N min ago"
// style relative timestamp.
//
// When alarms fire but every physical Link State check is green,
// the message calls out the mismatch and explains it (usually the
// overlay/VPN mesh is flapping while the physical circuits stay up).
// This addresses the "looks green but 20 alarms" cognitive dissonance
// operators would otherwise have to reason through themselves.
import { maybeSkippedByKillSwitch } from './_helpers.js'; import { maybeSkippedByKillSwitch } from './_helpers.js';
import {
categorizeAlarm,
humanizeAlarmCode,
rollupAlarms,
countByCategory,
humanizeAge,
humanizeWindow,
} from '../../../enrichment/alarmSemantics.js';
export const WAN_ALARMS_STANDARDS = Object.freeze({ export const WAN_ALARMS_STANDARDS = Object.freeze({
criticalAllowed: 0, criticalAllowed: 0,
@ -22,7 +45,11 @@ export const WAN_ALARMS_STANDARDS = Object.freeze({
export const wanAlarmsCheck = { export const wanAlarmsCheck = {
id: 'wanAlarms', id: 'wanAlarms',
label: 'SD-WAN Alarms (last 1h)', // Label is intentionally generic (no window suffix) — the effective
// window is dynamic (comes from ctx.sdwanData.window.alarmMinutes)
// and gets baked into the message rather than the label, since
// labels are static across the process lifetime.
label: 'SD-WAN Alarms',
requires: ['sdwanSite'], requires: ['sdwanSite'],
scope: null, scope: null,
standards: WAN_ALARMS_STANDARDS, standards: WAN_ALARMS_STANDARDS,
@ -43,18 +70,42 @@ export const wanAlarmsCheck = {
const { critical = 0, major = 0, minor = 0 } = alarms.last1h || {}; const { critical = 0, major = 0, minor = 0 } = alarms.last1h || {};
const samples = Array.isArray(alarms.samples) ? alarms.samples : []; const samples = Array.isArray(alarms.samples) ? alarms.samples : [];
const alarmWindowMin = ctx.sdwanData?.window?.alarmMinutes || 60;
const windowLabel = humanizeWindow(alarmWindowMin);
// Roll up samples so 20 identical NETWORK_ANYNETLINK_DOWN alarms
// become 1 line with "×20" — same shape the WAN follow-up renderer
// uses, kept consistent by pulling the helper from a shared module.
const rollups = rollupAlarms(samples);
const byCategory = countByCategory(samples);
// Physical vs overlay heuristic. Only ever a HINT — a store where
// Link State also reports paths down doesn't need us telling them
// "physical is up".
const linksAllUp = allPhysicalLinksUp(ctx?.sdwanData?.links);
const details = { const details = {
critical, major, minor, critical, major, minor,
recentSamples: samples.slice(0, 3), window: windowLabel,
byCategory,
recentSamples: rollups.slice(0, 3).map((r) => ({
code: r.code,
humanized: humanizeAlarmCode(r.code),
category: categorizeAlarm(r.code),
severity: r.severity,
count: r.count,
age: humanizeAge(r.newestTs),
ts: r.newestTs,
})),
}; };
if (critical > 0) { if (critical > 0) {
return { return {
status: 'error', status: 'error',
message: message: buildMessage({
`${critical} critical alarm${critical === 1 ? '' : 's'} raised in the last hour` + count: critical, severityLabel: 'critical',
(samples[0]?.message ? ` — most recent: ${samples[0].code}: ${samples[0].message}` : '.'), windowLabel, rollups, byCategory, linksAllUp,
}),
details, details,
remediation: null, remediation: null,
}; };
@ -63,9 +114,10 @@ export const wanAlarmsCheck = {
if (major > 0) { if (major > 0) {
return { return {
status: 'warn', status: 'warn',
message: message: buildMessage({
`${major} major alarm${major === 1 ? '' : 's'} raised in the last hour` + count: major, severityLabel: 'major',
(samples[0]?.message ? ` — most recent: ${samples[0].code}: ${samples[0].message}` : '.'), windowLabel, rollups, byCategory, linksAllUp,
}),
details, details,
remediation: null, remediation: null,
}; };
@ -74,7 +126,9 @@ export const wanAlarmsCheck = {
if (minor > 0) { if (minor > 0) {
return { return {
status: 'ok', status: 'ok',
message: `${minor} minor alarm${minor === 1 ? '' : 's'} in the last hour, no critical or major.`, message:
`${minor} minor alarm${minor === 1 ? '' : 's'} in the last ` +
`${windowLabel}, no critical or major.`,
details, details,
remediation: null, remediation: null,
}; };
@ -82,9 +136,76 @@ export const wanAlarmsCheck = {
return { return {
status: 'ok', status: 'ok',
message: 'No alarms raised in the last hour.', message: `No alarms raised in the last ${windowLabel}.`,
details, details,
remediation: null, remediation: null,
}; };
}, },
}; };
/**
* Build the WARN / ERROR message body. Structure:
* <N> <sev> alarm(s) in last <window> most common: <humanized code>
* (×N, latest <N> min ago). <optional physical-vs-overlay hint>.
*/
function buildMessage({
count, severityLabel, windowLabel, rollups, byCategory, linksAllUp,
}) {
// Top rollup entry drives the human summary. Preserves severity
// ordering (critical > major > minor) so the "most notable" alarm
// is what we cite even if a lower-severity code has a higher count.
const top = rollups[0];
const parts = [
`${count} ${severityLabel} alarm${count === 1 ? '' : 's'} in the last ${windowLabel}`,
];
if (top) {
const label = humanizeAlarmCode(top.code);
const suffix = [];
if (top.count > 1) suffix.push(`×${top.count}`);
if (top.newestTs) suffix.push(`latest ${humanizeAge(top.newestTs)}`);
const suffixStr = suffix.length > 0 ? ` (${suffix.join(', ')})` : '';
parts.push(` — most common: ${label}${suffixStr}`);
}
// Physical-vs-overlay clarifier — the most common source of
// "looks green but 20 alarms" confusion. Only emit when we
// can confidently distinguish the two cases.
const overlayHeavy = byCategory.overlay > 0
&& byCategory.overlay >= byCategory.physical;
if (overlayHeavy && linksAllUp === true) {
parts.push(
'. These affect SD-WAN overlay/VPN tunnels between sites; ' +
'physical WAN paths are all up per the Link State check.',
);
} else if (byCategory.physical > 0 && linksAllUp === false) {
parts.push(
'. Physical WAN interfaces are affected — see the Link State ' +
'check for the impacted path(s).',
);
} else if (byCategory.device > 0) {
parts.push('. Device / ION appliance affected.');
} else {
parts.push('.');
}
return parts.join('');
}
/**
* True iff every physical link on the site reports up. Returns
* `null` when the link array is missing or empty so the caller can
* decide whether to emit the physical-vs-overlay hint at all.
*/
function allPhysicalLinksUp(links) {
if (!Array.isArray(links) || links.length === 0) return null;
let seen = 0;
for (const l of links) {
// Unknown state → treat as "cannot confirm" and short-circuit
// rather than lie about the state.
if (l?.up === null || l?.up === undefined) continue;
seen += 1;
if (l.up === false) return false;
}
return seen > 0 ? true : null;
}

View file

@ -0,0 +1,179 @@
// Unit tests for services/enrichment/alarmSemantics.js
//
// Coverage targets:
// - Known codes map to their expected category
// - Unknown codes fall through to 'other' (never throw)
// - Humanizer returns a friendly string; unknown codes become
// Title-Cased words rather than a raw constant
// - rollupAlarms collapses (code+severity) tuples, preserves the
// newest timestamp, and orders critical → major → minor
// - countByCategory sums correctly and defaults missing fields to 0
// - humanizeAge produces the expected "just now / Nm / Nh / Nd" bands
// - humanizeWindow formats 15/60/360/1440 minutes as 15m/1h/6h/1d
import test from 'node:test';
import assert from 'node:assert/strict';
import {
categorizeAlarm,
humanizeAlarmCode,
rollupAlarms,
countByCategory,
humanizeAge,
humanizeWindow,
} from '../services/enrichment/alarmSemantics.js';
// ─── categorizeAlarm ─────────────────────────────────────────────────
test('categorizeAlarm: overlay codes bucket as overlay', () => {
for (const code of [
'NETWORK_ANYNETLINK_DOWN',
'NETWORK_VPNLINK_DOWN',
'NETWORK_VPNLINK_FLAP',
'NETWORK_SITE_UNREACHABLE',
'NETWORK_STANDBY_LINK_DOWN',
]) {
assert.equal(categorizeAlarm(code), 'overlay', `${code} → overlay`);
}
});
test('categorizeAlarm: physical WAN codes bucket as physical', () => {
for (const code of [
'NETWORK_INTERNET_DOWN',
'DEVICE_INTERFACE_DOWN',
'DEVICE_INTERFACE_STATE_CHANGED',
'NETWORK_LTE_LINK_DOWN',
]) {
assert.equal(categorizeAlarm(code), 'physical', `${code} → physical`);
}
});
test('categorizeAlarm: device / ION codes bucket as device', () => {
for (const code of [
'DEVICE_HB_MISSED',
'DEVICE_UNREACHABLE',
'DEVICE_HIGH_CPU',
'DEVICE_REBOOT',
]) {
assert.equal(categorizeAlarm(code), 'device');
}
});
test('categorizeAlarm: unknown / falsy → other (never throws)', () => {
assert.equal(categorizeAlarm('SOMETHING_BRAND_NEW'), 'other');
assert.equal(categorizeAlarm(''), 'other');
assert.equal(categorizeAlarm(null), 'other');
assert.equal(categorizeAlarm(undefined), 'other');
});
test('categorizeAlarm: case-insensitive on input', () => {
assert.equal(categorizeAlarm('network_anynetlink_down'), 'overlay');
});
// ─── humanizeAlarmCode ───────────────────────────────────────────────
test('humanizeAlarmCode: known codes → friendly labels', () => {
assert.equal(humanizeAlarmCode('NETWORK_ANYNETLINK_DOWN'), 'SD-WAN overlay tunnel down');
assert.equal(humanizeAlarmCode('DEVICE_HB_MISSED'), 'ION heartbeat missed');
assert.equal(humanizeAlarmCode('NETWORK_INTERNET_DOWN'), 'Internet WAN circuit down');
});
test('humanizeAlarmCode: unknown code → Title Cased fallback (not raw constant)', () => {
assert.equal(humanizeAlarmCode('SOMETHING_BRAND_NEW_DOWN'), 'Something Brand New Down');
});
test('humanizeAlarmCode: falsy → generic fallback', () => {
assert.equal(humanizeAlarmCode(''), 'Unknown alarm');
assert.equal(humanizeAlarmCode(null), 'Unknown alarm');
});
// ─── rollupAlarms ────────────────────────────────────────────────────
test('rollupAlarms: collapses (code + severity) tuples, keeps newest ts', () => {
const samples = [
{ code: 'X', severity: 'major', ts: '2026-07-09T10:00:00Z' },
{ code: 'X', severity: 'major', ts: '2026-07-09T12:00:00Z' },
{ code: 'X', severity: 'major', ts: '2026-07-09T09:00:00Z' },
{ code: 'Y', severity: 'critical', ts: '2026-07-09T08:00:00Z' },
];
const rollups = rollupAlarms(samples);
assert.equal(rollups.length, 2);
// Critical severity floats to the top regardless of count.
assert.equal(rollups[0].code, 'Y');
assert.equal(rollups[0].count, 1);
assert.equal(rollups[1].code, 'X');
assert.equal(rollups[1].count, 3);
// Newest ts survives.
assert.equal(rollups[1].newestTs, '2026-07-09T12:00:00Z');
});
test('rollupAlarms: sort — critical > major > minor, ties broken by count desc', () => {
const rollups = rollupAlarms([
{ code: 'A', severity: 'minor', ts: 't1' },
{ code: 'B', severity: 'major', ts: 't2' },
{ code: 'C', severity: 'critical', ts: 't3' },
{ code: 'D', severity: 'major', ts: 't4' },
{ code: 'D', severity: 'major', ts: 't5' },
]);
assert.deepEqual(
rollups.map((r) => r.code),
['C', 'D', 'B', 'A'],
'critical (C), then major sorted by count (D×2, B×1), then minor',
);
});
test('rollupAlarms: null/empty → []', () => {
assert.deepEqual(rollupAlarms(null), []);
assert.deepEqual(rollupAlarms([]), []);
assert.deepEqual(rollupAlarms(undefined), []);
});
// ─── countByCategory ─────────────────────────────────────────────────
test('countByCategory: counts overlay/physical/device/other + total', () => {
const c = countByCategory([
{ code: 'NETWORK_ANYNETLINK_DOWN' },
{ code: 'NETWORK_ANYNETLINK_DOWN' },
{ code: 'NETWORK_INTERNET_DOWN' },
{ code: 'DEVICE_HB_MISSED' },
{ code: 'UNRECOGNIZED_FOO' },
]);
assert.deepEqual(c, {
overlay: 2, physical: 1, device: 1, other: 1, total: 5,
});
});
test('countByCategory: empty → all zeros', () => {
assert.deepEqual(countByCategory([]), {
overlay: 0, physical: 0, device: 0, other: 0, total: 0,
});
});
// ─── humanizeAge ─────────────────────────────────────────────────────
test('humanizeAge: coarse buckets', () => {
const now = Date.parse('2026-07-09T14:00:00Z');
const at = (offsetSec) =>
humanizeAge(new Date(now - offsetSec * 1000).toISOString(), now);
assert.equal(at(10), 'just now');
assert.equal(at(120), '2m ago');
assert.equal(at(3600 * 2), '2h ago');
assert.equal(at(86400 * 3), '3d ago');
});
test('humanizeAge: missing / invalid ts → empty string', () => {
assert.equal(humanizeAge(null), '');
assert.equal(humanizeAge(undefined), '');
assert.equal(humanizeAge('not-a-date'), '');
});
// ─── humanizeWindow ──────────────────────────────────────────────────
test('humanizeWindow: canonical values', () => {
assert.equal(humanizeWindow(15), '15m');
assert.equal(humanizeWindow(45), '45m');
assert.equal(humanizeWindow(60), '1h');
assert.equal(humanizeWindow(360), '6h');
assert.equal(humanizeWindow(1440), '1d');
assert.equal(humanizeWindow(2880), '2d');
});

View file

@ -287,20 +287,86 @@ test('wanAlarms: minor only → ok (informational)', async () => {
assert.equal(r.details.minor, 3); assert.equal(r.details.minor, 3);
}); });
test('wanAlarms: major → warn', async () => { test('wanAlarms: major → warn (message cites humanized code, never raw info blob)', async () => {
const r = await wanAlarmsCheck.run(mkWanCtx({ const r = await wanAlarmsCheck.run(mkWanCtx({
alarms: { last1h: { critical: 0, major: 1, minor: 0 }, samples: [{ code: 'M1', message: 'wobble' }] }, alarms: {
last1h: { critical: 0, major: 1, minor: 0 },
samples: [{
code: 'NETWORK_ANYNETLINK_DOWN',
severity: 'major',
message: '{"vpn_reasons":[{"code":"NETWORK_VPNLINK_DOWN","element_id":"…"}]}',
ts: new Date().toISOString(),
}],
},
})); }));
assert.equal(r.status, 'warn'); assert.equal(r.status, 'warn');
assert.match(r.message, /wobble/); // Humanized code appears — no raw JSON blob.
assert.match(r.message, /SD-WAN overlay tunnel down/);
assert.equal(r.message.includes('vpn_reasons'), false,
'raw Prisma info JSON must never leak into the chat message');
assert.equal(r.message.includes('{"'), false,
'no stringified object leakage');
// Category count captured in details.
assert.equal(r.details.byCategory.overlay, 1);
}); });
test('wanAlarms: critical → error', async () => { test('wanAlarms: critical → error, cites the highest-severity rollup', async () => {
const r = await wanAlarmsCheck.run(mkWanCtx({ const r = await wanAlarmsCheck.run(mkWanCtx({
alarms: { last1h: { critical: 2, major: 0, minor: 0 }, samples: [{ code: 'C1', message: 'boom' }] }, alarms: {
last1h: { critical: 2, major: 0, minor: 0 },
samples: [
{ code: 'DEVICE_UNREACHABLE', severity: 'critical', ts: '2026-07-09T14:00:00Z' },
{ code: 'DEVICE_UNREACHABLE', severity: 'critical', ts: '2026-07-09T14:05:00Z' },
],
},
})); }));
assert.equal(r.status, 'error'); assert.equal(r.status, 'error');
assert.match(r.message, /boom/); assert.match(r.message, /2 critical alarms/);
assert.match(r.message, /ION unreachable/);
assert.match(r.message, /×2/);
});
test('wanAlarms: rollup groups near-identical alarms in details.recentSamples', async () => {
const samples = Array.from({ length: 20 }).map((_, i) => ({
code: 'NETWORK_ANYNETLINK_DOWN',
severity: 'major',
ts: new Date(Date.now() - i * 60_000).toISOString(),
}));
const r = await wanAlarmsCheck.run(mkWanCtx({
alarms: { last1h: { critical: 0, major: 20, minor: 0 }, samples },
}));
assert.equal(r.status, 'warn');
assert.equal(r.details.recentSamples.length, 1,
'20 identical alarms should collapse to a single rolled-up row');
assert.equal(r.details.recentSamples[0].count, 20);
assert.equal(r.details.recentSamples[0].category, 'overlay');
assert.equal(r.details.recentSamples[0].humanized, 'SD-WAN overlay tunnel down');
});
test('wanAlarms: overlay-only + all physical links up → adds clarifying hint', async () => {
const r = await wanAlarmsCheck.run(mkWanCtx({
links: [link(), link({ interfaceId: 'if-bb', up: true })],
alarms: {
last1h: { critical: 0, major: 5, minor: 0 },
samples: Array.from({ length: 5 }).map(() => ({
code: 'NETWORK_VPNLINK_DOWN', severity: 'major',
})),
},
}));
assert.equal(r.status, 'warn');
assert.match(r.message, /physical WAN paths are all up/);
});
test('wanAlarms: physical alarms + a physical link down → different clarifier', async () => {
const r = await wanAlarmsCheck.run(mkWanCtx({
links: [link({ up: false })],
alarms: {
last1h: { critical: 1, major: 0, minor: 0 },
samples: [{ code: 'NETWORK_INTERNET_DOWN', severity: 'critical' }],
},
}));
assert.equal(r.status, 'error');
assert.match(r.message, /Physical WAN interfaces are affected/);
}); });
test('wanAlarms: no alarms feed → skipped', async () => { test('wanAlarms: no alarms feed → skipped', async () => {
@ -310,6 +376,19 @@ test('wanAlarms: no alarms feed → skipped', async () => {
assert.equal(r.status, 'skipped'); assert.equal(r.status, 'skipped');
}); });
test('wanAlarms: message reflects the effective alarm window (not hardcoded 1h)', async () => {
const ctx = mkWanCtx({
alarms: {
last1h: { critical: 0, major: 1, minor: 0 },
samples: [{ code: 'NETWORK_ANYNETLINK_DOWN', severity: 'major' }],
},
});
// Simulate a 24h alarm window (matches the shipping default)
ctx.sdwanData.window = { minutes: 1440, alarmMinutes: 1440 };
const r = await wanAlarmsCheck.run(ctx);
assert.match(r.message, /in the last 1d/);
});
// ─── standards regression + registry ordering ────────────────────── // ─── standards regression + registry ordering ──────────────────────
test('every WAN check exposes a standards object', () => { test('every WAN check exposes a standards object', () => {

View file

@ -220,7 +220,7 @@ test('renderer: link-state shape → up/down/unknown roll-up + offender list', (
assert.equal(md.includes('"offenders"'), false); assert.equal(md.includes('"offenders"'), false);
}); });
test('renderer: alarm shape → counts one-liner + recent alarm list', () => { test('renderer: alarm shape → counts one-liner + recent alarm list (raw code fallback)', () => {
const results = [ const results = [
R('wanAlarms', 'warn', '2 major alarms in last hour', null, { R('wanAlarms', 'warn', '2 major alarms in last hour', null, {
critical: 0, major: 2, minor: 1, critical: 0, major: 2, minor: 1,
@ -237,6 +237,36 @@ test('renderer: alarm shape → counts one-liner + recent alarm list', () => {
assert.match(md, /DEVICE_HB_MISSED \(minor\)/); assert.match(md, /DEVICE_HB_MISSED \(minor\)/);
}); });
test('renderer: alarm shape → new byCategory + humanized rollup format', () => {
const results = [
R('wanAlarms', 'warn', '20 major alarms in the last 1d — most common: SD-WAN overlay tunnel down (×20, latest 3h ago). These affect SD-WAN overlay/VPN tunnels between sites; physical WAN paths are all up per the Link State check.', null, {
critical: 0, major: 20, minor: 0,
window: '1d',
byCategory: { overlay: 20, physical: 0, device: 0, other: 0, total: 20 },
recentSamples: [
{
code: 'NETWORK_ANYNETLINK_DOWN',
humanized: 'SD-WAN overlay tunnel down',
category: 'overlay',
severity: 'major',
count: 20,
age: '3h ago',
},
],
}),
];
const md = renderVoiceDiagMarkdown(results, { storeNum: '782', detailed: true });
// Counts line still present.
assert.match(md, /Counts: 🔴 0 critical · 🟠 20 major · 🟡 0 minor/);
// Category breakdown appears when byCategory is populated.
assert.match(md, /Category: 20 overlay\/VPN/);
// Humanized label first, raw code in backticks, count, and age.
assert.match(md, /SD-WAN overlay tunnel down/);
assert.match(md, /`NETWORK_ANYNETLINK_DOWN`/);
assert.match(md, /×20/);
assert.match(md, /3h ago/);
});
test('renderer: healthscore shape → breakdown only (value already in message)', () => { test('renderer: healthscore shape → breakdown only (value already in message)', () => {
const results = [ const results = [
R('wanHealthscore', 'ok', 'Healthscore 100/100 (>=80).', null, { R('wanHealthscore', 'ok', 'Healthscore 100/100 (>=80).', null, {