The wanAlarms check was pasting raw Prisma `info` JSON blobs (nested
`vpn_reasons` arrays with element/site/vpnlink ids) into the chat
message field. On a store with 20 NETWORK_ANYNETLINK_DOWN flaps this
produced a wall of unreadable stringified JSON where the actual
signal ("SD-WAN overlay tunnels are flapping") was lost.
Introduces a shared alarmSemantics module that:
- Buckets each code into overlay / physical / device / other so
the check + renderer stay consistent
- Humanizes codes (NETWORK_ANYNETLINK_DOWN → "SD-WAN overlay
tunnel down") with a Title-Cased fallback for unknown codes
- Rolls up (code + severity) tuples so 20 identical alarms show as
a single line with ×20 and a "just now / Nm / Nh / Nd" age
Rewrites wanAlarms.run() to use those helpers + cross-reference the
site's physical link state so operators aren't left wondering why 20
alarms fired while every metric shows green: overlay flaps get a
"physical WAN paths are all up per Link State" clarifier, and
physical alarms point back at the Link State check. The label loses
its hardcoded "(last 1h)" suffix since the alarm window is now
dynamic (defaults to 24h to match the WAN window).
The follow-up renderer used by /phonestatus imports the same helpers
so the two surfaces cannot drift.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|---|---|---|
| .. | ||
| checks | ||
| README.md | ||
| voiceDiagService.js | ||
/voicediag
Rules-engine style diagnostic for the store's Webex Calling
per-user configuration. The orchestrator (voiceDiagService.js)
walks a registry of check modules, gathers their CheckResult
objects, and hands them to the renderer + adaptive-card layer in
commands/voiceDiag.js.
Command
/voicediag <storeNum> default: hides OK checks, posts fixable cards
/voicediag <storeNum> detail include OK checks + expand every details block
/voicediag <storeNum> --only dnd,callForwarding
/voicediag <storeNum> --window 15m narrow the WAN look-back (default 24h)
/voicediag list-checks enumerate every registered check + its scope
HTTP path: GET /voicediag?storeNum=<n>[&detailed=true][&only=dnd,callWaiting][&window=15m].
--window accepts Nm / Nh / Nd shorthand (e.g. 15m, 1h, 6h,
24h, 1d) or a bare integer of minutes. Applies to the WAN
healthscore + LQM (per-path latency/jitter/loss/MOS) fetches. Alarms
are floored at 60m regardless — sub-hour alarm queries are usually
too noisy to be actionable. Global default is
WAN_STANDARD_WINDOW_MINUTES (env, defaults to 1440 / 24h). The
same default is used by the /phonestatus WAN follow-up (no CLI
override on that surface — set the env if you want a different value
globally).
HTTP callers get the markdown snapshot only — remediation cards are
chat-only.
Prerequisites
All checks call /v1/people/{personId}/features/{feature} (with a
few Webex-namespace variants noted inline in each check module).
The scopes required are:
spark-admin:people_read— every read (all checks)spark-admin:people_write— remediations (disable DND, clear forwarding, enable call waiting, disable call intercept)
These are the same scopes /webexhost and /offboarduser already
use, so no operator work is needed to enable /voicediag on an
existing deployment. If the token ever loses those scopes,
each check self-reports as skipped: scope missing — the run still
completes, and the phone-online summary keeps working.
About the URL scheme
Early drafts of this feature documented /v1/telephony/config/people/{id}/callSettings/*.
That path family returns 404 "no static resource" from the Webex API
gateway — it is not a live surface. The correct admin path is
/v1/people/{id}/features/{feature}, per Webex's User Call Settings
docs. A couple of features use different feature-name segments than
their check id — most notably intercept (not callIntercept);
see each check module for the exact endpoint. The runner
distinguishes routing-404s ("URL moved, update the check") from
"not applicable" 404s ("this person isn't a calling user") using
the no static resource marker in the response body.
Store voice standards
The store phone standard is enforced by the checks below. Every
check descriptor exports a standards object so the desired state
is legible from the check file without reading run(). Changes
here are meant to be a two-step change: update standards, then
teach run() to interpret it — the checks compare the live state
against standards and emit the severity in the "when non-compliant"
column.
Per-user Webex Calling standards
| Check | Standard | Non-compliant severity | Auto-remediation |
|---|---|---|---|
dnd |
enabled: false |
warn | disable_dnd |
callForwarding |
{always, busy, noAnswer}.enabled: false |
error | clear_call_forwarding |
callWaiting |
enabled: true |
warn | enable_call_waiting |
callIntercept |
enabled: false |
error | disable_call_intercept |
voicemail |
enabled: true, all three send*Calls.enabled: false, mwiEnabled: true |
error on send-to-VM, warn on MWI-off / disabled / off-org email | stop_sending_to_voicemail for the send-to-VM path only |
hoteling |
enabled: false |
warn | disable_hoteling |
executiveAssistant |
type: 'UNASSIGNED' |
warn | none — Control Hub cleanup |
outgoingPermission |
no high-impact call type BLOCKED (LOCAL, NATIONAL, TOLL_FREE, TOLL, INTERNATIONAL) | warn | none — location-scoped fix |
callForwarding is intentionally at error severity: forwarding
active on a store line silently drops customer calls, and it's the
single most common voice ticket. Voicemail's error path is
narrower — only the three send-* triggers upgrade to error, since
that also silently swallows calls. MWI-off, off-org email
forwarding, and voicemail-disabled all stay at warn.
Switchport / port-hygiene standards
These reuse the phone-status snapshot from /phonestatus — no
extra Webex API calls — and cross-reference against the Meraki
port config that already flows through services/enrichment/merakiEnrichment.js.
Trunk uplinks are treated as a visibility boundary: the phone is
behind a non-Meraki switch (typically a Cisco stack in a store)
and per-port policy isn't ours to enforce, so downstream VLAN /
PoE / admin-state checks defer to the operator.
| Check | Standard | Non-compliant severity | Notes |
|---|---|---|---|
portType |
portType: 'access' on every wired phone / DECT base |
warn | Trunk uplinks flagged so operator checks the downstream switch |
portVlan |
vlan === VOICE_STANDARD_PHONE_VLAN (default 102) |
warn | Env-configurable — VLAN may move from data-side to a proper voice VLAN in the future |
portPoe |
poeEnabled: true |
warn | Skipped for trunk-uplinked devices |
portEnabled |
portEnabled: true |
error | Admin-disabled port → phone is dead |
No auto-remediation on any port check — Meraki port-config PUTs are a separate scope of work; the operator handles fixes in the Meraki Dashboard.
Environment overrides
Both port-hygiene knobs live in .env:
| Var | Default | Purpose |
|---|---|---|
VOICE_STANDARD_PHONE_VLAN |
102 |
Expected VLAN for a store phone. Set per-site if the fleet moves onto a proper voice VLAN. |
VOICE_STANDARD_ENABLED |
true |
Global kill-switch for the port-hygiene bucket. false silences portType / portVlan / portPoe / portEnabled while Meraki cleanup is in progress. Feature-config checks always run. |
WAN standards (Prisma SD-WAN)
Sources Prisma SD-WAN metrics (see integrations/paloalto/) for
the store's site and grades per-path LQM + healthscore + alarms
against ITU-T G.114 / RFC 3550 references. Diagnostic-only — no
auto-remediation, since WAN config PUTs are the operator's job in
the Prisma portal, not the bot's. All checks skip cleanly with
"not applicable — missing sdwanSite" for non-Prisma-managed sites,
so the WAN bucket adds no noise to stores that live entirely
behind Meraki.
Data flow: voiceDiagService.buildContext() calls
collectSdwanForStore(storeNum) in parallel with the Webex +
Meraki fetches. That composer resolves the store to a Prisma site
(CG${pad5(storeNum)} — e.g. store 782 → CG00782), pulls
elements + healthscore + LinkState + 4 LQM metrics + recent alarms
in parallel, and normalises to a stable shape the checks + the
/phonestatus follow-up renderer share.
| Check | Standard | Non-compliant severity | Notes |
|---|---|---|---|
wanSite |
Prisma site resolves for the store | skipped when unresolved | Info-only anchor — surfaces site + element + link counts |
wanHealthscore |
>= WAN_STANDARD_HEALTHSCORE_WARN (default 80) |
warn < 80, error < 60 | Prisma AIOps composite (0-100) |
wanLinkState |
Every WAN path up | error if any path down | No remediation — physical / carrier work |
wanLatency |
<= WAN_STANDARD_LATENCY_WARN_MS (default 150ms) |
warn > 150ms, error > 400ms | ITU-T G.114 one-way reference. Worst path drives severity. |
wanJitter |
<= WAN_STANDARD_JITTER_WARN_MS (default 30ms) |
warn > 30ms, error > 50ms | RFC 3550 jitter-buffer envelope |
wanLoss |
<= WAN_STANDARD_LOSS_WARN_PCT (default 1%) |
warn > 1%, error > 3% | G.711 PLC tolerance |
wanMos |
>= WAN_STANDARD_MOS_WARN (default 4.0) |
warn < 4.0, error < 3.5 | ITU-T P.800 MOS scale. LOW is bad — the accessor flips the comparison. |
wanAlarms |
Zero critical + zero major alarms in the last hour | warn on major, error on critical | Minor alarms are info-only. Pass-through of Prisma severity. |
Environment overrides (all optional — defaults match the standards above):
| Var | Default | Purpose |
|---|---|---|
WAN_STANDARD_LATENCY_WARN_MS |
150 |
Latency warn threshold, ms |
WAN_STANDARD_LATENCY_ERROR_MS |
400 |
Latency error threshold, ms |
WAN_STANDARD_JITTER_WARN_MS |
30 |
Jitter warn threshold, ms |
WAN_STANDARD_JITTER_ERROR_MS |
50 |
Jitter error threshold, ms |
WAN_STANDARD_LOSS_WARN_PCT |
1 |
Packet-loss warn threshold, % |
WAN_STANDARD_LOSS_ERROR_PCT |
3 |
Packet-loss error threshold, % |
WAN_STANDARD_MOS_WARN |
4.0 |
MOS warn threshold (LOW is bad) |
WAN_STANDARD_MOS_ERROR |
3.5 |
MOS error threshold |
WAN_STANDARD_HEALTHSCORE_WARN |
80 |
Site healthscore warn threshold (0-100) |
WAN_STANDARD_HEALTHSCORE_ERROR |
60 |
Site healthscore error threshold |
WAN_STANDARD_ENABLED |
true |
Global kill-switch for the WAN bucket. false silences all 8 WAN checks while a Prisma cleanup / integration validation is in flight. |
Prisma credentials and auth-mode selection live under the
Palo Alto Prisma SD-WAN block in .env.example (SASE OAuth 2.0
recommended, legacy CloudGenix session-token supported for
backward compatibility). See that block for the full env
inventory.
Debugging Prisma schema drift
Prisma SD-WAN's monitor API schemas vary between tenants — request
body shapes documented on pan.dev or in LIVEcommunity examples
frequently fail schema check on this tenant with cryptic 400
messages like "$.filter.wan_interfaces: is not defined in the schema" or "$.view: does not have a value in the enumeration [summary, timeseries]". Iterating on those in-band via
/phonestatus is painful (full DECT + Meraki + WAN pipeline runs
every attempt).
Use the standalone probe script instead:
node scripts/prismaProbe.js discover 782
node scripts/prismaProbe.js try-shapes health <siteId>
node scripts/prismaProbe.js try-shapes lqm <siteId> <wiId1,wiId2>
node scripts/prismaProbe.js raw POST /sdwan/v3.7/api/events/query --body '{"limit":{"count":5}}'
try-shapes fires a curated list of candidate request bodies and
prints which pass schema check (with response-shape summary) and
which fail (with the exact SCHEMA_CHECK_FAIL field). When you find
a winning shape, port it into integrations/paloalto/metrics.js
and add a regression test in tests/paloalto.metrics.test.js so
the tenant's quirk doesn't get "helpfully cleaned up" by a future
refactor. Same auth stack as the bot, no side effects.
Apply-all-N-fixes card
When two or more checks return fixable results, /voicediag
posts one extra adaptive card at the bottom offering to apply the
whole batch in a single click. The individual per-issue cards stay
on-screen so operators can still pick and choose; the combined
card is a shortcut for the common "everything looks right, do it
all" case. The batch executes each fix in sequence (not parallel)
so audit lines stay readable and per-person Webex API write
throttling doesn't stack; failures accumulate into a final summary
line rather than aborting the run.
Dispatcher-side, the combined card uses the same
pendingVoiceFixes map and the same voicediag branch in
index.js — the payload's combined: true flag + the
confirm_voicediag_all / cancel_voicediag_all action ids are
what pick the batch handler over the single-fix handler.
Adding a new check
-
Create
services/voiceDiag/checks/<newCheck>.jsand export a descriptor:// The standards block is *the* source of truth for the desired // state. Keep the run() comparison in sync — regression tests // in tests/voiceDiag.checks.test.js assert every check has one. export const MY_NEW_STANDARDS = Object.freeze({ enabled: false }); export const myNewCheck = { id: 'myNew', label: 'My New Check', requires: ['personId'], // subset of ['personId','phoneStatus','telephonyProfile'] scope: 'spark-admin:people_read', standards: MY_NEW_STANDARDS, async run(ctx) { const data = await ctx.webex.request('GET', `people/${ctx.personId}/features/whatever`); if (!!data?.enabled === MY_NEW_STANDARDS.enabled) { return { status: 'ok', message: 'Compliant.', details: null, remediation: null }; } return { status: 'warn', // 'ok' | 'warn' | 'error' | 'skipped' message: 'Human sentence for the chat row.', details: { relevant: 'facts' }, remediation: { // optional action: 'fix_my_thing', // globally unique remediation id title: 'Fix My Thing', summary: 'One-line description of what the fix will do.', payload: { personId: ctx.personId, personLabel: ctx.personLabel, storeNum: ctx.storeNum }, }, }; }, remediations: { // optional async fix_my_thing(bot, data, requester) { // do the PUT, audit, reply }, }, }; -
Register it in
services/voiceDiag/checks/index.jsat the position you want the renderer to output it. -
Add a
describe-level block totests/voiceDiag.checks.test.jscovering the enabled / disabled / 403 branches.
That's it — no changes to the runner, renderer, command handler,
index.js dispatcher, or commands/registry.js. The single
confirm_voicediag / cancel_voicediag action namespace routes
through the shared dispatcher, which looks the remediation up in the
central registry built at module load. Remediation id collisions
throw at import so you'll notice immediately.
CheckResult shape
type Severity = 'ok' | 'warn' | 'error' | 'skipped';
type Remediation = {
action: string; // globally unique across all checks
title: string; // button label
summary?: string; // optional card body line
payload: Record<string, unknown>; // handed to remediations[action](bot, payload, requester)
};
type CheckResult = {
id: string; // set from check.id automatically
label: string; // set from check.label automatically
status: Severity;
message: string;
details?: object | null; // shown under `detail` mode
remediation?: Remediation | null;
};
Error handling
Never throw from run(). The runner wraps every check in a
try/catch and converts:
- HTTP 401/403/404 →
status: 'skipped'with a "missing scope" hint pointing atcheck.scope. - Any other error →
status: 'error'with the API message.
So if you can't reason about a specific 5xx, just let it bubble — the runner will surface it.