The Cisco Webex SDK's Mercury WebSocket (used by the bot in WebSocket
mode) can die silently — network blip, WDM device TTL expiring, Cisco-
side hiccup — without any error the framework surfaces. When this
happens, the HTTP paths (webhook → Webex message API) keep working but
the bot silently stops receiving commands. This is the classic
"webhook alerts still arrive but the bot ignores me" failure mode.
Detect it by polling the SDK's live `webex.internal.mercury.connected`
boolean every 60s. Track consecutive misses:
- After ~2 min disconnected: log a warning
- After ~5 min disconnected: log an error and trigger graceful
shutdown, so Docker's `restart: unless-stopped` policy brings us
back with a fresh Mercury socket
Complementary changes:
- /health now returns 503 when the watchdog considers the bot dead,
with `bot.mercuryConnected`, `consecutiveFailures`, `lastHealthyAt`
in the JSON body. Docker HEALTHCHECK will start failing too, which
helps external autoheal / K8s liveness probes catch it before the
in-process exit fires. During SMOKE_TEST=true the bot state is
reported as "skipped-smoke-test" so smoke tests still pass.
- Framework 'log' events are forwarded into our structured logger so
framework-internal diagnostics (device registration issues,
membership rule denials, etc.) are actually visible in logs.
- `removeDeviceRegistrationsOnStart: true` cleans up WDM device
registrations left behind by previous silently-dead instances so
they don't accumulate over time. Safe for single-instance
deployments; comment flags the multi-instance caveat.
README updated to document the new health semantics and watchdog.
Co-authored-by: Cursor <cursoragent@cursor.com>
Reliability / correctness:
- Always arm the graceful-shutdown safety timeout. Previously
`shutdown(force=true)` (called from uncaughtException) skipped the
timeout entirely, so a hung `framework.stop()` after a crash would
wedge the process until Docker's SIGKILL. Now uses 3s when forced,
8s otherwise, and .unref()s so it never blocks a clean exit.
- Attach a `.catch()` to `framework.start()` so a bad Webex token or
WebSocket handshake failure produces a clear "Webex framework failed
to start" error line instead of a bare Unhandled Rejection while the
bot silently stays dead.
- Rename MDM timestamp labels from "(EDT)" to "(ET)" since the
formatter uses DST-aware America/New_York (half the year it's EST).
Cleanup:
- Drop `body-parser` in favor of the built-in `express.json()`
(Express 4.16+). Removes one direct dep; still present as a
transitive dep of express itself.
- Remove orphaned JSDoc block referring to a helper that no longer
exists.
- Delete legacy `query-offline.js` (marked deprecated since the bot
`offline` command shipped) and remove its `APPSPACE_API_TOKEN` /
`APPSPACE_BASE_URL` env vars from `.env.example` and the
`offline:legacy` npm script from `package.json`.
Config / metadata:
- Add `"engines": { "node": ">=20" }` to package.json so npm warns on
the wrong Node version instead of just the README saying so.
- Document `SMOKE_TEST=true` in `.env.example`.
Docs:
- Rewrite README to document the `restart-offline` command (iOS
Supervised requirement, 50-device cap, concurrency, audit log
fields, fresh-at-execute semantics), the ET-not-EDT labeling,
structured error logging, character-budget rendering, and the
new CI workflow. Refresh the TODO section to reflect what has
actually shipped.
CI:
- Add `.gitea/workflows/ci.yml` with two jobs: syntax check
(`node --check` on index.js and mdm.js) and a Docker smoke test
that builds the production image, boots it with dummy credentials
+ SMOKE_TEST=true, and waits up to 30s for the container's
built-in healthcheck to reach `healthy`. Dumps container logs
on failure.
Co-authored-by: Cursor <cursoragent@cursor.com>
Node/Express service that:
- Receives Appspace outbound webhooks, enriches with Workspace ONE MDM
data (matched by serial), and posts Adaptive Card alerts to Webex.
- Runs a Webex bot in WebSocket mode with two commands:
* `offline [filter]` - lists currently offline / lost / failed
Appspace devices, enriched with per-device MDM facts + console links.
* `restart-offline [filter]` - sends WS1 SoftReset (reboot) to every
currently-offline device that has a WS1 record. Capped at 50 per
invocation with bounded concurrency to protect the WS1 API.
Notes on hardening already applied:
- In-flight promise coalescing in mdm.js and index.js so burst webhook
traffic can't stampede the WS1 token / device-cache refresh or the
Appspace token refresh.
- Structured logger that serializes Error instances (message, stack,
code, axios response.status/data) instead of stringifying to "{}".
- Webex 7439-char message-limit handling: `offline` builds its body
incrementally against a character budget and reports accurate
"N more not shown" truncation.
- Uses string phrases for `framework.hears(...)` so the framework's
`(^| )phrase($| )` wrapper handles group-space @mentions correctly,
and a shared `extractFilterArg()` helper so filter parsing works
identically in DMs and mentioned messages.
Config, Docker, smoke-test profile, and healthcheck included.
Secrets are managed via `.env` (gitignored); see `.env.example`.
Co-authored-by: Cursor <cursoragent@cursor.com>
