Appspace Webex Alerts

Lightweight Node/Express service that bridges Appspace device health events to Cisco Webex via Adaptive Cards, with optional enrichment (and remediation) from Workspace ONE MDM.

It also runs an interactive Webex bot (WebSocket mode) for on-demand queries and actions.

What it does

• Listens for Appspace outbound webhooks (DEVICE.HEALTHSTATUS.* and DEVICE.UNREGISTERED).
• Ignores PWA devices.
• Enriches alerts with current MDM data (model, OS version, compliance, last sample/seen timestamps in Eastern Time — DST-aware).
• Posts formatted Adaptive Cards to a configured Webex room (with direct links to both consoles).
• Provides a Webex bot with commands (see below).

Requirements

• Node 20+ (enforced via engines in package.json)
• Appspace instance with outbound webhook + Application refresh token
• Webex bot token + room ID
• (Optional but recommended) Workspace ONE MDM OAuth client for enrichment and reboot

Quick Start (Docker — recommended)

1. Copy env:

   cp .env.example .env
   # or .env.dev for the dev profile
   

2. Fill in the required values (see .env.example for descriptions).

3. Run:

   # Production profile
   npm run docker:prod
   
   # Development (with live reload + volume mount)
   npm run docker:dev
   
   # Smoke test (builds image + verifies /health responds "healthy" inside container)
   npm run docker:smoke
   

Port mapping notes: The container always listens internally on port 3000 (hardened default). Host port 1889 is used for both dev (docker compose --profile dev up -d app-dev) and prod (docker compose up -d). The PORT env inside the container is forced to 3000 via compose. You cannot run both profiles at the same time due to the shared host port.

Direct (no Docker):

npm install
npm start

Health check: GET /health

Bot Commands

In the Webex space where the bot is added — either as a DM to the bot, or @mention in a group space:

Command	What it does
offline	Snapshot of all currently offline / lost / failed Appspace devices, grouped by location, enriched with per-device MDM facts and clickable Appspace + Workspace ONE console links.
offline <filter>	Same, narrowed to devices whose deviceType or name contains the filter substring. E.g. offline ios, offline lobby.
restart-offline	Sends a Workspace ONE SoftReset (reboot) to every currently-offline device that maps to a WS1 record by serial. Capped at 50 devices per invocation. Skipped devices (no WS1 record) are reported separately.
restart-offline <filter>	Same, narrowed to devices matching the filter. Use this to get under the 50-device cap for large fleets (e.g. restart-offline ios).
help	Print the command list.

Notes on restart-offline:

• Fresh at execute time. The command re-queries Appspace at the moment of execution, so devices that came back online after you last looked are automatically excluded.
• No confirmation prompt. Configured for immediate execution per project preference — narrow with a filter if you want to limit scope.
• iOS Supervised requirement. WS1's SoftReset only actually reboots iOS devices that are Supervised (DEP-enrolled). Non-Supervised devices will surface a clean WS1 error in the failure list; they will not silently appear to succeed.
• Rate-limited. Reboots are sent with concurrency = 3 to avoid hammering the WS1 API.
• Auditable. Every invocation emits a JSON log line with user, filter, matched, withMdm, withoutMdm, succeeded, and failed counts.

The bot runs in WebSocket mode (no public webhook required).

Environment Variables

See .env.example for the full documented list.

Key ones:

• WEBEX_BOT_TOKEN, WEBEX_ROOM_ID
• APPSPACE_INSTANCE_URL, APPSPACE_SUBJECT_ID, APPSPACE_REFRESH_TOKEN, APPSPACE_API_BASE_URL
• APPSPACE_CONSOLE_BASE_URL (used to build per-device Appspace links in Webex cards)
• WS1_BASE_URL (API hostname, e.g. as1991.awmdm.com)
• WS1_CONSOLE_BASE_URL (console hostname, e.g. cn1896.awmdm.com — different from the API host; used for clickable console links)
• WS1_CLIENT_ID, WS1_CLIENT_SECRET, WS1_TENANT_CODE
• WEBHOOK_SECRET (recommended for the Appspace webhook; validated via x-webhook-secret header)
• DEBUG, DEBUG_WEBHOOK, LOG_FORMAT (see Debugging)

Debugging

• DEBUG=true — verbose logging for MDM lookups, ignored events, command handling, etc. (very useful in dev, noisy in prod).
• DEBUG_WEBHOOK=true — log the full incoming Appspace webhook payload (contains device details; do not leave on in production).
• LOG_FORMAT=json (or NODE_ENV=production) — output structured JSON logs (ideal for Docker/K8s log collectors).
• SMOKE_TEST=true — skips Webex bot framework startup so the container can reach healthy status with dummy credentials. Used only by the smoke test and CI; do not set in prod.

Production & Docker Notes

• Graceful shutdown: The service handles SIGTERM (used by docker stop, Kubernetes, etc.) and SIGINT. It will:
  1. Stop the Webex WebSocket framework (important to avoid "excessive device registrations").
  2. Close the HTTP server.
  3. Exit cleanly. A hard safety timeout forces exit after ~8s (or ~3s from a crash handler).
• Healthcheck: /health returns:
  • 200 when the HTTP server is up AND the Webex bot's Mercury WebSocket appears healthy (or the framework is still initializing during the startup grace window).
  • 503 when a background watchdog has detected the bot's Mercury WebSocket is dead (bot commands would be silently failing even though webhooks still work). Response body includes bot.mercuryConnected, bot.consecutiveFailures, bot.lastHealthyAt, etc. for debugging.
  • bot is reported as "skipped-smoke-test" when SMOKE_TEST=true.
• Bot watchdog: The Cisco Webex SDK's Mercury WebSocket (used for the bot in WebSocket mode) can die silently — a network blip, WDM device TTL expiring, or a Cisco-side hiccup — while the HTTP paths (webhook → Webex message API) keep working. This creates the classic "webhook alerts still fire but the bot doesn't respond to commands" symptom. A watchdog polls webex.internal.mercury.connected every 60s. If it stays disconnected for ~1 minute, a warning is logged; at ~2 minutes an in-process framework restart is attempted; after ~4 minutes without recovery the process exits so Docker's restart: unless-stopped policy brings us back with a fresh Mercury socket. Mercury offline/online events are also wired for faster detection. On startup, removeDeviceRegistrationsOnStart cleans up any WDM device registrations left behind by previous instances (safe for single-instance deployments; revisit if you ever run multiple instances against the same bot token).
• Logging: Logs go to stdout/stderr (12-factor / Docker friendly). Use LOG_FORMAT=json or NODE_ENV=production for structured JSON. Use DEBUG=true in non-prod for detail. Pipe to a collector (Loki, CloudWatch, etc.) as needed. Errors (including axios failures) are serialized with message, stack, code, responseStatus, and responseData so failures are actually visible in logs.
• Secrets: Never bake secrets into the image. Use:
  • env_file for compose (dev/staging only)
  • Docker secrets, Kubernetes Secrets, or a secrets manager (Vault, AWS Secrets Manager) for production.
• Ports: Container always listens on 3000 internally. Map host ports as needed (see docker-compose.yml).
• Non-root: Production image runs as the node user.
• Resources: In production, set CPU/memory limits in your orchestrator. The bot command does a full device list scan (limit 500) — monitor for large fleets.

Architecture Notes

• Appspace token uses refresh-token flow with cooldown, safety buffer, and in-flight promise coalescing so burst traffic doesn't stampede the token endpoint.
• MDM uses a 24h serial→Id cache + fresh detail lookup by Id on every alert (for up-to-date compliance/last-seen). Cache refresh and OAuth token fetch are also coalesced.
• WebSocket mode for the bot avoids the WebSocket-restart rate limits associated with the webhook mode.
• All enrichment is best-effort; alerts are never blocked by MDM or token issues.
• Webex message rendering respects the 7439-character pre-encryption limit by building bodies incrementally against a character budget, with accurate "N more not shown" truncation notes.
• Bot command matching uses string phrases so the framework's (^| )phrase($| ) wrapper handles group-space @mentions correctly. Filter parsing works identically in DMs and mentioned messages via a shared helper.

CI

.gitea/workflows/ci.yml builds the Docker image on every push to main and every pull request, then boots the container with dummy credentials (SMOKE_TEST=true) and verifies the built-in healthcheck reaches healthy. Fails the run and dumps container logs if it doesn't.

You can reproduce the same check locally with npm run docker:smoke.

License / Support

Internal tool. Tweak as needed.

TODO / Future

• Server-side filtering for the devices list when the Appspace API supports it reliably.
• Optional metrics endpoint (/metrics in Prometheus format).
• Support for more Appspace event types (e.g. content push failures, if useful).
• Multi-stage Dockerfile for even smaller prod images (current image is already Alpine + prod-only npm deps).
• Optional email allowlist for restart-offline (env-driven), if the current "any user in the bot's space can invoke it" policy becomes too permissive.