jmcqueen
f7cda8f0c1
Reliability / correctness:
- Always arm the graceful-shutdown safety timeout. Previously
`shutdown(force=true)` (called from uncaughtException) skipped the
timeout entirely, so a hung `framework.stop()` after a crash would
wedge the process until Docker's SIGKILL. Now uses 3s when forced,
8s otherwise, and .unref()s so it never blocks a clean exit.
- Attach a `.catch()` to `framework.start()` so a bad Webex token or
WebSocket handshake failure produces a clear "Webex framework failed
to start" error line instead of a bare Unhandled Rejection while the
bot silently stays dead.
- Rename MDM timestamp labels from "(EDT)" to "(ET)" since the
formatter uses DST-aware America/New_York (half the year it's EST).
Cleanup:
- Drop `body-parser` in favor of the built-in `express.json()`
(Express 4.16+). Removes one direct dep; still present as a
transitive dep of express itself.
- Remove orphaned JSDoc block referring to a helper that no longer
exists.
- Delete legacy `query-offline.js` (marked deprecated since the bot
`offline` command shipped) and remove its `APPSPACE_API_TOKEN` /
`APPSPACE_BASE_URL` env vars from `.env.example` and the
`offline:legacy` npm script from `package.json`.
Config / metadata:
- Add `"engines": { "node": ">=20" }` to package.json so npm warns on
the wrong Node version instead of just the README saying so.
- Document `SMOKE_TEST=true` in `.env.example`.
Docs:
- Rewrite README to document the `restart-offline` command (iOS
Supervised requirement, 50-device cap, concurrency, audit log
fields, fresh-at-execute semantics), the ET-not-EDT labeling,
structured error logging, character-budget rendering, and the
new CI workflow. Refresh the TODO section to reflect what has
actually shipped.
CI:
- Add `.gitea/workflows/ci.yml` with two jobs: syntax check
(`node --check` on index.js and mdm.js) and a Docker smoke test
that builds the production image, boots it with dummy credentials
+ SMOKE_TEST=true, and waits up to 30s for the container's
built-in healthcheck to reach `healthy`. Dumps container logs
on failure.
Co-authored-by: Cursor <cursoragent@cursor.com>
137 lines
8.1 KiB
Markdown
137 lines
8.1 KiB
Markdown
# Appspace Webex Alerts
|
|
|
|
Lightweight Node/Express service that bridges **Appspace** device health events to **Cisco Webex** via Adaptive Cards, with optional enrichment (and remediation) from **Workspace ONE MDM**.
|
|
|
|
It also runs an interactive Webex bot (WebSocket mode) for on-demand queries and actions.
|
|
|
|
## What it does
|
|
|
|
- Listens for Appspace outbound webhooks (`DEVICE.HEALTHSTATUS.*` and `DEVICE.UNREGISTERED`).
|
|
- Ignores PWA devices.
|
|
- Enriches alerts with current MDM data (model, OS version, compliance, last sample/seen timestamps in Eastern Time — DST-aware).
|
|
- Posts formatted Adaptive Cards to a configured Webex room (with direct links to both consoles).
|
|
- Provides a Webex bot with commands (see below).
|
|
|
|
## Requirements
|
|
|
|
- Node 20+ (enforced via `engines` in `package.json`)
|
|
- Appspace instance with outbound webhook + Application refresh token
|
|
- Webex bot token + room ID
|
|
- (Optional but recommended) Workspace ONE MDM OAuth client for enrichment and reboot
|
|
|
|
## Quick Start (Docker — recommended)
|
|
|
|
1. Copy env:
|
|
```bash
|
|
cp .env.example .env
|
|
# or .env.dev for the dev profile
|
|
```
|
|
|
|
2. Fill in the required values (see `.env.example` for descriptions).
|
|
|
|
3. Run:
|
|
```bash
|
|
# Production profile
|
|
npm run docker:prod
|
|
|
|
# Development (with live reload + volume mount)
|
|
npm run docker:dev
|
|
|
|
# Smoke test (builds image + verifies /health responds "healthy" inside container)
|
|
npm run docker:smoke
|
|
```
|
|
|
|
**Port mapping notes**: The container always listens internally on port 3000 (hardened default). Host port 1889 is used for both dev (`docker compose --profile dev up -d app-dev`) and prod (`docker compose up -d`). The `PORT` env inside the container is forced to 3000 via compose. You cannot run both profiles at the same time due to the shared host port.
|
|
|
|
Direct (no Docker):
|
|
```bash
|
|
npm install
|
|
npm start
|
|
```
|
|
|
|
Health check: `GET /health`
|
|
|
|
## Bot Commands
|
|
|
|
In the Webex space where the bot is added — either as a DM to the bot, or `@mention` in a group space:
|
|
|
|
| Command | What it does |
|
|
| --- | --- |
|
|
| `offline` | Snapshot of all currently offline / lost / failed Appspace devices, grouped by location, enriched with per-device MDM facts and clickable Appspace + Workspace ONE console links. |
|
|
| `offline <filter>` | Same, narrowed to devices whose `deviceType` or name contains the filter substring. E.g. `offline ios`, `offline lobby`. |
|
|
| `restart-offline` | Sends a Workspace ONE **SoftReset** (reboot) to every currently-offline device that maps to a WS1 record by serial. Capped at **50** devices per invocation. Skipped devices (no WS1 record) are reported separately. |
|
|
| `restart-offline <filter>` | Same, narrowed to devices matching the filter. Use this to get under the 50-device cap for large fleets (e.g. `restart-offline ios`). |
|
|
| `help` | Print the command list. |
|
|
|
|
Notes on `restart-offline`:
|
|
|
|
- **Fresh at execute time.** The command re-queries Appspace at the moment of execution, so devices that came back online after you last looked are automatically excluded.
|
|
- **No confirmation prompt.** Configured for immediate execution per project preference — narrow with a filter if you want to limit scope.
|
|
- **iOS Supervised requirement.** WS1's SoftReset only actually reboots iOS devices that are Supervised (DEP-enrolled). Non-Supervised devices will surface a clean WS1 error in the failure list; they will not silently appear to succeed.
|
|
- **Rate-limited.** Reboots are sent with concurrency = 3 to avoid hammering the WS1 API.
|
|
- **Auditable.** Every invocation emits a JSON log line with `user`, `filter`, `matched`, `withMdm`, `withoutMdm`, `succeeded`, and `failed` counts.
|
|
|
|
The bot runs in WebSocket mode (no public webhook required).
|
|
|
|
## Environment Variables
|
|
|
|
See `.env.example` for the full documented list.
|
|
|
|
Key ones:
|
|
- `WEBEX_BOT_TOKEN`, `WEBEX_ROOM_ID`
|
|
- `APPSPACE_INSTANCE_URL`, `APPSPACE_SUBJECT_ID`, `APPSPACE_REFRESH_TOKEN`, `APPSPACE_API_BASE_URL`
|
|
- `APPSPACE_CONSOLE_BASE_URL` (used to build per-device Appspace links in Webex cards)
|
|
- `WS1_BASE_URL` (API hostname, e.g. `as1991.awmdm.com`)
|
|
- `WS1_CONSOLE_BASE_URL` (console hostname, e.g. `cn1896.awmdm.com` — different from the API host; used for clickable console links)
|
|
- `WS1_CLIENT_ID`, `WS1_CLIENT_SECRET`, `WS1_TENANT_CODE`
|
|
- `WEBHOOK_SECRET` (recommended for the Appspace webhook; validated via `x-webhook-secret` header)
|
|
- `DEBUG`, `DEBUG_WEBHOOK`, `LOG_FORMAT` (see Debugging)
|
|
|
|
## Debugging
|
|
|
|
- `DEBUG=true` — verbose logging for MDM lookups, ignored events, command handling, etc. (very useful in dev, noisy in prod).
|
|
- `DEBUG_WEBHOOK=true` — log the **full** incoming Appspace webhook payload (contains device details; do **not** leave on in production).
|
|
- `LOG_FORMAT=json` (or `NODE_ENV=production`) — output structured JSON logs (ideal for Docker/K8s log collectors).
|
|
- `SMOKE_TEST=true` — skips Webex bot framework startup so the container can reach healthy status with dummy credentials. Used only by the smoke test and CI; do not set in prod.
|
|
|
|
## Production & Docker Notes
|
|
|
|
- **Graceful shutdown**: The service handles `SIGTERM` (used by `docker stop`, Kubernetes, etc.) and `SIGINT`. It will:
|
|
1. Stop the Webex WebSocket framework (important to avoid "excessive device registrations").
|
|
2. Close the HTTP server.
|
|
3. Exit cleanly. A hard safety timeout forces exit after ~8s (or ~3s from a crash handler).
|
|
- **Healthcheck**: `/health` returns 200 with basic status. Used by Docker and orchestrators.
|
|
- **Logging**: Logs go to stdout/stderr (12-factor / Docker friendly). Use `LOG_FORMAT=json` or `NODE_ENV=production` for structured JSON. Use `DEBUG=true` in non-prod for detail. Pipe to a collector (Loki, CloudWatch, etc.) as needed. Errors (including axios failures) are serialized with `message`, `stack`, `code`, `responseStatus`, and `responseData` so failures are actually visible in logs.
|
|
- **Secrets**: Never bake secrets into the image. Use:
|
|
- `env_file` for compose (dev/staging only)
|
|
- Docker secrets, Kubernetes Secrets, or a secrets manager (Vault, AWS Secrets Manager) for production.
|
|
- **Ports**: Container always listens on 3000 internally. Map host ports as needed (see docker-compose.yml).
|
|
- **Non-root**: Production image runs as the `node` user.
|
|
- **Resources**: In production, set CPU/memory limits in your orchestrator. The bot command does a full device list scan (limit 500) — monitor for large fleets.
|
|
|
|
## Architecture Notes
|
|
|
|
- Appspace token uses refresh-token flow with cooldown, safety buffer, and in-flight promise coalescing so burst traffic doesn't stampede the token endpoint.
|
|
- MDM uses a 24h serial→Id cache + fresh detail lookup by Id on every alert (for up-to-date compliance/last-seen). Cache refresh and OAuth token fetch are also coalesced.
|
|
- WebSocket mode for the bot avoids the WebSocket-restart rate limits associated with the webhook mode.
|
|
- All enrichment is best-effort; alerts are never blocked by MDM or token issues.
|
|
- Webex message rendering respects the 7439-character pre-encryption limit by building bodies incrementally against a character budget, with accurate "N more not shown" truncation notes.
|
|
- Bot command matching uses string phrases so the framework's `(^| )phrase($| )` wrapper handles group-space `@mentions` correctly. Filter parsing works identically in DMs and mentioned messages via a shared helper.
|
|
|
|
## CI
|
|
|
|
`.gitea/workflows/ci.yml` builds the Docker image on every push to `main` and every pull request, then boots the container with dummy credentials (`SMOKE_TEST=true`) and verifies the built-in healthcheck reaches `healthy`. Fails the run and dumps container logs if it doesn't.
|
|
|
|
You can reproduce the same check locally with `npm run docker:smoke`.
|
|
|
|
## License / Support
|
|
|
|
Internal tool. Tweak as needed.
|
|
|
|
## TODO / Future
|
|
|
|
- Server-side filtering for the devices list when the Appspace API supports it reliably.
|
|
- Optional metrics endpoint (`/metrics` in Prometheus format).
|
|
- Support for more Appspace event types (e.g. content push failures, if useful).
|
|
- Multi-stage Dockerfile for even smaller prod images (current image is already Alpine + prod-only npm deps).
|
|
- Optional email allowlist for `restart-offline` (env-driven), if the current "any user in the bot's space can invoke it" policy becomes too permissive.
|