appspace/README.md

# Appspace Webex Alerts

Lightweight Node/Express service that bridges **Appspace** device health events to **Cisco Webex** via Adaptive Cards, with optional enrichment (and remediation) from **Workspace ONE MDM**.

It also runs an interactive Webex bot (WebSocket mode) for on-demand queries and actions.

## What it does

- Listens for Appspace outbound webhooks (`DEVICE.HEALTHSTATUS.*` and `DEVICE.UNREGISTERED`).
- Ignores PWA devices.
- Enriches alerts with current MDM data (model, OS version, compliance, last sample/seen timestamps in Eastern Time — DST-aware).
- Posts formatted Adaptive Cards to a configured Webex room (with direct links to both consoles).
- Provides a Webex bot with commands (see below).

## Requirements

- Node 20+ (enforced via `engines` in `package.json`)
- Appspace instance with outbound webhook + Application refresh token
- Webex bot token + room ID
- (Optional but recommended) Workspace ONE MDM OAuth client for enrichment and reboot

## Quick Start (Docker — recommended)

1. Copy env:
   ```bash
   cp .env.example .env
   # or .env.dev for the dev profile
   ```

2. Fill in the required values (see `.env.example` for descriptions).

3. Run:
   ```bash
   # Production profile
   npm run docker:prod

   # Development (with live reload + volume mount)
   npm run docker:dev

   # Smoke test (builds image + verifies /health responds "healthy" inside container)
   npm run docker:smoke
   ```

**Port mapping notes**: The container always listens internally on port 3000 (hardened default). Host port 1889 is used for both dev (`docker compose --profile dev up -d app-dev`) and prod (`docker compose up -d`). The `PORT` env inside the container is forced to 3000 via compose. You cannot run both profiles at the same time due to the shared host port.

Direct (no Docker):
```bash
npm install
npm start
```

Health check: `GET /health`

## Bot Commands

In the Webex space where the bot is added — either as a DM to the bot, or `@mention` in a group space:

| Command | What it does |
| --- | --- |
| `offline` | Snapshot of all currently offline / lost / failed Appspace devices, grouped by location, enriched with per-device MDM facts and clickable Appspace + Workspace ONE console links. |
| `offline <filter>` | Same, narrowed to devices whose `deviceType` or name contains the filter substring. E.g. `offline ios`, `offline lobby`. |
| `restart-offline` | Sends a Workspace ONE **SoftReset** (reboot) to every currently-offline device that maps to a WS1 record by serial. Capped at **50** devices per invocation. Skipped devices (no WS1 record) are reported separately. |
| `restart-offline <filter>` | Same, narrowed to devices matching the filter. Use this to get under the 50-device cap for large fleets (e.g. `restart-offline ios`). |
| `help` | Print the command list. |

Notes on `restart-offline`:

- **Fresh at execute time.** The command re-queries Appspace at the moment of execution, so devices that came back online after you last looked are automatically excluded.
- **No confirmation prompt.** Configured for immediate execution per project preference — narrow with a filter if you want to limit scope.
- **iOS Supervised requirement.** WS1's SoftReset only actually reboots iOS devices that are Supervised (DEP-enrolled). Non-Supervised devices will surface a clean WS1 error in the failure list; they will not silently appear to succeed.
- **Rate-limited.** Reboots are sent with concurrency = 3 to avoid hammering the WS1 API.
- **Auditable.** Every invocation emits a JSON log line with `user`, `filter`, `matched`, `withMdm`, `withoutMdm`, `succeeded`, and `failed` counts.

The bot runs in WebSocket mode (no public webhook required).

## Environment Variables

See `.env.example` for the full documented list.

Key ones:
- `WEBEX_BOT_TOKEN`, `WEBEX_ROOM_ID`
- `APPSPACE_INSTANCE_URL`, `APPSPACE_SUBJECT_ID`, `APPSPACE_REFRESH_TOKEN`, `APPSPACE_API_BASE_URL`
- `APPSPACE_CONSOLE_BASE_URL` (used to build per-device Appspace links in Webex cards)
- `WS1_BASE_URL` (API hostname, e.g. `as1991.awmdm.com`)
- `WS1_CONSOLE_BASE_URL` (console hostname, e.g. `cn1896.awmdm.com` — different from the API host; used for clickable console links)
- `WS1_CLIENT_ID`, `WS1_CLIENT_SECRET`, `WS1_TENANT_CODE`
- `WEBHOOK_SECRET` (recommended for the Appspace webhook; validated via `x-webhook-secret` header)
- `DEBUG`, `DEBUG_WEBHOOK`, `LOG_FORMAT` (see Debugging)

## Debugging

- `DEBUG=true` — verbose logging for MDM lookups, ignored events, command handling, etc. (very useful in dev, noisy in prod).
- `DEBUG_WEBHOOK=true` — log the **full** incoming Appspace webhook payload (contains device details; do **not** leave on in production).
- `LOG_FORMAT=json` (or `NODE_ENV=production`) — output structured JSON logs (ideal for Docker/K8s log collectors).
- `SMOKE_TEST=true` — skips Webex bot framework startup so the container can reach healthy status with dummy credentials. Used only by the smoke test and CI; do not set in prod.

## Production & Docker Notes

- **Graceful shutdown**: The service handles `SIGTERM` (used by `docker stop`, Kubernetes, etc.) and `SIGINT`. It will:
  1. Stop the Webex WebSocket framework (important to avoid "excessive device registrations").
  2. Close the HTTP server.
  3. Exit cleanly. A hard safety timeout forces exit after ~8s (or ~3s from a crash handler).
- **Healthcheck**: `/health` returns:
  - **200** when the HTTP server is up AND the Webex bot's Mercury WebSocket appears healthy (or the framework is still initializing during the startup grace window).
  - **503** when a background watchdog has detected the bot's Mercury WebSocket is dead (bot commands would be silently failing even though webhooks still work). Response body includes `bot.mercuryConnected`, `bot.consecutiveFailures`, `bot.lastHealthyAt`, etc. for debugging.
  - `bot` is reported as `"skipped-smoke-test"` when `SMOKE_TEST=true`.
- **Bot watchdog**: The Cisco Webex SDK's Mercury WebSocket (used for the bot in WebSocket mode) can die silently — a network blip, WDM device TTL expiring, or a Cisco-side hiccup — while the HTTP paths (webhook → Webex message API) keep working. This creates the classic "webhook alerts still fire but the bot doesn't respond to commands" symptom. A watchdog polls `webex.internal.mercury.connected` every 60s. If it stays disconnected for ~1 minute, a warning is logged; at ~2 minutes an in-process framework restart is attempted; after ~4 minutes without recovery the process exits so Docker's `restart: unless-stopped` policy brings us back with a fresh Mercury socket. Mercury `offline`/`online` events are also wired for faster detection. On startup, `removeDeviceRegistrationsOnStart` cleans up any WDM device registrations left behind by previous instances (safe for single-instance deployments; revisit if you ever run multiple instances against the same bot token).
- **Logging**: Logs go to stdout/stderr (12-factor / Docker friendly). Use `LOG_FORMAT=json` or `NODE_ENV=production` for structured JSON. Use `DEBUG=true` in non-prod for detail. Pipe to a collector (Loki, CloudWatch, etc.) as needed. Errors (including axios failures) are serialized with `message`, `stack`, `code`, `responseStatus`, and `responseData` so failures are actually visible in logs.
- **Secrets**: Never bake secrets into the image. Use:
  - `env_file` for compose (dev/staging only)
  - Docker secrets, Kubernetes Secrets, or a secrets manager (Vault, AWS Secrets Manager) for production.
- **Ports**: Container always listens on 3000 internally. Map host ports as needed (see docker-compose.yml).
- **Non-root**: Production image runs as the `node` user.
- **Resources**: In production, set CPU/memory limits in your orchestrator. The bot command does a full device list scan (limit 500) — monitor for large fleets.

## Architecture Notes

- Appspace token uses refresh-token flow with cooldown, safety buffer, and in-flight promise coalescing so burst traffic doesn't stampede the token endpoint.
- MDM uses a 24h serial→Id cache + fresh detail lookup by Id on every alert (for up-to-date compliance/last-seen). Cache refresh and OAuth token fetch are also coalesced.
- WebSocket mode for the bot avoids the WebSocket-restart rate limits associated with the webhook mode.
- All enrichment is best-effort; alerts are never blocked by MDM or token issues.
- Webex message rendering respects the 7439-character pre-encryption limit by building bodies incrementally against a character budget, with accurate "N more not shown" truncation notes.
- Bot command matching uses string phrases so the framework's `(^| )phrase($| )` wrapper handles group-space `@mentions` correctly. Filter parsing works identically in DMs and mentioned messages via a shared helper.

## CI

`.gitea/workflows/ci.yml` builds the Docker image on every push to `main` and every pull request, then boots the container with dummy credentials (`SMOKE_TEST=true`) and verifies the built-in healthcheck reaches `healthy`. Fails the run and dumps container logs if it doesn't.

You can reproduce the same check locally with `npm run docker:smoke`.

## License / Support

Internal tool. Tweak as needed.

## TODO / Future

- Server-side filtering for the devices list when the Appspace API supports it reliably.
- Optional metrics endpoint (`/metrics` in Prometheus format).
- Support for more Appspace event types (e.g. content push failures, if useful).
- Multi-stage Dockerfile for even smaller prod images (current image is already Alpine + prod-only npm deps).
- Optional email allowlist for `restart-offline` (env-driven), if the current "any user in the bot's space can invoke it" policy becomes too permissive.