jmcqueen/wxccai
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

[workaround] Cache Assets store lookups via user PAT #8

New issue
Closed
opened 2026-07-07 10:11:59 -04:00 by jmcqueen · 2 comments
jmcqueen commented 2026-07-07 10:11:59 -04:00
Owner
Copy link

Workaround for #1 while the object-type-level permission on Store Address / Hierarchy (109) is being investigated by the Assets admin.

Context

The service account (webex-bot-fzjq885stn) has schema-level read on schema 68 but is silently filtered out for objects in object type 109. Every direct API path (raw Assets API, site gateway, servicedeskapi CMDB proxy) is either 403 or total=0. The customer-portal autocomplete endpoint the JSM UI itself uses (found via HAR) works but requires a browser session cookie — no path with Basic Auth or the OAuth-gateway token gets past the permission wall.

The workaround

A personal Atlassian PAT (Joe's account, verified to have read access to type 109) periodically syncs storeNumber -> objectId into a local cache. resolveStoreAssetReference reads from the cache first; the raw Assets AQL path becomes a fallback.

Stores are the perfect data for this: small (≈ thousands), stable (only adds/removes for new/closed stores), and the lookup is a plain key-value.

Design

Auth compartmentalization — two Atlassian identities in the app:

Identity Used for Where token lives
Service account (webex-bot-fzjq885stn) Everything except the Assets store sync (ticket create/update/comment/attach) .env (existing)
Personal PAT (Joe's account) ONLY the periodic store cache sync + optional live cache-miss lookups macOS Keychain (security add-generic-password -s jira-assets-sync -a mcqueenj@ae.com); env-loaded via helper script; never in .env

Cache layout — two-tier:

  • In-memory Map<paddedStoreNumber, { objectId, objectKey, label, syncedAt }> — sub-ms lookups
  • On-disk JSON at data/stores.json (gitignored) — survives restart, avoids full resync every boot

Sync mechanics:

  • On startup: load data/stores.json; if missing or older than N hours, kick off background full refresh
  • Scheduled: full resync every 24h (configurable via env)
  • On-demand: POST /api/wxccai/admin/storesCache/refresh for immediate refresh
  • Status: GET /api/wxccai/admin/storesCache/status returns { storeCount, lastSyncAt, ageSeconds, syncing, lastError, assetsSyncConfigured }

Updated resolveStoreAssetReference behavior:

  1. Cache hit → return objectId immediately (no external call)
  2. Cache miss + sync PAT configured → one live AQL lookup via the PAT (handles brand-new stores that landed between syncs)
  3. Live PAT miss → fall back to existing service-account AQL path (works once #1 is resolved)
  4. Total miss → clear error naming the store number and the cache's last-sync time

Trade-offs

  • Personal account dependency: sync breaks if Joe's account is offboarded. Mitigation: status endpoint exposes lastSyncAt and lastError so health checks can flag ">48h stale".
  • Doesn't fix the underlying issue: #1 stays open as the real fix. This is expressly a workaround, though the cache is also a legitimate performance/resilience layer that's worth keeping even after #1 lands.
  • Audit trail: full sync shows up in Assets audit logs as "mcqueenj queried objectSchemaId=68". Acceptable.

Acceptance

  • POST /api/wxccai/admin/storesCache/refresh populates the cache successfully with more than 0 stores using Joe's PAT.
  • Cache survives a restart (loaded from data/stores.json).
  • A createSSRequest call for a known store (e.g. 514, 782) succeeds using the cached objectId, without needing the admin fix on #1.
  • Status endpoint returns the expected fields.
  • No secrets committed; PAT lives in Keychain, loaded via helper script.

Non-goals

  • Fixing the object-type-109 permission (that's #1)
  • Changing how tickets are created (still the same JSM POST /request shape)
**Workaround for #1** while the object-type-level permission on `Store Address / Hierarchy (109)` is being investigated by the Assets admin. ### Context The service account (`webex-bot-fzjq885stn`) has schema-level read on schema 68 but is silently filtered out for objects in object type 109. Every direct API path (raw Assets API, site gateway, servicedeskapi CMDB proxy) is either 403 or `total=0`. The customer-portal autocomplete endpoint the JSM UI itself uses (found via HAR) works but requires a browser session cookie — no path with Basic Auth or the OAuth-gateway token gets past the permission wall. ### The workaround A **personal Atlassian PAT** (Joe's account, verified to have read access to type 109) periodically syncs `storeNumber -> objectId` into a local cache. `resolveStoreAssetReference` reads from the cache first; the raw Assets AQL path becomes a fallback. Stores are the perfect data for this: small (≈ thousands), stable (only adds/removes for new/closed stores), and the lookup is a plain key-value. ### Design **Auth compartmentalization** — two Atlassian identities in the app: | Identity | Used for | Where token lives | | -------- | -------- | ----------------- | | Service account (`webex-bot-fzjq885stn`) | Everything except the Assets store sync (ticket create/update/comment/attach) | `.env` (existing) | | Personal PAT (Joe's account) | ONLY the periodic store cache sync + optional live cache-miss lookups | macOS Keychain (`security add-generic-password -s jira-assets-sync -a mcqueenj@ae.com`); env-loaded via helper script; never in `.env` | **Cache layout** — two-tier: - In-memory `Map<paddedStoreNumber, { objectId, objectKey, label, syncedAt }>` — sub-ms lookups - On-disk JSON at `data/stores.json` (gitignored) — survives restart, avoids full resync every boot **Sync mechanics:** - On startup: load `data/stores.json`; if missing or older than N hours, kick off background full refresh - Scheduled: full resync every 24h (configurable via env) - On-demand: `POST /api/wxccai/admin/storesCache/refresh` for immediate refresh - Status: `GET /api/wxccai/admin/storesCache/status` returns `{ storeCount, lastSyncAt, ageSeconds, syncing, lastError, assetsSyncConfigured }` **Updated `resolveStoreAssetReference` behavior:** 1. Cache hit → return objectId immediately (no external call) 2. Cache miss + sync PAT configured → one live AQL lookup via the PAT (handles brand-new stores that landed between syncs) 3. Live PAT miss → fall back to existing service-account AQL path (works once #1 is resolved) 4. Total miss → clear error naming the store number and the cache's last-sync time ### Trade-offs - **Personal account dependency**: sync breaks if Joe's account is offboarded. Mitigation: status endpoint exposes `lastSyncAt` and `lastError` so health checks can flag ">48h stale". - **Doesn't fix the underlying issue**: #1 stays open as the real fix. This is expressly a workaround, though the cache is also a legitimate performance/resilience layer that's worth keeping even after #1 lands. - **Audit trail**: full sync shows up in Assets audit logs as "mcqueenj queried objectSchemaId=68". Acceptable. ### Acceptance - `POST /api/wxccai/admin/storesCache/refresh` populates the cache successfully with more than 0 stores using Joe's PAT. - Cache survives a restart (loaded from `data/stores.json`). - A `createSSRequest` call for a known store (e.g. 514, 782) succeeds using the cached objectId, without needing the admin fix on #1. - Status endpoint returns the expected fields. - No secrets committed; PAT lives in Keychain, loaded via helper script. ### Non-goals - Fixing the object-type-109 permission (that's #1) - Changing how tickets are created (still the same JSM `POST /request` shape)
jmcqueen added this to the v1: Jira lifecycle GA milestone 2026-07-07 10:11:59 -04:00
jmcqueen added the
enhancement
 label 2026-07-07 10:11:59 -04:00
jmcqueen commented 2026-07-07 10:19:15 -04:00
Author
Owner
Copy link

Shipped on cursor/stores-cache at fa06538a.

What landed

  • src/services/jira/assetsSyncClient.js — Basic-auth axios at api.atlassian.com/jsm/assets/workspace/{ws}/v1, sourced from ASSETS_SYNC_EMAIL / ASSETS_SYNC_TOKEN.
  • src/services/jira/storesCache.js — in-memory Map + on-disk data/stores.json (gitignored, atomic write). Paginated full sync via objectTypeId = 109. Boot-time load + background refresh if stale, scheduled resync every STORES_CACHE_REFRESH_HOURS.
  • bin/load-assets-sync-secret.sh — wraps the process so the PAT is pulled from Keychain (security find-generic-password -s jira-assets-sync -a <email>) and never sits in .env.
  • resolveStoreAssetReference now resolves in this order: cache → live PAT lookup → service-account AQL (existing fallback). The fallback is intentionally preserved so it just becomes the primary path again once the permission on #1 is fixed — nothing here has to be ripped out.
  • Admin endpoints: GET /api/wxccai/admin/storesCache/status, POST /api/wxccai/admin/storesCache/refresh.

Verified on boot

$ npm start
info: Server running on port 1866 in development mode
info: No stores cache on disk yet; will populate on first sync
warn: Assets sync credentials not configured (ASSETS_SYNC_EMAIL / ASSETS_SYNC_TOKEN)… fall back to the service-account AQL path.
warn: Stores cache is missing or stale but assets sync is not configured; skipping initial refresh
info: Stores cache: periodic refresh scheduled { intervalHours: 24 }

$ curl -s :1866/api/wxccai/admin/storesCache/status | jq
{ storeCount: 0, assetsSyncConfigured: false, refreshIntervalHours: 24, staleAfterHours: 48, ... }

$ curl -s -X POST :1866/api/wxccai/admin/storesCache/refresh | jq
{ success: true, skipped: true, reason: 'not_configured', status: { ... } }

Real-credentials smoke test still to run (this is a workflow, not blocker code):

  1. security add-generic-password -s jira-assets-sync -a mcqueenj@ae.com -w '<PAT>' -U
  2. ASSETS_SYNC_EMAIL=mcqueenj@ae.com ./bin/load-assets-sync-secret.sh npm start
  3. curl -X POST :1866/api/wxccai/admin/storesCache/refresh → expect ~thousands of stores populated in a few seconds.
  4. createSSRequest for a known store (e.g. 514) → expect success without an Assets API call.

Closing this issue on merge. #1 stays open as the real fix; the cache path deactivates itself in favor of the service-account AQL as soon as that permission lands.

Shipped on `cursor/stores-cache` at [fa06538a](https://git.joesjavajoint.com/jmcqueen/wxccai/commit/fa06538aa40179a970a3e922aec3299aebb715c5). **What landed** - `src/services/jira/assetsSyncClient.js` — Basic-auth axios at `api.atlassian.com/jsm/assets/workspace/{ws}/v1`, sourced from `ASSETS_SYNC_EMAIL` / `ASSETS_SYNC_TOKEN`. - `src/services/jira/storesCache.js` — in-memory `Map` + on-disk `data/stores.json` (gitignored, atomic write). Paginated full sync via `objectTypeId = 109`. Boot-time load + background refresh if stale, scheduled resync every `STORES_CACHE_REFRESH_HOURS`. - `bin/load-assets-sync-secret.sh` — wraps the process so the PAT is pulled from Keychain (`security find-generic-password -s jira-assets-sync -a <email>`) and never sits in `.env`. - `resolveStoreAssetReference` now resolves in this order: **cache → live PAT lookup → service-account AQL (existing fallback)**. The fallback is intentionally preserved so it just becomes the *primary* path again once the permission on #1 is fixed — nothing here has to be ripped out. - Admin endpoints: `GET /api/wxccai/admin/storesCache/status`, `POST /api/wxccai/admin/storesCache/refresh`. **Verified on boot** ``` $ npm start info: Server running on port 1866 in development mode info: No stores cache on disk yet; will populate on first sync warn: Assets sync credentials not configured (ASSETS_SYNC_EMAIL / ASSETS_SYNC_TOKEN)… fall back to the service-account AQL path. warn: Stores cache is missing or stale but assets sync is not configured; skipping initial refresh info: Stores cache: periodic refresh scheduled { intervalHours: 24 } $ curl -s :1866/api/wxccai/admin/storesCache/status | jq { storeCount: 0, assetsSyncConfigured: false, refreshIntervalHours: 24, staleAfterHours: 48, ... } $ curl -s -X POST :1866/api/wxccai/admin/storesCache/refresh | jq { success: true, skipped: true, reason: 'not_configured', status: { ... } } ``` **Real-credentials smoke test still to run** (this is a workflow, not blocker code): 1. `security add-generic-password -s jira-assets-sync -a mcqueenj@ae.com -w '<PAT>' -U` 2. `ASSETS_SYNC_EMAIL=mcqueenj@ae.com ./bin/load-assets-sync-secret.sh npm start` 3. `curl -X POST :1866/api/wxccai/admin/storesCache/refresh` → expect ~thousands of stores populated in a few seconds. 4. `createSSRequest` for a known store (e.g. 514) → expect success without an Assets API call. Closing this issue on merge. #1 stays open as the real fix; the cache path deactivates itself in favor of the service-account AQL as soon as that permission lands.
jmcqueen commented 2026-07-07 10:39:17 -04:00
Author
Owner
Copy link

End-to-end verification followup (b4bc646)

Real-credentials smoke test surfaced two bugs in the initial cut. Both fixed on the same branch, verified against a real store, real ticket.

Bug 1 — Assets AQL pagination silently capped at 25. POST /object/aql uses startAt + maxResults as URL query params, not page + resultPerPage in the body. Passing the wrong param names caused Atlassian to silently fall back to maxResults=25, so the first sync only cached 25 stores (out of 2,661). After fix: full sync in 5.3s across 6 pages, zero orphans.

Bug 2 — Wrong CMDB field shape for Jira Cloud. resolveStoreAssetReference was returning [{ objectId: "75974" }], which is the legacy Data Center / Server shape. Jira Cloud CMDB-object custom fields require [{ id: "<workspaceId>:<objectId>" }]. The old shape is silently accepted (HTTP 204 on REST PUT, no error on JSM POST) but the field is never persisted — confirmed via direct REST GET showing customfield_10261: [] on the first test ticket (SS-20943). After fix: SS-20948 was created for store 00782 (objectId 75974) via cache-only lookup and shows the field populated correctly:

[{
  "workspaceId": "42c7631b-13ba-4fe9-b468-0635785a52fa",
  "id": "42c7631b-13ba-4fe9-b468-0635785a52fa:75974",
  "objectId": "75974"
}]

Workaround status: proven working end-to-end. The service account can now create SS tickets with a populated Store Number field without needing direct access to Object Type 109.

Test artifacts (both moved to Waiting for customer with an internal explanatory comment; please close manually with the triage fields when you get to them):

  • SS-20943 (initial cut, field ended up empty due to Bug 2; field was later manually corrected during Format E probing)
  • SS-20948 (post-fix, field populated correctly on creation)

Follow-ups filed: #9 for a small close-route bug found during cleanup.

### End-to-end verification followup ([b4bc646](https://git.joesjavajoint.com/jmcqueen/wxccai/commit/b4bc646)) Real-credentials smoke test surfaced two bugs in the initial cut. Both fixed on the same branch, verified against a real store, real ticket. **Bug 1 — Assets AQL pagination silently capped at 25.** `POST /object/aql` uses `startAt` + `maxResults` as URL query params, not `page` + `resultPerPage` in the body. Passing the wrong param names caused Atlassian to silently fall back to `maxResults=25`, so the first sync only cached 25 stores (out of 2,661). After fix: full sync in 5.3s across 6 pages, zero orphans. **Bug 2 — Wrong CMDB field shape for Jira Cloud.** `resolveStoreAssetReference` was returning `[{ objectId: "75974" }]`, which is the legacy Data Center / Server shape. Jira Cloud CMDB-object custom fields require `[{ id: "<workspaceId>:<objectId>" }]`. The old shape is **silently accepted** (HTTP 204 on REST PUT, no error on JSM POST) but the field is never persisted — confirmed via direct REST GET showing `customfield_10261: []` on the first test ticket (SS-20943). After fix: SS-20948 was created for store 00782 (objectId 75974) via cache-only lookup and shows the field populated correctly: ```json [{ "workspaceId": "42c7631b-13ba-4fe9-b468-0635785a52fa", "id": "42c7631b-13ba-4fe9-b468-0635785a52fa:75974", "objectId": "75974" }] ``` **Workaround status: proven working end-to-end.** The service account can now create SS tickets with a populated Store Number field without needing direct access to Object Type 109. **Test artifacts** (both moved to `Waiting for customer` with an internal explanatory comment; please close manually with the triage fields when you get to them): - SS-20943 (initial cut, field ended up empty due to Bug 2; field was later manually corrected during Format E probing) - SS-20948 (post-fix, field populated correctly on creation) **Follow-ups filed:** #9 for a small close-route bug found during cleanup.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
cursor/init-repo-and-jira-lifecycle
cursor/close-tickets
cursor/stores-cache
cursor/hardening-4-5-6-7
No results found.
Labels
Clear labels   
blocker

Blocks other work

  
bug

Something is broken

  
cleanup

Housekeeping / hygiene

  
easy

Low-effort

  
enhancement

New capability / improvement

  
followup

Depends on other work

  
refactor

Non-behavioral code rework

No labels
blocker
bug
cleanup
easy
enhancement
followup
refactor
Milestone
Clear milestone
No items
No milestone
v1: Jira lifecycle GA
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: jmcqueen/wxccai#8
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?