Edge Connector — Full Specification

00 · Front matter

How to use this spec

• Three repos. aivis-edge-connector — public, open source (§12): worker/, ops/ CLI code, public docs. aivis-edge-fleet — private: per-customer configs, encrypted token refs, customer runbooks. The AIVIS-side additions (§06) land in the existing private epoint-digital/aivis repo, following its conventions (createApiHandler, Zod validators, one processor per JobKind, docs/v21 update rules).
• Requirement IDs are normative. W-* worker · C-* cache/invalidation · P-* provisioning · A-* analytics · V-* AIVIS-side · S-* security · F-* failure/SLO · T-* testing. Invariants I1–I8 come from the architecture overview and win over any conflicting requirement.
• Live test target. Dev instance https://aivis-new.dev.onepoint.ro; API reference at /api/public/v1/docs, machine spec at /api/public/v1/openapi.json. Daniel supplies an aivis_ bearer token. The endpoint returns the latest available artifact per URL.
• Anything marked verify at build is believed correct but must be confirmed against current Cloudflare docs/behaviour before relying on it.

The eight invariants (recap)

02 · Edge worker internals

The worker, precisely

W-01Runtime & repo layout

TypeScript on the Cloudflare Workers runtime, built with Wrangler. Tests run against workerd (Vitest + @cloudflare/vitest-pool-workers). One worker script serves one customer zone; multi-domain customers get one deployment per zone.

aivis-edge-connector/
  worker/
    src/index.ts          entry — fetch handler, top-level try/catch (I1)
    src/gate.ts           request/response gating (W-03)
    src/normalize.ts      URL normalization (C-01)
    src/resolve.ts        cache-first schema resolution (C-*)
    src/ingest.ts         zero-trust validation pipeline (S-*)
    src/inject.ts         HTMLRewriter injection (W-05)
    src/bots.ts           UA registry + classifier (A-*)
    src/events.ts         event buffer + flush (A-*)
    src/config.ts         env parsing + defaults (W-02)
    test/                 unit + integration fixtures
  ops/                    provisioning CLI (§04)
  docs/                   exported copies of these specs

W-02Configuration contract

All per-customer configuration arrives as Worker vars/secrets at deploy time. No runtime config fetches in v1 (keeps the hot path self-contained; the epoch var doubles as a coarse invalidation lever, C-08).

W-03Gating — when the worker acts at all

1. Request gates: method is GET; path not under EXCLUDE_PATHS; not a synthetic cache-key path (/__aivis/). Everything else: fetch(request) pass-through, zero further logic.
2. Response gates: status 200; Content-Type starts with text/html. Redirects, errors, JSON, assets: returned as-is. Gate on the response content type, not the request Accept header — origins lie less than clients.
3. HEAD requests: pass through (no body to rewrite).
4. Requests carrying ?__aivis=off bypass injection (debug aid; the parameter is stripped before origin fetch so caches don't fragment). confirm param name with Daniel

W-04Concurrency model per request

Origin fetch and schema resolution start simultaneously. The response is held until the schema promise settles or BUDGET_MS elapses — whichever is first. Warm path: cache answers in ~1 ms, so the hold is unmeasurable. Cold path: at most BUDGET_MS added to TTFB, once per URL per colo per TTL. On timeout: skip injection, let waitUntil() finish the API call and populate the cache for the next request (I2).

W-05Injection mechanics

• HTMLRewriter with a handler on head; insert via the element's end-tag hook so the script lands immediately before </head>. verify at build: onEndTag API shape
• Injected element, exactly one: <script type="application/ld+json" data-aivis="1" data-aivis-v="{workerVersion}">{payload}</script>. The data-aivis attribute is the idempotency + verification marker.
• {payload} is the sanitized re-serialization from the ingestion pipeline (S-04) — never bytes straight from the API.
• Idempotency: if the streamed document already contains data-aivis (origin accidentally shipped it, or double-proxying), do not inject again. Detection via a first-pass handler on script[data-aivis].
• Pages with no <head> element: skip injection (fragment/AMP/edge cases), count it in events as skip:no_head.
• Compression is transparent: the runtime decompresses upstream bodies for HTMLRewriter and re-encodes downstream. verify at build
• Charset: HTMLRewriter operates on UTF-8; non-UTF-8 origin pages (rare, legacy) are passed through unmodified with a skip:charset event. verify at build: rewriter charset behaviour

W-06Failure envelope

The entire pipeline runs inside one top-level try/catch that falls back to fetch(request). Rewriter stream errors after headers are sent are the one unfixable window — mitigated by doing all fallible work (resolution, validation) before constructing the rewriter, so the streaming phase touches no external systems. CPU-time and subrequest limits are respected by design: warm path ≈ 1 subrequest (origin only), cold path ≤ 2.

03 · Caching & invalidation

Freshness you can command, per URL and per domain

C-01URL normalization contract

One function, shared conceptually with the API (V-07), applied to every page URL before lookup or caching:

1. Parse with new URL(); invalid ⇒ skip injection.
2. Scheme ⇒ https; host lowercased; default ports stripped; fragment dropped.
3. Query: drop the tracking blocklist (utm_*, gclid, fbclid, msclkid, mc_eid, ref) — then, v1 default, drop the remaining query entirely (AIVIS stores canonical page URLs without queries). A per-customer keep-list can override for genuinely content-bearing params. confirm AIVIS canonical form incl. trailing-slash policy — Open Q-1
4. Path case preserved; trailing slash normalized to match AIVIS storage (pending Q-1).

C-02Cache design — purgeable by construction

Schema entries are stored in the Cloudflare cache under a synthetic URL on the customer's own zone:

cacheKey = https://{PRIMARY_HOST}/__aivis/c/{EPOCH}/{sha256(normalizedPageUrl)}

• Because the key is a zone-owned URL, Cloudflare single-file purge works on it — globally, across all data centers, in seconds. This is the entire trick. verify at build: Cache API + single-file purge interaction
• The worker's own cache.delete() clears only the local colo — it is never used as an invalidation mechanism, only the purge API is (C-07).
• The stored Response body is the post-validation, sanitized payload (S-04) plus metadata headers: x-aivis-generated-at, x-aivis-sig-status, x-aivis-stored-at. Cache lifetime is controlled via Cache-Control on the stored response.
• The worker returns 404 for any real visitor request under /__aivis/ — synthetic keys are cache addresses, not readable endpoints (S-09).

C-03Resolution algorithm

1. cache.match(cacheKey). Hit & fresh ⇒ use. Hit & in SWR window ⇒ use, and revalidate via waitUntil.
2. Miss ⇒ GET {AIVIS_API_BASE}/api/public/v1/jsonld?url={normalizedPageUrl}, bearer auth, AbortController at BUDGET_MS.
3. 200 ⇒ run ingestion pipeline (S-*) ⇒ pass ⇒ cache.put + inject. Fail ⇒ treat as upstream-invalid (C-06).
4. 404 ⇒ negative-cache marker for NEG_TTL_S (C-05); no injection.
5. Timeout / 5xx / network error ⇒ last-known-good if present and younger than LKG_MAX_S, else skip (C-06).
6. Best-effort request coalescing: an in-isolate in-flight map prevents duplicate API calls for the same key within one isolate.

C-04–C-06 TTLs, negatives, last-known-good

C-07Invalidation — the command surface

• Never purge_everything — it would flush the customer's real CDN cache (unacceptable blast radius).
• Purge requires the Cache Purge token scope (P-02). In self-managed mode without it: TTL-only freshness (≤ 5 min staleness) or an optional purge-only token — a far easier grant than deploy rights.
• Purge API rate limits are plan-dependent — the domain-purge batcher must throttle and report progress. verify at build: current limits per plan
• Post-purge cold starts are per-colo (first visitor per colo takes the ≤150 ms path once). Global pre-warming is not feasible with per-colo caches — accepted; SWR + budget make it invisible in practice.
• Note the customer's own HTML cache never needs purging: injection happens per-request after cache lookup, so schema freshness is independent of page caching.

C-09Existing on-page schema

v1 is additive (I4). The rewriter detects pre-existing application/ld+json blocks and reports conflict:onpage_schema through events (when analytics is on) so strategists see overlap. De-duplication/replacement is explicitly a v2 conversation.

C-10Retraction — deletion wins over resilience

Publishing has an inverse: a bad JSON-LD must be removable, not just replaceable. Flow: unpublish in AIVIS (V-08) ⇒ /jsonld returns 404 for that URL ⇒ the purge hook (V-03) clears the edge cache key globally ⇒ the worker's next lookup gets the 404 and overwrites the cached entry — including last-known-good — with a negative marker.

Precedence rule: last-known-good exists to absorb transport failures (timeout, 5xx, network). A confirmed 404 is an authoritative statement of absence and permanently clears the LKG entry. Without this rule, a retracted schema could resurrect hours later during an API outage via the stale-if-error path — the exact opposite of a retraction. Tested explicitly (T-02).

Distinct case: removing a bad JSON-LD that the customer's own CMS ships inside the origin HTML. That is markup modification, fenced out of v1 by I4/D6, and listed as the v2 suppression/replacement conversation (C-09).

Freshness summary

04 · Provisioning & operations

Two modes, one artifact, a clean handover

The worker always runs in the customer's Cloudflare account. The modes differ only in who holds deployment rights — which is why the Allianz handover is a permissions change, not a migration.

P-02Cloudflare token scopes (Mode A) — least privilege

Exact scope names to be confirmed against the current CF token UI at build time verify at build. Token should be zone-restricted and, where the plan allows, IP-restricted to AIVIS ops egress.

P-03Ops CLI command surface

P-04 Per-customer config lives as one reviewed YAML file in the private fleet repo (aivis-edge-fleet, §12 O-01), rendered to Worker vars at deploy; secrets in an encrypted store, never in the YAML — and never anywhere in the public connector repo.

P-05The self-managed deployment package

• Built worker bundle (dist/) + source, wrangler.toml template, Terraform module (optional apply path)
• Runbook: deploy, set secret, attach route, verify, pause, uninstall (§11 Pack C)
• Distributed as signed GitHub Releases from the public repo (§12 O-05): semver, CHANGELOG.md, SHA-256SUMS + signature, provenance attestation
• Support boundary statement: AIVIS supports the package; the customer operates it

P-06Handover procedure (Mode A → B)

1. Deliver package (P-05) + security pack (§11 Pack B) to customer IT.
2. Joint session: walk the runbook; customer IT performs a no-op redeploy themselves.
3. Customer revokes AIVIS's CF token. Nothing moves — the worker was in their account all along.
4. Agree freshness mode (TTL-only vs purge-only token) and update-notification channel.

05 · Analytics

Optional by design — off means off

A-02Event schema (analytics on)

{
  "ts": "2026-08-06T10:31:04Z",     // minute precision is enough
  "host": "www.example.com",
  "path": "/products/x",             // normalized path only, never query strings
  "bot": "GPTBot",                   // matched registry family, or "none"
  "cat": "training|search|live|none",
  "injected": true,                  // true | false | "origin" (block already shipped by origin, A-07)
  "cache": "hit|miss|skip",
  "skip": "no_head|charset|budget|no_schema|conflict:onpage_schema|null",
  "wv": "1.4.2"                      // worker version
}

• Never collected: IP addresses, raw user-agent strings, cookies, headers, query strings, anything about identifiable humans. v1 records bot traffic and injection health; human pageviews appear only as anonymous aggregate injection counts (sampled 1:100). confirm sampling with Daniel
• A-03 Buffering: per-isolate buffer, flush via waitUntil at 25 events or 30 s. Flush failure ⇒ drop (loss-tolerant; never retry on the visitor's clock). Target loss < 5%.
• A-04 Bot registry ships in bots.ts as data (family, operator, category, match rule). Updates ride worker releases; an EXTRA_BOTS var allows per-customer additions without a release.
• A-05 v1.1 option (spec'd, not built): ANALYTICS=local — write to a Workers Analytics Engine dataset inside the customer's account; AIVIS reads it via the CF GraphQL API with the token it already holds. Analytics without customer data ever landing on AIVIS servers — the enterprise middle path.

A-06Registry v1

Same table as the architecture overview §7: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Googlebot (note: Google-Extended is a robots directive, not a UA), Bingbot, CCBot, Bytespider, Amazonbot, Applebot, meta-externalagent, MistralAI-User, DuckAssistBot. Registry entries carry a verification hint (published IP ranges / reverse-DNS) for a later "verified bot" flag — matching alone is v1.

A-07Analytics-only mode & origin composition

With INJECT=off the worker becomes a pure measurement tap: no HTMLRewriter pass, no mutation of any response — only crawler classification and batched events. This is the recommended setup when the origin already delivers the JSON-LD, e.g. a WordPress site running the AIVIS WordPress Connector behind Cloudflare: the origin-shipped block rides inside cached HTML (zero added edge latency), and the worker's idempotency detection (W-05) recognizes data-aivis and records injected:"origin". Delivery at the origin, measurement at the edge — each layer where it is truthful. WordPress-side crawler analytics is permanently excluded (WP spec §22): PHP sees only cache-miss traffic, a systematically biased sample unfit for the Monitoring SKU.

06 · AIVIS-side work package

Changes in the aivis repo — all small, one prerequisite

Follow repo conventions: createApiHandler + Zod, jobs as one processor per JobKind, spec updates in docs/v21. Listed in build order:

V-02Business-scoped API tokens prerequisite for any external deploy

Today's aivis_ tokens are user-scoped: a worker secret in customer infrastructure could list every business of that user. Add a token type bound to one businessId with permission flags { jsonld:read, events:write }; /jsonld and /events enforce the binding. Existing user tokens keep working for the profile/API use case.

V-01Events ingest endpoint

POST /api/public/v1/events        Bearer (business-scoped, events:write)
body: { workerVersion, host, events: Event[≤100] }   → 202
storage: EdgeEvent (raw, 90-day retention) + EdgeEventDaily rollup (kept)
rate limit: per-token, generous (batches are small); reject >100 events/req

V-03Regenerate/retract→purge hook

New model EdgeConnection { businessId, mode(A|B), zoneId, accountId, cfTokenRef(encrypted), analytics, purgeEnabled, status }. After a successful generate_jsonld job — and after a retraction (V-08) — for a URL whose business has a connection with purgeEnabled: enqueue edge_purge (new JobKind) → single-file purge of that URL's synthetic key via the CF API. Domain-level purge = same job fed by the URL inventory. Failures retry ×3 then alert; purge failure is never user-facing breakage (TTL still bounds staleness).

V-04Serve-gate on /jsonld

Today the endpoint returns the latest artifact. With auto-purge, a bad regeneration reaches production in seconds — so add a per-business serve policy: latest (today's behaviour) or validated (only artifacts with validation.errors == 0; falls back to the previous passing artifact — requires keeping the previous artifact row instead of delete-and-recreate). Default validated for connected businesses. An explicit per-URL publish flag remains future scope.

V-05Artifact signing (enterprise option)

At artifact write time, sign businessId + "\n" + url + "\n" + generatedAt + "\n" + canonicalPayload with Ed25519; store { keyId, sig, canonicalPayload } on the artifact row. /jsonld returns the canonical string verbatim plus the signature block; the worker (when VERIFY_SIGNATURES=enforce) verifies against the pinned public key and parses that exact string. Signing keys live outside the app container (KMS or at minimum an isolated env not reachable from app code paths); honest limits of this scheme are stated in §07.

V-08Retract / unpublish action

Per-URL action (plus chain-level bulk) that withdraws a published JSON-LD without waiting for a replacement: set suppressedAt on the artifact — recommended over hard deletion (reversible, keeps the audit trail; the existing DELETE endpoint remains for true deletion). While suppressed, /jsonld returns 404 for that URL regardless of serve policy, and the purge hook (V-03) fires immediately. Combined with C-10 the block is off every page in seconds and cannot resurrect. Dashboard surface: an "Unpublish from edge" action next to regenerate.

V-06–V-07 Supporting endpoints

• V-06 GET /api/public/v1/jsonld/bulk?businessId=… — paginated ready-URL list + payload hashes; used by ops verify and future warm tooling.
• V-07 Tolerant lookup: /jsonld?url= applies the same normalization as C-01 server-side, so worker and API can never disagree about identity (resolves Open Q-1 permanently).

07 · Security, tenancy & privacy

Zero-trust ingestion: a hacked AIVIS cannot hurt the customer

S-01The ingestion pipeline — every payload, every time

1. S-02 Strict parse. JSON.parse or reject. No eval-anything, no lenient parsing.
2. S-03 Sanitizing re-serialization — the XSS killer. JSON.stringify the parsed value, then replace every < with \u003c. The output is byte-equivalent JSON-LD to any consumer, but can never contain </script>, <script, or <!-- — script-context breakout becomes impossible regardless of payload content.
3. S-04 Shape + size. Top level is an object or array; nodes carry @type/@id/@context keys consistent with Schema.org shape; serialized size ≤ 128 KiB; nesting depth ≤ 32. Violations reject.
4. S-05 Domain binding. Response businessId must equal BUSINESS_ID; response url host must match PRIMARY_HOST (www-equivalence configurable). A payload for another tenant or domain never gets injected — kills cross-tenant injection even if the API mixes up tenants.
5. S-06 Signature (enterprise, enforce). Ed25519 verify against the public key pinned in the customer's own deployment (V-05). The customer's account holds the trust anchor, not AIVIS's runtime.

S-07Honest threat model

S-09 The /__aivis/ namespace returns 404 to real visitors; synthetic keys are unguessable (hashed) and carry no secrets anyway. S-10 Events contain no personal data (A-02) — the GDPR story is "bot traffic statistics, no natural persons," and ANALYTICS=off removes even that. Data-processing summary lives in §11 Pack B.

08 · Failure modes & SLOs

Every way it breaks, and what happens instead

F-02Service-level objectives (targets to validate at build)

• Added latency: p50 ≤ 2 ms, p99 ≤ 10 ms on warm traffic; cold-miss requests ≤ BUDGET_MS + 10 ms, and ≤ 1% of requests take the cold path in steady state.
• Injection coverage: ≥ 99% of eligible pageviews (200 + HTML + schema exists) carry the block.
• Freshness: ≤ 60 s after purge; ≤ TTL without purge rights.
• Event loss: ≤ 5% (analytics is best-effort by contract).

F-03 Runbooks (short, in ops/runbooks/): retract a single bad JSON-LD (V-08 + purge — the scalpel, seconds); pause a customer (the hammer); purge-all for a customer; roll back a release; rotate AIVIS token; rotate CF token; "AIVIS compromise" drill = pause all Mode-A customers (route detach loop) + notify Mode-B customers to pause, then purge-all after all-clear.

09 · Testing & verification

Proving the invariants, not just the features

T-06 Each invariant I1–I8 must map to at least one automated test or checklist line; the mapping table lives in worker/test/INVARIANTS.md.

10 · Rollout milestones

Build order for the implementing agent

1. M1Worker core.Gates, normalization, resolution with cache states, ingestion pipeline S-02…S-05, injection, fail-open. The public repo exists from the first commit with license + hygiene guards in place (§12 O-01). Exit: T-01/T-02 green; live smoke T-03 against dev API; deployed on an AIVIS-owned test zone with a demo page.
2. M2Ops CLI.connect / deploy / verify / pause / resume / uninstall / invalidate. Exit: full lifecycle demonstrated on the test zone incl. per-URL and per-domain purge timing measurements.
3. M3AIVIS-side package.V-02 scoped tokens (prerequisite), V-01 ingest, V-03 purge hook + EdgeConnection, V-04 serve-gate, V-07 tolerant lookup. Exit: regenerate-in-AIVIS ⇒ live-on-page in seconds, end to end (T-04).
4. M4Analytics.Classifier, buffering, ingest storage + rollups; ANALYTICS on/off verified incl. the I8 egress test. Exit: bot hits from the test zone visible in AIVIS data.
5. M5Hardening + SMB pilot.V-05 signing path (warn mode), F-03 runbooks, §11 Packs A+B drafted, upgrade/rollback drill, OSS launch checklist complete (O-05 release pipeline + O-07 community files). Exit: first real SMB customer live in Mode A; first public tagged release.
6. M6Enterprise kit.P-05 package, P-06 handover procedure, signing in enforce mode, German documentation. Exit: a customer-IT team can deploy from the package alone — Allianz-ready.

M1–M2 and M3 can proceed in parallel after the C-01/V-07 normalization contract is agreed (Open Q-1 first).

11 · Customer documentation set

Three packs + a handover kit — EN master, DE for DACH

12 · Open source & supply chain

Public by default — the connector is not the moat

O-01Repo boundary

The rule: nothing customer-identifying and no real secrets ever enter the public repo — code, tests, fixtures, issues, commit messages, or history. Fixtures use invented domains and fake tokens. Public from the first commit means there is never a history to scrub.

O-02License & contributions

Apache-2.0 (recommended; explicit patent grant, enterprise-legal-friendly — MIT is the acceptable alternative, Q-8), with NOTICE file. Contributions under DCO sign-off (no CLA — needless friction at this size). The AIVIS name and logo remain trademarks: the code is free, the brand is not; a short trademark note lives in the README.

O-03The provider interface is documented, not gated

The worker consumes a documented HTTP contract — GET /jsonld?url=… with bearer auth, the response shape from the OpenAPI spec, and the optional signature block (V-05). Any conforming backend works; AIVIS is the reference provider. This is stated openly in the public docs: the connector is generic infrastructure, the value is in what AIVIS puts through it.

O-04–O-06 Supply-chain hardening, releases, CI

• O-04 Repo guards: protected default branch, required review before merge, maintainer 2FA, CODEOWNERS, GitHub secret scanning + a gitleaks CI step, lockfile-pinned dependencies with Renovate/Dependabot.
• O-05 Releases are the deployment artifact: semver git tags → GitHub Release carrying the built worker bundle, SHA-256SUMS, a signature (Sigstore/cosign or minisign — builder's choice verify at build), and GitHub artifact attestations (SLSA provenance). Mode B customers consume releases directly and verify checksums per the runbook; Mode A deploys the identical artifact via the fleet repo.
• O-06 CI split: public Actions run lint / type-check / tests / build on every PR with zero secrets available to fork PRs. Anything needing real tokens — live smoke (T-03), fleet deploys — runs only in the private fleet repo's protected environment.

O-07–O-08 Community files & public docs

• O-07 README (what it does + deploy-to-your-own-zone quickstart), CONTRIBUTING.md (dev setup, DCO), SECURITY.md (private disclosure email, no public vuln issues, 72 h acknowledgement target), CODE_OF_CONDUCT.md, CHANGELOG.md.
• O-08 Public docs = sanitized architecture overview, the self-managed deployment guide (Pack C lives here as the canonical copy), the provider interface contract, and bot-registry docs. Internal planning documents (these spec artifacts) and customer-specific Packs A/B stay private; public docs never name customers or internal URLs.

O-09–O-10 Consequences elsewhere

• O-09 The supply-chain row in the threat model (§07) exists because of this decision — public repo = public attack surface, answered by O-04/O-05 plus customers pinning versions. The data-aivis-v version attribute on pages is acceptable disclosure; versions are public anyway.
• O-10 Not open: the AIVIS platform and API implementation, fleet configs and customer identities, Packs A/B, signing private keys. Nothing in this section obliges opening any of those later.

Appendix · Open questions & risks

To resolve before / during M1