diff --git a/README.md b/README.md index 6b6cd46..2305e9a 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,7 @@ If you only want the skills, disable the `contentstack` MCP server in your tool' | `dx-delivery-sdk` | Delivery SDK | Developer Experience | Production-ready TypeScript with `@contentstack/delivery-sdk` for entries, assets, references, filters, sorting, pagination, locales, Live Preview, and Visual Builder. | | `dx-kickstart-next` | Contentstack Kickstart Next | Developer Experience | Maintaining Contentstack's Next.js kickstart: App Router structure, Delivery SDK setup, Live Preview, seeded stack alignment, env vars, validation, and PR reviews. | | `dx-mcp-server` | Contentstack MCP | Developer Experience | Connecting AI agents with the local `@contentstack/mcp` server or the hosted MCP Profile Hub: choosing, client setup, auth, tool groups and profiles, and least-privilege safety. | +| `dx-sap-spartacus-import` | SAP Spartacus Content Import | Developer Experience | Importing an SAP Commerce Cloud (Spartacus) storefront's CMS content over OCC into Contentstack for the Spartacus connector: dry run, two approval gates, idempotent seed and publish, round-trip verify, extra locales, and the SAP Commerce Cloud app's product picker for carousels. | | `dx-migrate-js-to-ts-sdk` | Migrate JS to TS SDK | Developer Experience | Migration from the JavaScript Contentstack SDK to the TypeScript Delivery SDK, including API mapping, rewrites, and unsupported pattern callouts. | | `launch-platform` | Launch Platform | Launch | Hosting on Launch: frameworks and builds, Cloud and Edge Functions, launch.json routing, caching and revalidation, deploy hooks, domains, access control, log targets, monorepos, and limits. | | `launch-sync-environment-variables-from-env-example` | Sync Launch environment variables from .env.example | Launch | Comparing a local `.env.example` with Launch environment variables and adding missing keys with a full-object update, without printing secrets. | diff --git a/docs/README.md b/docs/README.md index 19d0305..d692b48 100644 --- a/docs/README.md +++ b/docs/README.md @@ -52,7 +52,7 @@ Every skill is written once in the open [Agent Skills](https://agentskills.io) f | | | | ------------------------ | -------------------------------------------------------------------------------------------------------------- | -| **Skills included** | 33 (see the [skill reference](skills/index.md)) | +| **Skills included** | 34 (see the [skill reference](skills/index.md)) | | **Products covered** | CMS, Assets, Developer Experience, Studio, Launch, Brand Kit, Developer Hub, Marketplace, Personalize, Lytics, Automate | | **Supported tools** | Claude Code, claude.ai / Cowork, Cursor, Codex, GitHub Copilot, Gemini CLI, `skills` CLI | | **Source of truth** | `skills//SKILL.md` | diff --git a/docs/skills/dx-sap-spartacus-import.md b/docs/skills/dx-sap-spartacus-import.md new file mode 100644 index 0000000..36e952a --- /dev/null +++ b/docs/skills/dx-sap-spartacus-import.md @@ -0,0 +1,37 @@ +--- +title: SAP Spartacus Content Import +description: Import an SAP Commerce Cloud storefront's CMS content over OCC into Contentstack, shaped for the Contentstack Spartacus connector. +--- + +# SAP Spartacus Content Import + +**Slug:** `dx-sap-spartacus-import` · **Product:** Developer Experience · **Type:** Action (gated) · **Ships references & scripts** + +Takes the content an SAP Commerce Cloud (Spartacus / composable storefront) site serves over OCC and recreates it in a Contentstack stack, in the model `@contentstack/contentstack-spartacus-connector` reads. Pages, slots, banners, paragraphs, carousels, links, navigation, logo, footer and images come across. Products, prices, search, cart and checkout stay with SAP, following the connector's hybrid model. + +## When it triggers + +When a developer has added the connector to a Spartacus app (`ng add @contentstack/contentstack-spartacus-connector`) and wants the existing SAP storefront content in Contentstack. Also for seeding a demo stack from an SAP electronics or powertools site, re-running an import to update entries, or adding locales such as de-de, ja-jp and zh-cn. + +## What it covers + +- **Dry run first**: extracts the site and prints what would be written, without touching the stack. +- **Two approval gates**: unknown component types need a mapping decision, and nothing is written until the user approves the summary. +- **Starter-pack check**: the run stops if the plan needs a content type or field the installed connector doesn't ship. +- **Idempotent seed and publish**: entries upsert by their SAP uid, so re-runs update in place. +- **Verification**: a round-trip comparison against SAP and a Delivery API read of the home page. +- **Localization**: localized content and images per language from SAP's own localized responses. +- **SAP product picker**: optionally installs and configures the SAP Commerce Cloud marketplace app, so editors pick carousel products from SAP instead of typing SKUs. The agent checks the picker, repairs it at most twice, then lets the user fall back to the plain SKU list. +- **Your own login**: writes with a management token, or with the user's `csdx auth:login --oauth` session (required for the marketplace app). + +## Example prompts + +- "I just added the Spartacus connector. Import our SAP electronics storefront into my new stack." +- "Seed a demo stack from powertools-spa with German and Japanese." +- "Re-run the SAP import; we changed the homepage banners." +- "The import says a component type is unknown. What do I do?" +- "Let our editors pick the carousel products from SAP instead of typing SKUs." + +## References + +Ships `references/pipeline.md` (stages, gates, flags, run directory, exit codes), `references/mapping-profile.md` (how SAP templates, slots and typeCodes map to connector content types) and `references/sap-commerce-app.md` (the marketplace app stage and its verify, repair and fallback steps). The Python scripts in `scripts/` use only the standard library; the csdx login and the marketplace app step use Node.js and the Contentstack CLI. Built by Vaishnavi Kedar as the Spartacus content-modelling agent. diff --git a/docs/skills/index.md b/docs/skills/index.md index 691c027..af58f1e 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -5,7 +5,7 @@ description: The full catalog of Contentstack Agent Skills, grouped by product. # Skills reference -The bundle includes 33 skills across ten Contentstack products. Each links to a dedicated page with what it does, when it triggers, example prompts, and its safety notes. +The bundle includes 34 skills across ten Contentstack products. Each links to a dedicated page with what it does, when it triggers, example prompts, and its safety notes. > Each skill page mirrors its canonical `skills//SKILL.md`. The slug is the skill's identifier: use it with the [`skills` CLI](../get-started/skills-cli.md). @@ -37,6 +37,7 @@ The bundle includes 33 skills across ten Contentstack products. Each links to a | [Contentstack MCP](dx-mcp-server.md) | `dx-mcp-server` | Local MCP server vs hosted MCP Profile Hub, client setup, auth, tool scoping, safety. | | [Delivery SDK](dx-delivery-sdk.md) | `dx-delivery-sdk` | Production-ready TypeScript with `@contentstack/delivery-sdk` for entries, assets, queries, Live Preview. | | [Contentstack Kickstart Next](dx-kickstart-next.md) | `dx-kickstart-next` | Maintaining the Next.js kickstart: App Router, Delivery SDK setup, Live Preview, seed alignment, env vars, validation. | +| [SAP Spartacus Content Import](dx-sap-spartacus-import.md) | `dx-sap-spartacus-import` | Importing an SAP Commerce Cloud storefront's content into Contentstack for the Spartacus connector. | | [Migrate JS→TS SDK](dx-migrate-js-to-ts-sdk.md) | `dx-migrate-js-to-ts-sdk` | Migrating Delivery SDK code from the JavaScript SDK to the TypeScript SDK. | | [Live Preview & Visual Editor Support](cms-live-preview-visual-editor-support-assistant.md) | `cms-live-preview-visual-editor-support-assistant` | Setting up or debugging Live Preview, Visual Editor (Visual Builder) or Timeline across CSR, SSR, SSG, BFF. | | [Migration Companion](migration-companion.md) | `migration-companion` | Guided Contentful → Contentstack migration: content and application code. | diff --git a/evals/routing-dx-sap-spartacus-import-1/graders/loads-dx-sap-spartacus-import.md b/evals/routing-dx-sap-spartacus-import-1/graders/loads-dx-sap-spartacus-import.md new file mode 100644 index 0000000..cb33de8 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-1/graders/loads-dx-sap-spartacus-import.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?dx-sap-spartacus-import"' +min: 1 +--- +The agent loads the `dx-sap-spartacus-import` skill. diff --git a/evals/routing-dx-sap-spartacus-import-1/graders/skips-migration-companion.md b/evals/routing-dx-sap-spartacus-import-1/graders/skips-migration-companion.md new file mode 100644 index 0000000..a334b5e --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-1/graders/skips-migration-companion.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?migration-companion"' +min: 0 +max: 0 +--- +The agent doesn't load `migration-companion`, which covers Contentful migrations. diff --git a/evals/routing-dx-sap-spartacus-import-1/prompt.md b/evals/routing-dx-sap-spartacus-import-1/prompt.md new file mode 100644 index 0000000..9afa6f1 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-1/prompt.md @@ -0,0 +1,7 @@ +--- +name: routing-dx-sap-spartacus-import-1 +tags: [routing, dx-sap-spartacus-import] +max_turns: 3 +timeout_seconds: 180 +--- +I just ran ng add for the Contentstack Spartacus connector. Import the content from our SAP Commerce electronics storefront into my new stack. diff --git a/evals/routing-dx-sap-spartacus-import-2/graders/loads-dx-sap-spartacus-import.md b/evals/routing-dx-sap-spartacus-import-2/graders/loads-dx-sap-spartacus-import.md new file mode 100644 index 0000000..cb33de8 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-2/graders/loads-dx-sap-spartacus-import.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?dx-sap-spartacus-import"' +min: 1 +--- +The agent loads the `dx-sap-spartacus-import` skill. diff --git a/evals/routing-dx-sap-spartacus-import-2/graders/skips-migration-companion.md b/evals/routing-dx-sap-spartacus-import-2/graders/skips-migration-companion.md new file mode 100644 index 0000000..a334b5e --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-2/graders/skips-migration-companion.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?migration-companion"' +min: 0 +max: 0 +--- +The agent doesn't load `migration-companion`, which covers Contentful migrations. diff --git a/evals/routing-dx-sap-spartacus-import-2/prompt.md b/evals/routing-dx-sap-spartacus-import-2/prompt.md new file mode 100644 index 0000000..486d337 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-2/prompt.md @@ -0,0 +1,7 @@ +--- +name: routing-dx-sap-spartacus-import-2 +tags: [routing, dx-sap-spartacus-import] +max_turns: 3 +timeout_seconds: 180 +--- +Seed a Contentstack demo stack from our SAP composable storefront's OCC content, with German and Japanese as well. diff --git a/skills/dx-sap-spartacus-import/SKILL.md b/skills/dx-sap-spartacus-import/SKILL.md new file mode 100644 index 0000000..c05be3d --- /dev/null +++ b/skills/dx-sap-spartacus-import/SKILL.md @@ -0,0 +1,130 @@ +--- +name: dx-sap-spartacus-import +description: "Import the CMS content of a running SAP Commerce Cloud (Spartacus / composable storefront) site into a Contentstack stack, in the model the Contentstack Spartacus connector reads. Extracts pages, slots, components, navigation and images over SAP OCC, maps them onto the connector starter-pack content types, seeds and publishes them, verifies the round trip, and can localize into more languages. Can also install the SAP Commerce Cloud marketplace app so carousels use its product picker. Use when a developer has added @contentstack/contentstack-spartacus-connector to a Spartacus app (ng add) and wants to bring the existing SAP storefront content into Contentstack, seed a demo stack from an SAP electronics or powertools site, re-run the import to update entries, add de/ja/zh locales, or have editors pick carousel products from SAP. For Contentful-to-Contentstack migrations, use migration-companion; for general csdx import or export, use dx-cli; for content model design, use cms-data-modeling-best-practices." +allowed-tools: Read Grep Glob Bash +--- + +# SAP Spartacus content import + +Takes the content an SAP Commerce Cloud storefront serves over OCC and recreates it in Contentstack, shaped for `@contentstack/contentstack-spartacus-connector`. It follows the connector's hybrid model: Contentstack holds the authored slots (banners, paragraphs, carousels, links, navigation, logo, footer), while SAP keeps products, prices, search, cart, checkout and every functional component. + +## When to use + +- A Spartacus app has the connector (`ng add @contentstack/contentstack-spartacus-connector`) and the team wants the existing SAP storefront content in Contentstack. +- Seeding a demo or proof-of-concept stack from an SAP site such as `electronics-spa` or `powertools-spa`. +- Re-running the import after content changed in SAP (it updates entries in place). +- Adding locales (`de-de`, `ja-jp`, `zh-cn`, …) from SAP's localized content. + +Use another skill for: + +- Migrating from Contentful → `migration-companion` +- Plain `csdx` import, export, seed or bulk publish → `dx-cli` +- Designing a content model from scratch → `cms-data-modeling-best-practices` +- Live Preview or Visual Builder setup → `cms-live-preview-visual-editor-support-assistant` +- Other marketplace apps, or the SAP app's category picker and fields outside carousels → `marketplace-apps` + +## Before you answer + +Confirm, without asking for secret values: + +1. The current directory is the Spartacus project root and `node_modules/@contentstack/contentstack-spartacus-connector/import-export/starter-pack/content_types` exists. If not, the connector isn't installed: run `ng add @contentstack/contentstack-spartacus-connector` first, or point `CONNECTOR_STARTER_PACK` at a connector checkout. +2. The SAP baseSite uid (for example `electronics-spa`) and the OCC v2 base URL (`https://HOST/occ/v2`). List baseSites with `curl -s "$OCC_BASE_URL/basesites"`. +3. The target stack is **new or dedicated to this import**. The publish step publishes every entry and asset in the stack. +4. `python3 --version` is 3.9 or later. The Python scripts use only the standard library. +5. How the run writes: with `CS_MANAGEMENT_TOKEN`, or, without one, as the user's own `csdx auth:login --oauth` session. The SAP Commerce Cloud app step always needs that csdx session (a management token can't install apps), plus Node.js and `@contentstack/cli` on `PATH`. + +## Instructions + +1. Resolve ``: the folder this `SKILL.md` was loaded from. Run every script by absolute path, `python3 "/scripts/orchestrate.py"`, from the Spartacus project root. +2. Ask the user to export the variables under Environment Variables in their own shell. Never ask them to paste values into chat. +3. **Dry run** (writes nothing): + ```bash + python3 "/scripts/orchestrate.py" --site electronics-spa + ``` + It extracts the site, builds the plan and prints both gate reports. Summarize them for the user: pages, component entries, assets, which slots stay with SAP, any unknown component types, where carousel SKUs go (`products` or `sap_products`), and whether writes run as the management token or the csdx login. +4. **Gate 1, model approval.** If the report lists unknown typeCodes, stop. Each one is either authored (add it to `component_map` in `scripts/registry/connector.registry.json` with its connector content type and field mapping) or functional (add it to `occ_only_types.types`). Re-run the dry run. Pass `--approve-model` only after the user agrees. +5. **Gate 2, pre-publish.** Show the counts and the target stack and environment. The run stops here if the plan writes a content type or field the installed starter pack lacks. The fix is a newer connector (`npm i @contentstack/contentstack-spartacus-connector@latest`), not editing the stack by hand. +6. With the user's explicit yes, run the write: + ```bash + python3 "/scripts/orchestrate.py" --site electronics-spa \ + --approve-model --approve-publish --env development + ``` + This sets up the SAP Commerce Cloud app when carousels use the picker, imports the starter-pack content types, uploads images, upserts entries, creates the environment if needed, publishes, and runs the verify steps. +7. **Carousel products.** With `SAP_APP_API_BASE_URL`, `SAP_APP_BASE_SITE` and `SAP_APP_BACKOFFICE_URL` exported, carousels go into `sap_products`, the SAP Commerce Cloud app's product picker; otherwise into the plain `products` SKU list. Force either with `--carousel-field products|sap_products`. After a picker run, follow "Verify, retry, then ask" in [references/sap-commerce-app.md](references/sap-commerce-app.md): check the picker, repair at most twice, then let the user choose between fixing it and falling back to `products`. +8. For more languages, add `--locales de=de-de,ja=ja-jp,zh=zh-cn` (SAP isocode = Contentstack locale). Locales are created when missing, with the master locale as fallback. +9. Report the result from `manifest.json` in the run directory, then point the user at the app config: `cmsPageContentType: 'landing_page'`, a `localeMapping` that matches the locales imported, and `occFallback: true`. + +### How the pipeline works + +`extract → nav-resolve → model → gate 1 → gate 2 → sap-app → seed → publish → verify → localize`. Stage details, run-directory files and flags are in [references/pipeline.md](references/pipeline.md). The mapping rules (which SAP templates, slots and typeCodes become which content types) are in [references/mapping-profile.md](references/mapping-profile.md). The SAP Commerce Cloud app stage, its auth and its recovery steps are in [references/sap-commerce-app.md](references/sap-commerce-app.md). + +### Rules + +- **Idempotent.** Component entries upsert by `sap_uid`, pages by `url`, the shell entry by `title`, assets by a `sap_code` tag. Re-running updates in place; it never duplicates. +- **Starter pack is the model of record.** The skill never invents content types. It imports the connector's pack and drops any page slot the target page type lacks, leaving that slot to SAP. +- **Hybrid by design.** Login, account, cart, checkout, search and other functional pages get no Contentstack page. SiteContext and SiteLinks stay with SAP, so the language switcher and sign-in keep working. +- **Reuse an extract** with `--inventory /inventory.json` to re-plan without calling OCC again. +- **Self-signed SAP dev backends** fail TLS verification. Set `OCC_INSECURE_TLS=1` only for a backend the user owns and trusts; never for production. + +## MCP tools + +The import runs on the bundled Python scripts against the Content Management API; it doesn't need MCP. When the Contentstack MCP server is connected, its read tools are a way to check the result (local server group in brackets): + +- Inspect: `get_all_content_types`, `get_single_entry`, `get_all_entries`, `get_all_assets`, `get_all_languages`, `get_all_environments` (`cma`) + +Don't use MCP write or delete tools to patch imported content mid-run; fix the mapping and re-run instead. + +## Output + +After each run, give a short status: stage results from `manifest.json`, counts (pages, component entries, navigation entries, assets, locales), slots left to SAP, and the next command. On a failure, quote the failing stage and the last lines of its output, and say whether a re-run is safe (it always is: every step upserts). + +## Security + +### Defaults + +- Dry run first, every time. Writes happen only with both `--approve-model` and `--approve-publish`, after the user has seen the gate reports. +- OCC TLS is verified. `OCC_INSECURE_TLS=1` is an explicit opt-in for a trusted dev backend. +- Reads from SAP only. The skill never writes to SAP Commerce Cloud. + +### Destructive Actions + +- The seed step creates or updates content types, entries and assets in the target stack, and changes existing content types to match the starter pack. +- The publish step publishes **every** entry and asset in the stack to the chosen environment, not just the imported ones. +- The `sap-app` stage installs and configures the SAP Commerce Cloud app on the stack. `--reinstall-app` (or `sap_app.py --reinstall`) uninstalls and reinstalls it; `use_products.py` removes the `sap_products` field and its data from the carousel type. Run those only on the user's explicit choice. + +So before passing `--approve-publish`, name the stack (API key's stack name, not the key), the environment and the counts, warn that existing content in that stack will be published, and wait for a clear yes. Recommend a new, dedicated stack. Never delete entries, content types, assets or stacks. + +### Secrets + +Never print, echo, log or store the management token, delivery token, API key or csdx session token, and never write them into the project, the run directory or the final report. The user exports them in their own shell; refer to them by variable name only. + +### Environment Variables + +- `OCC_BASE_URL`: SAP OCC v2 base, `https://HOST/occ/v2` (or pass `--base`). +- `OCC_INSECURE_TLS`: `1` only for a trusted self-signed dev backend. +- `CS_API_KEY`, `CS_MANAGEMENT_TOKEN`: target stack and a management token with write access. Leave the token unset to write as the csdx login. +- `CS_AUTH`: `auto` (default: the token when set, else csdx), `token` or `csdx`. +- `CS_DELIVERY_TOKEN`, `CS_ENVIRONMENT`: used by the final delivery read check. +- `CS_REGION_BASE`, `CS_CDA_BASE`: management and delivery hosts outside North America, for example `https://eu-api.contentstack.com` and `https://eu-cdn.contentstack.com`. +- `CS_MASTER_LOCALE`: defaults to `en-us`. +- `CS_PREVIEW_URL`: storefront URL set on the environment for every locale, so Live Preview loads (default `http://localhost:4200`). +- `CONNECTOR_STARTER_PACK`: override the starter-pack path. +- `SAP_APP_API_BASE_URL` (host, no scheme), `SAP_APP_BASE_SITE`, `SAP_APP_BACKOFFICE_URL`: the SAP server the app's picker searches. Setting all three turns on the picker. +- `SAP_APP_API_ROUTE` (default `/occ/v2/`), `SAP_APP_CONFIG_NAME` (default `hti`), and optionally `SAP_APP_UID`, `CS_ORG_UID`. + +## References + +- [references/pipeline.md](references/pipeline.md): stages, gates, run directory and flags. +- [references/mapping-profile.md](references/mapping-profile.md): how SAP templates, slots and typeCodes map to connector content types, and how to extend it. +- [references/sap-commerce-app.md](references/sap-commerce-app.md): the SAP Commerce Cloud app stage, the picker value, verify-repair-fallback. +- Connector package: https://www.npmjs.com/package/@contentstack/contentstack-spartacus-connector +- Connector docs: https://www.contentstack.com/docs/marketplace/spartacus-contentstack-connector +- Author: Vaishnavi Kedar (Contentstack Developer Ecosystem), from the Spartacus content-modelling agent. + +## Examples + +User: Import our SAP electronics storefront into my new stack. +Assistant: Runs the dry run, then reports: "6 pages (/, /faq, /contact, /sale, shared /product and /category layouts), 34 components, 79 navigation entries, 37 images. SearchBox, MiniCart, SiteContext and SiteLinks stay with SAP. No unknown component types. Gate 2 will write to stack 'SAP demo' and publish everything in it to 'development'. Shall I run it?" and waits. + +User: Just publish it, skip the dry run. +Assistant: Explains that the write step only runs after the gate reports have been reviewed, runs the dry run (it writes nothing and takes about a minute), and asks for the go-ahead with the counts in front of the user. diff --git a/skills/dx-sap-spartacus-import/references/mapping-profile.md b/skills/dx-sap-spartacus-import/references/mapping-profile.md new file mode 100644 index 0000000..4ef0b11 --- /dev/null +++ b/skills/dx-sap-spartacus-import/references/mapping-profile.md @@ -0,0 +1,34 @@ +# Mapping profile: SAP OCC to the connector model + +`scripts/registry/connector.registry.json` is a translation table from what SAP serves over OCC to the content types the Contentstack Spartacus connector ships in its starter pack. It defines no schemas: those come from the connector package, so the imported content always matches what the connector reads. + +## The hybrid model + +SAP renders the base of every page. Contentstack overrides only the slots that are authored. Functional components, slots nobody authors, and pages that aren't content pages stay with SAP and aren't imported. The slot is the unit of control. + +## Sections of the profile + +| Key | Maps | Notes | +|---|---|---| +| `page_types` | SAP page template → page content type | `LandingPage2Template` and `ContentPage1Template` → `landing_page`; `ProductDetailsPageTemplate` → `product_page` (shared slug `/product`); `ProductListPageTemplate` → `category_page` (shared slug `/category`). Any other template gets no Contentstack page. | +| `slot_field_map` | SAP slot position → page reference field | For example `Section1` → `section1`, `BodyContent` → `body_content`, `HomepageNavLink` → `homepage_nav_link`. A position missing here, or a field the target page type lacks, stays with SAP. | +| `global_slots` | Shell positions → fields on the single `global_slots` entry | `SiteLogo`, `NavigationBar` and `Footer` are authored. `SiteContext` and `SiteLinks` stay with SAP so the language switcher and sign-in keep working. | +| `component_map` | SAP typeCode → component content type and field mapping | Banners, product carousel (SKUs in `products`, or in the SAP app picker field `sap_products` with `--carousel-field sap_products`), paragraph, link, flex component. Image fields become uploaded assets. | +| `navigation` | Navigation trees | Seeded flat: `nav_node_flat` entries with `node_id`, `parent_id` and `sort_order`, plus a `category_navigation_flat` or `footer_navigation_flat` entry that references them through `all_nodes`. The connector rebuilds the tree. | +| `occ_only_types` | Functional typeCodes | Mini cart, search box, breadcrumb, product list, refinements, add to cart and similar. Never imported. | + +## Keys used for re-runs + +- Pages: `url` (home is `/`). +- Components and navigation entries: `sap_uid`, the SAP component or node uid. +- Shell entry: `title`, set to the baseSite uid. +- Assets: a `sap_code:` tag. + +## Extending it + +When gate 1 reports an unknown typeCode, decide with the user: + +- **Editors should control it**: add it to `component_map` with the connector content type it maps to and a field mapping from OCC field to connector field. The content type must exist in the starter pack; gate 2 checks this. +- **SAP renders it**: add it to `occ_only_types.types`. + +Re-run the dry run. The mapping then applies to every later run. diff --git a/skills/dx-sap-spartacus-import/references/pipeline.md b/skills/dx-sap-spartacus-import/references/pipeline.md new file mode 100644 index 0000000..7168af8 --- /dev/null +++ b/skills/dx-sap-spartacus-import/references/pipeline.md @@ -0,0 +1,66 @@ +# Pipeline: stages, gates and run directory + +`scripts/orchestrate.py` runs each stage as its own script and stops at two gates. Every stage writes to the run directory (default `$TMPDIR/sap-spartacus-import-/`), so a failed run can be inspected and re-run. + +``` +extract ─▶ nav-resolve ─▶ model ─▶ [gate 1] ─▶ [gate 2 + pack check] ─▶ sap-app ─▶ seed ─▶ publish ─▶ verify ─▶ [localize per locale] +``` + +## Stages + +| # | Stage | Script | What it does | +|---|---|---|---| +| 1 | extract | `storefront-extract/extract.py` | Crawls the baseSite over OCC: content pages found by following seed labels and internal links, one product and one category layout, slots with their components (`fields=FULL`), navigation, and a manifest of every image. Writes `inventory.json` (schema: `storefront-extract/inventory.schema.json`). | +| 2 | nav-resolve | `storefront-extract/nav_resolve.py` | Resolves navigation link entries into full link detail and finds the navigation roots. Adds `navigationLinks` and `navigationRoots` to the inventory. Skipped when `--inventory` is reused without an OCC base. | +| 3 | model | `content-model/model_connector.py` | Maps the inventory onto the connector model with `registry/connector.registry.json`. Writes `connector-plan.json` (entries, assets, content types needed) and `connector-report.json`. Drops page slots the target content type lacks in the starter pack. With `--carousel-field sap_products`, carousel SKUs become a `sap_products` placeholder the seed resolves. | +| – | gate 1 | orchestrator | Blocks when the site uses a component typeCode the profile doesn't know. Pass with `--approve-model` once it's mapped. | +| – | gate 2 | orchestrator | Prints what will be written. Checks that every content type and field the plan writes exists in the starter pack, and exits with code 3 if not. Without `--approve-publish` the run ends here as a dry run. | +| 4 | sap-app | `marketplace-app/sap_app.py` (+ `sap_app.cjs`) | Only when carousels use `sap_products`: installs and configures the SAP Commerce Cloud app on the user's csdx login, finds its Product field extension, writes `sap-app.json`. See [sap-commerce-app.md](sap-commerce-app.md). | +| 5 | seed | `cs-seed/cma_seed_connector.py` | Imports the starter-pack content types (adds a `sap_uid` field where the pack lacks one, and the `sap_products` picker field on the carousel when the app is set up), uploads images, then upserts component entries, the `global_slots` shell entry and page entries in reference order. | +| 6 | publish | `cs-seed/publish.py` | Creates the environment when missing (with a Live Preview URL per locale) and publishes every entry and asset in the stack to it in the master locale. | +| 7 | verify, round trip | `verify/verify_connector.py` | Reads the seeded entries back and compares each with its original OCC component, including the rebuilt navigation tree. | +| 8 | verify, delivery | `verify/cda_read.py` | Reads the home page (`landing_page`, url `/`) through the Delivery API with `CS_DELIVERY_TOKEN`. | +| 9 | localize | `cs-seed/localize.py` | Per `--locales` pair: a localized extract and model, the locale's images, a localized version of every entry, and a publish in that locale. | + +`cs-seed/cma_seed.py` holds the shared Content Management API helpers the seed, publish and localize stages import. `cs-seed/cs_auth.py` picks the auth header: `CS_MANAGEMENT_TOKEN`, or the user's csdx session through `cs-seed/csdx_auth.cjs` (refreshed once after a 401). `marketplace-app/use_products.py` is the user-chosen fallback from the picker back to `products`. + +## Flags + +| Flag | Default | Notes | +|---|---|---| +| `--site` | required | SAP baseSite uid. | +| `--base` | `$OCC_BASE_URL` | OCC v2 base URL. | +| `--inventory` | none | Reuse a saved `inventory.json` and skip the OCC extract. | +| `--starter-pack` | `$CONNECTOR_STARTER_PACK`, else `./node_modules/@contentstack/contentstack-spartacus-connector/import-export/starter-pack/content_types` | The connector's content types. | +| `--env` | `development` | Target environment for publishing. | +| `--approve-model` | off | Pass gate 1. | +| `--approve-publish` | off | Pass gate 2 and write. | +| `--carousel-field` | `auto` | `sap_products` when the `SAP_APP_*` configuration is exported, else `products`. | +| `--skip-app-install` | off | The SAP app is already set up: only discover its field extension. | +| `--reinstall-app` | off | Uninstall, reinstall and configure the SAP app once (repairs a broken install). | +| `--locales` | none | `sapIso=csLocale` pairs, for example `de=de-de,ja=ja-jp,zh=zh-cn`. | +| `--run-dir` | `$TMPDIR/sap-spartacus-import-` | Where every stage writes. | + +## Run directory + +- `inventory.json`, `inventory-.json`: extracted content. +- `connector-plan.json`, `connector-report.json`: the plan and its summary; `model-/` per locale. +- `sap-app.json`: the app installation and its field extension uids, when the picker is used. +- `manifest.json`: status per stage (`ok`, `warn`, `error`, `blocked`, `dry-run`, `pass`, `check`). + +## Exit codes + +- `0`: done, or dry run complete. +- `2`: gate 1 blocked on unknown typeCodes. +- `3`: gate 2 blocked on a starter-pack mismatch. +- Anything else: a stage failed. Its output is printed and recorded in `manifest.json`. + +## Common problems + +- **`CERTIFICATE_VERIFY_FAILED`**: the SAP backend uses a self-signed certificate. Fix the certificate, or set `OCC_INSECURE_TLS=1` for a dev backend you trust. +- **Starter pack not found**: run from the Spartacus project root after `ng add @contentstack/contentstack-spartacus-connector`, or set `CONNECTOR_STARTER_PACK`. +- **Gate 2 pack mismatch**: the installed connector is older than the mapping profile expects. Update the connector package. +- **`csdx not found` or `not logged in`**: the run has no management token, or reached the SAP app stage. Install `@contentstack/cli` and run `csdx auth:login --oauth`, or export `CS_MANAGEMENT_TOKEN` and pass `--carousel-field products`. +- **SAP Products field errors or HTTP 414**: see "Verify, retry, then ask" in [sap-commerce-app.md](sap-commerce-app.md). +- **Empty carousels on the storefront**: the connector is older than the `sap_products` release. Update it, or re-run with `--carousel-field products`. +- **Blank storefront after import**: check the app's `cmsPageContentType` is `landing_page`, the `localeMapping` covers the imported locales, and the environment name matches `CS_ENVIRONMENT`. diff --git a/skills/dx-sap-spartacus-import/references/sap-commerce-app.md b/skills/dx-sap-spartacus-import/references/sap-commerce-app.md new file mode 100644 index 0000000..d4019eb --- /dev/null +++ b/skills/dx-sap-spartacus-import/references/sap-commerce-app.md @@ -0,0 +1,78 @@ +# SAP Commerce Cloud app: carousel product picker + +The optional `sap-app` stage runs between gate 2 and the seed. It sets up the **SAP Commerce Cloud** marketplace app on the target stack, so carousel products come from the app's product picker (`sap_products`) instead of a hand-typed SKU list (`products`). Editors then add, remove and reorder products visually, and Live Preview's Edit on a carousel opens the picker. + +The stage runs when carousels are seeded into `sap_products`: by default when `SAP_APP_API_BASE_URL`, `SAP_APP_BASE_SITE` and `SAP_APP_BACKOFFICE_URL` are all exported, or always with `--carousel-field sap_products`. With `--carousel-field products`, or without that configuration, it's skipped and carousels keep the `products` list. + +The storefront must run a connector release that reads `sap_products` (it falls back to `products`). Connector 0.1.0 reads only `products` and would render these carousels empty. + +## What it does + +1. **Install** the app on the stack if it isn't there yet. +2. **Configure** it, on a fresh install only: one configuration named `SAP_APP_CONFIG_NAME` (default `hti`), set as default, with Custom URL `SAP_APP_API_BASE_URL` + `SAP_APP_API_ROUTE`, base site `SAP_APP_BASE_SITE`, Backoffice `SAP_APP_BACKOFFICE_URL`, and **Keys to Save** = Custom Fields `code` + `name`. Other configurations already on the stack are kept. +3. **Discover** the app's `SAP Commerce Cloud - Product` field extension (it only shows with `?include_marketplace_extensions=true`) and write it to `/sap-app.json`. + +The seed step then adds `sap_products` (JSON, that extension) to `product_carousel_component` right after `products`, and fills it per carousel with exactly what the picker saves: + +```json +{"type": "sapcommercecloud_product", + "data": [{"code": "300938", "name": "Photosmart E317 Digital Camera", + "cs_metadata": {"multiConfigName": "hti", "isConfigDeleted": false}}]} +``` + +Names come from the app's SAP server (`scripts/marketplace-app/sap_products.py`); codes that server doesn't have are left out and reported. A value over the 20 KB JSON-field limit is refused, which is why the app saves only `code` + `name` (images would make each product about 3 KB). `localize.py` writes the same value to every locale: the picker stores the app's language, not the locale's. + +## Auth + +A **management token can't install or configure marketplace apps.** Steps 1 and 2 run on the user's own Contentstack CLI session, through csdx's bundled `@contentstack/cli-utilities`, so they need Node.js and `@contentstack/cli` on `PATH`: + +```bash +csdx auth:login --oauth # once; orgs with SSO reject a basic email/password login +``` + +Step 3 and all seeding use `CS_MANAGEMENT_TOKEN` when it's set, or the same csdx session when it isn't (`CS_AUTH=auto`). + +## Run it on its own + +```bash +python3 "/scripts/marketplace-app/sap_app.py" --status # read-only: installation + config +python3 "/scripts/marketplace-app/sap_app.py" --out /sap-app.json # install/configure once + discover +python3 "/scripts/marketplace-app/sap_app.py" --out /sap-app.json --skip-install # already set up: discover only +python3 "/scripts/marketplace-app/sap_app.py" --out /sap-app.json --reinstall # repair a broken install +python3 "/scripts/marketplace-app/use_products.py" --plan /connector-plan.json --env # fallback to products +``` + +## Verify, retry, then ask + +The pipeline can't tell on its own whether the picker works: the app only talks to its server with a token Contentstack issues inside the browser. So after every `sap_products` run the agent checks it in the browser (or asks the user to), repairs it at most twice, and then hands the decision to the user. Don't keep retrying or ask open-ended questions. + +**Check (one attempt):** + +1. Open a seeded carousel entry: `https://app.contentstack.com/#!/stack//content-type/product_carousel_component/en-us/entries` (use the region's app host outside North America). +2. Wait about 20 seconds. **Pass** if the SAP Products field shows "N Products" with names. **Fail** if it shows "Something went wrong while fetching data". +3. Click **Add Product(s)**. **Pass** if the picker lists products. To read the app server's status code, run this in the picker tab with a product code from the carousel: + ```js + const tk = new URLSearchParams(location.hash.split('?')[1]||'').get('authToken'); + const q = new URLSearchParams({query:'product','id:in':'[]',isOldUser:'false',configKey:'{"hti":{}}'}); + (await fetch('https://sapcc.contentstackmarket.com/api-auth?'+q,{method:'POST', + headers:{'Content-Type':'application/json',authToken:tk}, + body:JSON.stringify({config:{},ids:{hti:['']}})})).status // 200 ok, 414 = broken install + ``` + Replace `hti` with `SAP_APP_CONFIG_NAME` if it was changed. Close the picker without adding anything. + +**Attempts:** + +- Attempt 1: the check right after the run. If it fails, wait a minute and reload once (a fresh install can need a short warm-up). That's still attempt 1. +- Attempts 2 and 3: repair with `sap_app.py --reinstall --out /sap-app.json`, wait a minute, check again. The repair uninstalls the app, installs it, configures it once, and points the carousel's `sap_products` field at the new extension. Entry data is kept. + +**After three failed attempts, stop and ask the user.** Say briefly what failed (field or picker error, the status code) and what you tried; recommend installing and configuring the app once from Marketplace in the UI, or raising it with the SAP Commerce Cloud app team with the installation uid from `sap_app.py --status`; and offer the choice: **keep trying to fix the picker**, or **use the `products` field instead**. + +Explain the fallback in a sentence or two: the storefront shows the same products either way, because the connector reads `sap_products` first and falls back to `products`. With `products`, editors type or paste SAP product codes into a list instead of searching and picking them, and the entry shows codes rather than names. + +**If the user picks `products`**, run `use_products.py --plan /connector-plan.json --env `. It re-imports the carousel content type without `sap_products`, writes each carousel's SKUs into `products` (same order, every locale the entry exists in) and publishes them. Later runs for that stack: `--carousel-field products`. The app stays installed; the user can uninstall it from Marketplace. + +## Limits + +- **Warm-up:** for about a minute after a fresh install, the field can show "Something went wrong while fetching data". Reload the entry. +- **The configuration is written exactly once, on a fresh install.** Writing it again to an install that already has it, even with identical values, breaks that install on the app's server: the field and picker then fail with HTTP 414 from `sapcc.contentstackmarket.com/api-auth`, and it doesn't recover. So re-runs only check an existing install, and stop if its settings differ. To change settings, use the app's Configuration page; to repair a broken install, run with `--reinstall-app` (or `sap_app.py --reinstall`). +- The first install of an app in an org can need a one-time consent in the UI. If the API install is refused, the stage stops and says so: install it once from Marketplace, then re-run. diff --git a/skills/dx-sap-spartacus-import/scripts/content-model/model_connector.py b/skills/dx-sap-spartacus-import/scripts/content-model/model_connector.py new file mode 100644 index 0000000..15134ef --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/content-model/model_connector.py @@ -0,0 +1,349 @@ +#!/usr/bin/env python3 +""" +model_connector — content-model for the CONNECTOR reference-field profile (Phase 4b). + +Reads an inventory (from storefront-extract) + connector.registry.json and emits a +connector-shaped plan that conforms to the @contentstack/contentstack-spartacus-connector starter-pack model: +per-template page entries with named per-slot REFERENCE fields, authored component entries +only, and a single global_slots shell entry. Functional components + non-content pages are +left to OCC (hybrid islands model). Stack-agnostic — no API calls. + +Output: connector-plan.json (+ prints a summary). + +Usage: + python3 model_connector.py --inventory --profile --out +""" +import argparse, json, os +from collections import OrderedDict + + +def clean(d): + return {k: v for k, v in d.items() if not k.startswith("_")} + + +def as_int(v): + try: + return int(v) + except (TypeError, ValueError): + return None + + +def as_target(v): # OCC "true"/"false" -> normalized "true"/"false" string (connector reads strings) + if isinstance(v, bool): + return "true" if v else "false" + return "true" if str(v).strip().lower() == "true" else "false" + + +def ref(content_type, uid): + return {"_ref": True, "content_type": content_type, "by": "sap_uid", "value": uid} + + +def asset_ref(code): + return {"_asset": True, "code": code} if code else None + + +class ConnectorModeler: + def __init__(self, inv, prof, pack_fields=None, carousel_field="products"): + self.inv = inv + # content type uid -> field uids in the connector's starter pack. When given, a page + # slot is only authored if the target page type has that slot field; otherwise it is + # left to OCC (e.g. HomepageNavLink exists on landing_page but not product_page). + self.pack_fields = pack_fields + # Where carousel SKUs go: "products" (plain SKU text list) or "sap_products" (the SAP + # Commerce Cloud app's product picker; seeders resolve the placeholder to picker JSON). + self.carousel_field = carousel_field + self.page_types = clean(prof["page_types"]) + self.slot_map = clean(prof["slot_field_map"]) + self.global_pos = clean(prof["global_slots"]["positions"]) + self.global_ct = prof["global_slots"]["content_type"] + self.cmap = {k: v for k, v in prof["component_map"].items() if not k.startswith("_")} + self.occ_only = set(prof["occ_only_types"]["types"]) + self.components = OrderedDict() # uid -> entry + self.assets = OrderedDict() + self.skipped_deferred = set() + self.unmapped_slots = set() + + # ---- component field mapping ---- + def add_asset(self, code): + if code and code not in self.assets: + m = self.inv.get("mediaManifest", {}).get(code, {}) + self.assets[code] = {"code": code, "url": m.get("url"), + "mime": m.get("mime"), "alt_text": m.get("altText")} + + def map_component(self, c): + tc = c.get("typeCode") + spec = self.cmap.get(tc) + # Nav components are NOT regular slot components — build_nav() flattens + # them into the nav_node_flat pool + a flat nav component and wires them + # into the shell itself. Skip them here (both the legacy "deferred-nav" + # and the profile's "nav-pass" status) so process_page never wires the + # shell with a recursive nav type that build_nav then can't override. + if not spec or spec.get("status") in ("deferred-nav", "nav-pass"): + if spec: + self.skipped_deferred.add(tc) + return None # not authored (functional or nav) + uid = c.get("uid") + if not uid: + return None + if uid in self.components: + return uid + ct = spec["content_type"] + m = c.get("media") or {} + fields = {"sap_uid": uid} + if tc == "SimpleResponsiveBannerComponent": + for bp, fld in (("mobile", "media_mobile"), ("tablet", "media_tablet"), + ("desktop", "media_desktop"), ("widescreen", "media_widescreen")): + d = m.get(bp) + if isinstance(d, dict) and d.get("code"): + fields[fld] = asset_ref(d["code"]); self.add_asset(d["code"]) + d = m.get("desktop") if isinstance(m.get("desktop"), dict) else None + if d and d.get("code"): + fields["media"] = asset_ref(d["code"]) + fields["url_link"] = c.get("urlLink") + fields["title"] = c.get("name") or uid + elif tc == "SimpleBannerComponent": + if m.get("code"): + fields["media"] = asset_ref(m["code"]); self.add_asset(m["code"]) + fields["url_link"] = c.get("urlLink") + fields["title"] = c.get("name") or uid + elif tc == "ProductCarouselComponent": + fields["title"] = c.get("title") or c.get("name") or uid # rendered heading + codes = [x for x in (c.get("productCodes") or "").split() if x] + if self.carousel_field == "sap_products": + fields["sap_products"] = {"_sap_products": True, "codes": codes} + else: + fields["products"] = codes + elif tc == "CMSParagraphComponent": + fields["title"] = c.get("name") or uid + fields["content"] = c.get("content") + elif tc == "CMSLinkComponent": + fields["title"] = c.get("name") or uid + fields["link_name"] = c.get("linkName") + fields["url"] = c.get("url") + fields["target"] = as_target(c.get("target")) + elif tc == "CMSFlexComponent": + fields["title"] = c.get("name") or uid + fields["flex_type"] = c.get("flexType") + else: + fields["title"] = c.get("name") or uid + entry = {"_content_type": ct, "_key": {"field": "sap_uid", "value": uid}} + entry.update({k: v for k, v in fields.items() if v is not None}) + self.components[uid] = entry + return uid + + # ---- one page (content page or layout sample) ---- + SHARED_SLUG = {"product_page": "/product", "category_page": "/category"} + + def process_page(self, label, p, global_fields, url_override=None): + tpl = p.get("template") + pt = self.page_types.get(tpl) + if not pt: + return None # non-content page -> OCC, no CS page + slot_fields = {} + for s in p.get("slots", []): + pos = s.get("position") + if pos not in self.global_pos and pos in self.slot_map and self.pack_fields is not None \ + and self.slot_map[pos] not in self.pack_fields.get(pt["content_type"], set()): + self.unmapped_slots.add(f"{pos} ({pt['content_type']} has no {self.slot_map[pos]})") + continue # target type lacks the slot -> OCC + if pos not in self.global_pos and pos not in self.slot_map: + self.unmapped_slots.add(pos) # authored page, unknown slot -> OCC + continue + comp_uids = [u for u in (self.map_component(c) for c in s.get("components", [])) if u] + if not comp_uids: + continue + if pos in self.global_pos: # shell -> global_slots (dedupe by uid) + fld = self.global_pos[pos] + bucket = global_fields.setdefault(fld, []) + seen = {r["value"] for r in bucket} + for u in comp_uids: + if u not in seen: + bucket.append(ref(self.components[u]["_content_type"], u)); seen.add(u) + elif pos in self.slot_map: # page slot + fld = self.slot_map[pos] + slot_fields[fld] = [ref(self.components[u]["_content_type"], u) for u in comp_uids] + else: + self.unmapped_slots.add(pos) # authored page, unknown slot -> OCC + url = url_override or ("/" if label == "homepage" else "/" + label) + page = {"_content_type": pt["content_type"], "_key": {"field": "url", "value": url}, + "title": p.get("title") or label, "url": url, + "page_type": p.get("typeCode"), "template": tpl} + page.update(slot_fields) + return page + + # ---- navigation: link leaves -> FLAT nav_node_flat pool -> flat nav components ---- + def build_nav(self, global_fields): + """Emit navigation as a FLAT adjacency list, not a recursive tree. + + Each menu becomes a pool of `nav_node_flat` entries — every node points at + its parent by the plain-text `parent_id` (a node id, NOT a reference) and + orders siblings via `sort_order` — plus a `category/footer_navigation_flat` + component that references the whole pool through `all_nodes`. The connector + rebuilds the tree in code, so the menu resolves in one shallow, constant + include chain regardless of depth (no Delivery API include-depth cap). + + Every entry is keyed by `sap_uid` (the connector seeder only tracks + sap_uid-keyed entries in its ref map); nav_node_flat also carries node_id == + sap_uid and parent_id == parent's sap_uid for the connector to read. + Emission order (link leaves -> nodes -> component) keeps every _ref target + created before its referrer. + """ + roots = self.inv.get("navigationRoots", {}) + links = self.inv.get("navigationLinks", {}) + tree = self.inv.get("navigation", {}) + nav_entries = [] + if not roots: + return nav_entries + # 1. link leaves actually used by the nav trees + used = set() + def collect(node): + for e in (node.get("entries") or []): + if e.get("itemType") == "CMSLinkComponent" and e.get("itemId") in links: + used.add(e["itemId"]) + for ch in (node.get("children") or []): + collect(ch) + for r in roots.values(): + if tree.get(r["root"]): + collect(tree[r["root"]]) + for lid in sorted(used): + ld = links[lid] + nav_entries.append({"_content_type": "cms_link_component", + "_key": {"field": "sap_uid", "value": lid}, "sap_uid": lid, + "title": ld.get("name") or ld.get("link_name") or lid, + "link_name": ld.get("link_name"), "url": ld.get("url"), + "target": as_target(ld.get("target"))}) + # 2. flatten a menu tree into nav_node_flat entries (parent_id text + + # sort_order); append in pre-order so parents precede children. + def flatten(node, parent_uid, order, collected): + uid = node.get("uid") + if not uid: + return + node_links = [e["itemId"] for e in (node.get("entries") or []) + if e.get("itemType") == "CMSLinkComponent" and e.get("itemId") in links] + # A SAP leaf node usually has NO title — its label lives on the linked + # cms_link_component. Fall back to the link's name so the storefront + # shows "Contact Us", not the node uid "ContactUsNavNode". + leaf_label = links[node_links[0]].get("link_name") if node_links else None + entry = {"_content_type": "nav_node_flat", + "_key": {"field": "sap_uid", "value": uid}, "sap_uid": uid, + "node_id": uid, "parent_id": parent_uid, "sort_order": order, + "title": node.get("title") or node.get("name") or leaf_label or uid} + if node_links: + entry["links"] = [ref("cms_link_component", lid) for lid in node_links] + nav_entries.append(entry) + collected.append(uid) + for i, ch in enumerate(node.get("children") or [], start=1): + flatten(ch, uid, i, collected) + # 3. flat nav components -> placed into the shell (navigation_bar / footer) + for cuid, r in roots.items(): + root_node = tree.get(r["root"]) + if not root_node: + continue + node_uids = [] + # the root is a pure container: its children are the top-level items + for i, ch in enumerate(root_node.get("children") or [], start=1): + flatten(ch, "", i, node_uids) + ct = ("category_navigation_flat" + if r["typeCode"] == "CategoryNavigationComponent" else "footer_navigation_flat") + nav_entries.append({"_content_type": ct, "_key": {"field": "sap_uid", "value": cuid}, + "sap_uid": cuid, "title": cuid, "wrap_after": as_int(r.get("wrapAfter")), + "all_nodes": [ref("nav_node_flat", nid) for nid in node_uids]}) + fld = self.global_pos.get(r.get("position")) + if fld: + bucket = global_fields.setdefault(fld, []) + if cuid not in {x["value"] for x in bucket}: + bucket.append(ref(ct, cuid)) + return nav_entries + + def run(self): + pages, global_fields = [], {} + for label, p in self.inv.get("contentPages", {}).items(): + pg = self.process_page(label, p, global_fields) + if pg: + pages.append(pg) + for label, p in self.inv.get("layoutSamples", {}).items(): # product/category pages + pt = self.page_types.get(p.get("template")) + if not pt: + continue + pg = self.process_page(label, p, global_fields, + url_override=self.SHARED_SLUG.get(pt["content_type"])) + if pg: + pages.append(pg) + nav_entries = self.build_nav(global_fields) # after pages so shell fields exist + + global_entry = None + if global_fields: + global_entry = {"_content_type": self.global_ct, + "_key": {"field": "title", "value": self.inv.get("site")}, + "title": self.inv.get("site")} + # Clear any shell field we no longer author (e.g. site_context/site_links now + # left to OCC) so an update removes stale references instead of keeping them. + ALL_SHELL = ["site_logo", "search_box", "mini_cart", "navigation_bar", + "site_context", "site_links", "header_links", "footer"] + for fld in ALL_SHELL: + global_entry.setdefault(fld, []) + global_entry.update(global_fields) + + components_out = list(self.components.values()) + nav_entries # nav ordered: links->nodes->components + used_cts = sorted({e["_content_type"] for e in components_out} | + {p["_content_type"] for p in pages} | + ({self.global_ct} if global_entry else set())) + from collections import Counter + nav_by_ct = Counter(e["_content_type"] for e in nav_entries) + plan = { + "site": self.inv.get("site"), "profile": "connector-reference-fields", + "content_types_needed": used_cts, + "assets": list(self.assets.values()), + "entries": {"components": components_out, + "global_slots": global_entry, "pages": pages}, + } + report = { + "site": self.inv.get("site"), + "authored_pages": [(p["url"], p["_content_type"]) for p in pages], + "component_entries": len(self.components), + "nav_entries": dict(nav_by_ct), + "assets": len(self.assets), + "global_slots_fields": sorted(global_fields.keys()), + "unmapped_slot_positions": sorted(self.unmapped_slots), + } + return plan, report + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--inventory", required=True) + ap.add_argument("--profile", required=True) + ap.add_argument("--out", required=True) + ap.add_argument("--starter-pack", default="", help="connector starter-pack content_types dir (drops slots a page type lacks)") + ap.add_argument("--carousel-field", choices=["products", "sap_products"], default="products", + help="carousel SKUs -> plain `products` list or the SAP app picker field") + a = ap.parse_args() + inv = json.load(open(a.inventory)); prof = json.load(open(a.profile)) + os.makedirs(a.out, exist_ok=True) + pack_fields = None + if a.starter_pack: + pack_fields = {} + for f in os.listdir(a.starter_pack): + if f.endswith(".json") and f != "schema.json": + ct = json.load(open(os.path.join(a.starter_pack, f))) + if isinstance(ct, dict) and ct.get("uid"): + pack_fields[ct["uid"]] = {x["uid"] for x in ct.get("schema", [])} + plan, report = ConnectorModeler(inv, prof, pack_fields, a.carousel_field).run() + plan["carousel_field"] = a.carousel_field + json.dump(plan, open(os.path.join(a.out, "connector-plan.json"), "w"), indent=2) + json.dump(report, open(os.path.join(a.out, "connector-report.json"), "w"), indent=2) + print(f"[model_connector] {report['site']} (profile: connector-reference-fields)") + print(f" authored pages: {len(report['authored_pages'])} -> {report['authored_pages']}") + print(f" component entries: {report['component_entries']} assets: {report['assets']}") + print(f" content types needed: {plan['content_types_needed']}") + print(f" carousel SKUs -> {a.carousel_field}") + print(f" global_slots fields: {report['global_slots_fields']}") + if report.get("nav_entries"): + print(f" nav entries: {report['nav_entries']}") + if report["unmapped_slot_positions"]: + print(f" unmapped slots -> OCC: {report['unmapped_slot_positions']}") + print(f" -> {a.out}/connector-plan.json") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed.py new file mode 100644 index 0000000..b2bf398 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed.py @@ -0,0 +1,356 @@ +#!/usr/bin/env python3 +""" +cma_seed — live Contentstack Management API executor for the seed plan. + +Applies a content-model output (content-types.json + entry-plan.json) to a real stack, +idempotently and in dependency order. English (master locale) only unless --locales given. + +Reads credentials from env: CS_API_KEY, CS_REGION_BASE (default https://api.contentstack.io) +and either CS_MANAGEMENT_TOKEN or, without one, the user's csdx login (see cs_auth.py). +Never logs the token. + +Stages (comma list, default all): globals,content_types,assets,entries + python3 cma_seed.py --model-dir DIR --site electronics-spa --stages globals,content_types + python3 cma_seed.py --model-dir DIR --site electronics-spa # full run + +Idempotent: content types & global fields by uid; assets by a `sap_code:` tag; +entries by their unique key field. Re-running updates in place. +""" +import argparse, json, mimetypes, os, ssl, sys, time, urllib.parse, urllib.request +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import cs_auth + +API = os.environ.get("CS_REGION_BASE", "https://api.contentstack.io").rstrip("/") +KEY = os.environ.get("CS_API_KEY") +# Truthy whenever a credential is configured, so the scripts' `KEY and TOKEN` checks hold in +# csdx mode too; the actual header comes from cs_auth.headers(). +TOKEN = cs_auth.MANAGEMENT_TOKEN or ("csdx-session" if cs_auth.available() else None) +MASTER = os.environ.get("CS_MASTER_LOCALE", "en-us") +OCC_HOST = (os.environ.get("OCC_BASE_URL") or "").split("/occ/")[0] + + +def occ_ssl_context(): + """TLS context for OCC calls. Verified by default; set OCC_INSECURE_TLS=1 only for a + dev backend with a self-signed certificate you trust.""" + ctx = ssl.create_default_context() + if os.environ.get("OCC_INSECURE_TLS") == "1": + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + + +OCC_CTX = occ_ssl_context() + + +def die(msg): + sys.exit(f"[cma_seed] ERROR: {msg}") + + +def cma(method, path, body=None, raw=None, content_type=None, retry=3): + url = f"{API}/v3{path}" + data, ct = None, None + if raw is not None: + data, ct = raw, content_type + elif body is not None: + data, ct = json.dumps(body).encode(), "application/json" + refreshed = False + for attempt in range(retry): + req = urllib.request.Request(url, data=data, method=method) + req.add_header("api_key", KEY) + for h, v in cs_auth.headers().items(): + req.add_header(h, v) + if ct: + req.add_header("Content-Type", ct) + try: + with urllib.request.urlopen(req, timeout=60) as r: + return r.status, json.loads(r.read().decode() or "{}") + except urllib.error.HTTPError as e: + payload = e.read().decode() + if e.code == 429 and attempt < retry - 1: # rate limited + time.sleep(1.5 * (attempt + 1)); continue + if e.code == 401 and cs_auth.mode() == "csdx" and not refreshed: + refreshed = True # expired OAuth token: refresh once + cs_auth.headers(refresh=True); continue + try: + return e.code, json.loads(payload) + except Exception: + return e.code, {"_raw": payload} + except Exception as e: + if attempt < retry - 1: + time.sleep(1); continue + return 0, {"_error": str(e)} + return 0, {"_error": "unreachable"} + + +# ---------------- schema normalization ---------------- +def ensure_title(ct): + """Contentstack requires a text field with uid 'title'. Inject one if absent and + point options.title at it. Other fields (page_label, slot_id, sap_name...) stay.""" + fields = ct.setdefault("schema", []) + if not any(f.get("uid") == "title" for f in fields): + fields.insert(0, {"display_name": "Title", "uid": "title", "data_type": "text", + "mandatory": True, "unique": False, + "field_metadata": {"_default": True}, "multiple": False}) + ct.setdefault("options", {}) + ct["options"]["title"] = "title" + ct["options"].setdefault("singleton", False) + # scrub any leftover _default on non-title fields (only title may be default) + for f in fields: + if f.get("uid") != "title": + fm = f.get("field_metadata") or {} + fm.pop("_default", None) + return ct + + +def html_rte_field(uid, display): + return {"display_name": display, "uid": uid, "data_type": "text", + "field_metadata": {"allow_rich_text": True, "rich_text_type": "advanced", + "description": "", "multiline": False, "version": 3}, + "multiple": False, "mandatory": False, "unique": False} + + +def normalize_content_type(ct): + """Make the registry-derived schema CMA-safe.""" + ct = json.loads(json.dumps(ct)) # deep copy + # rich_text.content -> HTML-based advanced RTE (stores HTML directly; round-trips clean) + if ct["uid"] == "rich_text": + ct["schema"] = [f for f in ct["schema"] if f.get("uid") != "content"] + ct["schema"].append(html_rte_field("content", "Content")) + # passthrough config: CS has no generic JSON field -> store JSON as multiline text + if ct["uid"] == "passthrough_component": + for f in ct["schema"]: + if f.get("uid") == "config": + f["data_type"] = "text" + f["field_metadata"] = {"description": "JSON-encoded verbatim OCC fields", + "multiline": True} + f.pop("enum", None) + for f in ct["schema"]: + if f.get("data_type") in ("reference", "global_field"): + f.setdefault("field_metadata", {}) + if f.get("data_type") == "reference": + f["field_metadata"].setdefault("ref_multiple", bool(f.get("field_metadata", {}).get("ref_multiple"))) + # CS select fields (enum) require a display_type + if f.get("enum"): + f["display_type"] = "dropdown" + f.setdefault("field_metadata", {}) + return ensure_title(ct) + + +# ---------------- idempotent upserts ---------------- +def upsert_global_field(gf): + uid = gf["uid"] + st, _ = cma("GET", f"/global_fields/{uid}") + body = {"global_field": gf} + if st == 200: + st2, r = cma("PUT", f"/global_fields/{uid}", body) + return "updated" if st2 == 200 else f"ERR {st2}:{r}" + st2, r = cma("POST", "/global_fields", body) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r)[:300]}" + + +def upsert_content_type(ct): + uid = ct["uid"] + st, _ = cma("GET", f"/content_types/{uid}") + body = {"content_type": ct} + if st == 200: + st2, r = cma("PUT", f"/content_types/{uid}", body) + return "updated" if st2 == 200 else f"ERR {st2}:{json.dumps(r)[:400]}" + st2, r = cma("POST", "/content_types", body) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r)[:400]}" + + +def download_occ(url): + full = url if url.startswith("http") else OCC_HOST + url + with urllib.request.urlopen(urllib.request.Request(full), context=OCC_CTX, timeout=60) as r: + return r.read() + + +def multipart(fields, filename, filedata, mime): + boundary = "----csseed" + str(len(filedata)) + out = [] + for k, v in fields.items(): + out.append(f"--{boundary}\r\nContent-Disposition: form-data; name=\"{k}\"\r\n\r\n{v}\r\n".encode()) + out.append((f"--{boundary}\r\nContent-Disposition: form-data; name=\"asset[upload]\"; " + f"filename=\"{filename}\"\r\nContent-Type: {mime}\r\n\r\n").encode()) + out.append(filedata) + out.append(f"\r\n--{boundary}--\r\n".encode()) + return b"".join(out), f"multipart/form-data; boundary={boundary}" + + +def asset_lookup(code): + """Idempotency key = asset title (= the media code; no length limit, unlike tags).""" + q = urllib.parse.quote(json.dumps({"title": code})) + st, r = cma("GET", f"/assets?query={q}&limit=1") + if st == 200 and r.get("assets"): + return r["assets"][0]["uid"] + return None + + +def upsert_asset(a, cache): + code = a["code"] + existing = asset_lookup(code) + if existing: + cache[code] = existing + return "exists" + try: + data = download_occ(a["url"]) + except Exception as e: + return f"ERR download:{e}" + fname = code.split("/")[-1].split("?")[0] or "asset" + mime = a.get("mime") or mimetypes.guess_type(fname)[0] or "application/octet-stream" + short_tag = "sapmedia_" + str(abs(hash(code)) % 10**10) # <=50 chars, stable within run + raw, ct = multipart({"asset[title]": code, "asset[tags]": short_tag}, fname, data, mime) + st2, r2 = cma("POST", "/assets", raw=raw, content_type=ct) + if st2 in (200, 201): + cache[code] = r2["asset"]["uid"] + return "created" + return f"ERR {st2}:{json.dumps(r2)[:200]}" + + +def resolve_value(v, key2uid, asset_cache): + """Turn plan placeholders (_ref/_asset/_rte_from_html) into CMA entry values.""" + if isinstance(v, dict): + if v.get("_asset"): + return asset_cache.get(v["code"]) + if v.get("_ref"): + uid = key2uid.get((v["content_type"], v["value"])) + return [{"uid": uid, "_content_type_uid": v["content_type"]}] if uid else None + if "_rte_from_html" in v: + return v["_rte_from_html"] or "" + return {k: resolve_value(x, key2uid, asset_cache) for k, x in v.items()} + if isinstance(v, list): + out = [] + for x in v: + rv = resolve_value(x, key2uid, asset_cache) + if isinstance(x, dict) and x.get("_ref"): + if rv: + out.extend(rv) # flatten single-ref arrays into the list + else: + out.append(rv) + return out + return v + + +def entry_payload(e, key2uid, asset_cache): + skip = {"_content_type", "_key", "_localized"} + body = {} + for k, v in e.items(): + if k in skip: + continue + body[k] = resolve_value(v, key2uid, asset_cache) + # passthrough config is a multiline-text field holding JSON + if e.get("_content_type") == "passthrough_component" and isinstance(body.get("config"), (dict, list)): + body["config"] = json.dumps(body["config"], ensure_ascii=False) + return {"entry": body} + + +def find_entry_uid(ct, key_field, key_value): + q = urllib.parse.quote(json.dumps({key_field: key_value})) + st, r = cma("GET", f"/content_types/{ct}/entries?query={q}&limit=1") + if st == 200 and r.get("entries"): + return r["entries"][0]["uid"] + return None + + +def upsert_entry(e, key2uid, asset_cache): + ct = e["_content_type"] + kf, kv = e["_key"]["field"], e["_key"]["value"] + body = entry_payload(e, key2uid, asset_cache) + existing = find_entry_uid(ct, kf, kv) + if existing: + st, r = cma("PUT", f"/content_types/{ct}/entries/{existing}", body) + uid = existing + status = "updated" if st == 200 else f"ERR {st}:{json.dumps(r)[:200]}" + else: + st, r = cma("POST", f"/content_types/{ct}/entries", body) + uid = r.get("entry", {}).get("uid") + status = "created" if st in (200, 201) else f"ERR {st}:{json.dumps(r)[:200]}" + if uid: + key2uid[(ct, kv)] = uid + return status + + +# ---------------- driver ---------------- +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--model-dir", required=True) + ap.add_argument("--site", required=True) + ap.add_argument("--stages", default="globals,content_types,assets,entries") + ap.add_argument("--limit-entries", type=int, default=0, help="cap entries per group (smoke test)") + a = ap.parse_args() + if not (KEY and TOKEN): + die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set in the environment") + stages = set(s.strip() for s in a.stages.split(",")) + + ct_doc = json.load(open(os.path.join(a.model_dir, "content-types.json"))) + ep = json.load(open(os.path.join(a.model_dir, "entry-plan.json"))) + + # topo order for content types (reuse seed.py logic) + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + import seed + order = seed.topo_content_types(ct_doc["content_types"]) + by_uid = {c["uid"]: c for c in ct_doc["content_types"]} + + key2uid, asset_cache = {}, {} + + if "globals" in stages: + print("[globals]") + for gf in ct_doc.get("global_fields", []): + print(f" {gf['uid']}: {upsert_global_field(gf)}") + + if "content_types" in stages: + print("[content_types] (dependency order)") + for uid in order: + ct = normalize_content_type(by_uid[uid]) + print(f" {uid}: {upsert_content_type(ct)}") + + if "assets" in stages: + print(f"[assets] {len(ep['assets'])}") + ok = err = 0 + for asset in ep["assets"]: + r = upsert_asset(asset, asset_cache) + if r.startswith("ERR"): + err += 1; print(f" {asset['code']}: {r}") + else: + ok += 1 + print(f" assets: {ok} ok, {err} err, {len(asset_cache)} in cache") + + if "entries" in stages: + # assets must be resolvable for banner refs even if we skipped the stage this run + if not asset_cache and "assets" not in stages: + for asset in ep["assets"]: + uid = asset_lookup(asset["code"]) + if uid: + asset_cache[asset["code"]] = uid + # Nav components hold an all_nodes pool of nav_node_flat entries, which in + # turn reference `link` leaves (ordinary components). Refs resolve only + # against entries already created this run (key2uid), so seed in this + # order: non-nav components (incl. link leaves) -> nav_nodes -> nav + # components -> slots -> pages. + NAV_CTS = {"category_navigation_flat", "footer_navigation_flat"} + comps = ep["entries"].get("components", []) + seed_groups = [ + ("components", [e for e in comps if e["_content_type"] not in NAV_CTS]), + ("nav_nodes", ep["entries"].get("nav_nodes", [])), + ("nav_components", [e for e in comps if e["_content_type"] in NAV_CTS]), + ("slots", ep["entries"]["slots"]), + ("pages", ep["entries"]["pages"]), + ] + for group, items in seed_groups: + if a.limit_entries: + items = items[:a.limit_entries] + print(f"[entries:{group}] {len(items)}") + ok = err = 0 + for e in items: + r = upsert_entry(e, key2uid, asset_cache) + if r.startswith("ERR"): + err += 1 + if err <= 5: + print(f" {e['_key']['value']}: {r}") + else: + ok += 1 + print(f" {group}: {ok} ok, {err} err") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed_connector.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed_connector.py new file mode 100644 index 0000000..5881222 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed_connector.py @@ -0,0 +1,222 @@ +#!/usr/bin/env python3 +""" +cma_seed_connector — live seeder for the CONNECTOR reference-field profile (Phase 4b). + +Imports the connector's shipped starter-pack content types into the stack, then applies a +connector-plan.json (from model_connector) idempotently: + content types (starter pack, +sap_uid idempotency field on components) + -> assets -> component entries (by sap_uid) -> global_slots (by title) -> pages (by url). + +Reuses low-level CMA helpers from cma_seed.py. Creds from env (CS_API_KEY, +CS_MANAGEMENT_TOKEN, CS_REGION_BASE). English/master locale only. + +Usage: + python3 cma_seed_connector.py --plan /connector-plan.json \ + --starter-pack /import-export/starter-pack/content_types \ + --stages types,assets,entries +""" +import argparse, json, os, sys, urllib.parse +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "marketplace-app")) +import cma_seed as low # reuse cma(), asset_lookup(), upsert_asset(), download_occ, multipart +from sap_products import SapProductResolver, is_placeholder + +PAGE_TYPES = {"landing_page", "content_page", "product_page", "category_page"} +GLOBAL_TYPE = "global_slots" + +SAP_UID_FIELD = {"display_name": "SAP UID", "uid": "sap_uid", "data_type": "text", + "mandatory": False, "unique": True, "multiple": False, + "field_metadata": {"instruction": "Seed idempotency key = SAP component uid."}} + + +def sap_products_field(extension_uid): + """The SAP Commerce Cloud app's product picker, as a carousel field. The extension uid is + per stack (from sap_app.py), so it is injected at import time, not shipped in the pack.""" + return {"display_name": "SAP Products", "uid": "sap_products", "data_type": "json", + "extension_uid": extension_uid, "config": {}, + "field_metadata": {"extension": True, "description": + "Pick products from SAP Commerce Cloud. The storefront reads the product codes from here."}, + "mandatory": False, "unique": False, "multiple": False, "non_localizable": False} + + +def load_starter_types(dir_): + out = {} + for fn in os.listdir(dir_): + if fn.endswith(".json"): + ct = json.load(open(os.path.join(dir_, fn))) + out[ct["uid"]] = ct + return out + + +def topo_types(starter): + """Order all starter content types so every reference target precedes its user.""" + ordered, seen = [], set() + def deps(ct): + d = set() + for f in ct.get("schema", []): + if f.get("data_type") == "reference": + for t in (f.get("reference_to") or []): + if t in starter and t != ct["uid"]: + d.add(t) + return d + def visit(uid, stack): + if uid in seen or uid in stack: + return + stack.add(uid) + for d in deps(starter[uid]): + visit(d, stack) + stack.discard(uid); seen.add(uid); ordered.append(uid) + for uid in starter: + visit(uid, set()) + return ordered + + +def import_content_types(needed, starter, sap_ext=None, verbose=True): + # Import the FULL shipped model (topo-ordered) so every reference_to resolves, + # even for types whose entries we defer (nav). Inject sap_uid on component types, + # and the SAP app's product picker on the carousel when the app is set up. + order = topo_types(starter) + for uid in order: + ct = json.loads(json.dumps(starter[uid])) + is_component = uid not in PAGE_TYPES and uid != GLOBAL_TYPE + if is_component and not any(f.get("uid") == "sap_uid" for f in ct["schema"]): + ct["schema"].append(dict(SAP_UID_FIELD)) + if uid == "product_carousel_component" and sap_ext: + ct["schema"] = [f for f in ct["schema"] if f.get("uid") != "sap_products"] + at = next((i + 1 for i, f in enumerate(ct["schema"]) if f.get("uid") == "products"), + len(ct["schema"])) + ct["schema"].insert(at, sap_products_field(sap_ext)) + st, _ = low.cma("GET", f"/content_types/{uid}") + body = {"content_type": ct} + if st == 200: + st2, r = low.cma("PUT", f"/content_types/{uid}", body) + res = "updated" if st2 == 200 else f"ERR {st2}:{json.dumps(r)[:200]}" + else: + st2, r = low.cma("POST", "/content_types", body) + res = "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r)[:200]}" + if verbose: + print(f" {uid}: {res}") + + +def resolve(v, uid_map, asset_cache, sap=None): + if isinstance(v, dict): + if v.get("_asset"): + return asset_cache.get(v["code"]) + if v.get("_ref"): + uid = uid_map.get(v["value"]) + return [{"uid": uid, "_content_type_uid": v["content_type"]}] if uid else None + if is_placeholder(v): + return sap.resolve(v) + return {k: resolve(x, uid_map, asset_cache, sap) for k, x in v.items()} + if isinstance(v, list): + out = [] + for x in v: + rv = resolve(x, uid_map, asset_cache, sap) + if isinstance(x, dict) and x.get("_ref"): + if rv: + out.extend(rv) + else: + out.append(rv) + return out + return v + + +def entry_body(e, uid_map, asset_cache, sap=None): + return {"entry": {k: resolve(v, uid_map, asset_cache, sap) + for k, v in e.items() if not k.startswith("_")}} + + +def find_uid(ct, field, value): + q = urllib.parse.quote(json.dumps({field: value})) + st, r = low.cma("GET", f"/content_types/{ct}/entries?query={q}&limit=1") + return (r.get("entries") or [{}])[0].get("uid") if st == 200 else None + + +def upsert_entry(e, uid_map, asset_cache, sap=None): + ct = e["_content_type"]; kf, kv = e["_key"]["field"], e["_key"]["value"] + body = entry_body(e, uid_map, asset_cache, sap) + existing = find_uid(ct, kf, kv) + if existing: + st, r = low.cma("PUT", f"/content_types/{ct}/entries/{existing}", body) + uid, res = existing, ("updated" if st == 200 else f"ERR {st}:{json.dumps(r)[:160]}") + else: + st, r = low.cma("POST", f"/content_types/{ct}/entries", body) + uid, res = r.get("entry", {}).get("uid"), ("created" if st in (200, 201) else f"ERR {st}:{json.dumps(r)[:160]}") + if uid and kf == "sap_uid": + uid_map[kv] = uid + return res, uid + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--plan", required=True) + ap.add_argument("--starter-pack", required=True) + ap.add_argument("--stages", default="types,assets,entries") + ap.add_argument("--sap-app", help="sap-app.json from marketplace-app/sap_app.py " + "(required when the plan seeds carousels into sap_products)") + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set") + stages = set(s.strip() for s in a.stages.split(",")) + plan = json.load(open(a.plan)) + uid_map, asset_cache = {}, {} + sap_ext, sap = None, None + if plan.get("carousel_field") == "sap_products": + if not a.sap_app: + low.die("plan seeds carousels into sap_products — pass --sap-app ") + sap_ext = json.load(open(a.sap_app))["product_extension_uid"] + sap = SapProductResolver(site=plan.get("site")) + + if "types" in stages: + print("[content types] (starter pack, +sap_uid on components" + f"{', +sap_products on carousel' if sap_ext else ''})") + import_content_types(plan["content_types_needed"], load_starter_types(a.starter_pack), sap_ext) + + if "assets" in stages: + print(f"[assets] {len(plan['assets'])}") + ok = err = 0 + for asset in plan["assets"]: + r = low.upsert_asset(asset, asset_cache) + ok += 1 if not r.startswith("ERR") else 0 + err += 1 if r.startswith("ERR") else 0 + if r.startswith("ERR"): + print(f" {asset['code']}: {r}") + print(f" assets: {ok} ok, {err} err") + + if "entries" in stages: + if not asset_cache: # resolve existing assets for banner refs + for asset in plan["assets"]: + u = low.asset_lookup(asset["code"]) + if u: + asset_cache[asset["code"]] = u + comps = plan["entries"]["components"] + print(f"[components] {len(comps)}") + ok = err = 0 + for e in comps: + res, _ = upsert_entry(e, uid_map, asset_cache, sap) + ok += 1 if not res.startswith("ERR") else 0 + if res.startswith("ERR"): + err += 1 + if err <= 6: + print(f" {e['_key']['value']}: {res}") + print(f" components: {ok} ok, {err} err") + if sap: + print(f" sap_products: {len(sap.cache) - len(sap.missing)} products resolved on the app's SAP server" + + (f", NOT FOUND (left out): {[c for c, _ in sap.missing]}" if sap.missing else "")) + gs = plan["entries"].get("global_slots") + if gs: + res, _ = upsert_entry(gs, uid_map, asset_cache, sap) + print(f"[global_slots] {res}") + pages = plan["entries"]["pages"] + print(f"[pages] {len(pages)}") + ok = err = 0 + for e in pages: + res, _ = upsert_entry(e, uid_map, asset_cache, sap) + ok += 1 if not res.startswith("ERR") else 0 + if res.startswith("ERR"): + err += 1; print(f" {e['_key']['value']}: {res}") + print(f" pages: {ok} ok, {err} err") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/cs_auth.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/cs_auth.py new file mode 100644 index 0000000..f3f8807 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/cs_auth.py @@ -0,0 +1,55 @@ +""" +cs_auth — the Management API auth header for every pipeline call. + +Two modes (CS_AUTH = token | csdx | auto, default auto): + token CS_MANAGEMENT_TOKEN, sent as `authorization: ` (stack-scoped, as before) + csdx the user's own Contentstack CLI session (`csdx auth:login --oauth`), via csdx_auth.cjs; + OAuth tokens are refreshed by csdx, and once more after a 401 + auto the management token when CS_MANAGEMENT_TOKEN is set, else the csdx session + +With the csdx session, writes run as the logged-in user (the stack's audit log shows them). +""" +import json, os, subprocess + +HERE = os.path.dirname(os.path.abspath(__file__)) +MANAGEMENT_TOKEN = os.environ.get("CS_MANAGEMENT_TOKEN") +_MODE = os.environ.get("CS_AUTH", "auto").lower() +_cache = {} + + +def mode(): + if _MODE == "token" or (_MODE == "auto" and MANAGEMENT_TOKEN): + return "token" + return "csdx" + + +def available(): + """True when some credential is configured (the csdx session is checked on first use).""" + return bool(MANAGEMENT_TOKEN) if mode() == "token" else True + + +def headers(refresh=False): + if mode() == "token": + return {"authorization": MANAGEMENT_TOKEN} + if refresh or "h" not in _cache: + args = ["node", os.path.join(HERE, "csdx_auth.cjs")] + (["--refresh"] if refresh else []) + # When csdx refreshes an expired OAuth token, the helper can exit before printing; + # the refreshed token is saved, so a second call returns it. + for _ in range(3): + p = subprocess.run(args, capture_output=True, text=True) + if p.returncode: + raise SystemExit((p.stderr or "csdx_auth failed").strip()) + # csdx can print progress lines (e.g. "Forcing token refresh...") on stdout + # before the helper's JSON, so read only the JSON line. + line = next((l for l in reversed(p.stdout.splitlines()) if l.strip().startswith("{")), "") + if line: + _cache["h"] = json.loads(line) + break + args = args[:2] + else: + raise SystemExit("[cs_auth] csdx returned no session — run `csdx auth:login --oauth`") + return dict(_cache["h"]) + + +def describe(): + return "management token" if mode() == "token" else "csdx login (CS_AUTH=csdx)" diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/csdx_auth.cjs b/skills/dx-sap-spartacus-import/scripts/cs-seed/csdx_auth.cjs new file mode 100644 index 0000000..1678808 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/csdx_auth.cjs @@ -0,0 +1,44 @@ +#!/usr/bin/env node +/* + * csdx_auth — print the Contentstack Management API auth header of the user's csdx session, + * as JSON on stdout, for cs_auth.py. Uses csdx's own @contentstack/cli-utilities, which + * refreshes an expired OAuth access token first. + * + * node csdx_auth.cjs # {"authorization": "Bearer …"} (OAuth) or {"authtoken": "…"} (basic) + * node csdx_auth.cjs --refresh # force an OAuth refresh first (after a 401) + * + * The value goes to the calling process only; never log it. + */ +const fs = require('fs'); +const path = require('path'); +const cp = require('child_process'); + +function fail(msg) { + process.stderr.write(`[csdx_auth] ${msg}\n`); + process.exit(1); +} + +(async () => { + let csdx; + try { + csdx = fs.realpathSync(cp.execSync('command -v csdx', { encoding: 'utf8' }).trim()); + } catch { + fail('csdx not found on PATH — install @contentstack/cli, or set CS_MANAGEMENT_TOKEN'); + } + const root = path.resolve(path.dirname(csdx), '..'); + const U = require(require.resolve('@contentstack/cli-utilities', { paths: [root] })); + const type = U.configHandler.get('authorisationType'); + if (type === 'OAUTH') { + if (process.argv.includes('--refresh')) await U.authHandler.compareOAuthExpiry(true); + else await U.authHandler.compareOAuthExpiry(); + const token = U.configHandler.get('oauthAccessToken'); + if (!token) fail('csdx OAuth session has no access token — run `csdx auth:login --oauth`'); + process.stdout.write(JSON.stringify({ authorization: `Bearer ${token}` })); + } else if (type === 'BASIC') { + const token = U.configHandler.get('authtoken'); + if (!token) fail('csdx session has no authtoken — run `csdx auth:login --oauth`'); + process.stdout.write(JSON.stringify({ authtoken: token })); + } else { + fail('not logged in — run `csdx auth:login --oauth`, or set CS_MANAGEMENT_TOKEN'); + } +})().catch((e) => fail(e?.message || String(e))); diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/localize.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/localize.py new file mode 100644 index 0000000..efa74c9 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/localize.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +""" +localize — full per-locale localization from the FULLY LOCALIZED OCC response (i18n v2). + +Unlike a field-diff, this drives off a complete localized inventory (extract.py run with +lang=de/ja/zh + nav_resolve --lang), so pages, components, nav AND per-locale media are all +localized. For one target locale it: + 1. uploads the locale's images (different per language, e.g. ...DE...) as CS assets, + 2. localizes EVERY authored entry (components + pages + global_slots + nav) by writing a + locale variant (PUT ?locale) with localized scalar/text fields + localized asset refs; + reference fields are left to inherit from master so they resolve to localized targets, + 3. publishes each localized entry to the environment in that locale (?locale=, required — + else CS 141 "Localised entries can not be published from master locale"). + +The base (master en-us) entries must already be seeded. Matches entries by their key +(sap_uid / url / title). Idempotent. Creds from env (CS_API_KEY, CS_MANAGEMENT_TOKEN). + +Usage: + python3 localize.py --locale de-de --plan /connector-plan.json --env development +""" +import argparse, json, os, sys, urllib.parse +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "marketplace-app")) +import cma_seed as low # cma(), upsert_asset(), asset_lookup(), download_occ +from sap_products import SapProductResolver, is_placeholder +from publish import ensure_environment + + +_LOCALE_NAMES = {"de-de": "German - Germany", "ja-jp": "Japanese - Japan", + "zh-cn": "Chinese - China", "fr-fr": "French - France", + "es-es": "Spanish - Spain", "it-it": "Italian - Italy"} + + +def ensure_locale(code, master): + """Create the target locale (fallback = master) if the stack doesn't have it.""" + st, r = low.cma("GET", "/locales") + if code in {l["code"] for l in (r.get("locales") or [])}: + return "exists" + st2, r2 = low.cma("POST", "/locales", + {"locale": {"code": code, "fallback_locale": master, + "name": _LOCALE_NAMES.get(code, code)}}) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r2)[:120]}" + + +def find_uid(ct, field, value, master): + q = urllib.parse.quote(json.dumps({field: value})) + st, r = low.cma("GET", f"/content_types/{ct}/entries?query={q}&locale={master}&limit=1") + es = r.get("entries") or [] + return es[0]["uid"] if es else None + + +def loc_fields(e, asset_cache, ref_uid, sap=None): + """Full localized payload. IMPORTANT: a Contentstack localized entry is standalone — + it does NOT per-field-inherit from master, so we must send EVERY field, including + reference fields (resolved to the shared entry uids, which point to the localized + versions at query time) and localized asset uids. A `sap_products` placeholder becomes + the same picker value as master (the picker stores the app's language, not the locale's).""" + def resolve(v): + if isinstance(v, dict): + if v.get("_asset"): + return asset_cache.get(v["code"]) + if v.get("_ref"): + uid = ref_uid(v["content_type"], v["value"]) + return [{"uid": uid, "_content_type_uid": v["content_type"]}] if uid else None + if is_placeholder(v): + return sap.resolve(v) + return {k: resolve(x) for k, x in v.items()} + if isinstance(v, list): + out = [] + for x in v: + rv = resolve(x) + if isinstance(x, dict) and x.get("_ref"): + if rv: + out.extend(rv) + elif rv is not None: + out.append(rv) + return out + return v + return {k: resolve(v) for k, v in e.items() if not k.startswith("_") and k != "sap_meta"} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--locale", required=True, help="target CS locale, e.g. de-de") + ap.add_argument("--plan", required=True, help="connector-plan.json from the LOCALIZED inventory") + ap.add_argument("--env", default="development") + ap.add_argument("--master", default="en-us") + ap.add_argument("--preview-url", default=os.environ.get("CS_PREVIEW_URL", "http://localhost:4200")) + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set") + loc = a.locale + plan = json.load(open(a.plan)) + sap = SapProductResolver(site=plan.get("site")) if plan.get("carousel_field") == "sap_products" else None + print(f"[{loc}] locale: {ensure_locale(loc, a.master)}") + print(f"[{loc}] live preview URL: {ensure_environment(a.env, [loc], a.preview_url)}") + + # 1) localized assets (different image per language) -> code->uid, then PUBLISH them + # (unpublished assets are not delivered -> nested banner media resolves to null). + asset_cache = {} + ok = err = 0 + for asset in plan.get("assets", []): + r = low.upsert_asset(asset, asset_cache) # downloads localized bytes, uploads by title + ok += 1 if not r.startswith("ERR") else 0 + err += 1 if r.startswith("ERR") else 0 + pub = 0 + for aid in set(asset_cache.values()): + st, _ = low.cma("POST", f"/assets/{aid}/publish", + {"asset": {"environments": [a.env], "locales": [a.master, loc]}}) + pub += 1 if st in (200, 201) else 0 + print(f"[{loc}] assets: {ok} ok, {err} err, {len(asset_cache)} resolved, {pub} published") + + # resolve a reference target's entry uid by sap_uid (shared across locales), cached + ref_cache = {} + def ref_uid(ct, sap_uid): + key = (ct, sap_uid) + if key not in ref_cache: + ref_cache[key] = find_uid(ct, "sap_uid", sap_uid, a.master) + return ref_cache[key] + + def do(entries, label): + put_ok = pub_ok = miss = e_err = 0 + for e in entries: + ct = e["_content_type"]; kf = e["_key"]["field"]; kv = e["_key"]["value"] + uid = find_uid(ct, kf, kv, a.master) + if not uid: + miss += 1; continue + fields = loc_fields(e, asset_cache, ref_uid, sap) + st, r = low.cma("PUT", f"/content_types/{ct}/entries/{uid}?locale={loc}", {"entry": fields}) + if st not in (200, 201): + e_err += 1 + if e_err <= 4: + print(f" {loc} {ct}/{kv}: PUT {st} {json.dumps(r)[:120]}") + continue + put_ok += 1 + st2, _ = low.cma("POST", f"/content_types/{ct}/entries/{uid}/publish?locale={loc}", + {"entry": {"environments": [a.env], "locales": [loc]}}) + pub_ok += 1 if st2 in (200, 201) else 0 + print(f" [{loc}] {label}: localized {put_ok}, published {pub_ok}, missing {miss}, err {e_err}") + + do(plan["entries"]["components"], "components+nav") + if plan["entries"].get("global_slots"): + do([plan["entries"]["global_slots"]], "global_slots") + do(plan["entries"]["pages"], "pages") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/publish.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/publish.py new file mode 100644 index 0000000..6a1152d --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/publish.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +""" +publish — publish seeded entries + assets to a Contentstack environment (Phase 4b). + +Publishing makes entries readable by a delivery token (a storefront/CDA). Publishes every +entry of every content type on the stack, plus every asset, to the given environment/locale. +Idempotent (re-publishing is safe). Creds from env (CS_API_KEY, CS_MANAGEMENT_TOKEN). + +Usage: python3 publish.py --env development [--locale en-us] +""" +import argparse, json, os, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import cma_seed as low + + +def ensure_environment(name, locales=("en-us",), url="http://localhost:4200"): + """Create the target environment if missing (fresh-stack support) and give it a base URL + for every locale. Live Preview is locale-based: a locale without a URL shows "Unable to + load the preview". Existing URLs are kept; only missing locales are added.""" + st, r = low.cma("GET", "/environments") + env = next((e for e in (r.get("environments") or []) if e["name"] == name), None) + if not env: + st2, r2 = low.cma("POST", "/environments", {"environment": { + "name": name, "urls": [{"locale": l, "url": url} for l in locales]}}) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r2)[:160]}" + urls = {u["locale"]: u.get("url") for u in env.get("urls") or []} + missing = [l for l in locales if not urls.get(l)] + if not missing: + return "exists" + urls.update({l: url for l in missing}) + st2, r2 = low.cma("PUT", f"/environments/{name}", {"environment": { + "name": name, "urls": [{"locale": l, "url": u} for l, u in urls.items()]}}) + return f"exists, added URL for {missing}" if st2 == 200 else f"ERR {st2}:{json.dumps(r2)[:160]}" + + +def list_all(path, key): + out, skip = [], 0 + while True: + st, r = low.cma("GET", f"{path}{'&' if '?' in path else '?'}limit=100&skip={skip}&include_count=true") + items = r.get(key) or [] + out += items; skip += len(items) + if len(items) < 100 or skip >= (r.get("count") or 0): + break + return out + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--env", required=True) + ap.add_argument("--locale", default="en-us") + ap.add_argument("--preview-locales", default="", + help="comma list of extra locales that need a Live Preview URL, e.g. de-de,ja-jp") + ap.add_argument("--preview-url", default=os.environ.get("CS_PREVIEW_URL", "http://localhost:4200"), + help="storefront base URL for Live Preview (default $CS_PREVIEW_URL or localhost:4200)") + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set") + + envs, loc = [a.env], [a.locale] + preview_locales = [a.locale] + [l for l in a.preview_locales.split(",") if l and l != a.locale] + print(f"[environment '{a.env}'] {ensure_environment(a.env, preview_locales, a.preview_url)}") + # assets first, then entries (components/shell) then pages last + assets = list_all("/assets", "assets") + print(f"[publish assets] {len(assets)}") + ok = err = 0 + for x in assets: + st, r = low.cma("POST", f"/assets/{x['uid']}/publish", + {"asset": {"environments": envs, "locales": loc}}) + ok += st in (200, 201); err += st not in (200, 201) + print(f" assets: {ok} ok, {err} err") + + cts = [c["uid"] for c in list_all("/content_types", "content_types")] + # order: components, then global_slots, then pages + page_types = {"landing_page", "content_page", "product_page", "category_page"} + order = ([c for c in cts if c not in page_types and c != "global_slots"] + + [c for c in cts if c == "global_slots"] + + [c for c in cts if c in page_types]) + for ct in order: + entries = list_all(f"/content_types/{ct}/entries", "entries") + if not entries: + continue + ok = err = 0 + for e in entries: + st, r = low.cma("POST", f"/content_types/{ct}/entries/{e['uid']}/publish", + {"entry": {"environments": envs, "locales": loc}}) + ok += st in (200, 201); err += st not in (200, 201) + if st not in (200, 201) and err <= 3: + print(f" {ct}/{e['uid']}: {st} {json.dumps(r)[:120]}") + print(f"[publish {ct}] {ok} ok, {err} err") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_app.cjs b/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_app.cjs new file mode 100644 index 0000000..3a62fc5 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_app.cjs @@ -0,0 +1,234 @@ +#!/usr/bin/env node +/* + * sap_app — install + configure the SAP Commerce Cloud marketplace app on a stack. + * + * A management token cannot install or configure marketplace apps, so this step runs on the + * user's own Contentstack CLI session (`csdx auth:login --oauth`), through csdx's bundled + * @contentstack/cli-utilities (the same client csdx plugins use). Nothing is stored here. + * + * node sap_app.cjs status --api-key # read-only: installation + config summary + * node sap_app.cjs ensure --api-key # install + configure once if missing; + * # an existing install is only checked + * node sap_app.cjs ensure --api-key --reinstall # uninstall, install, configure once + * + * The configuration applied is one named multi-config (SAP_APP_CONFIG_NAME, default "hti"), + * set as default, using a Custom URL, plus "Keys to Save" = Custom Fields: code + name + * (keeps each product ~110 B so carousels stay under the 20 KB JSON-field limit). + * + * Env (see SKILL.md › Environment Variables): + * SAP_APP_UID marketplace app uid (default: looked up by name) + * SAP_APP_NAME default "SAP Commerce Cloud" + * SAP_APP_CONFIG_NAME default "hti" + * SAP_APP_API_BASE_URL SAP OCC host, no scheme (required for ensure) + * SAP_APP_API_ROUTE default "/occ/v2/" + * SAP_APP_BASE_SITE e.g. "electronics" (required for ensure) + * SAP_APP_BACKOFFICE_URL backoffice host or URL (required for ensure) + * CS_ORG_UID optional; otherwise read from the stack + * + * Prints one JSON object on stdout; exits non-zero with a readable message on failure. + */ +const fs = require('fs'); +const path = require('path'); +const cp = require('child_process'); + +function loadCliUtilities() { + let csdx; + try { + csdx = fs.realpathSync(cp.execSync('command -v csdx', { encoding: 'utf8' }).trim()); + } catch { + fail('csdx not found on PATH — install @contentstack/cli and run `csdx auth:login --oauth`'); + } + const root = path.resolve(path.dirname(csdx), '..'); + return require(require.resolve('@contentstack/cli-utilities', { paths: [root] })); +} + +function fail(msg, extra) { + process.stderr.write(`[sap_app] ERROR: ${msg}\n`); + if (extra) process.stderr.write(`${extra}\n`); + process.exit(1); +} + +function arg(name, def) { + const i = process.argv.indexOf(`--${name}`); + return i > -1 ? process.argv[i + 1] : def; +} + +function errText(e) { + return [e?.status, e?.errorMessage || e?.error_message || e?.message].filter(Boolean).join(' '); +} + +function wantedConfig() { + const need = ['SAP_APP_API_BASE_URL', 'SAP_APP_BASE_SITE', 'SAP_APP_BACKOFFICE_URL']; + const missing = need.filter((k) => !process.env[k]); + if (missing.length) fail(`missing env: ${missing.join(', ')}`); + return { + name: process.env.SAP_APP_CONFIG_NAME || 'hti', + apiBaseUrl: process.env.SAP_APP_API_BASE_URL.replace(/^https?:\/\//, '').replace(/\/+$/, ''), + apiRoute: process.env.SAP_APP_API_ROUTE || '/occ/v2/', + baseSite: process.env.SAP_APP_BASE_SITE, + backofficeUrl: process.env.SAP_APP_BACKOFFICE_URL, + keys: ['code', 'name'], + }; +} + +/* + * The app splits each named configuration in two, exactly as its Configuration screen saves it + * (read from a working "hti" installation): + * configuration multi_config_keys[name] = {is_custom_baseUrl, base_url, api_route, backoffice_url} + * + custom_keys (Keys to Save), default/active_multi_config, is_custom_json + * server config multi_config_keys[name] = {base_url, api_route, base_site_id} + * Both merge into what is already there, so other configurations on the stack are kept. + */ +function buildConfiguration(current, w) { + const cfg = JSON.parse(JSON.stringify(current || {})); + cfg.multi_config_keys = { + ...(cfg.multi_config_keys || {}), + [w.name]: { + is_custom_baseUrl: true, + base_url: w.apiBaseUrl, + api_route: w.apiRoute, + backoffice_url: w.backofficeUrl, + }, + }; + cfg.custom_keys = w.keys.map((k) => ({ label: k, value: k, searchLabel: k, isDisabled: true })); + cfg.is_custom_json = true; + cfg.keypath_options = cfg.keypath_options || []; + cfg.default_multi_config_key = w.name; + cfg.active_multi_config = w.name; + return cfg; +} + +function buildServerConfig(current, w) { + const cfg = JSON.parse(JSON.stringify(current || {})); + cfg.multi_config_keys = { + ...(cfg.multi_config_keys || {}), + [w.name]: { base_url: w.apiBaseUrl, api_route: w.apiRoute, base_site_id: w.baseSite }, + }; + return cfg; +} + +// The public "SAP Commerce Cloud" marketplace app (same uid in every org). +const DEFAULT_SAP_APP_UID = '6388916afd520e0019bb13bc'; + +async function findAppUid(mp, appName) { + if (appName === 'SAP Commerce Cloud') return DEFAULT_SAP_APP_UID; + for (let skip = 0; skip < 500; skip += 50) { // API caps limit at 50 + const items = (await mp.findAllApps({ limit: 50, skip })).items || []; + const hit = items.find((a) => (a.name || '').trim() === appName); + if (hit) return hit.uid; + if (items.length < 50) break; + } + fail(`marketplace app "${appName}" not found; set SAP_APP_UID`); +} + +function summarize(cfg) { + const out = {}; + for (const [k, v] of Object.entries(cfg || {})) { + out[k] = /token|secret|password|private/i.test(k) ? '' : v; + } + return out; +} + +async function main() { + const cmd = process.argv[2]; + const apiKey = arg('api-key', process.env.CS_API_KEY); + if (!['status', 'ensure'].includes(cmd) || !apiKey) { + fail('usage: node sap_app.cjs --api-key '); + } + const U = loadCliUtilities(); + const authType = U.configHandler.get('authorisationType'); + if (!authType) fail('not logged in — run `csdx auth:login --oauth` first'); + + // Org of the stack: CS_ORG_UID, else read from the stack, else the org the OAuth login was + // granted for (an OAuth session may not be allowed to read stack details). + let org = process.env.CS_ORG_UID; + if (!org) { + try { + const mgmt = await U.managementSDKClient({}); + org = (await mgmt.stack({ api_key: apiKey }).fetch()).org_uid; + } catch (e) { + org = authType === 'OAUTH' ? U.configHandler.get('oauthOrgUid') : undefined; + if (!org) { + fail(`could not read stack ${apiKey} with the csdx session (${errText(e)})`, + authType === 'BASIC' + ? 'Your csdx session is a basic (email/password) login; orgs with SSO reject it. Run `csdx auth:login --oauth`.' + : 'Run `csdx auth:login --oauth` again, or set CS_ORG_UID.'); + } + } + } + const mp = (await U.marketplaceSDKClient({})).marketplace(org); + const appName = process.env.SAP_APP_NAME || 'SAP Commerce Cloud'; + + const installs = (await mp.installation().fetchAll({ target_uids: apiKey })).items || []; + let inst = installs.find((i) => (process.env.SAP_APP_UID && i.manifest?.uid === process.env.SAP_APP_UID) + || (i.manifest?.name || '').trim() === appName); + let appUid = process.env.SAP_APP_UID || inst?.manifest?.uid; + + if (cmd === 'status') { + const h = inst && mp.installation(inst.uid); + const config = h ? (await h.configuration())?.data : null; + const server = h ? (await h.serverConfig())?.data : null; + console.log(JSON.stringify({ org_uid: org, app_uid: appUid || null, + installation_uid: inst?.uid || null, + configuration: config && summarize(config), server_configuration: server && summarize(server) }, null, 1)); + return; + } + + const wanted = wantedConfig(); + let installed = false; + const reinstall = process.argv.includes('--reinstall'); + + // Writing the configuration again on an install that already has it breaks the install on + // the app's server: every request then fails with HTTP 414 and it never recovers. So an + // existing install is only ever read here — never re-configured. To change or repair one, + // --reinstall uninstalls it and configures the new install exactly once. + if (inst && !reinstall) { + const h = mp.installation(inst.uid); + const cfg = (await h.configuration())?.data || {}; + const srv = (await h.serverConfig())?.data || {}; + const same = deepEqual(buildConfiguration(cfg, wanted), cfg) + && deepEqual(buildServerConfig(srv, wanted), srv); + if (!same) { + fail(`the app is already installed on ${apiKey} with a different configuration`, + 'Writing it again breaks the install. Change it in the app\'s Configuration page, or re-run with --reinstall.'); + } + console.log(JSON.stringify({ org_uid: org, app_uid: appUid, installation_uid: inst.uid, + installed: false, configured: wanted.name, keys: wanted.keys, unchanged: true }, null, 1)); + return; + } + if (inst && reinstall) { + await mp.installation(inst.uid).uninstall(); + inst = null; + } + if (!inst) { + if (!appUid) appUid = await findAppUid(mp, appName); + try { + await mp.app(appUid).install({ targetUid: apiKey, targetType: 'stack' }); + installed = true; + } catch (e) { + fail(`install failed (${errText(e)})`, + 'If the app needs consent, install it once from Marketplace in the UI, then re-run.'); + } + // The install response doesn't carry a usable installation uid; read it back. + inst = ((await mp.installation().fetchAll({ target_uids: apiKey })).items || []) + .find((i) => i.manifest?.uid === appUid); + if (!inst) fail('app installed, but the installation was not found on the stack — re-run'); + } + // Fresh install: configure it exactly once (see above). + const handle = mp.installation(inst.uid); + await handle.setConfiguration(buildConfiguration({}, wanted)); + await handle.setServerConfig(buildServerConfig({}, wanted)); + console.log(JSON.stringify({ org_uid: org, app_uid: appUid, installation_uid: inst.uid, + installed, reinstalled: reinstall, configured: wanted.name, keys: wanted.keys }, null, 1)); +} + +function deepEqual(a, b) { + const norm = (v) => JSON.stringify(v, (k, x) => (x && typeof x === 'object' && !Array.isArray(x) + ? Object.fromEntries(Object.entries(x).sort(([p], [q]) => p.localeCompare(q))) : x)); + return norm(a) === norm(b); +} + +if (require.main === module) { + main().catch((e) => fail(errText(e) || String(e))); +} +module.exports = { buildConfiguration, buildServerConfig }; diff --git a/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_app.py b/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_app.py new file mode 100644 index 0000000..03eed3e --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_app.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +""" +sap_app — pipeline stage: make sure the SAP Commerce Cloud marketplace app is installed and +configured on the target stack, then find its Product custom-field extension uid. + + 1. install + configure the app (sap_app.cjs, on the user's csdx login — a management token + cannot install or configure marketplace apps), + 2. look up the app's "SAP Commerce Cloud - Product" field extension with the management + token. Marketplace extensions only appear with ?include_marketplace_extensions=true. + +Writes {installation_uid, product_extension_uid, category_extension_uid, ...} to --out, which +cma_seed_connector.py reads to add the `sap_products` field to product_carousel_component. + +Usage: + python3 sap_app.py --out /sap-app.json # install/configure + discover + python3 sap_app.py --out /sap-app.json --skip-install # discover only (app already set up) + python3 sap_app.py --status # read-only summary +""" +import argparse, json, os, subprocess, sys +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "cs-seed")) +import cma_seed as low # cma() + +HERE = os.path.dirname(os.path.abspath(__file__)) +APP_NAME = os.environ.get("SAP_APP_NAME", "SAP Commerce Cloud") + + +def node(cmd, *extra): + p = subprocess.run(["node", os.path.join(HERE, "sap_app.cjs"), cmd, "--api-key", low.KEY, *extra], + capture_output=True, text=True) + if p.returncode: + low.die((p.stderr or p.stdout).strip()) + return json.loads(p.stdout) + + +def find_extensions(): + st, r = low.cma("GET", "/extensions?include_marketplace_extensions=true") + if st != 200: + low.die(f"could not list extensions ({st}): {json.dumps(r)[:160]}") + out = {} + for x in r.get("extensions") or []: + title = " ".join((x.get("title") or "").split()) # app titles have a double space + if not title.startswith(APP_NAME): + continue + if x.get("type") == "field" and title.endswith("Product"): + out["product_extension_uid"] = x["uid"] + elif x.get("type") == "field" and title.endswith("Category"): + out["category_extension_uid"] = x["uid"] + out.setdefault("app_installation_uid", x.get("app_installation_uid") or x.get("app_uid")) + return out + + +def repoint_carousel_field(ext_uid, ct_uid="product_carousel_component"): + """A reinstall creates a new Product field extension; point an existing `sap_products` + field at it (entry data is kept). No-op when the type or field doesn't exist yet.""" + st, r = low.cma("GET", f"/content_types/{ct_uid}") + if st != 200: + return None + ct = r["content_type"] + field = next((f for f in ct["schema"] if f.get("uid") == "sap_products"), None) + if not field or field.get("extension_uid") == ext_uid: + return None + field["extension_uid"] = ext_uid + body = {"content_type": {k: ct[k] for k in ("title", "uid", "schema", "options", "description") if k in ct}} + st, r = low.cma("PUT", f"/content_types/{ct_uid}", body) + return ext_uid if st == 200 else low.die(f"could not re-point {ct_uid}.sap_products ({st}): {json.dumps(r)[:160]}") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--out", help="write the result JSON here") + ap.add_argument("--skip-install", action="store_true", + help="don't touch the app; only discover its extensions") + ap.add_argument("--status", action="store_true", help="read-only summary") + ap.add_argument("--reinstall", action="store_true", + help="uninstall the app on the stack, install it again and configure it once " + "(repairs an install whose picker fails with HTTP 414)") + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set") + + if a.status: + print(json.dumps({**node("status"), **find_extensions()}, indent=1)) + return + result = {} if a.skip_install else node("ensure", *(["--reinstall"] if a.reinstall else [])) + if result: + how = ("already set up — left unchanged" if result.get("unchanged") + else "reinstalled + configured" if result.get("reinstalled") + else "installed + configured") + print(f" app {how}: config '{result.get('configured')}', keys {result.get('keys')}") + result.update(find_extensions()) + if not result.get("product_extension_uid"): + low.die(f"'{APP_NAME} - Product' field extension not found on the stack — " + f"is the app installed with its Custom Field location enabled?") + print(f" product field extension: {result['product_extension_uid']}") + repointed = repoint_carousel_field(result["product_extension_uid"]) + if repointed: + print(f" carousel sap_products field -> {repointed}") + if a.out: + json.dump(result, open(a.out, "w"), indent=2) + print(f" -> {a.out}") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_products.py b/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_products.py new file mode 100644 index 0000000..fc9c2bc --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/marketplace-app/sap_products.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +""" +sap_products — build the value of a `sap_products` field (the SAP Commerce Cloud marketplace +app's product picker) from plain SKU codes, exactly as the picker itself saves it. + +The picker stores, per entry: + + {"type": "sapcommercecloud_product", + "data": [{"code": "300938", "name": "Photosmart E317 Digital Camera", + "cs_metadata": {"multiConfigName": "hti", "isConfigDeleted": false}}, ...]} + +with only the keys the app's "Keys to Save" setting allows (this skill configures the app to +save `code` + `name`, see sap_app.cjs). Names are looked up on the SAP server the APP is +configured for (SAP_APP_* env), not the storefront's OCC, because that is what the picker +reads; product codes are the same on both. + +Contentstack caps a JSON field at 20 KB, so build() refuses a value over that limit. + +Used by cma_seed_connector.py (master locale) and localize.py (other locales): the model step +emits a `{"_sap_products": true, "codes": [...]}` placeholder and the seeders call resolve(). + +Env: + SAP_APP_API_BASE_URL host of the app's SAP OCC (no scheme), e.g. api.<...>.commerce.ondemand.com + SAP_APP_API_ROUTE default /occ/v2/ + SAP_APP_BASE_SITE e.g. electronics + SAP_APP_CONFIG_NAME the app configuration name stored in cs_metadata, default hti +Falls back to OCC_BASE_URL + the run's site when SAP_APP_API_BASE_URL is unset. +""" +import json, os, ssl, urllib.parse, urllib.request + +FIELD_TYPE = "sapcommercecloud_product" +MAX_BYTES = 20 * 1024 + +def _occ_ctx(): + """TLS for the storefront's OCC fallback: verified unless OCC_INSECURE_TLS=1 (a trusted + self-signed dev backend). The app's own SAP server is always verified.""" + ctx = ssl.create_default_context() + if os.environ.get("OCC_INSECURE_TLS") == "1": + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + + +def occ_product_base(site=None): + """OCC products URL prefix the picker reads from (ends with '/products/').""" + host = (os.environ.get("SAP_APP_API_BASE_URL") or "").strip().rstrip("/") + if host: + if not host.startswith("http"): + host = "https://" + host + route = "/" + (os.environ.get("SAP_APP_API_ROUTE") or "/occ/v2/").strip("/") + "/" + base_site = os.environ.get("SAP_APP_BASE_SITE") or site + return f"{host}{route}{base_site}/products/", True + occ = (os.environ.get("OCC_BASE_URL") or "").rstrip("/") + if not occ: + raise SystemExit("[sap_products] set SAP_APP_API_BASE_URL (or OCC_BASE_URL) to look up product names") + return f"{occ}/{site}/products/", False + + +class SapProductResolver: + def __init__(self, site=None, lang="en"): + self.base, self.verified_tls = occ_product_base(site) + self.lang = lang + self.config_name = os.environ.get("SAP_APP_CONFIG_NAME", "hti") + self.cache = {} + self.missing = [] + + def item(self, code): + if code not in self.cache: + url = f"{self.base}{urllib.parse.quote(code)}?fields=code,name&lang={self.lang}" + ctx = None if self.verified_tls else _occ_ctx() + try: + with urllib.request.urlopen(url, context=ctx, timeout=30) as r: + p = json.loads(r.read().decode()) + self.cache[code] = {"code": p["code"], "name": p.get("name") or p["code"], + "cs_metadata": {"multiConfigName": self.config_name, + "isConfigDeleted": False}} + except Exception as e: # unknown code on the app's server -> skip it + self.cache[code] = None + self.missing.append((code, str(e)[:80])) + return self.cache[code] + + def build(self, codes): + data = [i for i in (self.item(c) for c in codes) if i] + value = {"data": data, "type": FIELD_TYPE} + size = len(json.dumps(value, separators=(",", ":")).encode()) + if size > MAX_BYTES: + raise ValueError(f"sap_products value is {size} B (> 20 KB JSON field limit); " + f"save fewer keys in the app or split the carousel") + return value + + def resolve(self, v): + """Turn a model placeholder {"_sap_products": true, "codes": [...]} into the field value.""" + return self.build(v.get("codes") or []) + + +def is_placeholder(v): + return isinstance(v, dict) and v.get("_sap_products") is True + + +def placeholder(codes): + return {"_sap_products": True, "codes": list(codes)} + + +if __name__ == "__main__": + import sys + r = SapProductResolver(site=os.environ.get("SAP_APP_BASE_SITE")) + print(json.dumps(r.build(sys.argv[1:]), indent=1)) + if r.missing: + print("missing:", r.missing, file=sys.stderr) diff --git a/skills/dx-sap-spartacus-import/scripts/marketplace-app/use_products.py b/skills/dx-sap-spartacus-import/scripts/marketplace-app/use_products.py new file mode 100644 index 0000000..a0af496 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/marketplace-app/use_products.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +""" +use_products — fallback when the SAP Commerce Cloud product picker can't be made to work: +switch a seeded stack's product carousels back to the plain `products` SKU list, as they were +before the picker was introduced. + +Only run this after the user has chosen the fallback (see SKILL.md › Verify, retry, then ask). + +What it does, for product_carousel_component: + 1. re-imports the content type from the starter pack WITHOUT `sap_products` + (this removes the picker field and the picker data stored in it), + 2. writes each carousel's SKUs into `products`, in the same order, in every locale the + entry exists in, + 3. publishes each of those locale versions to the environment. + +The storefront renders the same products either way — the connector reads `sap_products` +first and falls back to `products` — editors just type/paste SKU codes instead of picking. +The SAP app itself stays installed; uninstall it from Marketplace if it isn't wanted. + +Usage: + python3 use_products.py --plan /connector-plan.json --env development +""" +import argparse, json, os, sys, urllib.parse +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "cs-seed")) +import cma_seed as low +import cma_seed_connector as seeder +from sap_products import is_placeholder + +CT = "product_carousel_component" +# Same default as orchestrate.py: the starter pack `ng add` installed in the current project. +DEFAULT_STARTER = os.environ.get("CONNECTOR_STARTER_PACK", os.path.join( + os.getcwd(), "node_modules", "@contentstack", "contentstack-spartacus-connector", + "import-export", "starter-pack", "content_types")) + + +def carousel_codes(plan): + out = {} + for e in plan["entries"]["components"]: + if e["_content_type"] != CT: + continue + v = e.get("sap_products") + out[e["_key"]["value"]] = list(v["codes"]) if is_placeholder(v) else list(e.get("products") or []) + return out + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--plan", required=True, help="connector-plan.json of the run (master locale)") + ap.add_argument("--env", default="development") + ap.add_argument("--starter-pack", default=DEFAULT_STARTER) + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN (or a csdx login) must be set") + + codes = carousel_codes(json.load(open(a.plan))) + if not codes: + low.die("the plan has no product carousels") + + print(f"[{CT}] re-import without sap_products") + starter = seeder.load_starter_types(a.starter_pack) + seeder.import_content_types([CT], {CT: starter[CT]}, sap_ext=None) + + st, r = low.cma("GET", "/locales") + locales = [l["code"] for l in (r.get("locales") or [])] or [low.MASTER] + for sap_uid, skus in codes.items(): + q = urllib.parse.quote(json.dumps({"sap_uid": sap_uid})) + st, r = low.cma("GET", f"/content_types/{CT}/entries?query={q}&locale={low.MASTER}&limit=1") + found = r.get("entries") or [] + if not found: + print(f" {sap_uid}: not found on the stack, skipped"); continue + uid = found[0]["uid"] + done = [] + for loc in locales: + st, r = low.cma("GET", f"/content_types/{CT}/entries/{uid}?locale={loc}") + e = r.get("entry") or {} + if e.get("locale") != loc: # not localized in this locale: inherits master + continue + st, r = low.cma("PUT", f"/content_types/{CT}/entries/{uid}?locale={loc}", + {"entry": {"products": skus}}) + if st != 200: + print(f" {sap_uid} [{loc}]: PUT {st} {json.dumps(r)[:120]}"); continue + ver = r["entry"]["_version"] + st, r = low.cma("POST", f"/content_types/{CT}/entries/{uid}/publish?locale={loc}", + {"entry": {"environments": [a.env], "locales": [loc]}, + "locale": loc, "version": ver}) + done.append(loc if st in (200, 201) else f"{loc}(publish {st})") + print(f" {sap_uid}: {len(skus)} SKUs in products -> {done}") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/orchestrate.py b/skills/dx-sap-spartacus-import/scripts/orchestrate.py new file mode 100644 index 0000000..8966dd6 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/orchestrate.py @@ -0,0 +1,371 @@ +#!/usr/bin/env python3 +""" +orchestrate — the content-modeling AGENT: one governed command that runs the whole +pipeline for a storefront, with two human gates. + + extract ──▶ nav-resolve ──▶ model ──▶ [GATE 1: model approval] + ──▶ [GATE 2: pre-publish] ──▶ SAP app ──▶ seed ──▶ publish ──▶ verify + +The SAP app stage runs only when carousels are seeded into `sap_products` (the default when +the SAP_APP_* configuration is exported; --carousel-field overrides): it installs + configures +the SAP Commerce Cloud marketplace app on the stack using the user's csdx login, so editors +pick carousel products from SAP instead of typing SKUs. Otherwise carousels keep the plain +`products` SKU list. + +It sequences the skills (each a standalone, tested script), enforces the gates, and +handles the one judgment call the skills don't: an UNKNOWN typeCode (Gate 1). Everything +is idempotent — re-running updates in place — and resumable via the run directory. + +GATES (fail-closed): the pipeline STOPS at each gate unless the matching approval flag is +passed, so nothing is created or published without an explicit human go. + Gate 1 model approval -> pass --approve-model (only blocks if unknown typeCodes found) + Gate 2 pre-publish -> pass --approve-publish (always blocks before any write) + +Default (no approval flags) = a full DRY RUN: extract→model + both gate reports, no writes. + +Usage (from the Spartacus project root, after `ng add @contentstack/contentstack-spartacus-connector`): + # dry run (safe): see the plan + gate reports, write nothing + python3 orchestrate.py --site electronics-spa + + # full governed run (requires CS_* env for seed/publish/verify) + python3 orchestrate.py --site electronics-spa --approve-model --approve-publish --env development + +Written by Vaishnavi Kedar (spartacus-content-modelling-agent); packaged as the +dx-sap-spartacus-import skill. +""" +import argparse, json, os, subprocess, sys, datetime, tempfile + +ROOT = os.path.dirname(os.path.abspath(__file__)) +SKILLS = ROOT +PROFILE = os.path.join(ROOT, "registry", "connector.registry.json") +# The connector's shipped starter-pack content types are the model of record. Default: the +# copy `ng add` installed in the current project; override with --starter-pack or +# CONNECTOR_STARTER_PACK (e.g. a connector checkout). +_INSTALLED = os.path.join(os.getcwd(), "node_modules", "@contentstack", "contentstack-spartacus-connector", + "import-export", "starter-pack", "content_types") +DEFAULT_STARTER = os.environ.get("CONNECTOR_STARTER_PACK", _INSTALLED) +DEFAULT_OCC = os.environ.get("OCC_BASE_URL") +# The SAP Commerce Cloud marketplace app configuration (the SAP server its product picker +# searches). With all three set, `--carousel-field auto` seeds carousels into the picker field +# `sap_products`; without them, carousels keep the plain `products` SKU list. +SAP_APP_ENV = ("SAP_APP_API_BASE_URL", "SAP_APP_BASE_SITE", "SAP_APP_BACKOFFICE_URL") + +C = {"h": "\033[1m", "ok": "\033[32m", "warn": "\033[33m", "err": "\033[31m", + "gate": "\033[35m", "z": "\033[0m"} + + +def banner(txt, color="h"): + line = "─" * (len(txt) + 2) + print(f"{C[color]}┌{line}┐\n│ {txt} │\n└{line}┘{C['z']}") + + +def step(txt): + print(f"{C['h']}▶ {txt}{C['z']}") + + +OCC_BASE = None # set in main(); exported to every stage so asset downloads hit the same OCC + + +def run(script, args, env=None): + """Run a skill script; stream nothing, return (rc, output).""" + env = dict(env or os.environ) + if OCC_BASE: + env["OCC_BASE_URL"] = OCC_BASE + p = subprocess.run([sys.executable, os.path.join(SKILLS, script), *args], + capture_output=True, text=True, env=env) + out = (p.stdout or "") + (p.stderr or "") + return p.returncode, out + + +def known_typecodes(profile): + comp = {k for k in profile["component_map"] if not k.startswith("_")} + occ = set(profile["occ_only_types"]["types"]) + return comp | occ + + +def load_pack(dir_): + """uid -> content type JSON for every type in the starter pack (schema.json skipped).""" + pack = {} + for f in sorted(os.listdir(dir_)): + if f.endswith(".json") and f != "schema.json": + ct = json.load(open(os.path.join(dir_, f))) + if isinstance(ct, dict) and ct.get("uid"): + pack[ct["uid"]] = ct + return pack + + +def pack_mismatches(plan, pack): + """Every content type and field the plan writes must exist in the starter pack, or the + seed fails half-way. sap_uid is allowed: the seeder adds it when the pack lacks it.""" + problems = [] + for ct in plan.get("content_types_needed", []): + if ct not in pack: + problems.append(f"content type '{ct}' is not in the starter pack") + entries = plan.get("entries", {}) + groups = [entries.get("components", []), entries.get("pages", []), entries.get("nav", [])] + if entries.get("global_slots"): + groups.append([entries["global_slots"]]) + for group in groups: + for e in group: + ct = e.get("_content_type") + if ct not in pack: + continue + fields = {f["uid"] for f in pack[ct].get("schema", [])} | {"sap_uid", "title"} + if ct == "product_carousel_component": + fields.add("sap_products") # the seeder adds the SAP app picker field per stack + for k in e: + if not k.startswith("_") and k not in fields: + problems.append(f"{ct}.{k} is not a field in the starter pack") + return sorted(set(problems)) + + +def main(): + global OCC_BASE + ap = argparse.ArgumentParser() + ap.add_argument("--site", required=True) + ap.add_argument("--inventory", default="", help="reuse a saved inventory.json instead of extracting") + ap.add_argument("--base", default=DEFAULT_OCC, help="OCC v2 base URL, e.g. https://HOST/occ/v2 (or set OCC_BASE_URL)") + ap.add_argument("--starter-pack", default=DEFAULT_STARTER, help="connector starter-pack content_types dir") + ap.add_argument("--env", default="development") + ap.add_argument("--approve-model", action="store_true", help="pass Gate 1") + ap.add_argument("--approve-publish", action="store_true", help="pass Gate 2 (enables writes)") + ap.add_argument("--locales", default="", help="optional i18n: sapIso=csLocale,... e.g. de=de-de,ja=ja-jp,zh=zh-cn") + ap.add_argument("--carousel-field", choices=["auto", "products", "sap_products"], default="auto", + help="where carousel SKUs go; auto = sap_products for electronics-spa, products otherwise") + ap.add_argument("--skip-app-install", action="store_true", + help="SAP app already installed + configured: only discover its field extension") + ap.add_argument("--reinstall-app", action="store_true", + help="uninstall + reinstall the SAP app and configure it once (repairs a broken install)") + ap.add_argument("--run-dir", default="") + a = ap.parse_args() + if not a.base and (not a.inventory or a.locales or a.approve_publish): + # --inventory alone can re-plan offline; asset uploads and localized extracts need OCC. + sys.exit("Set OCC_BASE_URL (or pass --base https://HOST/occ/v2).") + OCC_BASE = a.base + if not os.path.isdir(a.starter_pack): + sys.exit(f"Starter pack not found at {a.starter_pack}. Run from the Spartacus project root after " + "`ng add @contentstack/contentstack-spartacus-connector`, or set CONNECTOR_STARTER_PACK.") + locale_map = dict(p.split("=", 1) for p in a.locales.split(",") if "=" in p) if a.locales else {} + carousel_field = a.carousel_field + if carousel_field == "auto": + carousel_field = "sap_products" if all(os.environ.get(k) for k in SAP_APP_ENV) else "products" + + run_dir = a.run_dir or os.path.join(tempfile.gettempdir(), f"sap-spartacus-import-{a.site}") + os.makedirs(run_dir, exist_ok=True) + inv_path = os.path.join(run_dir, "inventory.json") + profile = json.load(open(PROFILE)) + manifest = {"site": a.site, "started": datetime.datetime.now().isoformat(timespec="seconds"), + "stages": {}} + + def record(stage, status, note=""): + manifest["stages"][stage] = {"status": status, "note": note} + json.dump(manifest, open(os.path.join(run_dir, "manifest.json"), "w"), indent=2) + + banner(f"CONTENT-MODELING AGENT · {a.site} · profile connector-reference-fields") + print(f" run dir: {run_dir} env: {a.env} starter pack: {a.starter_pack}") + print(f" carousel SKUs -> {carousel_field}") + print(f" occ: {a.base} region: {os.environ.get('CS_REGION_BASE', 'https://api.contentstack.io')}") + print(f" approvals: model={'YES' if a.approve_model else 'no'} " + f"publish={'YES' if a.approve_publish else 'no'}" + f"{' (DRY RUN — no writes)' if not a.approve_publish else ''}\n") + + # 1. EXTRACT (or load fixture) + step("1/8 extract") + if a.inventory: + json.dump(json.load(open(a.inventory)), open(inv_path, "w")) + print(f" reused inventory {a.inventory} -> {inv_path}") + else: + rc, out = run("storefront-extract/extract.py", + ["--site", a.site, "--base", a.base, "--out", inv_path, "--quiet"]) + print(" " + out.strip().splitlines()[-1] if out.strip() else "") + if rc: + record("extract", "error", out[-300:]); sys.exit("extract failed") + record("extract", "ok") + + # 2. NAV RESOLVE + step("2/8 nav-resolve") + if a.base: + rc, out = run("storefront-extract/nav_resolve.py", ["--inventory", inv_path, "--site", a.site, "--base", a.base]) + print(" " + (out.strip().splitlines()[-1] if out.strip() else "")) + record("nav-resolve", "ok" if not rc else "warn") + else: # reused inventory, no OCC: keep the navigation it already resolved + has_nav = "navigationLinks" in json.load(open(inv_path)) + print(f" skipped (no OCC base){'; the inventory already has resolved navigation' if has_nav else ''}") + record("nav-resolve", "skipped" if has_nav else "warn", "" if has_nav else "no OCC base, navigation unresolved") + + # 3. MODEL + step("3/8 model") + rc, out = run("content-model/model_connector.py", + ["--inventory", inv_path, "--profile", PROFILE, "--out", run_dir, + "--starter-pack", a.starter_pack, "--carousel-field", carousel_field]) + print(" " + "\n ".join(out.strip().splitlines())) + if rc: + record("model", "error", out[-300:]); sys.exit("model failed") + record("model", "ok") + plan = json.load(open(os.path.join(run_dir, "connector-plan.json"))) + + # 4. GATE 1 — model approval (unknown typeCodes) + inv = json.load(open(inv_path)) + seen = set(inv.get("componentTypeInventory", {})) + unknown = sorted(seen - known_typecodes(profile)) + print() + banner("GATE 1 · MODEL APPROVAL", "gate") + if unknown: + print(f" {C['warn']}UNKNOWN typeCodes (not in profile): {unknown}{C['z']}") + print(" These need a human decision: add to component_map (authored) or occ_only_types (OCC).") + if not a.approve_model: + record("gate1", "blocked", f"unknown: {unknown}") + print(f"\n {C['err']}STOP{C['z']} — resolve the mapping, then re-run with --approve-model.") + sys.exit(2) + print(f" {C['ok']}approved (--approve-model){C['z']} — proceeding.") + else: + print(f" {C['ok']}no unknown typeCodes — all {len(seen)} classified.{C['z']}") + record("gate1", "passed", f"unknown={unknown}") + + # 5. GATE 2 — pre-publish summary + n_comp = len(plan["entries"]["components"]) + n_pages = len(plan["entries"]["pages"]) + print() + banner("GATE 2 · PRE-PUBLISH", "gate") + print(f" will UPSERT to stack + publish to '{a.env}':") + print(f" content types : {len(plan['content_types_needed'])}") + print(f" assets : {len(plan['assets'])}") + print(f" component entries: {n_comp} pages: {n_pages} global_slots: " + f"{'1' if plan['entries'].get('global_slots') else '0'}") + print(f" pages: {[p['_key']['value'] for p in plan['entries']['pages']]}") + if locale_map: + print(f" + localize into: {list(locale_map.values())} " + f"(fully-localized OCC extract per language, incl. per-locale images)") + mismatches = pack_mismatches(plan, load_pack(a.starter_pack)) + if mismatches: + print(f" {C['err']}starter pack mismatch ({len(mismatches)}){C['z']} — the plan writes types/fields the pack lacks:") + for m in mismatches[:25]: + print(f" - {m}") + print(" Update the connector (npm i @contentstack/contentstack-spartacus-connector@latest) or the " + "mapping profile before writing.") + record("gate2", "blocked", f"pack mismatch: {len(mismatches)}") + sys.exit(3) + print(f" {C['ok']}starter pack check: every planned type and field exists{C['z']}") + print(f" carousel SKUs -> {carousel_field}") + auth = os.environ.get("CS_AUTH", "auto").lower() + uses_token = auth == "token" or (auth == "auto" and os.environ.get("CS_MANAGEMENT_TOKEN")) + print(f" writes as: {'management token' if uses_token else 'your csdx login (csdx auth:login --oauth)'}") + if carousel_field == "sap_products": + if a.skip_app_install: + print(" SAP Commerce Cloud app: already set up (--skip-app-install) — only find its Product field") + else: + print(f" SAP Commerce Cloud app: install if missing + config " + f"'{os.environ.get('SAP_APP_CONFIG_NAME', 'hti')}' -> " + f"{os.environ.get('SAP_APP_API_BASE_URL', '')}" + f"{os.environ.get('SAP_APP_API_ROUTE', '/occ/v2/')}" + f"{os.environ.get('SAP_APP_BASE_SITE', '')}" + f", keys code+name (uses your csdx login)") + missing = [k for k in SAP_APP_ENV if not os.environ.get(k)] + if missing: + print(f" {C['warn']}missing env for the SAP app: {missing}{C['z']}") + if not a.approve_publish: + record("gate2", "dry-run") + print(f"\n {C['warn']}DRY RUN complete{C['z']} — nothing written. " + f"Re-run with --approve-publish (and CS_* env) to seed + publish.") + sys.exit(0) + if not os.environ.get("CS_API_KEY"): + record("gate2", "error", "no creds") + sys.exit(f"{C['err']}--approve-publish set but CS_API_KEY missing.{C['z']}") + if os.environ.get("CS_AUTH", "auto").lower() == "token" and not os.environ.get("CS_MANAGEMENT_TOKEN"): + record("gate2", "error", "no creds") + sys.exit(f"{C['err']}CS_AUTH=token but CS_MANAGEMENT_TOKEN missing.{C['z']}") + print(f" {C['ok']}approved (--approve-publish){C['z']} — writing.") + record("gate2", "passed") + + plan_path = os.path.join(run_dir, "connector-plan.json") + # 6. SAP APP — install/configure the marketplace app + find its Product field extension + sap_app_path = os.path.join(run_dir, "sap-app.json") + seed_args = ["--plan", plan_path, "--starter-pack", a.starter_pack, "--stages", "types,assets,entries"] + step("4/8 SAP Commerce Cloud app") + if carousel_field == "sap_products": + missing = [k for k in SAP_APP_ENV if not os.environ.get(k)] + if missing and not a.skip_app_install: + record("sap-app", "error", f"missing env {missing}") + sys.exit(f"{C['err']}SAP app env missing: {missing} — export them, or pass " + f"--carousel-field products.{C['z']}") + rc, out = run("marketplace-app/sap_app.py", + ["--out", sap_app_path] + (["--skip-install"] if a.skip_app_install else []) + + (["--reinstall"] if a.reinstall_app else [])) + print(" " + "\n ".join(out.strip().splitlines()[-4:])) + record("sap-app", "ok" if not rc else "error", out[-300:]) + if rc: + sys.exit("SAP app stage failed") + seed_args += ["--sap-app", sap_app_path] + else: + print(" skipped (carousels use the plain products SKU list)") + record("sap-app", "skipped") + + # 7. SEED + step("5/8 seed (types + assets + entries)") + rc, out = run("cs-seed/cma_seed_connector.py", seed_args) + tail = "\n ".join(out.strip().splitlines()[-6:]) + print(" " + tail) + record("seed", "ok" if not rc else "error", out[-300:]) + if rc: + sys.exit("seed failed") + + # 7. PUBLISH + step("6/8 publish") + rc, out = run("cs-seed/publish.py", ["--env", a.env, "--locale", "en-us"]) + print(" " + "\n ".join(out.strip().splitlines()[-4:])) + record("publish", "ok" if not rc else "error") + + # 8. VERIFY + step("7/8 verify — round-trip") + rc, out = run("verify/verify_connector.py", ["--inventory", inv_path, "--plan", plan_path]) + print(" " + "\n ".join(out.strip().splitlines()[-3:])) + rt_ok = rc == 0 + record("verify-roundtrip", "pass" if rt_ok else "check") + + step("8/8 verify — delivery read (landing page)") + venv = os.environ.copy(); venv["CS_ENVIRONMENT"] = a.env + rc, out = run("verify/cda_read.py", ["--content-type", "landing_page", "--url", "/"], env=venv) + print(" " + "\n ".join(out.strip().splitlines()[-2:])) + record("verify-cda", "pass" if rc == 0 else "check") + + # 9. LOCALIZE (optional) — full localized OCC extract per language, then localize+publish. + for sap_iso, cs_loc in locale_map.items(): + print() + banner(f"LOCALIZE · {sap_iso} → {cs_loc}", "gate") + linv = os.path.join(run_dir, f"inventory-{sap_iso}.json") + lmodel = os.path.join(run_dir, f"model-{sap_iso}") + # localized inventory (always live — i18n needs the localized OCC responses) + rc, out = run("storefront-extract/extract.py", + ["--site", a.site, "--base", a.base, "--langs", sap_iso, "--primary-only", "--out", linv, "--quiet"]) + if rc: + record(f"localize:{cs_loc}", "error", "extract failed"); continue + run("storefront-extract/nav_resolve.py", ["--inventory", linv, "--site", a.site, "--base", a.base, "--lang", sap_iso]) + rc, out = run("content-model/model_connector.py", ["--inventory", linv, "--profile", PROFILE, "--out", lmodel, + "--starter-pack", a.starter_pack, + "--carousel-field", carousel_field]) + if rc: + record(f"localize:{cs_loc}", "error", "model failed"); continue + rc, out = run("cs-seed/localize.py", + ["--locale", cs_loc, "--plan", os.path.join(lmodel, "connector-plan.json"), "--env", a.env]) + print(" " + "\n ".join(out.strip().splitlines()[-4:])) + record(f"localize:{cs_loc}", "ok" if not rc else "error", out[-200:]) + + print() + loc_ok = [l for l in locale_map.values() if manifest["stages"].get(f"localize:{l}", {}).get("status") == "ok"] + loc_failed = [l for l in locale_map.values() if l not in loc_ok] + banner(f"DONE · {a.site} · round-trip {'PASS' if rt_ok else 'CHECK'}" + f"{' · +' + str(len(loc_ok)) + ' locales' if loc_ok else ''}" + f"{' · FAILED locales: ' + ', '.join(loc_failed) if loc_failed else ''}", + "ok" if rt_ok and not loc_failed else "warn") + print(f" manifest: {run_dir}/manifest.json") + if carousel_field == "sap_products": + print(f"\n {C['gate']}NEXT · verify the SAP product picker in the browser{C['z']} — " + f"references/sap-commerce-app.md › 'Verify, retry, then ask'.\n" + f" Repair (max 2): python3 {os.path.join(SKILLS, 'marketplace-app', 'sap_app.py')} " + f"--reinstall --out {sap_app_path}\n" + f" Fallback (user's choice): python3 {os.path.join(SKILLS, 'marketplace-app', 'use_products.py')} " + f"--plan {plan_path} --env {a.env} --starter-pack {a.starter_pack}") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/registry/connector.registry.json b/skills/dx-sap-spartacus-import/scripts/registry/connector.registry.json new file mode 100644 index 0000000..5da296e --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/registry/connector.registry.json @@ -0,0 +1,179 @@ +{ + "meta": { + "name": "sap-occ-to-connector-mapping-profile", + "version": "0.3.1", + "status": "draft", + "changelog": [ + "0.3.1 — un-deferred navigation (nav pass: nav_resolve.py + model_connector.build_nav seed nav_node tree + link leaves + category/footer_navigation_component into global_slots). Added product_page/category_page authoring from layoutSamples (shared slugs /product, /category). Refactor also stopped over-seeding components from non-authored pages.", + "0.3.0 — initial connector reference-field mapping profile." + ], + "profile": "connector-reference-fields", + "generated_from": [ + "electronics-spa", + "powertools-spa" + ], + "connector": "@contentstack/contentstack-spartacus-connector", + "content_types_source": "node_modules/@contentstack/contentstack-spartacus-connector/import-export/starter-pack/content_types/", + "philosophy": "Hybrid islands model (verified in connector code + CONTENT-MODEL.md). OCC renders the base of every page; Contentstack overrides ONLY authored slots. Functional components, unauthored slots, and non-content pages stay in OCC and are NOT seeded. Unit of control = the slot.", + "important": "This profile MAPS OCC inventory onto the connector's SHIPPED content types. It does NOT define content-type schemas — those are imported from the starter pack (via csdx or POSTing the JSON files). Idempotency: pages upsert by `url`; component entries upsert by `title` (set to the SAP component uid, which is unique).", + "differs_from_v1": "registry.json (v0.2.1) was a free-form model with a cms_content_slot entry type, a unified `banner`, and a passthrough catch-all. This profile instead conforms to the connector: named per-slot reference fields on per-template pages, one content type per SAP typecode, functional types left to OCC." + }, + "page_types": { + "_note": "OCC page template -> connector page content type. Only these templates are authored in Contentstack; every other template (Login/Account/Cart/Checkout/SearchResults/Error/StoreFinder/...) is left to OCC and produces NO Contentstack page.", + "_slug_rule": "url = '/' for the homepage label; otherwise '/' + page label. page_type <- page.typeCode; template <- page.template.", + "LandingPage2Template": { + "content_type": "landing_page", + "slug_field": "url" + }, + "ContentPage1Template": { + "content_type": "landing_page", + "slug_field": "url" + }, + "ProductDetailsPageTemplate": { + "content_type": "product_page", + "slug_field": "url" + }, + "ProductListPageTemplate": { + "content_type": "category_page", + "slug_field": "url" + }, + "_note_content_pages": "Content pages (ContentPage1Template) land on landing_page: the connector resolves every per-route page against cmsPageContentType (landing_page), and the starter pack carries body_content/side_content there (connector PR #35, DX-31800)." + }, + "slot_field_map": { + "_note": "SAP slot POSITION (from inventory slot.position) -> connector page reference-field uid. Reverse of the connector's SLOT_FIELD_TO_SAP_NAME. Positions not listed here are left to OCC. Shell positions (see global_slot_positions) are authored on the single global_slots entry, not on the page.", + "Section1": "section1", + "Section2": "section2", + "Section2A": "section2_a", + "Section2B": "section2_b", + "Section2C": "section2_c", + "Section3": "section3", + "Section4": "section4", + "Section5": "section5", + "BodyContent": "body_content", + "SideContent": "side_content", + "BottomHeaderSlot": "bottom_header_slot", + "ProductLeftRefinements": "product_left_refinements", + "ProductListSlot": "product_list_slot", + "ProductGridSlot": "product_grid_slot", + "SearchResultsListSlot": "search_results_list_slot", + "SearchResultsGridSlot": "search_results_grid_slot", + "Summary": "summary", + "UpSelling": "up_selling", + "CrossSelling": "cross_selling", + "Tabs": "tabs", + "PlaceholderContentSlot": "placeholder_content_slot", + "TopContent": "top_content", + "MiddleContent": "middle_content", + "BottomContent": "bottom_content", + "CenterRightContentSlot": "center_right_content_slot", + "CenterRightContent": "center_right_content", + "EmptyCartMiddleContent": "empty_cart_middle_content", + "LeftContentSlot": "left_content_slot", + "RightContentSlot": "right_content_slot", + "HomepageNavLink": "homepage_nav_link" + }, + "global_slots": { + "_note": "Author ONLY content-managed shell slots. SiteContext (theme+language switcher) and SiteLinks (login/register + utility links) are FUNCTIONAL — authoring them partially overrode OCC's full shell and dropped the language dropdown + sign-in/register. Leave those to OCC (occFallback). Author logo (branding), category nav, and footer nav only.", + "content_type": "global_slots", + "upsert_by": "title", + "positions": { + "SiteLogo": "site_logo", + "NavigationBar": "navigation_bar", + "Footer": "footer" + }, + "positions_left_to_occ": { + "SiteContext": "site_context (theme+language -> OCC)", + "SiteLinks": "site_links (login/register+links -> OCC)", + "SearchBox": "search_box (functional -> OCC)", + "MiniCart": "mini_cart (functional -> OCC)", + "HeaderLinks": "header_links (functional -> OCC)" + } + }, + "component_map": { + "_note": "OCC typeCode -> connector component content type + field mapping (connector field uid <- OCC source). Only authored/content-bearing types appear here. title <- component.name (entry title is set to component.uid for idempotency; name is kept where a display field exists).", + "SimpleResponsiveBannerComponent": { + "content_type": "simple_responsive_banner_component", + "assets": true, + "fields": { + "url_link": "urlLink", + "media_mobile": "media.mobile", + "media_tablet": "media.tablet", + "media_desktop": "media.desktop", + "media_widescreen": "media.widescreen", + "media": "media.desktop" + } + }, + "SimpleBannerComponent": { + "content_type": "simple_banner_component", + "assets": true, + "fields": { + "url_link": "urlLink", + "media": "media" + } + }, + "ProductCarouselComponent": { + "content_type": "product_carousel_component", + "fields": { + "title": "title", + "products": "productCodes[space-split]" + }, + "_sap_products": "with --carousel-field sap_products (the default when the SAP_APP_* configuration is exported) the SKUs go to `sap_products` (SAP Commerce Cloud app picker JSON) instead of `products`" + }, + "CMSParagraphComponent": { + "content_type": "cms_paragraph_component", + "fields": { + "content": "content" + } + }, + "CMSLinkComponent": { + "content_type": "cms_link_component", + "fields": { + "link_name": "linkName", + "url": "url", + "target": "target" + } + }, + "CMSFlexComponent": { + "content_type": "cms_flex_component", + "fields": { + "flex_type": "flexType" + } + }, + "CategoryNavigationComponent": { + "content_type": "category_navigation_component", + "status": "nav-pass", + "_note": "NOT a regular slot component. Handled by the nav pass: nav_resolve.py resolves link leaves from OCC, model_connector.build_nav emits the nav_node tree + cms_link_component leaves + this component, placed into global_slots.navigation_bar. map_component skips it (status != mapped)." + }, + "FooterNavigationComponent": { + "content_type": "footer_navigation_component", + "status": "nav-pass", + "_note": "see CategoryNavigationComponent; placed into global_slots.footer." + } + }, + "navigation": { + "_note": "Navigation is seeded flat: nav_node_flat entries (node_id/parent_id/sort_order) plus a category_navigation_flat or footer_navigation_flat component that references the pool via all_nodes; the connector rebuilds the tree. See model_connector.build_nav.", + "nav_node": "nav_node_flat", + "category": "category_navigation_flat", + "footer": "footer_navigation_flat", + "link_leaf": "cms_link_component" + }, + "occ_only_types": { + "_note": "Functional / commerce-driven components with no editorial content. Per the hybrid model these hydrate from OCC and are NOT seeded into Contentstack. Slots containing only these are left empty in CS (OCC fills them).", + "types": [ + "MiniCartComponent", + "SearchBoxComponent", + "BreadcrumbComponent", + "CMSSiteContextComponent", + "ProductRefinementComponent", + "SearchResultsListComponent", + "CartSuggestionComponent", + "JspIncludeComponent", + "MerchandisingCarouselComponent", + "BundleCarouselComponent", + "CMSProductListComponent", + "CMSTabParagraphContainer", + "ProductAddToCartComponent", + "ProductReferencesComponent" + ] + } +} diff --git a/skills/dx-sap-spartacus-import/scripts/storefront-extract/extract.py b/skills/dx-sap-spartacus-import/scripts/storefront-extract/extract.py new file mode 100644 index 0000000..d28a51f --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/storefront-extract/extract.py @@ -0,0 +1,345 @@ +#!/usr/bin/env python3 +""" +storefront-extract — Phase 2 of the SAP Spartacus -> Contentstack content-modeling agent. + +Crawls one SAP Commerce baseSite over the OCC API and emits a normalized "content +inventory" JSON that conforms to inventory.schema.json. That inventory is the single +contract every downstream skill (content-model, cs-seed, verify) consumes. + +Design facts this encodes (all validated live against the SAP backend): + * OCC /cms/pages is PAGE-BY-LABEL, not a bulk list -> content pages are found by a + guided BFS from known labels + internal urlLinks. + * fields=FULL inlines each component's fields, media (responsive variants) AND the + full navigationNode tree. + * Product/Category page LAYOUTS are fetched with pageType + code= (not pageLabelOrId). + * Media bytes download from host + media.url (the ?context= token is required). + * Content is per content-catalog + version; we read the published Online version. + +Deterministic: no wall-clock timestamps in the output, stable key order -> fixture-diffable. + +Usage: + python3 extract.py --site electronics-spa [--base URL] [--langs en,de] \ + [--out inventory.json] [--download-media DIR] [--quiet] +""" +import argparse, json, os, re, ssl, sys, urllib.parse, urllib.request +from collections import OrderedDict, deque + +DEFAULT_BASE = os.environ.get("OCC_BASE_URL") + + +def occ_ssl_context(): + """TLS context for OCC calls. Verified by default; set OCC_INSECURE_TLS=1 only for a + dev backend with a self-signed certificate you trust.""" + ctx = ssl.create_default_context() + if os.environ.get("OCC_INSECURE_TLS") == "1": + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + +# Standard Spartacus content-page labels used to seed BFS discovery. +SEED_LABELS = [ + "homepage", "login", "register", "cart", "checkout", "orderConfirmation", + "search", "faq", "contact", "terms", "sale", "store-finder", "storefinder", + "wishlist", "notFound", "myaccount", "address-book", "update-profile", + "order-history", "my-coupons", "consents", "close-account", "payment-details", +] +# Component field names that carry localizable text (captured per language for i18n). +I18N_FIELDS = ("content", "title", "linkName", "name", "urlLink", "notice", "altText") +LABEL_RE = re.compile(r"^/([A-Za-z0-9\-_]+)/?$") + + +class Occ: + """Thin OCC client (TLS per occ_ssl_context).""" + def __init__(self, base, site): + self.base, self.site = base.rstrip("/"), site + self.host = base.split("/occ/")[0] + self.ctx = occ_ssl_context() + self.calls = 0 + + def get(self, path, **params): + self.calls += 1 + url = f"{self.base}/{self.site}/{path}?" + urllib.parse.urlencode(params) + req = urllib.request.Request(url, headers={"Accept": "application/json"}) + try: + with urllib.request.urlopen(req, context=self.ctx, timeout=40) as r: + return json.loads(r.read().decode()) + except Exception as e: + return {"__error__": str(e)} + + def download(self, media_url): + self.calls += 1 + url = media_url if media_url.startswith("http") else self.host + media_url + req = urllib.request.Request(url, headers={"Accept": "*/*"}) + with urllib.request.urlopen(req, context=self.ctx, timeout=60) as r: + return r.read() + + def page(self, page_type, key_field, key, lang): + return self.get("cms/pages", pageType=page_type, fields="FULL", lang=lang, + **{key_field: key}) + + +def sorted_json(o): + """Recursively order dict keys for deterministic, diff-stable output.""" + if isinstance(o, dict): + return OrderedDict((k, sorted_json(o[k])) for k in sorted(o)) + if isinstance(o, list): + return [sorted_json(v) for v in o] + return o + + +class Extractor: + def __init__(self, occ, langs, quiet=False): + self.occ = occ + self.langs = langs + self.primary = langs[0] + self.quiet = quiet + self.components_by_type = {} # typeCode -> [normalized component] + self.media = OrderedDict() # media code -> {url, mime, altText} + self.slots = OrderedDict() # slotId -> meta + self.nav_nodes = OrderedDict() # nodeUid -> raw navigationNode tree + self.warnings = [] + + def log(self, *a): + if not self.quiet: + print(*a, file=sys.stderr) + + # ---- media & link scanning ------------------------------------------------- + def record_media(self, m): + if not isinstance(m, dict): + return + if "code" in m and "url" in m: + self.media.setdefault(m["code"], { + "url": m.get("url"), "mime": m.get("mime"), "altText": m.get("altText")}) + else: + for v in m.values(): + if isinstance(v, dict): + self.record_media(v) + + def scan_labels(self, obj, out): + if isinstance(obj, dict): + for k, v in obj.items(): + if k in ("url", "urlLink", "linkTo") and isinstance(v, str): + mm = LABEL_RE.match(v.strip()) + if mm: + out.add(mm.group(1)) + self.scan_labels(v, out) + elif isinstance(obj, list): + for v in obj: + self.scan_labels(v, out) + + # ---- component normalization ---------------------------------------------- + DROP = ("uuid", "modifiedtime", "synchronizationBlocked") + + def normalize_component(self, c): + tc = c.get("typeCode", "") + if "media" in c: + self.record_media(c["media"]) + if "navigationNode" in c and isinstance(c["navigationNode"], dict): + nn = c["navigationNode"] + if nn.get("uid"): + self.nav_nodes.setdefault(nn["uid"], self.clean(nn)) + keep = self.clean(c) + self.components_by_type.setdefault(tc, []).append(keep) + return keep + + def clean(self, o): + if isinstance(o, dict): + return {k: self.clean(v) for k, v in o.items() if k not in self.DROP} + if isinstance(o, list): + return [self.clean(v) for v in o] + return o + + def normalize_page(self, label, raw): + slots = [] + for s in (raw.get("contentSlots") or {}).get("contentSlot", []): + sid = s.get("slotId") + self.slots.setdefault(sid, { + "position": s.get("position"), "shared": s.get("slotShared"), + "name": s.get("name"), "firstSeenOn": label}) + comps = (s.get("components") or {}).get("component", []) + slots.append({ + "slotId": sid, "position": s.get("position"), + "shared": s.get("slotShared"), + "components": [self.normalize_component(c) for c in comps]}) + return { + "label": raw.get("label") or label, "uid": raw.get("uid"), + "title": raw.get("title"), "template": raw.get("template"), + "typeCode": raw.get("typeCode"), "robotTag": raw.get("robotTag"), + "slots": slots} + + # ---- discovery ------------------------------------------------------------- + def discover_content_pages(self): + pages, visited = OrderedDict(), set() + frontier = deque(dict.fromkeys(SEED_LABELS)) + while frontier: + label = frontier.popleft() + if label in visited: + continue + visited.add(label) + raw = self.occ.page("ContentPage", "pageLabelOrId", label, self.primary) + if raw.get("__error__") or not raw.get("uid"): + continue + pages[label] = self.normalize_page(label, raw) + found = set() + self.scan_labels(raw, found) + for f in found: + if f not in visited: + frontier.append(f) + self.log(f" content pages: {len(pages)}") + return pages + + def sample_layout_pages(self): + out = OrderedDict() + # product code from search + s = self.occ.get("products/search", query="", pageSize="1", + fields="DEFAULT", lang=self.primary, curr="USD") + if not s.get("products"): + s = self.occ.get("products/search", query="a", pageSize="1", + fields="DEFAULT", lang=self.primary, curr="USD") + code = (s.get("products") or [{}])[0].get("code") + if code: + pp = self.occ.page("ProductPage", "code", code, self.primary) + if pp.get("uid"): + out["ProductPage"] = self.normalize_page(f"product:{code}", pp) + else: + self.warnings.append("no product code resolved for ProductPage layout") + # category code: try a handful + for cat in ("575", "576", "1", "brands", "1358"): + cp = self.occ.page("CategoryPage", "code", cat, self.primary) + if cp.get("uid"): + out[f"CategoryPage:{cat}"] = self.normalize_page(f"category:{cat}", cp) + break + else: + self.warnings.append("no category code resolved for CategoryPage layout") + self.log(f" layout samples: {list(out)}") + return out + + # ---- i18n ------------------------------------------------------------------ + def capture_i18n(self, page_labels): + i18n = OrderedDict() + for lang in self.langs[1:]: + per = OrderedDict() + for label in page_labels: + raw = self.occ.page("ContentPage", "pageLabelOrId", label, lang) + if raw.get("__error__"): + continue + for s in (raw.get("contentSlots") or {}).get("contentSlot", []): + for c in (s.get("components") or {}).get("component", []): + uid = c.get("uid") + if not uid: + continue + vals = {f: c[f] for f in I18N_FIELDS + if isinstance(c.get(f), str) and c[f]} + if vals: + per.setdefault(uid, {}).update(vals) + i18n[lang] = per + self.log(f" i18n[{lang}]: {len(per)} components") + return i18n + + # ---- media download (optional) -------------------------------------------- + def download_media(self, out_dir): + os.makedirs(out_dir, exist_ok=True) + ok = 0 + for code, m in self.media.items(): + if not m.get("url"): + continue + try: + data = self.occ.download(m["url"]) + fname = re.sub(r"[^A-Za-z0-9._-]", "_", code) + with open(os.path.join(out_dir, fname), "wb") as f: + f.write(data) + m["bytes"] = len(data) + ok += 1 + except Exception as e: + self.warnings.append(f"media download failed {code}: {e}") + self.log(f" media downloaded: {ok}/{len(self.media)}") + return ok + + # ---- assemble -------------------------------------------------------------- + def run(self, download_dir=None): + self.log(f"[extract] {self.occ.site} langs={self.langs}") + pages = self.discover_content_pages() + layouts = self.sample_layout_pages() + i18n = self.capture_i18n(list(pages)) + if download_dir: + self.download_media(download_dir) + + type_inventory = { + tc: {"instances": len(v), + "fields": sorted({k for c in v for k in c.keys()})} + for tc, v in sorted(self.components_by_type.items(), + key=lambda kv: -len(kv[1]))} + inv = { + "schemaVersion": "1.0", + "site": self.occ.site, + "source": self.occ.base, + "catalogVersion": "Online", + "primaryLanguage": self.primary, + "languages": self.langs, + "counts": { + "contentPages": len(pages), + "layoutSamples": len(layouts), + "uniqueSlots": len(self.slots), + "sharedSlots": sum(1 for s in self.slots.values() + if s["shared"] in (True, "true")), + "componentTypes": len(self.components_by_type), + "componentInstances": sum(len(v) for v in self.components_by_type.values()), + "navigationNodes": len(self.nav_nodes), + "mediaAssets": len(self.media), + }, + "componentTypeInventory": type_inventory, + "slots": self.slots, + "contentPages": pages, + "layoutSamples": layouts, + "navigation": self.nav_nodes, + "i18n": i18n, + "mediaManifest": self.media, + "warnings": self.warnings, + } + return inv + + +def resolve_languages(occ, arg): + if arg: + return [x.strip() for x in arg.split(",") if x.strip()] + d = occ.get("languages") + langs = [l["isocode"] for l in d.get("languages", []) if l.get("isocode")] + # put 'en' first if present for a stable primary + if "en" in langs: + langs = ["en"] + [l for l in langs if l != "en"] + return langs or ["en"] + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--site", required=True) + ap.add_argument("--base", default=DEFAULT_BASE, required=DEFAULT_BASE is None, + help="OCC v2 base URL, e.g. https://HOST/occ/v2 (or set OCC_BASE_URL)") + ap.add_argument("--langs", default="", help="comma list; default = site languages") + ap.add_argument("--primary-only", action="store_true", + help="crawl only the primary language (skip i18n)") + ap.add_argument("--out", default="inventory.json") + ap.add_argument("--download-media", default="", help="dir to save media bytes") + ap.add_argument("--quiet", action="store_true") + a = ap.parse_args() + + occ = Occ(a.base, a.site) + langs = resolve_languages(occ, a.langs) + if a.primary_only: + langs = langs[:1] + inv = Extractor(occ, langs, quiet=a.quiet).run(download_dir=a.download_media or None) + + with open(a.out, "w") as f: + json.dump(sorted_json(inv), f, indent=2) + + c = inv["counts"] + print(f"[extract] {a.site}: pages={c['contentPages']} layouts={c['layoutSamples']} " + f"types={c['componentTypes']} instances={c['componentInstances']} " + f"nav={c['navigationNodes']} media={c['mediaAssets']} " + f"| {occ.calls} OCC calls -> {a.out}") + if inv["warnings"]: + print(" warnings: " + "; ".join(inv["warnings"])) + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/storefront-extract/inventory.schema.json b/skills/dx-sap-spartacus-import/scripts/storefront-extract/inventory.schema.json new file mode 100644 index 0000000..8382d40 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/storefront-extract/inventory.schema.json @@ -0,0 +1,149 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://contentstack/sap-spartacus/inventory.schema.json", + "title": "Storefront Content Inventory", + "description": "Normalized output of storefront-extract. The single contract consumed by content-model, cs-seed, and verify. One file per baseSite.", + "type": "object", + "required": [ + "schemaVersion", "site", "primaryLanguage", "languages", "counts", + "componentTypeInventory", "slots", "contentPages", "navigation", + "i18n", "mediaManifest" + ], + "additionalProperties": true, + "properties": { + "schemaVersion": { "type": "string", "const": "1.0" }, + "site": { "type": "string", "description": "baseSite uid, e.g. electronics-spa" }, + "source": { "type": "string", "description": "OCC base url the inventory was read from" }, + "catalogVersion": { "type": "string", "description": "Online (published) unless overridden" }, + "primaryLanguage": { "type": "string" }, + "languages": { "type": "array", "items": { "type": "string" }, "minItems": 1 }, + + "counts": { + "type": "object", + "required": ["contentPages", "componentTypes", "componentInstances"], + "properties": { + "contentPages": { "type": "integer" }, + "layoutSamples": { "type": "integer" }, + "uniqueSlots": { "type": "integer" }, + "sharedSlots": { "type": "integer" }, + "componentTypes": { "type": "integer" }, + "componentInstances": { "type": "integer" }, + "navigationNodes": { "type": "integer" }, + "mediaAssets": { "type": "integer" } + } + }, + + "componentTypeInventory": { + "type": "object", + "description": "typeCode -> {instances, fields[]}. Used to diff against the registry to find unknown types (Gate 1).", + "additionalProperties": { + "type": "object", + "required": ["instances", "fields"], + "properties": { + "instances": { "type": "integer" }, + "fields": { "type": "array", "items": { "type": "string" } } + } + } + }, + + "slots": { + "type": "object", + "description": "slotId -> slot metadata. shared=true marks a global slot reused across pages.", + "additionalProperties": { + "type": "object", + "properties": { + "position": { "type": ["string", "null"] }, + "shared": { "type": ["boolean", "string", "null"] }, + "name": { "type": ["string", "null"] }, + "firstSeenOn": { "type": "string" } + } + } + }, + + "contentPages": { + "type": "object", + "description": "page label -> normalized page.", + "additionalProperties": { "$ref": "#/definitions/page" } + }, + + "layoutSamples": { + "type": "object", + "description": "One representative Product/Category page each (layout structure only; records stay in commerce).", + "additionalProperties": { "$ref": "#/definitions/page" } + }, + + "navigation": { + "type": "object", + "description": "navigationNode uid -> raw resolved node tree (children + entry references). Captured now; activated in registry v0.2.", + "additionalProperties": { "type": "object" } + }, + + "i18n": { + "type": "object", + "description": "language -> { componentUid -> { field -> localized string } } for non-primary languages.", + "additionalProperties": { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": { "type": "string" } + } + } + }, + + "mediaManifest": { + "type": "object", + "description": "media code -> {url, mime, altText, bytes?}. url needs the host prefix and carries the ?context= download token.", + "additionalProperties": { + "type": "object", + "required": ["url"], + "properties": { + "url": { "type": ["string", "null"] }, + "mime": { "type": ["string", "null"] }, + "altText": { "type": ["string", "null"] }, + "bytes": { "type": "integer" } + } + } + }, + + "warnings": { "type": "array", "items": { "type": "string" } } + }, + + "definitions": { + "page": { + "type": "object", + "required": ["label", "template", "slots"], + "properties": { + "label": { "type": ["string", "null"] }, + "uid": { "type": ["string", "null"] }, + "title": { "type": ["string", "null"] }, + "template": { "type": ["string", "null"] }, + "typeCode": { "type": ["string", "null"] }, + "robotTag": { "type": ["string", "null"] }, + "slots": { + "type": "array", + "items": { + "type": "object", + "required": ["slotId", "components"], + "properties": { + "slotId": { "type": ["string", "null"] }, + "position": { "type": ["string", "null"] }, + "shared": { "type": ["boolean", "string", "null"] }, + "components": { + "type": "array", + "items": { + "type": "object", + "required": ["typeCode"], + "properties": { + "uid": { "type": ["string", "null"] }, + "typeCode": { "type": "string" } + }, + "additionalProperties": true + } + } + } + } + } + } + } + } +} diff --git a/skills/dx-sap-spartacus-import/scripts/storefront-extract/nav_resolve.py b/skills/dx-sap-spartacus-import/scripts/storefront-extract/nav_resolve.py new file mode 100644 index 0000000..593062d --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/storefront-extract/nav_resolve.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +""" +nav_resolve — the deferred navigation resolve pass (enriches an inventory in place). + +storefront-extract captures the navigationNode trees, but each link leaf is only a +reference ({itemId, itemType: CMSLinkComponent}). This pass batch-resolves those leaves +from OCC (/cms/components) into full link detail, so the connector model can seed a +nav_node tree with real cms_link_component leaves. + +Adds two keys to the inventory: + navigationLinks : { itemId -> {link_name, url, target, external, name} } + navigationRoots : { componentUid -> rootNodeUid } (from CategoryNavigation/FooterNavigation) + +Usage: python3 nav_resolve.py --inventory --site electronics-spa [--base URL] +""" +import argparse, json, os, ssl, sys, urllib.parse, urllib.request + +DEFAULT_BASE = os.environ.get("OCC_BASE_URL") + + +def occ_ssl_context(): + """TLS context for OCC calls. Verified by default; set OCC_INSECURE_TLS=1 only for a + dev backend with a self-signed certificate you trust.""" + ctx = ssl.create_default_context() + if os.environ.get("OCC_INSECURE_TLS") == "1": + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + + +CTX = occ_ssl_context() + + +def _get(url): + req = urllib.request.Request(url, headers={"Accept": "application/json"}) + with urllib.request.urlopen(req, context=CTX, timeout=40) as r: + return json.loads(r.read().decode()) + + +def occ_components(base, site, ids, lang="en"): + out = {} + for i in range(0, len(ids), 10): + chunk = ids[i:i + 10] + url = f"{base}/{site}/cms/components?" + urllib.parse.urlencode( + {"componentIds": ",".join(chunk), "lang": lang}) + try: + d = _get(url) + for c in (d.get("component") or d.get("components") or []): + out[c.get("uid")] = c + except Exception as e: + print(f" batch chunk failed: {e}", file=sys.stderr) + # single-GET fallback for anything the batch didn't return + missing = [i for i in ids if i not in out] + for iid in missing: + try: + c = _get(f"{base}/{site}/cms/components/{iid}?lang={lang}") + if c.get("uid"): + out[c["uid"]] = c + except Exception: + pass + return out + + +def collect_link_ids(node, ids): + for e in (node.get("entries") or []): + if isinstance(e, dict) and e.get("itemType") == "CMSLinkComponent" and e.get("itemId"): + ids.add(e["itemId"]) + for ch in (node.get("children") or []): + collect_link_ids(ch, ids) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--inventory", required=True) + ap.add_argument("--site", required=True) + ap.add_argument("--base", default=DEFAULT_BASE, required=DEFAULT_BASE is None, + help="OCC v2 base URL (or set OCC_BASE_URL)") + ap.add_argument("--lang", default="en", help="OCC language for localized nav labels") + a = ap.parse_args() + inv = json.load(open(a.inventory)) + + nav = inv.get("navigation", {}) + ids = set() + for node in nav.values(): + collect_link_ids(node, ids) + print(f"[nav_resolve] {a.site} (lang={a.lang}): {len(nav)} root nodes, {len(ids)} link leaves to resolve") + + resolved = occ_components(a.base, a.site, sorted(ids), lang=a.lang) + links = {} + for iid, c in resolved.items(): + links[iid] = {"link_name": c.get("linkName"), "url": c.get("url"), + "target": c.get("target"), "external": c.get("external"), + "name": c.get("name")} + inv["navigationLinks"] = links + + # component uid -> root node uid, from the nav components in the inventory + roots = {} + for p in inv.get("contentPages", {}).values(): + for s in p.get("slots", []): + for comp in s.get("components", []): + if comp.get("typeCode") in ("CategoryNavigationComponent", "FooterNavigationComponent"): + nn = comp.get("navigationNode") or {} + if nn.get("uid"): + roots[comp["uid"]] = {"root": nn["uid"], "typeCode": comp["typeCode"], + "position": s.get("position"), + "wrapAfter": comp.get("wrapAfter")} + inv["navigationRoots"] = roots + + json.dump(inv, open(a.inventory, "w"), indent=2) + print(f" resolved {len(links)}/{len(ids)} links; {len(roots)} nav components " + f"-> {sorted((r['typeCode'], r['position']) for r in roots.values())}") + print(f" enriched -> {a.inventory}") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/verify/cda_read.py b/skills/dx-sap-spartacus-import/scripts/verify/cda_read.py new file mode 100644 index 0000000..29520d8 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/verify/cda_read.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +""" +cda_read — simulate the connector's delivery-time fetch (Phase 4b render pre-check). + +Uses the DELIVERY token against the Content Delivery API to fetch a page by its `url` slug +with slot references resolved (`include[]=`) — exactly what the connector's +ContentstackClientService.getPageBySlug does. Confirms the published content is readable by +a storefront and that slot references hydrate to real component entries. + +Read-only. Env: CS_API_KEY, CS_DELIVERY_TOKEN, CS_CDA_BASE (default https://cdn.contentstack.io), +CS_ENVIRONMENT (default development). + +Usage: python3 cda_read.py --content-type landing_page --url / [--preview] +""" +import argparse, json, os, sys, urllib.parse, urllib.request + +CDA = os.environ.get("CS_CDA_BASE", "https://cdn.contentstack.io").rstrip("/") +KEY = os.environ.get("CS_API_KEY") +DELIVERY = os.environ.get("CS_DELIVERY_TOKEN") +ENV = os.environ.get("CS_ENVIRONMENT", "development") + +SLOT_FIELDS = ["section1", "section2", "section2_a", "section2_b", "section2_c", + "section3", "section4", "section5", "body_content", "side_content", + "summary", "up_selling", "cross_selling", "tabs", "placeholder_content_slot", + "product_left_refinements", "product_list_slot", "product_grid_slot", + "search_results_list_slot", "search_results_grid_slot", + "top_content", "bottom_content", "header", "footer"] + + +def cda(path): + req = urllib.request.Request(f"{CDA}/v3{path}") + req.add_header("api_key", KEY) + req.add_header("access_token", DELIVERY) + with urllib.request.urlopen(req, timeout=40) as r: + return json.loads(r.read().decode()) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--content-type", default="landing_page") + ap.add_argument("--url", default="/") + a = ap.parse_args() + if not (KEY and DELIVERY): + sys.exit("set CS_API_KEY and CS_DELIVERY_TOKEN") + + params = [("environment", ENV), ("query", json.dumps({"url": a.url}))] + for f in SLOT_FIELDS: + params.append(("include[]", f)) + qs = urllib.parse.urlencode(params) + r = cda(f"/content_types/{a.content_type}/entries?{qs}") + entries = r.get("entries") or [] + print(f"[cda_read] {a.content_type} url={a.url} @ {CDA} env={ENV}") + if not entries: + print(" no entry found (published? correct env/token?)"); sys.exit(1) + e = entries[0] + print(f" title: {e.get('title')} template: {e.get('template')} page_type: {e.get('page_type')}") + total_refs = 0 + for f in SLOT_FIELDS: + v = e.get(f) + if isinstance(v, list) and v: + resolved = [x for x in v if isinstance(x, dict) and x.get("uid")] + total_refs += len(resolved) + types = sorted({x.get("_content_type_uid") or "?" for x in resolved}) + sample = resolved[0].get("title") if resolved else "" + print(f" {f:14s}: {len(resolved)} component(s) {types} e.g. \"{sample}\"") + print(f" RESULT: {'PASS — page reads with ' + str(total_refs) + ' resolved components' if total_refs else 'entry found but no references resolved (publish components too?)'}") + sys.exit(0 if total_refs else 1) + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/verify/verify_connector.py b/skills/dx-sap-spartacus-import/scripts/verify/verify_connector.py new file mode 100644 index 0000000..5206c98 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/verify/verify_connector.py @@ -0,0 +1,288 @@ +#!/usr/bin/env python3 +""" +verify_connector — round-trip verification for the CONNECTOR profile (Phase 4b). + +Reads the seeded connector-model entries back from the stack, reverse-maps each authored +component to its original OCC shape, and diffs against the original inventory. Only the +AUTHORED components (those the connector profile seeds) are checked — functional components +are intentionally left to OCC and not in Contentstack. + +Read-only. Creds from env: CS_API_KEY, CS_MANAGEMENT_TOKEN, CS_REGION_BASE. + +Usage: python3 verify_connector.py --inventory --plan [--verbose] +""" +import argparse, json, os, re, sys, urllib.parse, urllib.request +from collections import defaultdict +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "cs-seed")) +import cs_auth # management token, or the user's csdx login + +API = os.environ.get("CS_REGION_BASE", "https://api.contentstack.io").rstrip("/") +KEY, TOKEN = os.environ.get("CS_API_KEY"), (cs_auth.MANAGEMENT_TOKEN or cs_auth.available()) + +# connector content type -> OCC typeCode (subset we author) +CT_TO_TYPECODE = { + "simple_responsive_banner_component": "SimpleResponsiveBannerComponent", + "simple_banner_component": "SimpleBannerComponent", + "product_carousel_component": "ProductCarouselComponent", + "cms_paragraph_component": "CMSParagraphComponent", + "cms_link_component": "CMSLinkComponent", + "cms_flex_component": "CMSFlexComponent", +} + + +def cma(path): + req = urllib.request.Request(f"{API}/v3{path}") + req.add_header("api_key", KEY) + for h, v in cs_auth.headers().items(): + req.add_header(h, v) + with urllib.request.urlopen(req, timeout=40) as r: + return json.loads(r.read().decode()) + + +def all_entries(ct): + out, skip = [], 0 + while True: + r = cma(f"/content_types/{ct}/entries?limit=100&skip={skip}&include_count=true") + es = r.get("entries") or [] + out += es; skip += len(es) + if len(es) < 100 or skip >= (r.get("count") or 0): + break + return out + + +def asset_titles(): + out, skip = {}, 0 + while True: + r = cma(f"/assets?limit=100&skip={skip}&include_count=true") + a = r.get("assets") or [] + for x in a: + out[x["uid"]] = x.get("title") + skip += len(a) + if len(a) < 100 or skip >= (r.get("count") or 0): + break + return out + + +def norm_html(s): + return re.sub(r"\s+", " ", (s or "").strip()) + + +def as_target(v): + return "true" if str(v).strip().lower() == "true" else "false" + + +def _uid(v): + if isinstance(v, list) and v: + return v[0] if isinstance(v[0], str) else v[0].get("uid") + return v if isinstance(v, str) else (v.get("uid") if isinstance(v, dict) else None) + + +def orig_proj(c): + tc = c["typeCode"] + m = c.get("media") or {} + if tc == "SimpleResponsiveBannerComponent": + codes = sorted(m[b]["code"] for b in ("mobile", "tablet", "desktop", "widescreen") + if isinstance(m.get(b), dict) and m[b].get("code")) + return {"media": codes, "url_link": c.get("urlLink") or ""} + if tc == "SimpleBannerComponent": + return {"media": [m["code"]] if m.get("code") else [], "url_link": c.get("urlLink") or ""} + if tc == "ProductCarouselComponent": + return {"title": c.get("title") or "", "products": sorted((c.get("productCodes") or "").split())} + if tc == "CMSParagraphComponent": + return {"content": norm_html(c.get("content"))} + if tc == "CMSLinkComponent": + return {"link_name": c.get("linkName") or "", "url": c.get("url") or "", + "target": as_target(c.get("target"))} + if tc == "CMSFlexComponent": + return {"flex_type": c.get("flexType") or ""} + return {} + + +def cs_proj(e, ct, assets): + if ct == "simple_responsive_banner_component": + codes = sorted(assets.get(_uid(e.get(f))) for f in + ("media_mobile", "media_tablet", "media_desktop", "media_widescreen") if e.get(f)) + return {"media": [c for c in codes if c], "url_link": e.get("url_link") or ""} + if ct == "simple_banner_component": + c = assets.get(_uid(e.get("media"))) + return {"media": [c] if c else [], "url_link": e.get("url_link") or ""} + if ct == "product_carousel_component": + # Same precedence as the connector: SAP app picker codes first, legacy SKU list second. + sap = [i.get("code") for i in ((e.get("sap_products") or {}).get("data") or []) if i.get("code")] + return {"title": e.get("title") or "", "products": sorted(sap or e.get("products") or [])} + if ct == "cms_paragraph_component": + return {"content": norm_html(e.get("content"))} + if ct == "cms_link_component": + return {"link_name": e.get("link_name") or "", "url": e.get("url") or "", + "target": as_target(e.get("target"))} + if ct == "cms_flex_component": + return {"flex_type": e.get("flex_type") or ""} + return {} + + +def flatten_source_tree(root): + """Flatten a source OCC navigationNode tree into + {node_uid: (parent_uid, order, link_itemId)}. The root node is a pure + container, so its children are the top level (parent '').""" + flat = {} + def walk(node, parent, order): + uid = node.get("uid") + if not uid: + return + entries = [e.get("itemId") for e in (node.get("entries") or []) + if e.get("itemType") == "CMSLinkComponent" and e.get("itemId")] + flat[uid] = (parent, order, entries[0] if entries else None) + for i, ch in enumerate(node.get("children") or [], start=1): + walk(ch, uid, i) + for i, ch in enumerate(root.get("children") or [], start=1): + walk(ch, "", i) + return flat + + +def verify_nav(inv, nav_orig, verbose): + """Read the seeded FLAT nav back and diff each menu against its source tree. + + For every CategoryNavigation/FooterNavigation component: fetch the flat + component, walk its all_nodes pool into nav_node_flat entries, reassemble the + tree from node_id/parent_id, and diff each node's (parent, sort_order, link) + against the original OCC navigationNode tree. Proves the flat model faithfully + represents the source hierarchy — the same reassembly the connector does at + runtime. Returns (checked, ok, issues).""" + FLAT_CT = ("category_navigation_flat", "footer_navigation_flat") + comps = {} + for ct in FLAT_CT: + for e in all_entries(ct): + if e.get("sap_uid"): + comps[e["sap_uid"]] = e + nodes_by_csuid = {e["uid"]: e for e in all_entries("nav_node_flat")} + link_sap_by_csuid = {e["uid"]: e.get("sap_uid") for e in all_entries("cms_link_component")} + + def node_link_sap(n): + u = _uid(n.get("links")) if n.get("links") else None + return link_sap_by_csuid.get(u) if u else None + + checked = ok = 0 + issues = [] + for cuid, c in nav_orig.items(): + checked += 1 + root = c.get("navigationNode") or {} + if not root.get("children"): + root = (inv.get("navigation") or {}).get(root.get("uid"), root) + src = flatten_source_tree(root) + cs = comps.get(cuid) + if not cs: + issues.append((cuid, ["flat component missing in stack"])); continue + got = {} + for r in cs.get("all_nodes") or []: + n = nodes_by_csuid.get(r.get("uid") if isinstance(r, dict) else r) + if n and n.get("node_id"): + got[n["node_id"]] = (n.get("parent_id") or "", n.get("sort_order"), node_link_sap(n)) + diffs = [] + for nid, (p, o, lk) in src.items(): + if nid not in got: + diffs.append(f"missing node {nid}"); continue + gp, go, glk = got[nid] + if gp != p: + diffs.append(f"{nid}: parent {gp!r}!={p!r}") + if go != o: + diffs.append(f"{nid}: sort {go}!={o}") + if glk != lk: + diffs.append(f"{nid}: link {glk}!={lk}") + extra = sorted(set(got) - set(src)) + if extra: + diffs.append(f"extra nodes {extra}") + if not diffs: + ok += 1 + else: + issues.append((cuid, diffs)) + if verbose: + print(f" nav {cuid}: source {len(src)} nodes, stack {len(got)} " + f"-> {'OK' if not diffs else diffs[:4]}") + return checked, ok, issues + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--inventory", required=True) + ap.add_argument("--plan", required=True) + ap.add_argument("--verbose", action="store_true") + a = ap.parse_args() + if not (KEY and TOKEN): + sys.exit("set CS_API_KEY and CS_MANAGEMENT_TOKEN") + inv = json.load(open(a.inventory)) + plan = json.load(open(a.plan)) + + # original components by uid + orig = {} + for p in inv["contentPages"].values(): + for s in p["slots"]: + for c in s["components"]: + if c.get("uid"): + orig.setdefault(c["uid"], c) + + # which uids we actually seeded (from the plan) + seeded_uids = {e["_key"]["value"] for e in plan["entries"]["components"]} + + assets = asset_titles() + # fetch seeded component entries by content type, index by sap_uid + cs_by_uid, cs_ct = {}, {} + for ct in CT_TO_TYPECODE: + for e in all_entries(ct): + u = e.get("sap_uid") + if u: + cs_by_uid[u] = e; cs_ct[u] = ct + + per = defaultdict(lambda: {"n": 0, "match": 0}) + mismatches, missing = [], [] + NAV_TYPES = {"CategoryNavigationComponent", "FooterNavigationComponent"} + for uid in seeded_uids: + c = orig.get(uid) + if not c: + continue + tc = c["typeCode"] + if tc in NAV_TYPES: # nav components verified separately (nav tree read), not here + continue + cs = cs_by_uid.get(uid) + if not cs: + missing.append(uid); continue + want, got = orig_proj(c), cs_proj(cs, cs_ct[uid], assets) + per[tc]["n"] += 1 + if want == got: + per[tc]["match"] += 1 + else: + mismatches.append((tc, uid, {k: (want.get(k), got.get(k)) + for k in set(want) | set(got) if want.get(k) != got.get(k)})) + + total = sum(r["n"] for r in per.values()) + matched = sum(r["match"] for r in per.values()) + print(f"[verify_connector] {inv['site']} @ {API}") + print(f" authored components checked: {total} matched: {matched} " + f"mismatched: {total - matched} missing-in-stack: {len(missing)}") + for tc in sorted(per): + print(f" {tc:34s} {per[tc]['match']}/{per[tc]['n']}") + if missing: + print(" MISSING:", missing[:10]) + if mismatches and a.verbose: + for tc, uid, d in mismatches[:15]: + print(f" {tc} {uid}: {json.dumps(d)[:200]}") + + # flat navigation: reassemble the seeded nav_node_flat pool and diff the tree + # against the source (skipped in the per-component loop above). + nav_orig = {uid: c for uid, c in orig.items() if c["typeCode"] in NAV_TYPES} + nav_checked, nav_ok, nav_issues = verify_nav(inv, nav_orig, a.verbose) + if nav_checked: + print(f" nav menus (flat) checked: {nav_checked} trees match: {nav_ok}") + for cuid, diffs in nav_issues: + print(f" NAV MISMATCH {cuid}: {diffs[:6]}") + + comp_ok = (matched == total and not missing and total > 0) + nav_pass = (nav_ok == nav_checked) + ok = comp_ok and nav_pass + print(" RESULT:", "PASS — connector model round-trips (components + nav)" if ok + else f"{total-matched} comp mismatch / {len(missing)} missing / " + f"{nav_checked-nav_ok} nav mismatch") + sys.exit(0 if ok else 1) + + +if __name__ == "__main__": + main()