Skip to content

feat(connectors): export a single connector as JSON/YAML and import JSON/YAML files - #1202

Merged
helpcodeai merged 18 commits into
HelpCode-ai:mainfrom
GallaGit:feat/1004-single-connector-export-yaml
Oct 24, 2026
Merged

helpcodeai merged 18 commits into
HelpCode-ai:mainfrom
GallaGit:feat/1004-single-connector-export-yaml

Conversation

@GallaGit

Copy link
Copy Markdown
Contributor

Summary

This PR lets you export one connector and its tools as JSON or YAML, and import a JSON or YAML file back into the caller's organization. The export uses the same envelope as export-all, so a single-connector file also restores through import-all. Both import endpoints share one code path and the existing DTO validation.

Closes #1004. Part of the October Challenge #997.

Changes

  • GET /api/connectors/:id/export?format=json|yaml exports one connector in the export-all envelope (version, exportedAt, secretsIncluded, connectors: [entry]).
    • JSON is the default, and an unknown format returns 400.
    • YAML is written with js-yaml CORE_SCHEMA (noRefs, lineWidth: -1) and sent as application/yaml; charset=utf-8.
    • The file is sent as an attachment named <slug>.anythingmcp.<json|yaml>.
    • Access is checked with assertOrgMatch: another org's connector returns 403, and an unknown id returns 404.
  • POST /api/connectors/import takes { "content": "<file text>" }.
    • It checks assertCanCreate before parsing.
    • It parses the text with yaml.load(..., { schema: CORE_SCHEMA }), which also reads JSON. A syntax error returns 400 with the line and column.
    • A bare connector object or a bare array of connectors is wrapped in the export envelope for both JSON and YAML.
    • The result is validated against ImportAllDto with the same options as the global ValidationPipe (transform, whitelist, forbidNonWhitelisted).
  • Shared code:
    • toExportEntry and toExportEnvelope (connector-export.util.ts, reusing toPublicConnector from connector-secrets.util.ts) are now shared by export-all and :id/export.
    • The import-all loop moved to a private importConnectors() that both import endpoints use, so they apply the same licence/trial check per connector, skip duplicates by name within the caller's org, and return the same result shape.
    • Backend behaviour of export-all and import-all is unchanged.
  • Frontend:
    • The connector detail page has an Export menu (JSON / YAML) that downloads through fetch and a Blob. After export, a notice says the file holds secrets in plain text (admins) or had them removed (other roles), and that OAuth credentials are not exported.
    • Behaviour change in the import dialog on the connectors list: it is renamed "Import from file", accepts .json,.yaml,.yml, and now sends the raw text to POST /api/connectors/import instead of parsing JSON in the browser and calling import-all.
    • New api.ts helpers: exportConnector and importFile.
  • Docs: docs/api-reference.md lists export-all, import-all, :id/export and import, with a YAML example.

Validation that already existed still applies to the new endpoint. A file carrying organizationId or authConfig on a connector, or an unknown tool property such as useProxy, is rejected with 400 by forbidNonWhitelisted on the existing import DTOs, the same way as in import-all and identically for JSON and YAML. That behaviour comes from the existing DTOs; this PR adds no new rule for it.

Type

  • Bug fix
  • New feature
  • Documentation
  • Refactoring
  • Performance improvement
  • Other (describe below)

Testing

  • Added/updated unit tests

  • Tested manually (describe steps below)

  • No tests needed (explain why)

  • New connectors.export-import.spec.ts (32 tests) together with connectors.import-all.spec.ts (16 tests): 48/48 pass. They cover:

    • JSON and YAML round trips into a second org, using an OAuth connector fixture with tool-level fields.
    • Non-admin masking, and authConfig absent in both formats.
    • 403 for another org's connector, 403 for a VIEWER (nothing is parsed), and 404 for an unknown id.
    • 400 with line and column for a YAML syntax error.
    • Bare connector objects and arrays in JSON and YAML.
    • Unknown-property rejection in both formats and in bare files.
    • Duplicate skipping within the caller's org, and the licence limit.
  • cd packages/backend && npx jest src/connectors: 77 suites passed, 1 failed, 2 skipped. Tests: 1549 passed, 1 failed, 11 skipped (1561 total).

    • The only failure is engines/database.engine.ssrf.spec.ts › allows a blocked host when explicitly allowlisted (5000 ms timeout).
    • It fails the same way on clean main at 99f04a1: 76 suites passed, 1 failed, 2 skipped. Tests: 1517 passed, 1 failed, 11 skipped (1529 total). Same test, same timeout.
    • The cause is my environment, not this change. The test expects the connection to db.internal to fail quickly after it passes the SSRF guard. In my sandbox the DNS resolves db.internal to 198.18.0.1 (getent hosts db.internal), so the connection attempt hangs until Jest's 5 s timeout. In a normal environment the name does not resolve and the test fails fast as intended.
  • Backend tsc --noEmit: passes. eslint on the changed files: 0 errors, 0 warnings.

  • Frontend tsc --noEmit: passes. eslint src/: 0 errors, and the changed files have no new warnings (the existing warnings are unrelated).

  • Playwright tests/e2e/connector-export-import.spec.ts: 2/2 pass. They check that the Export menu requests format=yaml and downloads the file, and that the import dialog accepts a .yaml file and POSTs its text unchanged as { content }.

Checklist

  • My code follows the existing code style
  • I have tested my changes locally
  • I have updated documentation (if applicable)
  • All existing tests pass (npm test). All pass except the pre-existing environment-dependent SSRF test described above, which also fails on clean main here.
  • This PR has a descriptive title

Known limits

  • OAuth: authConfig is never exported, as the issue requires. That includes its non-secret parts (tokenUrl, authorizationUrl, grant, client id), so an imported OAuth connector keeps its authType but has to be configured and authorized again.
  • Not exported: useProxy, admin annotations overrides and tool role access. Only the 7 tool fields that export-all already writes are exported (name, description, isEnabled, parameters, endpointMapping, responseMapping, outputSchema), and ImportToolDto still rejects the others. Per the issue, annotations need their own design.
  • Secrets: following the issue, an admin export contains secret env vars and headers in plain text (same as export-all), and the UI warns about it. Other roles get them masked. authConfig is never included.
  • Route name: POST /api/connectors/import sits next to the existing POST /api/connectors/:id/import (spec → tools). They don't clash, but the names are close. I can rename it to import-file if you prefer.
  • Size: content is capped at 5,000,000 characters, within the existing 10 MB body limit.

Related Issues

Closes #1004
Part of the October Challenge #997

Pure export helper reused by export-all; the import-all loop moves to a private method so a new import endpoint can share it. No behaviour change. Refs HelpCode-ai#1004
Single-connector export in the export-all envelope, as JSON or YAML (js-yaml core schema). Same secret masking for non-admins, authConfig never exported, 403 for another organization's connector. Refs HelpCode-ai#1004
Accepts { content } with the text of an export file, parses it with js-yaml (core schema), validates it against ImportAllDto with the global pipe options and imports it into the caller's organization through the shared importConnectors. Syntax errors return 400 with line and column; viewers get 403 before parsing. Refs HelpCode-ai#1004
Round trip into another organization for JSON and YAML with an OAuth connector and tool-level settings, masking for non-admins, authConfig never written, cross-org 403, viewer 403, line/column on YAML errors and unknown properties rejected as in import-all. Refs HelpCode-ai#1004
…files

Connector page gets an Export menu (JSON or YAML) with a notice on secrets; the import dialog accepts .json/.yaml/.yml and sends the file text to POST /api/connectors/import. Refs HelpCode-ai#1004
A file holding a single connector object or an array of connectors is
wrapped in the export envelope after yaml.load, for JSON and YAML alike,
and then validated as before.

Refs HelpCode-ai#1004

@helpcodeai helpcodeai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @GallaGit, thanks a lot, this is a really clean PR. The shared toExportEntry / importConnectors refactor is exactly what we wanted, the round-trip tests are good, and the e2e covers the right things. CI is green and I ran the two connector specs locally too.

Quick answers to your questions first:

  1. OAuth: keep it as it is. Exporting the non-secret OAuth keys is a separate piece of work, we'll open an issue for it.
  2. Tool overrides: fine as a known limit for this PR.
  3. Keep POST /api/connectors/import, it matches the issue and there's no routing clash.
  4. You read it right. Admin exports carry secret env vars and headers like export-all does, and the warning in the UI covers it. When I said secrets never end up in the file I meant authConfig and tokens.

One thing has to change before we can merge, plus a smaller one:

  1. YAML aliases. js-yaml resolves *alias to the same object, so a few hundred bytes of nested anchors parse instantly but blow up once class-transformer and Prisma walk the tree. I tried a 438 byte file with 6 levels of &l1 [*l0, *l0, ...] under config: it blocked the process for about 17 seconds and stored a 69 MB config, and one more level would run the server out of memory. On Cloud that's one process for every workspace, and any trial user can call this endpoint. A self-referencing anchor (config: &c {self: *c}) also ends in a stack overflow and a 500. Exports never write anchors (noRefs: true), so the simplest fix is to walk the parsed value once in parseImportFile and return 400 if any object or array shows up twice. That catches both cases. Please add a spec for each.
  2. Hand-edited YAML can put numbers or booleans into envVars and headers (PORT: 5432 ends up stored as a number). Could you either turn those scalars into strings after parsing or reject them with a clear 400? Round trips are fine already, this is just for people who edit the file by hand.

Small optional stuff: the secretsIncluded regex in api.ts could be a response header from exportOne instead, and there's a stray blank line in connectors/page.tsx. No need to rebase, we'll update the branch before merging.

Thanks again, this is very close!

…vVars/headers on import

parseImportFile now walks the parsed value once and returns 400 if it meets the same object or array twice. That covers alias bombs and self-referencing anchors. Number and boolean values in envVars and headers become strings, and object or array values are rejected with a 400.
The frontend reads the header instead of matching the file text with a regex. CORS now exposes Content-Disposition and X-Secrets-Included.
@GallaGit

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Changes are in the latest commits:

  • YAML alias protection: parseImportFile now walks the parsed value once, iteratively, with a Set of seen objects checked before descending. If any object or array is reached twice, the import fails with a 400. This rejects billion-laughs-style nested anchors and self-referencing anchors (config: &c {self: *c}). The walk is linear and always terminates. Export never writes anchors (noRefs), so real export files are unaffected. New specs cover both cases: the 6-level alias bomb returns 400 well under 1 s, and the self-reference returns 400 instead of a 500.
  • envVars / headers scalars: numbers and booleans in hand-edited YAML (e.g. PORT: 5432) are now turned into strings after parsing. An object or array value is rejected with a 400 naming the key (e.g. connectors[0].envVars.NESTED). Specs added.
  • Optional: GET /connectors/:id/export now sends an X-Secrets-Included header, and the frontend reads it instead of the regex. CORS exposes it together with Content-Disposition, so the browser can read both. I also removed the stray blank line in connectors/page.tsx.

Backend typecheck and lint are clean, and all 36 export/import specs pass. Locally, database.engine.ssrf.spec.ts ("allows a blocked host when explicitly allowlisted") fails for me with or without these changes, so it looks unrelated.

@GallaGit
GallaGit requested a review from helpcodeai October 23, 2026 00:44

@helpcodeai helpcodeai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @GallaGit, the object alias fix and the scalar handling look good, and the specs are solid. I ran one more round on the YAML path and found a hole I should have mentioned the first time, sorry about that.

The alias check in connector-export.util.ts skips strings, so an anchored long string referenced many times still gets through. With a 1 MB string referenced 200k times inside config, the 1.6 MB file blocks the event loop for about 50 s and RSS goes to ~2.7 GB before Prisma gives up. A smaller one aliased inside a tool's endpointMapping (~100 KB file) is accepted and stores a ~100 MB value, which then gets loaded into the tool registry on every boot. Same thing through envVars and headers.

Could you give the walk a budget on the expanded size? Count string lengths and key lengths for every slot you visit (so each alias use counts again), and return a 400 once the total passes IMPORT_FILE_MAX_CHARS. Rejecting aliases altogether would also be fine with me, nobody needs them in a connector export. Please add a spec with a string alias bomb next to the existing ones.

Two smaller things while you're in there, both optional:

  • importConnectors could call assertUsableBaseUrl on each connector's baseUrl. The request-time SSRF guard still protects us, but rejecting a metadata IP at import time is nicer.
  • In the connector page, URL.revokeObjectURL right after a.click() can cancel the download in some browsers. A setTimeout of a few seconds avoids it.

After that I'll update the branch and merge.

…on import

The import walk now adds up the length of every string and object key, once per place it appears, and returns 400 once the total passes IMPORT_FILE_MAX_CHARS. This catches an anchored string repeated through aliases. importConnectors runs assertUsableBaseUrl on every connector before writing anything and returns 400 naming connectors[i].baseUrl.
@GallaGit

Copy link
Copy Markdown
Contributor Author

Thanks again for the careful review! Addressed in two new commits:

  • String alias bomb: the import walk (assertNoSharedReferences) now also keeps a running total of the length of every string and object key, counted once per place it appears. Since strings are primitives, each alias use counts again. Once the total passes IMPORT_FILE_MAX_CHARS, the import fails with a 400 ("too large once YAML aliases are expanded"). Repeated objects are still rejected by identity. The walk stays linear and stops at the first node over the budget. New specs reuse an anchored ~100 KB string 60 times under config, envVars, and a tool's endpointMapping. Each returns 400 quickly.
  • baseUrl on import: importConnectors now runs assertUsableBaseUrl on every connector before anything is written. A rejection becomes a 400 naming connectors[i].baseUrl, and a bad file imports nothing. A spec covers this. Note that validateBaseUrl checks shape and scheme (and, on cloud, that the host looks real). It is not an IP/SSRF filter, so 169.254.169.254 would pass it. The spec uses an ftp:// URL instead. Runtime SSRF protection stays in the engines.
  • Frontend: URL.revokeObjectURL now runs in a 5 s setTimeout after a.click(), in both connectors/page.tsx and connectors/[id]/page.tsx.

Tests: backend typecheck and eslint are clean. The export/import spec passes, including the existing round-trip specs. In the connectors folder, 1557 tests pass. The one failure, database.engine.ssrf.spec.ts ("allows a blocked host when explicitly allowlisted"), also fails on the branch without these changes. Frontend typecheck passes, and eslint shows no new warnings.

@helpcodeai helpcodeai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the new walk: strings and keys now count once per place they appear, so the anchored-string case gets a 400 well before it hurts, and the baseUrl check runs before anything is written. Thanks for sticking with this through three rounds, merging.

@helpcodeai
helpcodeai enabled auto-merge (squash) October 24, 2026 07:47
@helpcodeai
helpcodeai merged commit a23cf86 into HelpCode-ai:main Oct 24, 2026
14 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 24, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🎃 Export a single connector as JSON or YAML, and accept YAML on import

2 participants