MSSQL testdb: podman CLI engine, container user, schema hook, snapshot reset

README.md

@@ -276,7 +276,13 @@ changed any of these on a machine that already has one.
 Container management goes through the Docker API (the `docker` package,
 `docker.from_env()`, falling back to Podman's rootful/rootless socket) — it
 works against a real Docker daemon or Podman transparently, no CLI binary
-required either way.
+required either way. If no API socket is reachable (typical for rootless
+Podman without a systemd user session, where there is no `podman.sock` and
+`DOCKER_HOST` is unset), pgdevkit falls back to the `podman`/`docker` CLI
+(first one found on `PATH`, `podman` first). Set
+`PGDEVKIT_CONTAINER_ENGINE=podman` (or `docker`) to skip the socket probing
+and always use that CLI; `auto`/unset is the default behaviour above. This
+applies to both the Postgres and the MSSQL container.
 
 To point at a local Postgres install instead of the container — useful when
 neither is available, or you'd rather use peer authentication as the
@@ -309,6 +315,58 @@ take effect when pgdevkit creates the container, not when restarting an
 existing stopped one (`docker rm -f pgdevkit-mssql` first if you've changed
 any of them).
 
+`PGDEVKIT_TESTDB_MSSQL_CONTAINER_USER` (unset by default, i.e. the image's own
+user) is passed as the container's `--user`.
+
+**Rootless Podman recipe.** The SQL Server image's default user can't start
+there (`exec container process /opt/mssql/bin/launch_sqlservr.sh: Permission
+denied`); running the container as root *inside* the user namespace fixes it
+and doesn't give the container any real host privileges:
+
+```bash
+export PGDEVKIT_CONTAINER_ENGINE=podman          # no podman.sock needed
+export PGDEVKIT_TESTDB_MSSQL_CONTAINER_USER=0
+```
+
+**Generated schemas.** When the DDL is produced by code (e.g. from a metadata
+export) instead of `database/` files, pass a hook to `ensure_testdb` /
+`reset_testdb`. It receives an autocommit `mssql_python` connection to this
+workspace's database, after the `database/` files are applied — or instead of
+them with `apply_scripts=False`:
+
+```python
+def create_tables(conn) -> None:
+    cur = conn.cursor()
+    for ddl in generate_ddl():  # your generator
+        cur.execute(ddl)
+
+ensure_testdb(apply=create_tables, apply_scripts=False)
+```
+
+Like the files, the hook runs on every call, so make it idempotent (or check
+whether the objects exist first). Not available for Postgres.
+
+**Fast reset with database snapshots.** Build the schema (and seed data) once,
+then capture it and revert to it per test instead of rebuilding:
+
+```python
+from pgdevkit.testdb import ensure_testdb, restore_testdb, snapshot_testdb
+
+ensure_testdb(apply=create_tables, apply_scripts=False, force_reset=True)
+snapshot_testdb()          # once per session; name="template" by default
+
+restore_testdb()           # per test: kicks other sessions off, reverts, keeps the snapshot
+```
+
+This uses SQL Server database snapshots (Developer/Enterprise edition — what
+the managed container runs). A revert is only possible from a database's *sole*
+snapshot, so `restore_testdb` raises if another snapshot of the database
+exists, instead of dropping it. Snapshot files (sparse) are written next to the
+database's data files, so they follow it onto a `PGDEVKIT_TESTDB_MSSQL_TMPFS`
+mount; `PGDEVKIT_TESTDB_MSSQL_SNAPSHOT_DIR` (a path inside the container)
+overrides that. Snapshots are dropped together with their database
+(`force_reset`, `clean_testdb`) and never reported as orphans.
+
 `pgdb testdb shell` execs into
 [`sqlcmd`](https://github.com/microsoft/go-sqlcmd) (an external prerequisite,
 the same category as `psql` for the Postgres path) rather than a Python

pgdevkit/diff.py

@@ -192,11 +192,11 @@ def _parse_index_def(definition: str, dialect: Dialect = POSTGRES) -> dict | Non
 
     columns = []
     for col in (params.args.get("columns") if params else None) or []:
-        columns.append(_norm_sql(col.sql(dialect=dialect.sqlglot_name)))
+        columns.append(_norm_sql(_unquote(col).sql(dialect=dialect.sqlglot_name)))
 
     where_node = params.args.get("where") if params else None
-    where_ast = _unwrap_paren(where_node.this) if where_node else None
-    where_display = _norm_sql(where_node.this.sql(dialect=dialect.sqlglot_name)) if where_node else None
+    where_ast = _unwrap_paren(_unquote(where_node.this)) if where_node else None
+    where_display = _norm_sql(_unquote(where_node.this).sql(dialect=dialect.sqlglot_name)) if where_node else None
 
     return {
         "unique": bool(parsed.args.get("unique")),
@@ -207,8 +207,25 @@ def _parse_index_def(definition: str, dialect: Dialect = POSTGRES) -> dict | Non
     }
 
 
+def _unquote(node: exp.Expression) -> exp.Expression:
+    """A copy with every identifier's quoting dropped, so `[Posting Date]`
+    in a script (SSMS and generators bracket everything) and the same column
+    as the catalog reports it compare equal, however each side quoted it."""
+    node = node.copy()
+    for ident in node.find_all(exp.Identifier):
+        ident.set("quoted", False)
+    return node
+
+
 def _unwrap_paren(node):
-    return node.transform(lambda n: n.this if isinstance(n, exp.Paren) else n)
+    # transform() doesn't descend into a node it replaced, so a Paren nested
+    # inside another one ("([b]>(1))", how SQL Server stores a filter) needs
+    # another pass.
+    previous = None
+    while node != previous:
+        previous = node
+        node = node.transform(lambda n: n.this if isinstance(n, exp.Paren) else n)
+    return node
 
 
 def _norm_sql(s: str) -> str:
@@ -232,7 +249,8 @@ def _norm_type(t: str, dialect: Dialect = POSTGRES) -> str:
         s = s[:-2].strip()
 
     match = re.search(r"\(([^)]*)\)", s)
-    params = f"({match.group(1)})" if match else ""
+    # sqlglot renders "decimal(38, 20)", the catalogs report "decimal(38,20)".
+    params = f"({''.join(match.group(1).split())})" if match else ""
     base = (s[: match.start()] + s[match.end() :]).strip() if match else s
     base = " ".join(base.split())
 

pgdevkit/mssql_introspect.py

@@ -6,6 +6,7 @@
 import sqlglot
 import sqlglot.expressions as exp
 
+from .db.mssql_sql import ident
 from .models import (
     ColumnDef, ConstraintDef, DatabaseSchema, FunctionDef, IndexDef, TableDef, ViewDef,
 )
@@ -264,12 +265,14 @@ def _load_indexes(conn: Any, db: DatabaseSchema) -> None:
             WHERE ic.object_id = ? AND ic.index_id = ? AND ic.is_included_column = 0
             ORDER BY ic.key_ordinal
         """, (r["object_id"], r["index_id"]))
-        col_list = ", ".join(c["name"] for c in cols)
         unique = "UNIQUE " if r["is_unique"] else ""
         where_clause = f" WHERE {r['filter_definition']}" if r["filter_definition"] else ""
+        # Bracket-quoted: object names (NAV-style "Company$Cust. Ledger Entry")
+        # don't parse back unquoted, which would drop the diff to a raw text compare.
+        col_list = ", ".join(ident(c["name"]) for c in cols)
         definition = (
-            f"create {unique}index {r['index_name']} "
-            f"on {r['schema']}.{r['table_name']} ({col_list}){where_clause}"
+            f"create {unique}index {ident(r['index_name'])} "
+            f"on {ident(r['schema'])}.{ident(r['table_name'])} ({col_list}){where_clause}"
         ).lower()
         idx = IndexDef(schema=r["schema"], table=r["table_name"], name=r["index_name"], definition=definition)
         db.indexes[idx.qualified_name] = idx

pgdevkit/testdb/__init__.py

@@ -4,8 +4,10 @@
     ensure_testdb,
     find_orphaned_dbs,
     reset_testdb,
+    restore_testdb,
     run_sql,
     shell_argv,
+    snapshot_testdb,
     status,
     workspace_db_names,
 )
@@ -16,8 +18,10 @@
     "ensure_testdb",
     "find_orphaned_dbs",
     "reset_testdb",
+    "restore_testdb",
     "run_sql",
     "shell_argv",
+    "snapshot_testdb",
     "status",
     "workspace_db_names",
 ]

pgdevkit/testdb/_docker.py

@@ -1,8 +1,11 @@
 from __future__ import annotations
 
 import os
+import shutil
+import subprocess
 
 import docker
+import docker.errors
 
 # Candidate Docker-API-compatible socket URLs tried after plain
 # docker.from_env() (which only looks at DOCKER_HOST / the default Docker
@@ -14,12 +17,110 @@
 ]
 
 
-def client() -> docker.DockerClient:
-    """A Docker-API client, working against a real Docker daemon or a
-    Podman one (Podman exposes the same API over its own socket) -- callers
-    never need to know or care which one is actually running. Shared by
-    both the Postgres and MSSQL container modules -- a container is a
-    container, regardless of which image runs inside it."""
+# Container engines whose CLI can stand in for the Docker SDK, in the order
+# tried by auto-detection. Podman first: it is the one that routinely has no
+# API socket (rootless without a systemd user session), while a `docker` CLI
+# needs a reachable daemon just like the SDK does.
+_CLI_ENGINES = ("podman", "docker")
+
+
+class CliContainer:
+    """The slice of docker-py's `Container` that the container modules use
+    (`status`, `start()`), backed by the engine's CLI."""
+
+    def __init__(self, engine: str, name: str, status: str) -> None:
+        self._engine = engine
+        self.name = name
+        self.status = status
+
+    def start(self) -> None:
+        _run_cli(self._engine, ["start", self.name])
+
+
+class CliContainers:
+    """The slice of docker-py's `ContainerCollection` that the container
+    modules use (`get`, `run`), implemented by shelling out to
+    `podman`/`docker`, for environments with no reachable API socket."""
+
+    def __init__(self, engine: str) -> None:
+        self._engine = engine
+
+    def get(self, name: str) -> CliContainer:
+        result = _run_cli(self._engine, ["inspect", "--format", "{{.State.Status}}", name], check=False)
+        if result.returncode != 0:
+            if "no such" in result.stderr.lower():
+                raise docker.errors.NotFound(result.stderr.strip())
+            raise RuntimeError(f"`{self._engine} inspect {name}` failed: {result.stderr.strip()}")
+        return CliContainer(self._engine, name, result.stdout.strip())
+
+    def run(
+        self,
+        image: str,
+        *,
+        name: str,
+        ports: dict[str, int],
+        environment: dict[str, str],
+        detach: bool = True,  # accepted for docker-py compatibility; the CLI always runs detached
+        command: list[str] | None = None,
+        user: str | None = None,
+        tmpfs: dict[str, str] | None = None,
+        shm_size: str | None = None,
+        mem_limit: str | None = None,
+        memswap_limit: str | None = None,
+        nano_cpus: int | None = None,
+    ) -> CliContainer:
+        # Values are passed as bare `-e KEY` with the real value in the child's
+        # environment, so e.g. the SA password never shows up in `ps` output.
+        args = ["run", "--detach", "--name", name]
+        for container_port, host_port in ports.items():
+            args += ["--publish", f"{host_port}:{container_port}"]
+        for key in environment:
+            args += ["--env", key]
+        if user:
+            args += ["--user", user]
+        for path, options in (tmpfs or {}).items():
+            args += ["--tmpfs", f"{path}:{options}" if options else path]
+        if shm_size:
+            args += ["--shm-size", shm_size]
+        if mem_limit:
+            args += ["--memory", mem_limit]
+        if memswap_limit:
+            args += ["--memory-swap", memswap_limit]
+        if nano_cpus:
+            args += ["--cpus", str(nano_cpus / 1_000_000_000)]
+        args.append(image)
+        args += command or []
+        result = _run_cli(self._engine, args, env=environment, check=False)
+        if result.returncode != 0:
+            # Same exception type docker-py raises, so the callers' "name already
+            # in use -> start the existing container" handling applies unchanged.
+            raise docker.errors.APIError(result.stderr.strip() or f"`{self._engine} run` failed")
+        return CliContainer(self._engine, name, "running")
+
+
+class CliClient:
+    """Stands in for `docker.DockerClient` when only a container CLI is available."""
+
+    def __init__(self, engine: str) -> None:
+        self.engine = engine
+        self.containers = CliContainers(engine)
+
+
+def _run_cli(
+    engine: str, args: list[str], *, env: dict[str, str] | None = None, check: bool = True
+) -> subprocess.CompletedProcess[str]:
+    result = subprocess.run(
+        [engine, *args],
+        capture_output=True,
+        text=True,
+        env={**os.environ, **env} if env else None,
+    )
+    if check and result.returncode != 0:
+        raise RuntimeError(f"`{engine} {args[0]}` failed: {result.stderr.strip()}")
+    return result
+
+
+def _sdk_client() -> docker.DockerClient | None:
     try:
         c = docker.from_env()
         c.ping()
@@ -35,9 +136,36 @@ def client() -> docker.DockerClient:
             return c
         except Exception:  # noqa: BLE001
             continue
+    return None
+
+
+def client() -> docker.DockerClient | CliClient:
+    """A container client, working against a real Docker daemon or a Podman
+    one (Podman exposes the same API over its own socket) -- callers never
+    need to know or care which one is actually running. Shared by both the
+    Postgres and MSSQL container modules -- a container is a container,
+    regardless of which image runs inside it.
+
+    PGDEVKIT_CONTAINER_ENGINE=docker|podman skips the SDK and drives that
+    engine's CLI directly. Unset (or "auto"), the SDK is tried first and the
+    first of `podman`/`docker` on PATH is the fallback when no API socket is
+    reachable -- e.g. rootless Podman without a systemd user session."""
+    engine = os.environ.get("PGDEVKIT_CONTAINER_ENGINE", "").strip().lower()
+    if engine in ("docker", "podman"):
+        if shutil.which(engine) is None:
+            raise RuntimeError(f"PGDEVKIT_CONTAINER_ENGINE={engine}, but `{engine}` is not on PATH.")
+        return CliClient(engine)
+    if engine not in ("", "auto"):
+        raise ValueError(f"PGDEVKIT_CONTAINER_ENGINE must be docker, podman or auto, not {engine!r}")
+    sdk = _sdk_client()
+    if sdk is not None:
+        return sdk
+    for cli_engine in _CLI_ENGINES:
+        if shutil.which(cli_engine):
+            return CliClient(cli_engine)
     raise RuntimeError(
-        "Could not reach a Docker-compatible API. Set DOCKER_HOST, or make sure "
-        "Docker or Podman's API socket is running."
+        "Could not reach a Docker-compatible API and found no `podman`/`docker` CLI. Set "
+        "DOCKER_HOST, make sure Docker or Podman's API socket is running, or install a container CLI."
     )