From 3f8970d4762c69d5d4f73a78114092d62f2efb5c Mon Sep 17 00:00:00 2001 From: dgunter Date: Sat, 5 Sep 2026 20:36:14 -0500 Subject: [PATCH 1/4] TDS through Zeek: a notebook on zeek-tds, a Docker image for it, and a Revisiting TDS draft tds-zeek.ipynb runs Zeek 8.2 with zeek-tds v0.2.0 in Docker over the 2009 capture, reads the logs with ParseZeekLogs and redoes the Part 4 analysis, showing the two structural cases tshark could not resolve: a call spanning two packets and a message holding two calls. The 2018 pyshark notebook stays. CI builds the image before executing notebooks. Co-Authored-By: Claude Fable 5.1 --- .github/workflows/notebooks.yml | 4 +- README.md | 14 +- TDS Analysis/tds-zeek.ipynb | 1257 +++++++++++++++++++++++++++++++ TDS Analysis/zeek/Dockerfile | 11 + docs/2026-09-revisiting-tds.md | 126 ++++ 5 files changed, 1406 insertions(+), 6 deletions(-) create mode 100644 TDS Analysis/tds-zeek.ipynb create mode 100644 TDS Analysis/zeek/Dockerfile create mode 100644 docs/2026-09-revisiting-tds.md diff --git a/.github/workflows/notebooks.yml b/.github/workflows/notebooks.yml index 02446f2..cc6e78c 100644 --- a/.github/workflows/notebooks.yml +++ b/.github/workflows/notebooks.yml @@ -25,10 +25,12 @@ jobs: run: uv sync --frozen - name: Lint code cells run: uv run ruff check . + - name: Build the Zeek image for the TDS notebook + run: docker build -q -t zeek-tds:0.2.0 "TDS Analysis/zeek" - name: Execute every notebook run: | uv run jupyter nbconvert --to notebook --execute --output-dir /tmp/executed \ - --ExecutePreprocessor.timeout=600 \ + --ExecutePreprocessor.timeout=1500 \ "Bro HTTP Log Analysis/"*.ipynb "TDS Analysis/"*.ipynb - name: Fail on cells that wrote to stderr run: | diff --git a/README.md b/README.md index a0548bf..b19a618 100644 --- a/README.md +++ b/README.md @@ -18,15 +18,18 @@ their images so the code has its narrative next to it. | [Detecting Nmap Behavior with Bro HTTP Logs](Bro%20HTTP%20Log%20Analysis/Detecting%20Nmap%20Behavior%20with%20Bro%20HTTP%20Logs.ipynb) | The same log as a time series: user agents and status codes per minute, with the scan window standing out once the busy browser is removed | [Part 2: Detecting Nmap Behavior with Bro HTTP Logs](docs/2017-11-threat-hunting-with-python-part-2-nmap-bro-http.md) (2017-11) | | [Detecting Nmap Behavior with ParseZeekLogs](Bro%20HTTP%20Log%20Analysis/Detecting%20Nmap%20Behavior%20with%20ParseZeekLogs.ipynb) | Both hunts above redone with [ParseZeekLogs](https://github.com/dgunter/ParseZeekLogs) and pandas, added in 2026 | same two posts | | [tds](TDS%20Analysis/tds.ipynb) | Pull TDS packet types, SQL batches and remote procedure calls out of a pcap with pyshark and look at what a historian's clients actually run | [Part 4: Examining Microsoft SQL Based Historian Traffic](docs/2018-03-threat-hunting-with-python-part-4-mssql-historian.md) (2018-03) | +| [tds-zeek](TDS%20Analysis/tds-zeek.ipynb) | The same capture through Zeek with the [zeek-tds](https://github.com/dgunter/zeek-tds) analyzer, added in 2026: reassembled messages, every procedure call with its parameters and prepared-statement SQL, read into pandas with ParseZeekLogs | [Revisiting TDS](docs/2026-09-revisiting-tds.md) (draft) | [Part 3: Taming SMB](docs/2018-02-threat-hunting-with-python-part-3-taming-smb.md) (2018-02) is preserved too; its examples were shown inline rather than as a notebook. -The TDS notebook works from the packet capture rather than from Zeek logs -because Zeek has no TDS analyzer, then or now: Zeek 8.2.2 run over the same -pcap produces only `conn.log` for the port 1433 sessions. pyshark hands the -fields Wireshark's TDS dissector decodes to pandas instead. +The 2018 TDS notebook works from the packet capture with pyshark because +Zeek had no usable TDS analyzer at the time. It now does: `tds-zeek.ipynb` +runs Zeek 8.2 with the [zeek-tds](https://github.com/dgunter/zeek-tds) +package in Docker (built from `TDS Analysis/zeek/Dockerfile` on first run, +which takes a few minutes) and finds the calls the packet-by-packet approach +missed. Docker is the only extra requirement. The 2017 notebooks split each log line on tabs and count values in dictionaries. They were written for Python 2.7 and have been ported to run on @@ -57,7 +60,8 @@ relative path. To execute everything from the command line, the way CI does: uv run jupyter nbconvert --to notebook --execute --inplace "Bro HTTP Log Analysis/"*.ipynb "TDS Analysis/"*.ipynb ``` -`uv run ruff check .` lints the code cells. The GitHub Actions workflow runs +`uv run ruff check .` lints the code cells. The Zeek notebook needs Docker; it +builds the `zeek-tds:0.2.0` image on first run. The GitHub Actions workflow runs the lint and executes every notebook on each push and pull request, and fails if any cell errors or writes to stderr. diff --git a/TDS Analysis/tds-zeek.ipynb b/TDS Analysis/tds-zeek.ipynb new file mode 100644 index 0000000..bd20f23 --- /dev/null +++ b/TDS Analysis/tds-zeek.ipynb @@ -0,0 +1,1257 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "23c16741", + "metadata": {}, + "source": [ + "# Examining MS SQL (TDS) traffic with Zeek\n", + "\n", + "This is the hunt from [*Threat Hunting with Python Part 4: Examining Microsoft SQL Based Historian Traffic*](../docs/2018-03-threat-hunting-with-python-part-4-mssql-historian.md), redone with Zeek instead of tshark. The original notebook in this folder, `tds.ipynb`, drives tshark through pyshark and picks fields out of each packet; it still runs and is kept as written.\n", + "\n", + "Here the same capture goes through Zeek with the [zeek-tds](https://github.com/dgunter/zeek-tds) analyzer, which understands the Tabular Data Stream protocol end to end: it reassembles messages across packets, resolves the numbered procedures drivers use for prepared statements, renders every parameter value by type, and writes logs that join to the rest of Zeek's output. [ParseZeekLogs](https://github.com/dgunter/ParseZeekLogs) then reads those logs into pandas. Zeek runs in Docker so nothing but Docker is needed on the machine; the same image is what you would run on a sensor." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "f8619977", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:48.469765Z", + "iopub.status.busy": "2026-09-06T01:34:48.469609Z", + "iopub.status.idle": "2026-09-06T01:34:48.731276Z", + "shell.execute_reply": "2026-09-06T01:34:48.730722Z" + } + }, + "outputs": [], + "source": [ + "import shutil\n", + "import subprocess\n", + "import tempfile\n", + "from pathlib import Path\n", + "\n", + "import pandas as pd\n", + "from parsezeeklogs import ZeekLog\n", + "\n", + "pd.set_option(\"display.max_colwidth\", 120)\n", + "pd.set_option(\"display.width\", 200)\n", + "\n", + "PCAP = Path(\"ms-sql-tds-rpc-requests.pcap\").resolve()\n", + "IMAGE = \"zeek-tds:0.2.0\"" + ] + }, + { + "cell_type": "markdown", + "id": "71975baa", + "metadata": {}, + "source": [ + "## Run Zeek over the capture\n", + "\n", + "The image is Zeek 8.2 with the zeek-tds package installed; `zeek/Dockerfile` builds it in a few minutes if it is not already present." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "e999596f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:48.732476Z", + "iopub.status.busy": "2026-09-06T01:34:48.732380Z", + "iopub.status.idle": "2026-09-06T01:34:49.244404Z", + "shell.execute_reply": "2026-09-06T01:34:49.243890Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['conn.log',\n", + " 'packet_filter.log',\n", + " 'tds.log',\n", + " 'tds_rpc.log',\n", + " 'tds_sql_batch.log']" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def ensure_image():\n", + " if subprocess.run([\"docker\", \"image\", \"inspect\", IMAGE], capture_output=True).returncode != 0:\n", + " subprocess.run(\n", + " [\"docker\", \"build\", \"-q\", \"-t\", IMAGE, str(Path(\"zeek\").resolve())], check=True\n", + " )\n", + "\n", + "\n", + "def run_zeek(pcap: Path) -> Path:\n", + " out = Path(tempfile.mkdtemp(prefix=\"zeek-tds-\"))\n", + " subprocess.run(\n", + " [\n", + " \"docker\",\n", + " \"run\",\n", + " \"--rm\",\n", + " \"-v\",\n", + " f\"{pcap.parent}:/pcap:ro\",\n", + " \"-v\",\n", + " f\"{out}:/data\",\n", + " \"-u\",\n", + " \"0\",\n", + " IMAGE,\n", + " \"-r\",\n", + " f\"/pcap/{pcap.name}\",\n", + " ],\n", + " check=True,\n", + " capture_output=True,\n", + " )\n", + " return out\n", + "\n", + "\n", + "ensure_image()\n", + "logs = run_zeek(PCAP)\n", + "sorted(p.name for p in logs.glob(\"*.log\"))" + ] + }, + { + "cell_type": "markdown", + "id": "07902a4c", + "metadata": {}, + "source": [ + "## Load the logs\n", + "\n", + "Every Zeek log becomes a typed DataFrame. The `uid` column is the connection identifier that ties the TDS logs to `conn.log` and to each other." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "a724be57", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.245740Z", + "iopub.status.busy": "2026-09-06T01:34:49.245568Z", + "iopub.status.idle": "2026-09-06T01:34:49.263052Z", + "shell.execute_reply": "2026-09-06T01:34:49.262623Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "12 connections, 29 TDS messages, 3 SQL batches, 17 procedure calls\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
tsuidid.orig_hid.orig_pid.resp_hid.resp_pserviceorig_bytesresp_bytes
02009-04-28 00:18:37.888015032+00:00CMV1YtcGQa28Aw3Li10.111.111.111111110.0.0.11433None482.0392.0
12009-12-02 14:00:00.004436970+00:00CyMGgL2gwVE3UoRR6410.111.111.111222210.0.0.11433None44.017.0
22009-12-02 14:00:00.022561073+00:00CKrjw32unPYVCZru3h10.111.111.111333310.0.0.11433NoneNaNNaN
32009-12-02 14:00:00.033700943+00:00C2FSPlZyZ8faGM5lk10.111.111.111444410.0.0.11433NoneNaNNaN
42009-12-02 14:00:00.716203928+00:00CxgidU3ajiUBltwFG410.111.111.111555510.0.0.11433None994.01143.0
52009-12-02 14:01:14.884130955+00:00CElNO63RG0vn3FqZne10.111.111.111666610.0.0.11433None8339.00.0
62009-12-02 14:01:17.536189079+00:00CIe7Xt2ea3S98hMYR610.111.111.111777710.0.0.11433NoneNaNNaN
72009-12-02 14:01:22.456089973+00:00CAVjrH15Sb48LKvXMj10.111.111.111888810.0.0.11433NoneNaNNaN
82010-07-02 11:00:44.666074991+00:00CLajdH1f5ARDC3FFCc10.111.111.1113333310.0.0.11433NoneNaNNaN
92010-07-02 11:00:44.650715113+00:00ChlDVK3XDEu7tWz8O710.111.111.1112222210.0.0.11433NoneNaNNaN
102010-07-02 11:00:44.614485025+00:00CQkAdl4KBxSHdq8Xxc10.111.111.1111111110.0.0.11433NoneNaNNaN
112010-07-02 11:00:44.584976912+00:00C730H12lAYINwsHsO310.111.111.111999910.0.0.11433NoneNaNNaN
\n", + "
" + ], + "text/plain": [ + " ts uid id.orig_h id.orig_p id.resp_h id.resp_p service orig_bytes resp_bytes\n", + "0 2009-04-28 00:18:37.888015032+00:00 CMV1YtcGQa28Aw3Li 10.111.111.111 1111 10.0.0.1 1433 None 482.0 392.0\n", + "1 2009-12-02 14:00:00.004436970+00:00 CyMGgL2gwVE3UoRR64 10.111.111.111 2222 10.0.0.1 1433 None 44.0 17.0\n", + "2 2009-12-02 14:00:00.022561073+00:00 CKrjw32unPYVCZru3h 10.111.111.111 3333 10.0.0.1 1433 None NaN NaN\n", + "3 2009-12-02 14:00:00.033700943+00:00 C2FSPlZyZ8faGM5lk 10.111.111.111 4444 10.0.0.1 1433 None NaN NaN\n", + "4 2009-12-02 14:00:00.716203928+00:00 CxgidU3ajiUBltwFG4 10.111.111.111 5555 10.0.0.1 1433 None 994.0 1143.0\n", + "5 2009-12-02 14:01:14.884130955+00:00 CElNO63RG0vn3FqZne 10.111.111.111 6666 10.0.0.1 1433 None 8339.0 0.0\n", + "6 2009-12-02 14:01:17.536189079+00:00 CIe7Xt2ea3S98hMYR6 10.111.111.111 7777 10.0.0.1 1433 None NaN NaN\n", + "7 2009-12-02 14:01:22.456089973+00:00 CAVjrH15Sb48LKvXMj 10.111.111.111 8888 10.0.0.1 1433 None NaN NaN\n", + "8 2010-07-02 11:00:44.666074991+00:00 CLajdH1f5ARDC3FFCc 10.111.111.111 33333 10.0.0.1 1433 None NaN NaN\n", + "9 2010-07-02 11:00:44.650715113+00:00 ChlDVK3XDEu7tWz8O7 10.111.111.111 22222 10.0.0.1 1433 None NaN NaN\n", + "10 2010-07-02 11:00:44.614485025+00:00 CQkAdl4KBxSHdq8Xxc 10.111.111.111 11111 10.0.0.1 1433 None NaN NaN\n", + "11 2010-07-02 11:00:44.584976912+00:00 C730H12lAYINwsHsO3 10.111.111.111 9999 10.0.0.1 1433 None NaN NaN" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def load(name: str) -> pd.DataFrame:\n", + " with ZeekLog(logs / f\"{name}.log\") as log:\n", + " df = pd.DataFrame(log)\n", + " if \"ts\" in df:\n", + " df[\"ts\"] = pd.to_datetime(df[\"ts\"], unit=\"s\", utc=True)\n", + " return df\n", + "\n", + "\n", + "conn = load(\"conn\")\n", + "tds = load(\"tds\")\n", + "batches = load(\"tds_sql_batch\")\n", + "rpc = load(\"tds_rpc\")\n", + "print(\n", + " f\"{len(conn)} connections, {len(tds)} TDS messages, \"\n", + " f\"{len(batches)} SQL batches, {len(rpc)} procedure calls\"\n", + ")\n", + "conn[\n", + " [\n", + " \"ts\",\n", + " \"uid\",\n", + " \"id.orig_h\",\n", + " \"id.orig_p\",\n", + " \"id.resp_h\",\n", + " \"id.resp_p\",\n", + " \"service\",\n", + " \"orig_bytes\",\n", + " \"resp_bytes\",\n", + " ]\n", + "]" + ] + }, + { + "cell_type": "markdown", + "id": "fac720a3", + "metadata": {}, + "source": [ + "## What kinds of messages, and which way\n", + "\n", + "`tds.log` is the skeleton: one line per message after reassembly. The original notebook counted TDS packet types; here the counts are per message, and responses carry how many rows they returned." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "94d82e3b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.264194Z", + "iopub.status.busy": "2026-09-06T01:34:49.264122Z", + "iopub.status.idle": "2026-09-06T01:34:49.272742Z", + "shell.execute_reply": "2026-09-06T01:34:49.272395Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
messagesbytespacketsrows
directionmsg_type
clientrpc1612030170.0
sql_batch340030.0
servertabular_result101472107.0
\n", + "
" + ], + "text/plain": [ + " messages bytes packets rows\n", + "direction msg_type \n", + "client rpc 16 12030 17 0.0\n", + " sql_batch 3 400 3 0.0\n", + "server tabular_result 10 1472 10 7.0" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tds[\"direction\"] = tds[\"is_orig\"].map({True: \"client\", False: \"server\"})\n", + "tds.groupby([\"direction\", \"msg_type\"]).agg(\n", + " messages=(\"uid\", \"size\"), bytes=(\"len\", \"sum\"), packets=(\"packets\", \"sum\"), rows=(\"rows\", \"sum\")\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "4ff1b22e", + "metadata": {}, + "source": [ + "## What they typed: SQL batches" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "66f58339", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.273809Z", + "iopub.status.busy": "2026-09-06T01:34:49.273741Z", + "iopub.status.idle": "2026-09-06T01:34:49.277094Z", + "shell.execute_reply": "2026-09-06T01:34:49.276809Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
tsid.orig_ptransaction_descriptorquery
02009-04-28 00:18:37.888015032+00:0011110set transaction isolation level  read committed  set implicit_transactions off
12009-12-02 14:00:00.004436970+00:0022220COMMIT TRANSACTION
22009-12-02 14:00:00.716203928+00:0055550set transaction isolation level  read committed  set implicit_transactions off
\n", + "
" + ], + "text/plain": [ + " ts id.orig_p transaction_descriptor query\n", + "0 2009-04-28 00:18:37.888015032+00:00 1111 0 set transaction isolation level read committed set implicit_transactions off \n", + "1 2009-12-02 14:00:00.004436970+00:00 2222 0 COMMIT TRANSACTION\n", + "2 2009-12-02 14:00:00.716203928+00:00 5555 0 set transaction isolation level read committed set implicit_transactions off " + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "batches[[\"ts\", \"id.orig_p\", \"transaction_descriptor\", \"query\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "f81f7114", + "metadata": {}, + "source": [ + "## What they called: procedures\n", + "\n", + "This is where the 2018 analysis ran out of road. Drivers ship prepared statements as calls to `sp_prepexec` and `sp_execute`, addressed by number on the wire; the original notebook mapped the numbers to names by hand and never saw the SQL inside them. Zeek resolves the procedure, lifts the SQL text into `statement`, and keeps the handle that ties an `sp_execute` back to the `sp_prepexec` that prepared it." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "7c117379", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.278195Z", + "iopub.status.busy": "2026-09-06T01:34:49.278134Z", + "iopub.status.idle": "2026-09-06T01:34:49.280841Z", + "shell.execute_reply": "2026-09-06T01:34:49.280497Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "procedure\n", + "sp_prepexec 5\n", + "sp_execute 3\n", + "proc_GetMyExampleTableSampleMetaData 2\n", + "p_GetBogusData 1\n", + "sp_executesql 1\n", + "p_SaveExample 1\n", + "p_SetBogusSample 1\n", + "p_GetMyExampleTableRowCount 1\n", + "dbo.proc_GetMySampleDataItems 1\n", + "proc_FetchMyExampleData 1\n", + "Name: count, dtype: int64" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rpc[\"procedure\"].value_counts()" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "4d14add7", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.281806Z", + "iopub.status.busy": "2026-09-06T01:34:49.281751Z", + "iopub.status.idle": "2026-09-06T01:34:49.285810Z", + "shell.execute_reply": "2026-09-06T01:34:49.285347Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
tsid.orig_pprocedureproc_idhandlestatement
02009-04-28 00:18:37.918653011+00:001111sp_prepexec13.00select * from test_table_1 where name = @P0 and id = @P1
12009-12-02 14:00:00.747371912+00:005555sp_prepexec13.00create table newsyb (column1 char(30) not null, column2 char(30) null,column3 char(30) null)
22009-12-02 14:00:01.226156950+00:005555sp_prepexec13.00insert INTO newsyb (column1, column2, column3) VALUES ('first', 'second', 'third')
32009-12-02 14:00:01.711920977+00:005555sp_execute12.02NaN
42009-12-02 14:00:01.711920977+00:005555sp_execute12.02NaN
52009-12-02 14:00:03.727473021+00:005555sp_prepexec13.00select * from newsyb
62009-12-02 14:00:05.827626944+00:005555sp_execute12.03NaN
72009-12-02 14:00:07.864718914+00:005555sp_prepexec13.00drop table newsyb
82009-12-02 14:00:00.022561073+00:003333p_GetBogusDataNaNNaNNaN
92009-12-02 14:00:00.033700943+00:004444sp_executesqlNaNNaNSELECT TOP 88 [dbo].[MyExampleTable].[ID], [dbo].[MyExampleTable].[EntityID], [dbo].[MyExampleTable].[EntityHistoryI...
102009-12-02 14:01:14.884730101+00:006666p_SaveExampleNaNNaNNaN
112009-12-02 14:01:17.536189079+00:007777p_SetBogusSampleNaNNaNNaN
122009-12-02 14:01:22.456089973+00:008888p_GetMyExampleTableRowCountNaNNaNNaN
132010-07-02 11:00:44.666074991+00:0033333dbo.proc_GetMySampleDataItemsNaNNaNNaN
142010-07-02 11:00:44.650715113+00:0022222proc_FetchMyExampleDataNaNNaNNaN
152010-07-02 11:00:44.614485025+00:0011111proc_GetMyExampleTableSampleMetaDataNaNNaNNaN
162010-07-02 11:00:44.584976912+00:009999proc_GetMyExampleTableSampleMetaDataNaNNaNNaN
\n", + "
" + ], + "text/plain": [ + " ts id.orig_p procedure proc_id handle \\\n", + "0 2009-04-28 00:18:37.918653011+00:00 1111 sp_prepexec 13.0 0 \n", + "1 2009-12-02 14:00:00.747371912+00:00 5555 sp_prepexec 13.0 0 \n", + "2 2009-12-02 14:00:01.226156950+00:00 5555 sp_prepexec 13.0 0 \n", + "3 2009-12-02 14:00:01.711920977+00:00 5555 sp_execute 12.0 2 \n", + "4 2009-12-02 14:00:01.711920977+00:00 5555 sp_execute 12.0 2 \n", + "5 2009-12-02 14:00:03.727473021+00:00 5555 sp_prepexec 13.0 0 \n", + "6 2009-12-02 14:00:05.827626944+00:00 5555 sp_execute 12.0 3 \n", + "7 2009-12-02 14:00:07.864718914+00:00 5555 sp_prepexec 13.0 0 \n", + "8 2009-12-02 14:00:00.022561073+00:00 3333 p_GetBogusData NaN NaN \n", + "9 2009-12-02 14:00:00.033700943+00:00 4444 sp_executesql NaN NaN \n", + "10 2009-12-02 14:01:14.884730101+00:00 6666 p_SaveExample NaN NaN \n", + "11 2009-12-02 14:01:17.536189079+00:00 7777 p_SetBogusSample NaN NaN \n", + "12 2009-12-02 14:01:22.456089973+00:00 8888 p_GetMyExampleTableRowCount NaN NaN \n", + "13 2010-07-02 11:00:44.666074991+00:00 33333 dbo.proc_GetMySampleDataItems NaN NaN \n", + "14 2010-07-02 11:00:44.650715113+00:00 22222 proc_FetchMyExampleData NaN NaN \n", + "15 2010-07-02 11:00:44.614485025+00:00 11111 proc_GetMyExampleTableSampleMetaData NaN NaN \n", + "16 2010-07-02 11:00:44.584976912+00:00 9999 proc_GetMyExampleTableSampleMetaData NaN NaN \n", + "\n", + " statement \n", + "0 select * from test_table_1 where name = @P0 and id = @P1 \n", + "1 create table newsyb (column1 char(30) not null, column2 char(30) null,column3 char(30) null) \n", + "2 insert INTO newsyb (column1, column2, column3) VALUES ('first', 'second', 'third') \n", + "3 NaN \n", + "4 NaN \n", + "5 select * from newsyb \n", + "6 NaN \n", + "7 drop table newsyb \n", + "8 NaN \n", + "9 SELECT TOP 88 [dbo].[MyExampleTable].[ID], [dbo].[MyExampleTable].[EntityID], [dbo].[MyExampleTable].[EntityHistoryI... \n", + "10 NaN \n", + "11 NaN \n", + "12 NaN \n", + "13 NaN \n", + "14 NaN \n", + "15 NaN \n", + "16 NaN " + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rpc[[\"ts\", \"id.orig_p\", \"procedure\", \"proc_id\", \"handle\", \"statement\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "4fedea56", + "metadata": {}, + "source": [ + "Every parameter is rendered by type, so the historian's own procedures show what they were asked for:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "183035c3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.287353Z", + "iopub.status.busy": "2026-09-06T01:34:49.287265Z", + "iopub.status.idle": "2026-09-06T01:34:49.290685Z", + "shell.execute_reply": "2026-09-06T01:34:49.290119Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "p_GetBogusData\n", + " @SearchType intn(1) = 1\n", + " @MaxWaitTimeInSeconds intn(4) = 0\n", + " @ProcessNegativeAck intn(1) = 0\n", + "p_SaveExample\n", + " @LongParam nvarchar(max) = Studenckie Koło Przewodników Turystycznych w Gdańsku\n", + "zaprasza na:\n", + "XXXV Nocne Marsze\n", + " @Operation intn(4) = 1\n", + "p_SetBogusSample\n", + " @BogusDetailsID intn(8) = 74565\n", + " @BogusStatusID intn(8) = 5\n", + " @ResultCode nvarchar(1) = NULL\n", + " @ResultMsg nvarchar(1) = NULL\n", + " @ErrorCode nvarchar(1) = NULL\n", + " @ErrorMsg nvarchar(1) = NULL\n", + " @ExampleBogusGeneratedID nvarchar(1) = NULL\n", + " @ExampleType intn(4) = 1\n", + "p_GetMyExampleTableRowCount\n", + "dbo.proc_GetMySampleDataItems\n", + " @SampleItemId uniqueidentifier = 4EC31A66-A214-4853-A77E-E7060FFFFF07\n", + " @DataId nvarchar(1) = NULL\n", + " @DataItemType uniqueidentifier = BCB9459B-83A8-4564-B1D3-E9E198478F4E\n", + " @TableId nvarchar(1) = NULL\n", + " @MaxFetchSize intn(4) = 100\n", + " @SomeOtherSampleId intn(4) = 0\n", + "proc_FetchMyExampleData\n", + " @p1 uniqueidentifier = 67452301-AB89-EFCD-0123-456789ABCDEF\n", + " @p2 nvarchar(0) = \n", + " @p3 nvarchar(5) = BOGUS\n", + " @p4 bitn = false\n", + " @p5 datetimen(8) = 1899-12-30 00:00:00.000\n", + " @p6 intn(4) = 0\n", + " @p7 intn(4) = 0\n", + " @p8 intn(4) = NULL\n", + " @p9 uniqueidentifier = NULL\n", + " @p10 bitn = NULL\n", + " @p11 intn(1) = 0\n", + " @p12 varbinary(28) = 0x0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF01234567\n", + " @p13 intn(4) = NULL\n", + " @p14 intn(1) = 1\n", + " @p15 intn(4) = 5242880\n", + " @p16 intn(8) = 45\n", + " @p17 intn(1) = 1\n", + " @p18 bitn = false\n", + " @p19 nvarchar(1) = NULL\n", + " @p20 intn(1) output = NULL\n", + "proc_GetMyExampleTableSampleMetaData\n", + " @p1 uniqueidentifier = 33221100-5544-7766-8899-AABBCCDDEEFF\n", + " @p2 null = NULL\n", + " @p3 nvarchar(5) = Bogus\n", + " @p4 null = NULL\n", + " @p5 intn(4) = 1\n", + " @p6 intn(8) = 45\n", + " @p7 varbinary(28) = 0x0123456789ABCDEFEDCBA9876543210123456789ABCDEFEDCBA98765\n", + " @p8 intn(4) = 18\n", + "proc_GetMyExampleTableSampleMetaData\n", + " @p1 uniqueidentifier = 00112233-4455-6677-8899-AABBCCDDEEFF\n", + " @p2 null = NULL\n", + " @p3 nvarchar(0) = \n", + " @p4 varchar(36) = ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghij\n", + " @p5 intn(4) = 1\n", + " @p6 intn(8) = 45\n", + " @p7 varbinary(12) = 0x0123456789ABCDEFFEDCBA98\n", + " @p8 intn(4) = 108\n" + ] + } + ], + "source": [ + "for _, row in rpc[~rpc[\"procedure\"].str.startswith(\"sp_\")].iterrows():\n", + " print(f\"{row['procedure']}\")\n", + " for p in row[\"parameters\"]:\n", + " print(f\" {p[:110]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "8508dd31", + "metadata": {}, + "source": [ + "## The questions from the post\n", + "\n", + "The post looked for two specific calls. Both are a filter away, and now the parameters come with them." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "4adbf596", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.292380Z", + "iopub.status.busy": "2026-09-06T01:34:49.292303Z", + "iopub.status.idle": "2026-09-06T01:34:49.295962Z", + "shell.execute_reply": "2026-09-06T01:34:49.295547Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
tsid.orig_hid.resp_hprocedureparameters
82009-12-02 14:00:00.022561073+00:0010.111.111.11110.0.0.1p_GetBogusData[@SearchType intn(1) = 1, @MaxWaitTimeInSeconds intn(4) = 0, @ProcessNegativeAck intn(1) = 0]
\n", + "
" + ], + "text/plain": [ + " ts id.orig_h id.resp_h procedure parameters\n", + "8 2009-12-02 14:00:00.022561073+00:00 10.111.111.111 10.0.0.1 p_GetBogusData [@SearchType intn(1) = 1, @MaxWaitTimeInSeconds intn(4) = 0, @ProcessNegativeAck intn(1) = 0]" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rpc.loc[\n", + " rpc[\"procedure\"] == \"p_GetBogusData\",\n", + " [\"ts\", \"id.orig_h\", \"id.resp_h\", \"procedure\", \"parameters\"],\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "2cc04fe8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.297175Z", + "iopub.status.busy": "2026-09-06T01:34:49.297107Z", + "iopub.status.idle": "2026-09-06T01:34:49.301123Z", + "shell.execute_reply": "2026-09-06T01:34:49.300767Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
tsid.orig_hid.resp_hprocedurestatementparameters
92009-12-02 14:00:00.033700943+00:0010.111.111.11110.0.0.1sp_executesqlSELECT TOP 88 [dbo].[MyExampleTable].[ID], [dbo].[MyExampleTable].[EntityID], [dbo].[MyExampleTable].[EntityHistoryI...[@p1 nvarchar(467) = SELECT TOP 88 [dbo].[MyExampleTable].[ID], [dbo].[MyExampleTable].[EntityID], [dbo].[MyExampleT...
112009-12-02 14:01:17.536189079+00:0010.111.111.11110.0.0.1p_SetBogusSampleNaN[@BogusDetailsID intn(8) = 74565, @BogusStatusID intn(8) = 5, @ResultCode nvarchar(1) = NULL, @ResultMsg nvarchar(1)...
\n", + "
" + ], + "text/plain": [ + " ts id.orig_h id.resp_h procedure \\\n", + "9 2009-12-02 14:00:00.033700943+00:00 10.111.111.111 10.0.0.1 sp_executesql \n", + "11 2009-12-02 14:01:17.536189079+00:00 10.111.111.111 10.0.0.1 p_SetBogusSample \n", + "\n", + " statement \\\n", + "9 SELECT TOP 88 [dbo].[MyExampleTable].[ID], [dbo].[MyExampleTable].[EntityID], [dbo].[MyExampleTable].[EntityHistoryI... \n", + "11 NaN \n", + "\n", + " parameters \n", + "9 [@p1 nvarchar(467) = SELECT TOP 88 [dbo].[MyExampleTable].[ID], [dbo].[MyExampleTable].[EntityID], [dbo].[MyExampleT... \n", + "11 [@BogusDetailsID intn(8) = 74565, @BogusStatusID intn(8) = 5, @ResultCode nvarchar(1) = NULL, @ResultMsg nvarchar(1)... " + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rpc.loc[\n", + " rpc[\"procedure\"].isin([\"p_SetBogusSample\", \"sp_executesql\"]),\n", + " [\"ts\", \"id.orig_h\", \"id.resp_h\", \"procedure\", \"statement\", \"parameters\"],\n", + "]" + ] + }, + { + "cell_type": "markdown", + "id": "88094f5c", + "metadata": {}, + "source": [ + "## What tshark did not see\n", + "\n", + "The original notebook's pyshark pass found 8 named procedures and 7 numbered ones. Zeek logs 17 procedure calls from the same 16 RPC messages. The two differences are structural, not a matter of field extraction:\n", + "\n", + "- **Messages that span packets.** One RPC (`p_SaveExample`, with an 8 KB `nvarchar(max)` argument) arrives in two TDS packets. tshark dissects packet by packet and does not reassemble TDS messages, so it saw two RPC frames with no procedure in either. Zeek reassembles the message first.\n", + "- **Messages that hold several calls.** One 53-byte message carries two `sp_execute` calls separated by a batch flag. tshark reports the first procedure per packet; Zeek logs each call.\n", + "\n", + "Both cases are visible in the Zeek output: RPC messages with more than one packet, and timestamps at which more than one procedure call was logged." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "286a9849", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.302076Z", + "iopub.status.busy": "2026-09-06T01:34:49.302012Z", + "iopub.status.idle": "2026-09-06T01:34:49.306666Z", + "shell.execute_reply": "2026-09-06T01:34:49.306299Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "RPC messages spanning several packets:\n", + " ts id.orig_p len packets\n", + "2009-12-02 14:01:14.884730101+00:00 6666 8323 2\n", + "\n", + "Timestamps with more than one procedure call in one message:\n", + " ts id.orig_p procedure parameters\n", + "2009-12-02 14:00:01.711920977+00:00 5555 sp_execute [@p1 intn(4) = 2]\n", + "2009-12-02 14:00:01.711920977+00:00 5555 sp_execute [@p1 intn(4) = 2]\n" + ] + } + ], + "source": [ + "multi_packet = tds[(tds[\"msg_type\"] == \"rpc\") & (tds[\"packets\"] > 1)][\n", + " [\"ts\", \"id.orig_p\", \"len\", \"packets\"]\n", + "]\n", + "multi_call = rpc.groupby(\"ts\").size()\n", + "multi_call = multi_call[multi_call > 1]\n", + "print(\"RPC messages spanning several packets:\")\n", + "print(multi_packet.to_string(index=False))\n", + "print(\"\\nTimestamps with more than one procedure call in one message:\")\n", + "print(\n", + " rpc[rpc[\"ts\"].isin(multi_call.index)][[\"ts\", \"id.orig_p\", \"procedure\", \"parameters\"]].to_string(\n", + " index=False\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "1921857c", + "metadata": {}, + "source": [ + "## Where this goes next\n", + "\n", + "On a sensor the same analyzer runs continuously and adds a `tds_login.log` of who connected with which driver, a `tds_error.log` of failed logins and permission errors, and notices for brute force, dangerous procedures and unusual result sizes. The capture here predates TLS on the wire; on a modern network the analyzer hands encrypted sessions to Zeek's SSL analyzer and keeps the login negotiation." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "68482006", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:34:49.307764Z", + "iopub.status.busy": "2026-09-06T01:34:49.307661Z", + "iopub.status.idle": "2026-09-06T01:34:49.309773Z", + "shell.execute_reply": "2026-09-06T01:34:49.309422Z" + } + }, + "outputs": [], + "source": [ + "shutil.rmtree(logs, ignore_errors=True)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.14.7" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/TDS Analysis/zeek/Dockerfile b/TDS Analysis/zeek/Dockerfile new file mode 100644 index 0000000..13829bf --- /dev/null +++ b/TDS Analysis/zeek/Dockerfile @@ -0,0 +1,11 @@ +# Zeek with the zeek-tds package, for running the TDS notebook's analysis. +# Build once: docker build -t zeek-tds:0.2.0 "TDS Analysis/zeek" +FROM zeek/zeek:8.2.2 +ARG ZEEK_TDS_VERSION=v0.2.0 +RUN apt-get update -q \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y -q --no-install-recommends g++ cmake make git libpcap-dev libssl-dev \ + && zkg autoconfig --force \ + && zkg install --force --skiptests --version "${ZEEK_TDS_VERSION}" https://github.com/dgunter/zeek-tds \ + && apt-get purge -y -q g++ cmake make && apt-get autoremove -y -q && rm -rf /var/lib/apt/lists/* +WORKDIR /data +ENTRYPOINT ["zeek", "-C", "packages", "LogAscii::use_json=F"] diff --git a/docs/2026-09-revisiting-tds.md b/docs/2026-09-revisiting-tds.md new file mode 100644 index 0000000..dd1f37a --- /dev/null +++ b/docs/2026-09-revisiting-tds.md @@ -0,0 +1,126 @@ +# Revisiting TDS: from tshark to a Zeek analyzer + +*Draft, September 2026. Follows [Threat Hunting with Python Part 4: Examining Microsoft SQL Based Historian Traffic](2018-03-threat-hunting-with-python-part-4-mssql-historian.md) (2018).* + +In 2018 I wrote about pulling Microsoft SQL Server traffic apart to see what a +historian's clients were actually asking it. The tool was tshark, driven from a +Jupyter notebook through pyshark, and the approach was to walk every packet, +read the TDS fields Wireshark exposed, and drop them into a pandas frame. It +worked, and it still runs today; the notebook is in this repository, ported to +Python 3. + +It also had a ceiling I did not see at the time. Eight years later I went back +to the same capture with a different tool and found calls I had never known were +there. This post is about what was hiding, why a packet dissector cannot show +it, and what a protocol analyzer written for Zeek does instead. + +## What the 2018 notebook saw + +Tabular Data Stream is the protocol every SQL Server client speaks. It has a +small header and a body whose meaning depends on the message type: a SQL batch +is UTF-16 text, a remote procedure call is a procedure name or number followed +by typed parameters, a response is a stream of tokens. Wireshark's dissector +handles all of that per packet, and the 2018 notebook picked out three things: +the TDS packet type, the query text of batches, and for RPCs the procedure name +or its number. + +That gave a table of 30 packets: three plain SQL batches, eight procedures +called by name, seven called by number, and a dozen server responses. The +numbers were the interesting part. `sp_prepexec` and `sp_execute` are how +drivers run prepared statements, so seven of the calls were application SQL +whose text I could not read from the columns I had extracted. I mapped the +numbers to names with a dictionary and moved on. + +## What was missing + +Two things, both structural. + +The first is reassembly. One of the calls in the capture, `p_SaveExample`, +carries an 8 KB `nvarchar(max)` argument and arrives in two TDS packets. +Wireshark dissects each packet on its own and does not reassemble TDS +messages, so it showed two RPC frames with no procedure name in either. The +notebook counted them as blanks. + +The second is that a TDS message can hold more than one call. A 53-byte RPC +message in the capture carries two `sp_execute` calls separated by a batch +flag. Wireshark reports the first procedure in a packet; the second never +appeared. + +Neither is a bug in tshark. A packet dissector shows you packets. The things I +wanted to count were messages and calls, and those are one layer up. + +## Doing it in Zeek + +Zeek reads a TCP connection as a stream and hands it to a protocol analyzer +that knows where messages begin and end. There was a TDS analyzer for Zeek, a +2019 BinPAC plugin, but it had fallen behind Zeek's build and misread the +numbered procedures, so I wrote a new one in Spicy, Zeek's current parser +language: [zeek-tds](https://github.com/dgunter/zeek-tds). + +It reassembles messages across packets and across MARS sessions, resolves the +numbered procedures, renders every parameter by type, follows the response +tokens, and hands sessions that negotiate encryption to Zeek's SSL analyzer. It +writes logs rather than a packet list: one line per login, per batch, per +procedure call, per server error, per result set. Every line carries Zeek's +connection identifier, so a SQL statement sits next to the connection it +travelled in, the TLS certificate if there was one, and the NTLM exchange if +the login was a Windows one. + +Run over the 2009 capture, `tds_rpc.log` has seventeen procedure calls: the +eight named ones, the seven numbered ones, the call that spanned two packets, +and the second call in the double message. For the prepared statements it +lifts the SQL text out of the parameter that carries it: + +``` +procedure sp_prepexec +statement select * from test_table_1 where name = @P0 and id = @P1 +handle 0 + +procedure sp_execute +handle 2 + +procedure sp_prepexec +statement create table newsyb (column1 char(30) not null, column2 char(30) null,column3 char(30) null) +``` + +And for the historian's own procedures it shows what they were asked for: + +``` +p_GetBogusData + @SearchType intn(1) = 1 + @MaxWaitTimeInSeconds intn(4) = 0 + @ProcessNegativeAck intn(1) = 0 + +proc_GetMyExampleTableSampleMetaData + @p1 uniqueidentifier = 00112233-4455-6677-8899-AABBCCDDEEFF + @p4 varchar(36) = ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghij + @p7 varbinary(12) = 0x0123456789ABCDEFFEDCBA98 +``` + +The notebook that does this, `tds-zeek.ipynb`, is next to the 2018 one. It +runs Zeek in Docker over the capture, reads the logs with ParseZeekLogs, and +reproduces the post's analysis in a handful of pandas calls. + +## Why it matters for a hunt + +The point of the 2018 post was that a historian's SQL traffic tells you what +the plant's systems are doing, and that anything outside the normal set of +procedures is worth a look. That is still true, and the analyzer is built for +it. On a sensor it answers, continuously and for every session: who connected, +from which host and with which driver; whether the login succeeded, and if +not why; what they ran and what they called, with the SQL inside prepared +statements; what the server refused; how much came back. It raises notices for +login brute force, `xp_cmdshell` and its relatives, PRELOGIN scanning, and +results far larger than normal, and a coverage table maps each log and notice +to ATT&CK and ICS ATT&CK techniques. + +The honest limit is encryption. Where client and server agree on TLS, only the +login negotiation and the certificate are visible. The plaintext decoding +matters most exactly where the 2018 post lived: control-system networks where +the historian's clients still speak in the clear. + +## Try it + +- The notebook: [TDS Analysis/tds-zeek.ipynb](../TDS%20Analysis/tds-zeek.ipynb), and the 2018 original beside it. +- The analyzer: [github.com/dgunter/zeek-tds](https://github.com/dgunter/zeek-tds), `zkg install zeek-tds` on Zeek 7 or 8. +- The log reader: [github.com/dgunter/ParseZeekLogs](https://github.com/dgunter/ParseZeekLogs). From c1309ed2ac173a7bdac06915fe8074564f6d495a Mon Sep 17 00:00:00 2001 From: dgunter Date: Sat, 5 Sep 2026 20:39:32 -0500 Subject: [PATCH 2/4] Keep the Revisiting TDS draft out of the repository for now --- .gitignore | 3 + README.md | 2 +- docs/2026-09-revisiting-tds.md | 126 --------------------------------- 3 files changed, 4 insertions(+), 127 deletions(-) delete mode 100644 docs/2026-09-revisiting-tds.md diff --git a/.gitignore b/.gitignore index baa960d..dc3d51e 100644 --- a/.gitignore +++ b/.gitignore @@ -96,3 +96,6 @@ ENV/ # Notebook execution artefacts /tmp/ + +# Unpublished drafts +docs/*-draft.md diff --git a/README.md b/README.md index b19a618..9c04c3b 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ their images so the code has its narrative next to it. | [Detecting Nmap Behavior with Bro HTTP Logs](Bro%20HTTP%20Log%20Analysis/Detecting%20Nmap%20Behavior%20with%20Bro%20HTTP%20Logs.ipynb) | The same log as a time series: user agents and status codes per minute, with the scan window standing out once the busy browser is removed | [Part 2: Detecting Nmap Behavior with Bro HTTP Logs](docs/2017-11-threat-hunting-with-python-part-2-nmap-bro-http.md) (2017-11) | | [Detecting Nmap Behavior with ParseZeekLogs](Bro%20HTTP%20Log%20Analysis/Detecting%20Nmap%20Behavior%20with%20ParseZeekLogs.ipynb) | Both hunts above redone with [ParseZeekLogs](https://github.com/dgunter/ParseZeekLogs) and pandas, added in 2026 | same two posts | | [tds](TDS%20Analysis/tds.ipynb) | Pull TDS packet types, SQL batches and remote procedure calls out of a pcap with pyshark and look at what a historian's clients actually run | [Part 4: Examining Microsoft SQL Based Historian Traffic](docs/2018-03-threat-hunting-with-python-part-4-mssql-historian.md) (2018-03) | -| [tds-zeek](TDS%20Analysis/tds-zeek.ipynb) | The same capture through Zeek with the [zeek-tds](https://github.com/dgunter/zeek-tds) analyzer, added in 2026: reassembled messages, every procedure call with its parameters and prepared-statement SQL, read into pandas with ParseZeekLogs | [Revisiting TDS](docs/2026-09-revisiting-tds.md) (draft) | +| [tds-zeek](TDS%20Analysis/tds-zeek.ipynb) | The same capture through Zeek with the [zeek-tds](https://github.com/dgunter/zeek-tds) analyzer, added in 2026: reassembled messages, every procedure call with its parameters and prepared-statement SQL, read into pandas with ParseZeekLogs | same post, revisited | [Part 3: Taming SMB](docs/2018-02-threat-hunting-with-python-part-3-taming-smb.md) (2018-02) is preserved too; its examples were shown inline rather than as a diff --git a/docs/2026-09-revisiting-tds.md b/docs/2026-09-revisiting-tds.md deleted file mode 100644 index dd1f37a..0000000 --- a/docs/2026-09-revisiting-tds.md +++ /dev/null @@ -1,126 +0,0 @@ -# Revisiting TDS: from tshark to a Zeek analyzer - -*Draft, September 2026. Follows [Threat Hunting with Python Part 4: Examining Microsoft SQL Based Historian Traffic](2018-03-threat-hunting-with-python-part-4-mssql-historian.md) (2018).* - -In 2018 I wrote about pulling Microsoft SQL Server traffic apart to see what a -historian's clients were actually asking it. The tool was tshark, driven from a -Jupyter notebook through pyshark, and the approach was to walk every packet, -read the TDS fields Wireshark exposed, and drop them into a pandas frame. It -worked, and it still runs today; the notebook is in this repository, ported to -Python 3. - -It also had a ceiling I did not see at the time. Eight years later I went back -to the same capture with a different tool and found calls I had never known were -there. This post is about what was hiding, why a packet dissector cannot show -it, and what a protocol analyzer written for Zeek does instead. - -## What the 2018 notebook saw - -Tabular Data Stream is the protocol every SQL Server client speaks. It has a -small header and a body whose meaning depends on the message type: a SQL batch -is UTF-16 text, a remote procedure call is a procedure name or number followed -by typed parameters, a response is a stream of tokens. Wireshark's dissector -handles all of that per packet, and the 2018 notebook picked out three things: -the TDS packet type, the query text of batches, and for RPCs the procedure name -or its number. - -That gave a table of 30 packets: three plain SQL batches, eight procedures -called by name, seven called by number, and a dozen server responses. The -numbers were the interesting part. `sp_prepexec` and `sp_execute` are how -drivers run prepared statements, so seven of the calls were application SQL -whose text I could not read from the columns I had extracted. I mapped the -numbers to names with a dictionary and moved on. - -## What was missing - -Two things, both structural. - -The first is reassembly. One of the calls in the capture, `p_SaveExample`, -carries an 8 KB `nvarchar(max)` argument and arrives in two TDS packets. -Wireshark dissects each packet on its own and does not reassemble TDS -messages, so it showed two RPC frames with no procedure name in either. The -notebook counted them as blanks. - -The second is that a TDS message can hold more than one call. A 53-byte RPC -message in the capture carries two `sp_execute` calls separated by a batch -flag. Wireshark reports the first procedure in a packet; the second never -appeared. - -Neither is a bug in tshark. A packet dissector shows you packets. The things I -wanted to count were messages and calls, and those are one layer up. - -## Doing it in Zeek - -Zeek reads a TCP connection as a stream and hands it to a protocol analyzer -that knows where messages begin and end. There was a TDS analyzer for Zeek, a -2019 BinPAC plugin, but it had fallen behind Zeek's build and misread the -numbered procedures, so I wrote a new one in Spicy, Zeek's current parser -language: [zeek-tds](https://github.com/dgunter/zeek-tds). - -It reassembles messages across packets and across MARS sessions, resolves the -numbered procedures, renders every parameter by type, follows the response -tokens, and hands sessions that negotiate encryption to Zeek's SSL analyzer. It -writes logs rather than a packet list: one line per login, per batch, per -procedure call, per server error, per result set. Every line carries Zeek's -connection identifier, so a SQL statement sits next to the connection it -travelled in, the TLS certificate if there was one, and the NTLM exchange if -the login was a Windows one. - -Run over the 2009 capture, `tds_rpc.log` has seventeen procedure calls: the -eight named ones, the seven numbered ones, the call that spanned two packets, -and the second call in the double message. For the prepared statements it -lifts the SQL text out of the parameter that carries it: - -``` -procedure sp_prepexec -statement select * from test_table_1 where name = @P0 and id = @P1 -handle 0 - -procedure sp_execute -handle 2 - -procedure sp_prepexec -statement create table newsyb (column1 char(30) not null, column2 char(30) null,column3 char(30) null) -``` - -And for the historian's own procedures it shows what they were asked for: - -``` -p_GetBogusData - @SearchType intn(1) = 1 - @MaxWaitTimeInSeconds intn(4) = 0 - @ProcessNegativeAck intn(1) = 0 - -proc_GetMyExampleTableSampleMetaData - @p1 uniqueidentifier = 00112233-4455-6677-8899-AABBCCDDEEFF - @p4 varchar(36) = ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghij - @p7 varbinary(12) = 0x0123456789ABCDEFFEDCBA98 -``` - -The notebook that does this, `tds-zeek.ipynb`, is next to the 2018 one. It -runs Zeek in Docker over the capture, reads the logs with ParseZeekLogs, and -reproduces the post's analysis in a handful of pandas calls. - -## Why it matters for a hunt - -The point of the 2018 post was that a historian's SQL traffic tells you what -the plant's systems are doing, and that anything outside the normal set of -procedures is worth a look. That is still true, and the analyzer is built for -it. On a sensor it answers, continuously and for every session: who connected, -from which host and with which driver; whether the login succeeded, and if -not why; what they ran and what they called, with the SQL inside prepared -statements; what the server refused; how much came back. It raises notices for -login brute force, `xp_cmdshell` and its relatives, PRELOGIN scanning, and -results far larger than normal, and a coverage table maps each log and notice -to ATT&CK and ICS ATT&CK techniques. - -The honest limit is encryption. Where client and server agree on TLS, only the -login negotiation and the certificate are visible. The plaintext decoding -matters most exactly where the 2018 post lived: control-system networks where -the historian's clients still speak in the clear. - -## Try it - -- The notebook: [TDS Analysis/tds-zeek.ipynb](../TDS%20Analysis/tds-zeek.ipynb), and the 2018 original beside it. -- The analyzer: [github.com/dgunter/zeek-tds](https://github.com/dgunter/zeek-tds), `zkg install zeek-tds` on Zeek 7 or 8. -- The log reader: [github.com/dgunter/ParseZeekLogs](https://github.com/dgunter/ParseZeekLogs). From 616ade6f9de9bb886e3840d61f5fc07f2c97fe2d Mon Sep 17 00:00:00 2001 From: dgunter Date: Sat, 5 Sep 2026 20:45:27 -0500 Subject: [PATCH 3/4] TDS notebook: show every zeek-tds log on the release corpus Fetches the small public captures from the zeek-tds v0.2.0 tag and walks tds_login, tds_error, ssl and ntlm under the same uid, MARS sessions in tds.log, the opt-in tds_result.log, ssrp.log and the notices. Co-Authored-By: Claude Fable 5.1 --- TDS Analysis/tds-zeek.ipynb | 1219 ++++++++++++++++++++++++++++++++--- 1 file changed, 1139 insertions(+), 80 deletions(-) diff --git a/TDS Analysis/tds-zeek.ipynb b/TDS Analysis/tds-zeek.ipynb index bd20f23..446aaf1 100644 --- a/TDS Analysis/tds-zeek.ipynb +++ b/TDS Analysis/tds-zeek.ipynb @@ -18,10 +18,10 @@ "id": "f8619977", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:48.469765Z", - "iopub.status.busy": "2026-09-06T01:34:48.469609Z", - "iopub.status.idle": "2026-09-06T01:34:48.731276Z", - "shell.execute_reply": "2026-09-06T01:34:48.730722Z" + "iopub.execute_input": "2026-09-06T01:45:07.826338Z", + "iopub.status.busy": "2026-09-06T01:45:07.826182Z", + "iopub.status.idle": "2026-09-06T01:45:08.064151Z", + "shell.execute_reply": "2026-09-06T01:45:08.063755Z" } }, "outputs": [], @@ -57,10 +57,10 @@ "id": "e999596f", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:48.732476Z", - "iopub.status.busy": "2026-09-06T01:34:48.732380Z", - "iopub.status.idle": "2026-09-06T01:34:49.244404Z", - "shell.execute_reply": "2026-09-06T01:34:49.243890Z" + "iopub.execute_input": "2026-09-06T01:45:08.065493Z", + "iopub.status.busy": "2026-09-06T01:45:08.065399Z", + "iopub.status.idle": "2026-09-06T01:45:08.508993Z", + "shell.execute_reply": "2026-09-06T01:45:08.508270Z" } }, "outputs": [ @@ -87,7 +87,7 @@ " )\n", "\n", "\n", - "def run_zeek(pcap: Path) -> Path:\n", + "def run_zeek(pcap: Path, *options: str) -> Path:\n", " out = Path(tempfile.mkdtemp(prefix=\"zeek-tds-\"))\n", " subprocess.run(\n", " [\n", @@ -103,6 +103,7 @@ " IMAGE,\n", " \"-r\",\n", " f\"/pcap/{pcap.name}\",\n", + " *options,\n", " ],\n", " check=True,\n", " capture_output=True,\n", @@ -131,10 +132,10 @@ "id": "a724be57", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.245740Z", - "iopub.status.busy": "2026-09-06T01:34:49.245568Z", - "iopub.status.idle": "2026-09-06T01:34:49.263052Z", - "shell.execute_reply": "2026-09-06T01:34:49.262623Z" + "iopub.execute_input": "2026-09-06T01:45:08.511082Z", + "iopub.status.busy": "2026-09-06T01:45:08.510984Z", + "iopub.status.idle": "2026-09-06T01:45:08.525881Z", + "shell.execute_reply": "2026-09-06T01:45:08.525487Z" } }, "outputs": [ @@ -181,7 +182,7 @@ " \n", " 0\n", " 2009-04-28 00:18:37.888015032+00:00\n", - " CMV1YtcGQa28Aw3Li\n", + " CFgl432RiV5QMf8kWl\n", " 10.111.111.111\n", " 1111\n", " 10.0.0.1\n", @@ -193,7 +194,7 @@ " \n", " 1\n", " 2009-12-02 14:00:00.004436970+00:00\n", - " CyMGgL2gwVE3UoRR64\n", + " CNYwCO2ekLU4aDsC65\n", " 10.111.111.111\n", " 2222\n", " 10.0.0.1\n", @@ -205,7 +206,7 @@ " \n", " 2\n", " 2009-12-02 14:00:00.022561073+00:00\n", - " CKrjw32unPYVCZru3h\n", + " CVNYx24sE1cSVevXz4\n", " 10.111.111.111\n", " 3333\n", " 10.0.0.1\n", @@ -217,7 +218,7 @@ " \n", " 3\n", " 2009-12-02 14:00:00.033700943+00:00\n", - " C2FSPlZyZ8faGM5lk\n", + " CvX7ZcsIuoRn5WDc8\n", " 10.111.111.111\n", " 4444\n", " 10.0.0.1\n", @@ -229,7 +230,7 @@ " \n", " 4\n", " 2009-12-02 14:00:00.716203928+00:00\n", - " CxgidU3ajiUBltwFG4\n", + " CkVlKs11iDPrCFBXl3\n", " 10.111.111.111\n", " 5555\n", " 10.0.0.1\n", @@ -241,7 +242,7 @@ " \n", " 5\n", " 2009-12-02 14:01:14.884130955+00:00\n", - " CElNO63RG0vn3FqZne\n", + " CRZiJOln0tcFoRmg\n", " 10.111.111.111\n", " 6666\n", " 10.0.0.1\n", @@ -253,7 +254,7 @@ " \n", " 6\n", " 2009-12-02 14:01:17.536189079+00:00\n", - " CIe7Xt2ea3S98hMYR6\n", + " CSKq7A1JndbWUM1xT1\n", " 10.111.111.111\n", " 7777\n", " 10.0.0.1\n", @@ -265,7 +266,7 @@ " \n", " 7\n", " 2009-12-02 14:01:22.456089973+00:00\n", - " CAVjrH15Sb48LKvXMj\n", + " CerKLf1w3qoFTVOCnb\n", " 10.111.111.111\n", " 8888\n", " 10.0.0.1\n", @@ -277,7 +278,7 @@ " \n", " 8\n", " 2010-07-02 11:00:44.666074991+00:00\n", - " CLajdH1f5ARDC3FFCc\n", + " CMrmlKeYGyZvEOqIe\n", " 10.111.111.111\n", " 33333\n", " 10.0.0.1\n", @@ -289,7 +290,7 @@ " \n", " 9\n", " 2010-07-02 11:00:44.650715113+00:00\n", - " ChlDVK3XDEu7tWz8O7\n", + " CHNGko1juvnYRD1XQh\n", " 10.111.111.111\n", " 22222\n", " 10.0.0.1\n", @@ -301,7 +302,7 @@ " \n", " 10\n", " 2010-07-02 11:00:44.614485025+00:00\n", - " CQkAdl4KBxSHdq8Xxc\n", + " C2O8qZ3wcBjl0P0hl5\n", " 10.111.111.111\n", " 11111\n", " 10.0.0.1\n", @@ -313,7 +314,7 @@ " \n", " 11\n", " 2010-07-02 11:00:44.584976912+00:00\n", - " C730H12lAYINwsHsO3\n", + " CR0JnB4dTciTHMUYq9\n", " 10.111.111.111\n", " 9999\n", " 10.0.0.1\n", @@ -328,18 +329,18 @@ ], "text/plain": [ " ts uid id.orig_h id.orig_p id.resp_h id.resp_p service orig_bytes resp_bytes\n", - "0 2009-04-28 00:18:37.888015032+00:00 CMV1YtcGQa28Aw3Li 10.111.111.111 1111 10.0.0.1 1433 None 482.0 392.0\n", - "1 2009-12-02 14:00:00.004436970+00:00 CyMGgL2gwVE3UoRR64 10.111.111.111 2222 10.0.0.1 1433 None 44.0 17.0\n", - "2 2009-12-02 14:00:00.022561073+00:00 CKrjw32unPYVCZru3h 10.111.111.111 3333 10.0.0.1 1433 None NaN NaN\n", - "3 2009-12-02 14:00:00.033700943+00:00 C2FSPlZyZ8faGM5lk 10.111.111.111 4444 10.0.0.1 1433 None NaN NaN\n", - "4 2009-12-02 14:00:00.716203928+00:00 CxgidU3ajiUBltwFG4 10.111.111.111 5555 10.0.0.1 1433 None 994.0 1143.0\n", - "5 2009-12-02 14:01:14.884130955+00:00 CElNO63RG0vn3FqZne 10.111.111.111 6666 10.0.0.1 1433 None 8339.0 0.0\n", - "6 2009-12-02 14:01:17.536189079+00:00 CIe7Xt2ea3S98hMYR6 10.111.111.111 7777 10.0.0.1 1433 None NaN NaN\n", - "7 2009-12-02 14:01:22.456089973+00:00 CAVjrH15Sb48LKvXMj 10.111.111.111 8888 10.0.0.1 1433 None NaN NaN\n", - "8 2010-07-02 11:00:44.666074991+00:00 CLajdH1f5ARDC3FFCc 10.111.111.111 33333 10.0.0.1 1433 None NaN NaN\n", - "9 2010-07-02 11:00:44.650715113+00:00 ChlDVK3XDEu7tWz8O7 10.111.111.111 22222 10.0.0.1 1433 None NaN NaN\n", - "10 2010-07-02 11:00:44.614485025+00:00 CQkAdl4KBxSHdq8Xxc 10.111.111.111 11111 10.0.0.1 1433 None NaN NaN\n", - "11 2010-07-02 11:00:44.584976912+00:00 C730H12lAYINwsHsO3 10.111.111.111 9999 10.0.0.1 1433 None NaN NaN" + "0 2009-04-28 00:18:37.888015032+00:00 CFgl432RiV5QMf8kWl 10.111.111.111 1111 10.0.0.1 1433 None 482.0 392.0\n", + "1 2009-12-02 14:00:00.004436970+00:00 CNYwCO2ekLU4aDsC65 10.111.111.111 2222 10.0.0.1 1433 None 44.0 17.0\n", + "2 2009-12-02 14:00:00.022561073+00:00 CVNYx24sE1cSVevXz4 10.111.111.111 3333 10.0.0.1 1433 None NaN NaN\n", + "3 2009-12-02 14:00:00.033700943+00:00 CvX7ZcsIuoRn5WDc8 10.111.111.111 4444 10.0.0.1 1433 None NaN NaN\n", + "4 2009-12-02 14:00:00.716203928+00:00 CkVlKs11iDPrCFBXl3 10.111.111.111 5555 10.0.0.1 1433 None 994.0 1143.0\n", + "5 2009-12-02 14:01:14.884130955+00:00 CRZiJOln0tcFoRmg 10.111.111.111 6666 10.0.0.1 1433 None 8339.0 0.0\n", + "6 2009-12-02 14:01:17.536189079+00:00 CSKq7A1JndbWUM1xT1 10.111.111.111 7777 10.0.0.1 1433 None NaN NaN\n", + "7 2009-12-02 14:01:22.456089973+00:00 CerKLf1w3qoFTVOCnb 10.111.111.111 8888 10.0.0.1 1433 None NaN NaN\n", + "8 2010-07-02 11:00:44.666074991+00:00 CMrmlKeYGyZvEOqIe 10.111.111.111 33333 10.0.0.1 1433 None NaN NaN\n", + "9 2010-07-02 11:00:44.650715113+00:00 CHNGko1juvnYRD1XQh 10.111.111.111 22222 10.0.0.1 1433 None NaN NaN\n", + "10 2010-07-02 11:00:44.614485025+00:00 C2O8qZ3wcBjl0P0hl5 10.111.111.111 11111 10.0.0.1 1433 None NaN NaN\n", + "11 2010-07-02 11:00:44.584976912+00:00 CR0JnB4dTciTHMUYq9 10.111.111.111 9999 10.0.0.1 1433 None NaN NaN" ] }, "execution_count": 3, @@ -348,8 +349,8 @@ } ], "source": [ - "def load(name: str) -> pd.DataFrame:\n", - " with ZeekLog(logs / f\"{name}.log\") as log:\n", + "def load(name: str, where: Path | None = None) -> pd.DataFrame:\n", + " with ZeekLog((where or logs) / f\"{name}.log\") as log:\n", " df = pd.DataFrame(log)\n", " if \"ts\" in df:\n", " df[\"ts\"] = pd.to_datetime(df[\"ts\"], unit=\"s\", utc=True)\n", @@ -395,10 +396,10 @@ "id": "94d82e3b", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.264194Z", - "iopub.status.busy": "2026-09-06T01:34:49.264122Z", - "iopub.status.idle": "2026-09-06T01:34:49.272742Z", - "shell.execute_reply": "2026-09-06T01:34:49.272395Z" + "iopub.execute_input": "2026-09-06T01:45:08.526918Z", + "iopub.status.busy": "2026-09-06T01:45:08.526861Z", + "iopub.status.idle": "2026-09-06T01:45:08.535530Z", + "shell.execute_reply": "2026-09-06T01:45:08.535064Z" } }, "outputs": [ @@ -500,10 +501,10 @@ "id": "66f58339", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.273809Z", - "iopub.status.busy": "2026-09-06T01:34:49.273741Z", - "iopub.status.idle": "2026-09-06T01:34:49.277094Z", - "shell.execute_reply": "2026-09-06T01:34:49.276809Z" + "iopub.execute_input": "2026-09-06T01:45:08.536711Z", + "iopub.status.busy": "2026-09-06T01:45:08.536637Z", + "iopub.status.idle": "2026-09-06T01:45:08.539657Z", + "shell.execute_reply": "2026-09-06T01:45:08.539320Z" } }, "outputs": [ @@ -592,10 +593,10 @@ "id": "7c117379", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.278195Z", - "iopub.status.busy": "2026-09-06T01:34:49.278134Z", - "iopub.status.idle": "2026-09-06T01:34:49.280841Z", - "shell.execute_reply": "2026-09-06T01:34:49.280497Z" + "iopub.execute_input": "2026-09-06T01:45:08.540894Z", + "iopub.status.busy": "2026-09-06T01:45:08.540834Z", + "iopub.status.idle": "2026-09-06T01:45:08.543684Z", + "shell.execute_reply": "2026-09-06T01:45:08.543249Z" } }, "outputs": [ @@ -631,10 +632,10 @@ "id": "4d14add7", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.281806Z", - "iopub.status.busy": "2026-09-06T01:34:49.281751Z", - "iopub.status.idle": "2026-09-06T01:34:49.285810Z", - "shell.execute_reply": "2026-09-06T01:34:49.285347Z" + "iopub.execute_input": "2026-09-06T01:45:08.544675Z", + "iopub.status.busy": "2026-09-06T01:45:08.544615Z", + "iopub.status.idle": "2026-09-06T01:45:08.548517Z", + "shell.execute_reply": "2026-09-06T01:45:08.548173Z" } }, "outputs": [ @@ -888,10 +889,10 @@ "id": "183035c3", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.287353Z", - "iopub.status.busy": "2026-09-06T01:34:49.287265Z", - "iopub.status.idle": "2026-09-06T01:34:49.290685Z", - "shell.execute_reply": "2026-09-06T01:34:49.290119Z" + "iopub.execute_input": "2026-09-06T01:45:08.549601Z", + "iopub.status.busy": "2026-09-06T01:45:08.549538Z", + "iopub.status.idle": "2026-09-06T01:45:08.553047Z", + "shell.execute_reply": "2026-09-06T01:45:08.552664Z" } }, "outputs": [ @@ -990,10 +991,10 @@ "id": "4adbf596", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.292380Z", - "iopub.status.busy": "2026-09-06T01:34:49.292303Z", - "iopub.status.idle": "2026-09-06T01:34:49.295962Z", - "shell.execute_reply": "2026-09-06T01:34:49.295547Z" + "iopub.execute_input": "2026-09-06T01:45:08.554748Z", + "iopub.status.busy": "2026-09-06T01:45:08.554642Z", + "iopub.status.idle": "2026-09-06T01:45:08.558671Z", + "shell.execute_reply": "2026-09-06T01:45:08.558133Z" } }, "outputs": [ @@ -1061,10 +1062,10 @@ "id": "2cc04fe8", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.297175Z", - "iopub.status.busy": "2026-09-06T01:34:49.297107Z", - "iopub.status.idle": "2026-09-06T01:34:49.301123Z", - "shell.execute_reply": "2026-09-06T01:34:49.300767Z" + "iopub.execute_input": "2026-09-06T01:45:08.559673Z", + "iopub.status.busy": "2026-09-06T01:45:08.559608Z", + "iopub.status.idle": "2026-09-06T01:45:08.563588Z", + "shell.execute_reply": "2026-09-06T01:45:08.563191Z" } }, "outputs": [ @@ -1167,10 +1168,10 @@ "id": "286a9849", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.302076Z", - "iopub.status.busy": "2026-09-06T01:34:49.302012Z", - "iopub.status.idle": "2026-09-06T01:34:49.306666Z", - "shell.execute_reply": "2026-09-06T01:34:49.306299Z" + "iopub.execute_input": "2026-09-06T01:45:08.564660Z", + "iopub.status.busy": "2026-09-06T01:45:08.564594Z", + "iopub.status.idle": "2026-09-06T01:45:08.569164Z", + "shell.execute_reply": "2026-09-06T01:45:08.568859Z" } }, "outputs": [ @@ -1207,29 +1208,1087 @@ }, { "cell_type": "markdown", - "id": "1921857c", + "id": "c90b6abf", + "metadata": {}, + "source": [ + "## The other logs, on today's traffic\n", + "\n", + "The 2009 capture has no logins in it, so three of the analyzer's logs stay empty. The zeek-tds test corpus, recorded against Azure SQL Edge (the SQL Server 2019 engine) with Python, FreeTDS and .NET clients, exercises the rest. The captures are small and public; this fetches them from the tagged release." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "a22c02b2", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:08.570178Z", + "iopub.status.busy": "2026-09-06T01:45:08.570108Z", + "iopub.status.idle": "2026-09-06T01:45:11.519036Z", + "shell.execute_reply": "2026-09-06T01:45:11.518153Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'sqledge-pytds-workload': ['tds.log',\n", + " 'tds_error.log',\n", + " 'tds_login.log',\n", + " 'tds_result.log',\n", + " 'tds_rpc.log',\n", + " 'tds_sql_batch.log'],\n", + " 'sqledge-pytds-failed-login': ['tds.log', 'tds_error.log', 'tds_login.log'],\n", + " 'sqledge-freetds': ['ssl.log',\n", + " 'tds.log',\n", + " 'tds_login.log',\n", + " 'tds_result.log',\n", + " 'tds_rpc.log',\n", + " 'tds_sql_batch.log',\n", + " 'x509.log'],\n", + " 'sqledge-pytds-ntlm': ['ntlm.log',\n", + " 'tds.log',\n", + " 'tds_error.log',\n", + " 'tds_login.log'],\n", + " 'sqledge-dotnet-mars': ['ssl.log',\n", + " 'tds.log',\n", + " 'tds_login.log',\n", + " 'tds_result.log',\n", + " 'tds_rpc.log',\n", + " 'tds_sql_batch.log',\n", + " 'x509.log'],\n", + " 'ssrp-browser': ['ssrp.log']}" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import urllib.request\n", + "\n", + "CORPUS = \"https://raw.githubusercontent.com/dgunter/zeek-tds/v0.2.0/testing/Traces\"\n", + "TRACES = [\n", + " \"sqledge-pytds-workload\",\n", + " \"sqledge-pytds-failed-login\",\n", + " \"sqledge-freetds\",\n", + " \"sqledge-pytds-ntlm\",\n", + " \"sqledge-dotnet-mars\",\n", + " \"ssrp-browser\",\n", + "]\n", + "captures = Path(tempfile.mkdtemp(prefix=\"zeek-tds-corpus-\"))\n", + "for name in TRACES:\n", + " urllib.request.urlretrieve(f\"{CORPUS}/{name}.pcap\", captures / f\"{name}.pcap\")\n", + "\n", + "runs = {\n", + " name: run_zeek(captures / f\"{name}.pcap\", \"TDS::log_results=T\", \"TDS::result_sample_rows=3\")\n", + " for name in TRACES\n", + "}\n", + "{\n", + " name: sorted(\n", + " p.name for p in out.glob(\"*.log\") if p.name not in (\"packet_filter.log\", \"conn.log\")\n", + " )\n", + " for name, out in runs.items()\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "f71e2c28", + "metadata": {}, + "source": [ + "### tds_login.log: who connected, with what\n", + "\n", + "One line per connection: the client's host name, account, application and driver from LOGIN7, the encryption both sides asked for, the server's product and version from its answer, and whether the login succeeded. The failed-login capture shows the other outcome: error 18456 recorded on the same line." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "bbe556e8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:11.520719Z", + "iopub.status.busy": "2026-09-06T01:45:11.520616Z", + "iopub.status.idle": "2026-09-06T01:45:11.531702Z", + "shell.execute_reply": "2026-09-06T01:45:11.531080Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
capturesqledge-pytds-workloadsqledge-pytds-failed-loginsqledge-freetdssqledge-pytds-ntlmsqledge-dotnet-mars
hostnameworkstation-01workstation-01Noneworkstation-01None
usernamesasaNoneNone
has_passwordTrueTrueNoneFalseNone
password_in_clearTrueTrueFalseFalseFalse
integrated_authFalseFalseNoneTrueNone
app_namezeek-tds-corpuszeek-tds-badpwNonezeek-tds-ntlmNone
libraryPython TDS LibraryPython TDS LibraryNonePython TDS LibraryNone
databasezeektdsmasterNonemasterNone
tds_version7.47.4None7.4None
client_encryptionnot_supportednot_supportedoffnot_supportedoff
server_encryptionnot_supportednot_supportedoffnot_supportedoff
encryptedFalseFalseTrueFalseTrue
server_productMicrosoft SQL ServerNoneMicrosoft SQL ServerNoneMicrosoft SQL Server
server_product_version15.0.2000None15.0.2000None15.0.2000
successTrueFalseTrueFalseTrue
error_numberNone18456None18452None
error_messageNoneLogin failed for user 'sa'.NoneLogin failed. The login is from an untrusted domain and cannot be used with Integrated authentication.None
\n", + "
" + ], + "text/plain": [ + "capture sqledge-pytds-workload sqledge-pytds-failed-login sqledge-freetds \\\n", + "hostname workstation-01 workstation-01 None \n", + "username sa sa None \n", + "has_password True True None \n", + "password_in_clear True True False \n", + "integrated_auth False False None \n", + "app_name zeek-tds-corpus zeek-tds-badpw None \n", + "library Python TDS Library Python TDS Library None \n", + "database zeektds master None \n", + "tds_version 7.4 7.4 None \n", + "client_encryption not_supported not_supported off \n", + "server_encryption not_supported not_supported off \n", + "encrypted False False True \n", + "server_product Microsoft SQL Server None Microsoft SQL Server \n", + "server_product_version 15.0.2000 None 15.0.2000 \n", + "success True False True \n", + "error_number None 18456 None \n", + "error_message None Login failed for user 'sa'. None \n", + "\n", + "capture sqledge-pytds-ntlm sqledge-dotnet-mars \n", + "hostname workstation-01 None \n", + "username None \n", + "has_password False None \n", + "password_in_clear False False \n", + "integrated_auth True None \n", + "app_name zeek-tds-ntlm None \n", + "library Python TDS Library None \n", + "database master None \n", + "tds_version 7.4 None \n", + "client_encryption not_supported off \n", + "server_encryption not_supported off \n", + "encrypted False True \n", + "server_product None Microsoft SQL Server \n", + "server_product_version None 15.0.2000 \n", + "success False True \n", + "error_number 18452 None \n", + "error_message Login failed. The login is from an untrusted domain and cannot be used with Integrated authentication. None " + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "login_cols = [\n", + " \"hostname\",\n", + " \"username\",\n", + " \"has_password\",\n", + " \"password_in_clear\",\n", + " \"integrated_auth\",\n", + " \"app_name\",\n", + " \"library\",\n", + " \"database\",\n", + " \"tds_version\",\n", + " \"client_encryption\",\n", + " \"server_encryption\",\n", + " \"encrypted\",\n", + " \"server_product\",\n", + " \"server_product_version\",\n", + " \"success\",\n", + " \"error_number\",\n", + " \"error_message\",\n", + "]\n", + "logins = pd.concat(\n", + " [\n", + " load(\"tds_login\", runs[n]).assign(capture=n)\n", + " for n in TRACES\n", + " if (runs[n] / \"tds_login.log\").exists()\n", + " ]\n", + ")\n", + "logins.set_index(\"capture\")[login_cols].T" + ] + }, + { + "cell_type": "markdown", + "id": "e0b2cf19", + "metadata": {}, + "source": [ + "### tds_error.log: what the server refused\n", + "\n", + "Every ERROR token, with number, severity and text. 208 is the \"invalid object name\" that reconnaissance trips; 18456 is a failed login; 18452 is a Windows login the server would not trust." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "f63927cc", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:11.533053Z", + "iopub.status.busy": "2026-09-06T01:45:11.532971Z", + "iopub.status.idle": "2026-09-06T01:45:11.539053Z", + "shell.execute_reply": "2026-09-06T01:45:11.538510Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
capturenumberclassstatemessageserver_name
0sqledge-pytds-workload208161Invalid object name 'dbo.does_not_exist'.sqledge01
1sqledge-pytds-workload50000161custom failure 42sqledge01
0sqledge-pytds-failed-login18456141Login failed for user 'sa'.sqledge01
0sqledge-pytds-ntlm18452141Login failed. The login is from an untrusted domain and cannot be used with Integrated authentication.sqledge01
\n", + "
" + ], + "text/plain": [ + " capture number class state message server_name\n", + "0 sqledge-pytds-workload 208 16 1 Invalid object name 'dbo.does_not_exist'. sqledge01\n", + "1 sqledge-pytds-workload 50000 16 1 custom failure 42 sqledge01\n", + "0 sqledge-pytds-failed-login 18456 14 1 Login failed for user 'sa'. sqledge01\n", + "0 sqledge-pytds-ntlm 18452 14 1 Login failed. The login is from an untrusted domain and cannot be used with Integrated authentication. sqledge01" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "errors = pd.concat(\n", + " [\n", + " load(\"tds_error\", runs[n]).assign(capture=n)\n", + " for n in TRACES\n", + " if (runs[n] / \"tds_error.log\").exists()\n", + " ]\n", + ")\n", + "errors[[\"capture\", \"number\", \"class\", \"state\", \"message\", \"server_name\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "62079e29", + "metadata": {}, + "source": [ + "### Encrypted sessions: ssl.log and ntlm.log under the same uid\n", + "\n", + "FreeTDS encrypts only the login by default. The TLS handshake travels inside TDS PRELOGIN packets; the analyzer unwraps it and hands it to Zeek's SSL analyzer, so `conn.log` shows both services and `ssl.log` has the cipher and certificate for the SQL session. A Windows login does the same with NTLM." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "0307f9ab", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:11.540517Z", + "iopub.status.busy": "2026-09-06T01:45:11.540424Z", + "iopub.status.idle": "2026-09-06T01:45:11.546595Z", + "shell.execute_reply": "2026-09-06T01:45:11.546215Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "sqledge-freetds: conn.log service = tds,ssl, uid = CePJz11u2dS55ptDUe\n", + " uid version cipher curve server_name established\n", + "CePJz11u2dS55ptDUe TLSv12 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 x25519 None True\n", + "\n", + "sqledge-pytds-ntlm: conn.log service = tds,ntlm, uid = Ckca431lidT25yN8Cj\n", + " uid hostname domainname username\n", + "Ckca431lidT25yN8Cj None None None\n", + "\n" + ] + } + ], + "source": [ + "for name in (\"sqledge-freetds\", \"sqledge-pytds-ntlm\"):\n", + " c = load(\"conn\", runs[name])\n", + " print(f\"{name}: conn.log service = {c['service'].iloc[0]}, uid = {c['uid'].iloc[0]}\")\n", + " for extra in (\"ssl\", \"ntlm\"):\n", + " if (runs[name] / f\"{extra}.log\").exists():\n", + " df = load(extra, runs[name])\n", + " cols = [\n", + " col\n", + " for col in (\n", + " \"uid\",\n", + " \"version\",\n", + " \"cipher\",\n", + " \"curve\",\n", + " \"server_name\",\n", + " \"established\",\n", + " \"hostname\",\n", + " \"domainname\",\n", + " \"username\",\n", + " )\n", + " if col in df\n", + " ]\n", + " print(df[cols].to_string(index=False))\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "3c2d8f9c", + "metadata": {}, + "source": [ + "### MARS: several sessions on one connection\n", + "\n", + "A .NET client with `MultipleActiveResultSets=True` multiplexes sessions over one TCP connection. The analyzer unwraps the SMP framing and `tds.log` records the session id, so interleaved requests and their responses can be paired." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "770fa31a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:11.547887Z", + "iopub.status.busy": "2026-09-06T01:45:11.547802Z", + "iopub.status.idle": "2026-09-06T01:45:11.553099Z", + "shell.execute_reply": "2026-09-06T01:45:11.552789Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
tsis_origmsg_typelensessionrowsrow_count
32026-09-06 00:06:55.269366026+00:00Truerpc2331.0NaNNaN
42026-09-06 00:06:55.288081884+00:00Falsetabular_result19831.045.045.0
52026-09-06 00:06:55.291708946+00:00Truerpc2122.0NaNNaN
62026-09-06 00:06:55.297415972+00:00Falsetabular_result512.01.01.0
72026-09-06 00:06:55.299412012+00:00Truerpc2122.0NaNNaN
82026-09-06 00:06:55.299654007+00:00Falsetabular_result512.01.01.0
92026-09-06 00:06:55.299787045+00:00Truerpc2122.0NaNNaN
102026-09-06 00:06:55.299958944+00:00Falsetabular_result512.01.01.0
112026-09-06 00:06:55.300045013+00:00Truerpc2122.0NaNNaN
122026-09-06 00:06:55.300224066+00:00Falsetabular_result512.01.01.0
132026-09-06 00:06:55.300334930+00:00Truerpc2122.0NaNNaN
142026-09-06 00:06:55.300542116+00:00Falsetabular_result512.01.01.0
\n", + "
" + ], + "text/plain": [ + " ts is_orig msg_type len session rows row_count\n", + "3 2026-09-06 00:06:55.269366026+00:00 True rpc 233 1.0 NaN NaN\n", + "4 2026-09-06 00:06:55.288081884+00:00 False tabular_result 1983 1.0 45.0 45.0\n", + "5 2026-09-06 00:06:55.291708946+00:00 True rpc 212 2.0 NaN NaN\n", + "6 2026-09-06 00:06:55.297415972+00:00 False tabular_result 51 2.0 1.0 1.0\n", + "7 2026-09-06 00:06:55.299412012+00:00 True rpc 212 2.0 NaN NaN\n", + "8 2026-09-06 00:06:55.299654007+00:00 False tabular_result 51 2.0 1.0 1.0\n", + "9 2026-09-06 00:06:55.299787045+00:00 True rpc 212 2.0 NaN NaN\n", + "10 2026-09-06 00:06:55.299958944+00:00 False tabular_result 51 2.0 1.0 1.0\n", + "11 2026-09-06 00:06:55.300045013+00:00 True rpc 212 2.0 NaN NaN\n", + "12 2026-09-06 00:06:55.300224066+00:00 False tabular_result 51 2.0 1.0 1.0\n", + "13 2026-09-06 00:06:55.300334930+00:00 True rpc 212 2.0 NaN NaN\n", + "14 2026-09-06 00:06:55.300542116+00:00 False tabular_result 51 2.0 1.0 1.0" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "mars = load(\"tds\", runs[\"sqledge-dotnet-mars\"])\n", + "mars[[\"ts\", \"is_orig\", \"msg_type\", \"len\", \"session\", \"rows\", \"row_count\"]].iloc[3:15]" + ] + }, + { + "cell_type": "markdown", + "id": "8ab717db", + "metadata": {}, + "source": [ + "### tds_result.log: what came back\n", + "\n", + "Opt-in, because it describes the data itself. Column names and types and the row count per result set; with `TDS::result_sample_rows` set, the first rows rendered by type." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "d493113e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:11.554292Z", + "iopub.status.busy": "2026-09-06T01:45:11.554218Z", + "iopub.status.idle": "2026-09-06T01:45:11.559094Z", + "shell.execute_reply": "2026-09-06T01:45:11.558453Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
columnstypesrowssample
0[id, tag, value, quality, ts, note, blob, amount, price, flag, guid, d, t, dto, big, xmlcol][int, nvarchar(64), floatn(8), intn(1), datetime2, varchar(200), varbinary(max), decimaln(12,4), moneyn(8), bitn, un...60[1|PUMP-000.FLOW|NULL|NULL|2026-09-05 12:00:00.000|NULL|0x01|1234.5678|99.9900|false|09BBB7AA-CE65-4CCF-803A-D2CE6C5...
1[][intn(4)]1[45]
2[id, tag, value][int, nvarchar(64), floatn(8)]5[10|PUMP-009.FLOW|21.5, 9|PUMP-008.FLOW|20.5, 8|PUMP-007.FLOW|NULL]
3[big, x][nvarchar(max), xml]1[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
\n", + "
" + ], + "text/plain": [ + " columns \\\n", + "0 [id, tag, value, quality, ts, note, blob, amount, price, flag, guid, d, t, dto, big, xmlcol] \n", + "1 [] \n", + "2 [id, tag, value] \n", + "3 [big, x] \n", + "\n", + " types rows \\\n", + "0 [int, nvarchar(64), floatn(8), intn(1), datetime2, varchar(200), varbinary(max), decimaln(12,4), moneyn(8), bitn, un... 60 \n", + "1 [intn(4)] 1 \n", + "2 [int, nvarchar(64), floatn(8)] 5 \n", + "3 [nvarchar(max), xml] 1 \n", + "\n", + " sample \n", + "0 [1|PUMP-000.FLOW|NULL|NULL|2026-09-05 12:00:00.000|NULL|0x01|1234.5678|99.9900|false|09BBB7AA-CE65-4CCF-803A-D2CE6C5... \n", + "1 [45] \n", + "2 [10|PUMP-009.FLOW|21.5, 9|PUMP-008.FLOW|20.5, 8|PUMP-007.FLOW|NULL] \n", + "3 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx... " + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "results = load(\"tds_result\", runs[\"sqledge-pytds-workload\"])\n", + "results[[\"columns\", \"types\", \"rows\", \"sample\"]].head(6)" + ] + }, + { + "cell_type": "markdown", + "id": "017d4e40", + "metadata": {}, + "source": [ + "### ssrp.log: how clients and scanners find instances\n", + "\n", + "Before the first TDS packet, a client that knows only an instance name asks the SQL Server Browser service on UDP 1434. So does every scanner. The reply is an inventory: each instance, its build and its port." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "07c0f274", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:11.560353Z", + "iopub.status.busy": "2026-09-06T01:45:11.560279Z", + "iopub.status.idle": "2026-09-06T01:45:11.564759Z", + "shell.execute_reply": "2026-09-06T01:45:11.564366Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
requestinstanceinstancesdac_portanswered
0enumerateNaN[SQLEDGE01\\MSSQLSERVER 15.0.2000.1574 tcp/1433, SQLEDGE01\\HISTORIAN 14.0.3456.2 tcp/1533]NaNTrue
1instanceHISTORIAN[SQLEDGE01\\HISTORIAN 14.0.3456.2 tcp/1533]NaNTrue
2instanceMSSQLSERVER[SQLEDGE01\\MSSQLSERVER 15.0.2000.1574 tcp/1433]NaNTrue
3dacHISTORIANNone1534.0True
4broadcastNaN[SQLEDGE01\\MSSQLSERVER 15.0.2000.1574 tcp/1433, SQLEDGE01\\HISTORIAN 14.0.3456.2 tcp/1533]NaNTrue
\n", + "
" + ], + "text/plain": [ + " request instance instances dac_port answered\n", + "0 enumerate NaN [SQLEDGE01\\MSSQLSERVER 15.0.2000.1574 tcp/1433, SQLEDGE01\\HISTORIAN 14.0.3456.2 tcp/1533] NaN True\n", + "1 instance HISTORIAN [SQLEDGE01\\HISTORIAN 14.0.3456.2 tcp/1533] NaN True\n", + "2 instance MSSQLSERVER [SQLEDGE01\\MSSQLSERVER 15.0.2000.1574 tcp/1433] NaN True\n", + "3 dac HISTORIAN None 1534.0 True\n", + "4 broadcast NaN [SQLEDGE01\\MSSQLSERVER 15.0.2000.1574 tcp/1433, SQLEDGE01\\HISTORIAN 14.0.3456.2 tcp/1533] NaN True" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "load(\"ssrp\", runs[\"ssrp-browser\"])[[\"request\", \"instance\", \"instances\", \"dac_port\", \"answered\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "7eecd543", + "metadata": {}, + "source": [ + "### Notices\n", + "\n", + "The package's detection script turns the common questions into Zeek notices. With default thresholds nothing fires on this corpus; here the large-result threshold is lowered to fifty rows and the brute-force threshold to one failure to show two of them." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "e26dfa01", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-06T01:45:11.565944Z", + "iopub.status.busy": "2026-09-06T01:45:11.565884Z", + "iopub.status.idle": "2026-09-06T01:45:12.383072Z", + "shell.execute_reply": "2026-09-06T01:45:12.382661Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
notemsg
0TDS::Cleartext_Password172.19.0.1 sent a SQL Server password outside TLS to 172.19.0.2 as sa
1TDS::Large_Result172.19.0.1 received a 60-row, 12995-byte result from 172.19.0.2
0TDS::Login_Bruteforce172.19.0.1 failed 1 SQL Server logins in 10.0 mins against 1 server(s)
\n", + "
" + ], + "text/plain": [ + " note msg\n", + "0 TDS::Cleartext_Password 172.19.0.1 sent a SQL Server password outside TLS to 172.19.0.2 as sa\n", + "1 TDS::Large_Result 172.19.0.1 received a 60-row, 12995-byte result from 172.19.0.2\n", + "0 TDS::Login_Bruteforce 172.19.0.1 failed 1 SQL Server logins in 10.0 mins against 1 server(s)" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "demo = run_zeek(\n", + " captures / \"sqledge-pytds-workload.pcap\",\n", + " \"TDS::large_result_rows=50\",\n", + " \"TDS::detect_cleartext_password=T\",\n", + ")\n", + "demo_fail = run_zeek(captures / \"sqledge-pytds-failed-login.pcap\", \"TDS::bruteforce_threshold=1.0\")\n", + "notices = pd.concat([load(\"notice\", d) for d in (demo, demo_fail) if (d / \"notice.log\").exists()])\n", + "notices[[\"note\", \"msg\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "7c9e4743", "metadata": {}, "source": [ "## Where this goes next\n", "\n", - "On a sensor the same analyzer runs continuously and adds a `tds_login.log` of who connected with which driver, a `tds_error.log` of failed logins and permission errors, and notices for brute force, dangerous procedures and unusual result sizes. The capture here predates TLS on the wire; on a modern network the analyzer hands encrypted sessions to Zeek's SSL analyzer and keeps the login negotiation." + "On a sensor the analyzer runs continuously, and these logs accumulate for every SQL Server session on the network: who connected with which driver, what they ran, what the server refused, how much came back, and, through `ssl.log` and `ntlm.log`, the certificate and the Windows account behind it. The 2009 capture predates TLS on the wire; on a modern network encrypted sessions keep only the login negotiation and the certificate, which is why the plaintext decoding matters most where this series started, on control-system networks." ] }, { "cell_type": "code", - "execution_count": 12, - "id": "68482006", + "execution_count": 20, + "id": "4dd5a92c", "metadata": { "execution": { - "iopub.execute_input": "2026-09-06T01:34:49.307764Z", - "iopub.status.busy": "2026-09-06T01:34:49.307661Z", - "iopub.status.idle": "2026-09-06T01:34:49.309773Z", - "shell.execute_reply": "2026-09-06T01:34:49.309422Z" + "iopub.execute_input": "2026-09-06T01:45:12.384842Z", + "iopub.status.busy": "2026-09-06T01:45:12.384703Z", + "iopub.status.idle": "2026-09-06T01:45:12.389770Z", + "shell.execute_reply": "2026-09-06T01:45:12.389420Z" } }, "outputs": [], "source": [ - "shutil.rmtree(logs, ignore_errors=True)" + "for d in [logs, captures, demo, demo_fail, *runs.values()]:\n", + " shutil.rmtree(d, ignore_errors=True)" ] } ], From 4e787deb92e7fcb7a22e1b88d5faac8d46cf3cf1 Mon Sep 17 00:00:00 2001 From: dgunter Date: Sat, 5 Sep 2026 20:48:38 -0500 Subject: [PATCH 4/4] Switch the license to Apache 2.0 --- LICENSE | 223 ++++++++++++++++++++++++++++++++++++++++++++----- README.md | 2 +- pyproject.toml | 2 +- 3 files changed, 204 insertions(+), 23 deletions(-) diff --git a/LICENSE b/LICENSE index 35ec442..d422422 100644 --- a/LICENSE +++ b/LICENSE @@ -1,21 +1,202 @@ -MIT License - -Copyright (c) 2017 Dan Gunter - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. \ No newline at end of file + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2018-2026 Dan Gunter + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md index 9c04c3b..4608501 100644 --- a/README.md +++ b/README.md @@ -84,5 +84,5 @@ if any cell errors or writes to stderr. ## License -MIT, see [LICENSE](LICENSE). The posts under `docs/` are my own writing, +Apache 2.0, see [LICENSE](LICENSE). The posts under `docs/` are my own writing, reproduced from the Wayback Machine. diff --git a/pyproject.toml b/pyproject.toml index d23471b..4ffbe08 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,7 +3,7 @@ name = "blog-code" version = "2.0.0" description = "Jupyter notebooks from the Threat Hunting with Python series at dgunter.com" readme = "README.md" -license = "MIT" +license = "Apache-2.0" license-files = ["LICENSE"] requires-python = ">=3.10" dependencies = [