DataAI

Language: English | 简体中文

English Overview

A local, governed workspace for conversational data analysis

DataAI combines DeepAnalyze’s autonomous data-science capabilities with WrenAI’s semantic layer and text-to-SQL workflow in one local environment. Users can ask questions in natural language and move from data querying and exploration to code execution, visualization, and report generation. Accounts, sessions, and model settings are persisted with per-user isolation.

Project Overview

This repository brings together three components to provide an end-to-end workflow from natural-language questions to reproducible analytical results:

• DeepAnalyze — RUC-DataLab’s open-source autonomous data-science agent (paper / model) for automated data exploration, code execution, visualization, and report generation.
• WrenAI — Canner’s open-source GenBI engine, providing governed text-to-SQL and a semantic layer (MDL). Its upstream source is included under WrenAI/ for local reference, integration, and Wren CLI builds.
• Core workspace — DeepAnalyze/demo/chat_v2/, an out-of-the-box conversational analytics application that connects DeepAnalyze and WrenAI with user authentication, isolated sessions, automatic semantic-layer construction, sandboxed code execution, and persistent model settings.

Supported sources include CSV, Excel, and SQLite files uploaded within a session. DataAI automatically builds DuckDB and a dynamic MDL, exposes them through the WrenAI semantic layer, and enables governed SQL queries through the injected wren_query() function.

---

✨ Highlights

1. One prompt, full pipeline — Upload a file, describe your goal in plain language, and the Agent writes Python/SQL, executes it, and produces charts and reports. No code, no SQL, no DuckDB knowledge required.
2. Governed semantic layer (WrenAI) — Each uploaded file auto-generates an MDL with column types, inferred PK/FK relations, and column meanings. All queries go through the semantic layer, catching the most common LLM mistakes (wrong columns, drifted joins, inconsistent metrics) before they happen.
3. Real results, not mockups — Every chart below was generated by the Agent running real Python in real sessions.

Six-country life expectancy (2019-2023)	Working-day vs non-working-day bike sharing	US college majors with high median & low unemployment
"2019-2023 life expectancy change and recovery across six countries (China, US, India, Japan, Brazil, South Africa)"	"Working-day vs non-working-day peak hours and 24h usage patterns"	"Find majors with median ≥ 50000 and unemployment rate < 0.05"

4. Multi-user isolation + cross-session recall — Accounts, sessions, and history are strictly per-user (path: workspace/u<user>__...). The same user's different sessions share a personal query memory: a successful NL→SQL pair from one session is automatically recalled in future sessions to avoid repeated trial-and-error.
5. Sandboxed execution — Default Docker sandbox (file-size limits, optional network). Local mode available for development. Per-step timeout 120s, max 12 rounds / 15 minutes per session.
6. Analyst-grade reports — One-click export to PDF (via Pandoc + xelatex) and Markdown, with charts, tables, code, and interpretation baked in.

---

Architecture

┌────────────────────────────────────────────────────────────┐
│  Browser · http://localhost:4000                           │
│  Next.js UI · authentication · chat · workspace files      │
└──────────────────────────┬─────────────────────────────────┘
                           │ same-origin /api proxy + auth cookie
┌──────────────────────────▼─────────────────────────────────┐
│  DeepAnalyze backend · http://localhost:9000               │
│  FastAPI · auth · sessions · code execution · exports      │
│  └─ starts the WrenAI query service on demand              │
└───────────────┬──────────────────────────┬────────────────┘
                │ OpenAI-compatible API     │ wren_query() bridge
┌───────────────▼──────────────┐  ┌────────▼─────────────────┐
│ LLM inference service         │  │ WrenAI environment        │
│ vLLM :8000 or custom API     │  │ semantic layer · MDL      │
└──────────────────────────────┘  └────────┬─────────────────┘
                                           │ DuckDB
                                  ┌────────▼─────────┐
                                  │ Session data      │
                                  │ CSV · Excel       │
                                  │ SQLite · DuckDB   │
                                  └───────────────────┘

WrenAI is not run as a separate permanent HTTP or MCP application. When the backend starts, it can launch an on-demand query service at 127.0.0.1:9471 to avoid the overhead of starting wren.exe query for every request. If the service is unavailable, the execution layer falls back to a CLI subprocess. In Docker mode, network isolation means the execution layer always uses the CLI fallback.

Quick Start

Prerequisites

• Windows with PowerShell; the included start and stop scripts use PowerShell.
• Python virtual environments for DeepAnalyze and WrenAI.
• Node.js and npm or pnpm.
• A local vLLM server, or any OpenAI-compatible API. The default endpoint is http://localhost:8000/v1.

1. Prepare the environments

# DeepAnalyze backend
python -m venv DeepAnalyze\.venv
DeepAnalyze\.venv\Scripts\pip install -r DeepAnalyze\requirements.txt

# WrenAI and the Wren CLI
cd WrenAI
python -m venv .venv
.\.venv\Scripts\pip install -e ".\core\wren-core-base" -e ".\core\wren-core-py" -e ".\core\wren"
cd ..

The backend defaults to D:\DataAI\WrenAI\.venv\Scripts\wren.exe. If your path differs, copy DeepAnalyze/demo/chat_v2/.env.example to .env and update DEEPANALYZE_WREN_CLI.

2. Start DataAI

powershell -ExecutionPolicy Bypass -File start-all.ps1

Service	Address	Role
DeepAnalyze backend	http://localhost:9000	API, sessions, execution, and exports
DataAI frontend	http://localhost:4000	Login, chat, and workspace UI
WrenAI query service	127.0.0.1:9471	Started on demand by the backend

Open http://localhost:4000 in your browser.

3. Stop DataAI

powershell -ExecutionPolicy Bypass -File stop-all.ps1

This stops the processes associated with ports 9000, 4000, and 9471.

Core Features

Authentication and session isolation

• Register an account on first use. Usernames support lowercase letters, numbers, underscores, and hyphens; valid length is 3–32 characters.
• Sessions are stored on the server under the current user and can be restored after signing in again.
• Cross-user access is rejected with 403.
• The first registered user can claim historical sessions that do not yet have an owner.
• The user menu provides session history, session switching, new sessions, and sign-out.

Per-user model settings

• Choose a local vLLM endpoint, the HeyWhale API, or another OpenAI-compatible custom endpoint.
• Save settings on the server and restore them automatically across browsers and devices.
• Test a connection with a real model request and inspect latency or the failure reason.
• Settings are isolated by username.

Data analysis workflow

• Upload CSV, Excel, or SQLite files, or load one of the four built-in example datasets.
• Describe an analytical task in natural language. The agent generates and executes Python code in a Docker sandbox by default, with an optional local mode.
• Store generated charts, tables, and reports in the session’s generated/ directory for preview and download.
• Use wren_query() and wren_dry_run() for semantic-layer queries, wren_describe() to register column meanings, and wren_remember() to reuse successful natural-language-to-SQL queries across sessions.
• Compress long context automatically to stay within the model context window.

WrenAI semantic layer

• On the first upload in a session, CSV and Excel files are imported into DuckDB and a dynamic MDL is generated.
• The per-session engine lives at workspace/<session>/.deepanalyze/wren/ and contains mdl.json, conn.json, and the DuckDB database.
• Original table and column names are retained after normalization to valid identifiers; Chinese names are preserved.
• Subsequent queries use the resident service at 127.0.0.1:9471 when available and fall back to the CLI when necessary.

Repository Layout

DataAI/
├── README.md                         # English documentation
├── README_ZH.md                      # Simplified Chinese documentation
├── start-all.ps1 / stop-all.ps1      # Start and stop the local stack
│
├── DeepAnalyze/                      # Upstream DeepAnalyze project
│   └── demo/chat_v2/                 # Core DataAI workspace
│       ├── backend_app/              # FastAPI backend
│       ├── frontend/                 # Next.js frontend
│       ├── demo_datasets_20260815/   # Built-in datasets
│       ├── tests/                    # Backend tests
│       ├── .env.example              # Configuration template
│       └── backend.py                # Backend entry point
│
└── WrenAI/                           # Included WrenAI source
    ├── core/                         # Wren core and CLI packages
    ├── docs/                         # Documentation
    ├── sdk/                          # SDKs
    ├── skills/                       # Agent skills
    ├── examples/                     # Semantic-layer examples
    └── evals/                        # Evaluation scripts and data

Local build products such as .venv/, target/, node_modules/, __pycache__/, compiled extensions, and temporary caches are excluded through .gitignore.

Configuration

Backend configuration lives in DeepAnalyze/demo/chat_v2/.env. Start from DeepAnalyze/demo/chat_v2/.env.example.

Variable	Description	Default
DEEPANALYZE_API_BASE	LLM inference endpoint	http://localhost:8000/v1
DEEPANALYZE_MODEL_PATH	Model identifier	DeepAnalyze-8B
DEEPANALYZE_EXECUTION_MODE	Code execution mode: docker or local	docker
DEEPANALYZE_EXECUTION_TIMEOUT_SEC	Per-execution timeout in seconds	120
DEEPANALYZE_BACKEND_PORT / FRONTEND_PORT	Service ports	9000 / 4000
DEEPANALYZE_CHAT_MAX_ROUNDS	Maximum agent loop rounds	12
DEEPANALYZE_CHAT_MAX_DURATION_SEC	Maximum session duration in seconds	900
DEEPANALYZE_WREN_CLI	Absolute path to wren.exe	Windows-specific
DEEPANALYZE_WREN_SERVICE	Enable the resident query service	true
DEEPANALYZE_WREN_SERVICE_PORT	Resident query service port	9471
DEEPANALYZE_MEMORY_SEMANTIC	Enable natural-language-to-SQL semantic retrieval	true

Logs and Tests

Area	Location
Backend	DeepAnalyze/demo/chat_v2/logs/backend.log and backend_err.log
WrenAI query service	DeepAnalyze/demo/chat_v2/logs/wren_service.log
Frontend	DeepAnalyze/demo/chat_v2/logs/frontend.log

Run the backend test suite with:

DeepAnalyze\.venv\Scripts\python.exe -m pytest DeepAnalyze\demo\chat_v2\tests -q

Acknowledgements

• DeepAnalyze by RUC-DataLab
• WrenAI by Canner