Skip to content
hellokiaqiPublic

About

Download Douyin videos, posts, profiles, and collections.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

138 Commits

Folders and files

Repository files navigation

DyFetch

DyFetch

English | 简体中文

DyFetch is a Python tool for downloading watermark-free videos, image/live-photo posts, user profiles, collections, and music-page videos from Douyin, with optional video frame extraction and a Tkinter desktop GUI.

What It Does

  • Media downloads: Single videos (/video/<id>), image and live-photo posts (/note/<id>), user profiles (/user/<sec_user_id>: published posts, liked posts, collections), collection pages (/collection/<id>, /mix/detail/<id>), music pages (/music/<id>), and v.douyin.com short links.
  • Extra assets & metadata: Saves background audio (_music.mp3), cover images (_cover.jpg), and raw metadata (_data.json) alongside downloaded posts.
  • Automatic frame extraction: Slices downloaded videos into JPG frames under frames/ at a configurable FPS using ffmpeg (bundled via imageio-ffmpeg out of the box), plus a standalone OpenCV frame tool (cut.py).
  • Browser fallback: When Douyin web APIs return empty bodies (for example when cookies expire), downloader.py can fall back to Playwright to inspect page network responses and DOM links so public posts remain downloadable.
  • Rolling session renewal: Reads sid_guard to report exact cookie expiry at startup and merges Set-Cookie headers back into local credential files on every request.
  • Messy input cleaning: Strips share captions, fixes full-width characters and HTML entities, deduplicates query parameters, and expands bare aweme_id or sec_user_id values into full URLs.

Current limitations: downloader.py does not record live streams (live.douyin.com URLs are skipped; the legacy DouYinCommand.py only parses live room info and prints FLV pull URLs). Full profile downloads require a real /user/<sec_user_id> URL rather than /user/self.

Entry Points in This Repository

Entry / Path Role
ui.py Tkinter desktop GUI (no local web server or port). Manages download queues, live logs, task history, and settings (ui_run_config.yml), driving downloader.py as a subprocess.
downloader.py Main async CLI downloader (UnifiedDownloader v3.2). Handles X-Bogus API requests, Playwright fallback, cookie renewal, SQLite incremental records (data.db), and ffmpeg frame slicing.
cookie_extractor.py Playwright login helper. Opens Chromium for QR/phone login (or connects via CDP port 9222) and writes cookies.txt, storage_state.json, and config_simple.yml.
get_cookies_manual.py Interactive helper for pasting cookies copied from browser DevTools without installing Playwright.
cut.py Optional standalone OpenCV frame exporter for slicing local video directories at custom frame intervals.
DouYinCommand.py Legacy synchronous CLI wrapper kept for compatibility (supports author avatar downloads, HTTP Range resume, and interactive live-stream URL lookup).
dy-downloader/ Standalone earlier modular async CLI package (v1.0.0) with its own run.py, requirements.txt, and dy_downloader.db. Supports single video, gallery, and user post downloads only; kept for reference.

Requirements & Installation

Requires Python 3.8+ (Tkinter is included with standard Windows/macOS Python installers for ui.py).

# 1. Install core dependencies for downloader.py and ui.py
pip install -r requirements.txt

# 2. (Recommended) Install Playwright + Chromium for cookie extraction and browser fallback
pip install playwright
playwright install chromium

# 3. (Optional) Install OpenCV + tqdm if you use the standalone cut.py script
pip install opencv-python tqdm

Frame slicing in downloader.py uses imageio-ffmpeg (already in requirements.txt) if no system ffmpeg binary is on PATH.

Getting Cookies

Douyin web APIs require a logged-in session for paging user profiles, liked posts, and non-public items.

Option 1: Browser Login via Playwright (Recommended)

python cookie_extractor.py
# Or non-interactive mode (opens Chromium directly for QR scan):
python cookie_extractor.py --auto

After signing in inside the opened browser window, credentials are saved to cookies.txt, storage_state.json, and optionally config_simple.yml. When downloader.py runs, it also syncs cookies.pkl.

Option 2: Paste from Browser DevTools (No Playwright)

python get_cookies_manual.py

Follow the on-screen F12 → Network instructions to paste the Cookie header from www.douyin.com. It validates key fields (ttwid, sessionid, msToken, etc.) and writes config_simple.yml, cookies.txt, and a timestamped cookies_backup_*.json.

Option 3: Edit Configuration File Directly

Set cookies in config_simple.yml (or config.yml) as auto, a raw cookie string, or a key-value map. With cookies: auto, downloader.py loads cookies.txt and cookies.pkl automatically (preferring whichever has a newer sid_guard expiry) and falls back to Playwright when needed.

Quick Start

1. Desktop GUI

python ui.py

Desktop Interface

Paste one or more Douyin links (or share text), press Enter to add them to the queue, and click 开始下载 (Ctrl+Enter to start, Esc to stop). GUI parameter overrides are saved to ui_run_config.yml without modifying your config_simple.yml or config.yml.

2. Command Line (downloader.py)

# Download a single video or image post
python downloader.py -c config_simple.yml -u "https://www.douyin.com/video/7320876060210373923"
python downloader.py -c config_simple.yml -u "https://www.douyin.com/note/7320876060210373923"

# Download all published posts on a user profile
python downloader.py -c config_simple.yml -u "https://www.douyin.com/user/MS4wLjABAAAA..."

# Override output directory or backfill frames for previously downloaded videos
python downloader.py -c config_simple.yml -u "https://www.douyin.com/user/MS4wLjABAAAA..." -p "./Downloaded/"
python downloader.py -c config_simple.yml -u "https://www.douyin.com/user/MS4wLjABAAAA..." --slice-backfill

# Renew and save session cookies without downloading media
python downloader.py -c config_simple.yml --refresh-session

CLI Options (downloader.py)

Flag Description
-c, --config YAML config path (default: config.yml, falls back to config_simple.yml if missing).
-u, --url One or more Douyin URLs, share strings, aweme_id, or sec_user_id values.
-p, --path Output directory (overrides path in config).
--auto-cookie Force cookies: auto and allow Playwright cookie lookup.
--cookie Pass a raw Cookie header string on the command line.
--slice-backfill / --no-slice-backfill Enable or skip frame extraction for existing video folders before downloading.
--refresh-session Refresh session cookies via browser/API and write them back to disk without downloading.

Configuration Overview

Everyday runs use config_simple.yml (copy from config.example.yml if needed; config.yml contains all incremental/limit keys):

link:
  - https://www.douyin.com/user/SEC_USER_ID

path: ./Downloaded/

music: true                        # Save background audio (video posts)
cover: true                        # Save cover image
json: true                         # Save raw metadata JSON

slice_videos: true                 # Extract JPG frames with ffmpeg
slice_fps: 10                      # Frames per second
slice_subdir: frames               # Subfolder name under each post folder
slice_backfill_on_run: false       # Backfill missing frames for older downloads
auto_install_ffmpeg: true          # Auto-install imageio-ffmpeg if ffmpeg is missing

start_time: ""                     # Date filter YYYY-MM-DD (inclusive), empty = no filter
end_time: ""

mode:                              # User profile sections: post / like / mix
  - post

number:                            # Item limits (0 = unlimited)
  post: 0
  like: 0
  allmix: 0
  mix: 0
  music: 0

database: true                     # Record downloads in SQLite (data.db)
increase:                          # Skip items already recorded in data.db
  post: false
  like: false
  mix: false
  music: false

retry_times: 3                     # API retry attempts
cookies: auto                      # auto / cookie string / cookie dict

profile_link_mode: all             # all = /user/... always downloads the profile
                                   # modal = download only the single modal_id/vid post

user_posts_browser_recovery: true  # Use Playwright when profile API paging is incomplete
user_posts_browser_headless: false
single_aweme_browser_recovery: true
single_aweme_browser_headless: true

See USAGE.md for the full configuration reference, link routing rules, and DouYinCommand.py / cut.py usage.

Output Layout

downloader.py writes each post into its own folder keyed by timestamp, sanitized description, and aweme_id:

<path>/<author_nickname>/<YYYY-MM-DD_HH-MM-SS>_<description>_<aweme_id>/
├── <folder_name>.mp4         # Video post
├── clip_1.mp4                # Motion/video clip inside an image post (if present)
├── image_1.jpg               # Static image inside an image post
├── <folder_name>_music.mp3   # Background audio (video posts)
├── <folder_name>_cover.jpg   # Cover image
├── <folder_name>_data.json   # Raw metadata JSON
└── frames/                   # Extracted video frames (when slice_videos: true)
    └── video_<stem>_000001.jpg

Existing media files on disk are skipped automatically on reruns.

Repository Structure

DyFetch/
├── ui.py                     # Tkinter desktop GUI (drives downloader.py)
├── downloader.py             # Main async CLI downloader (UnifiedDownloader v3.2)
├── cookie_extractor.py       # Playwright cookie login & CDP extractor
├── get_cookies_manual.py     # Manual DevTools cookie helper
├── cut.py                    # Standalone OpenCV frame exporter
├── DouYinCommand.py          # Legacy synchronous CLI wrapper
├── config_simple.yml         # Concise daily configuration
├── config.yml                # Full configuration with incremental & count fields
├── config.example.yml        # Configuration template
├── apiproxy/                 # Douyin API endpoints, X-Bogus signer, SQLite wrapper, Cookie manager
├── utils/                    # URL cleaner (urlfix.py) and rotating logger
├── tests/                    # Isolated regression suite (regress_downloader.py)
├── dy-downloader/            # Standalone earlier modular async CLI + pytest suite
├── claudedocs/               # Functional specification document
├── USAGE.md                  # Detailed usage & configuration manual
└── YOLO.md                   # Architecture & maintenance guide

Testing

The root regression script runs in a temporary directory without touching your local data.db or cookies:

python tests/regress_downloader.py            # 20 offline checks
python tests/regress_downloader.py --network  # + 4 live Douyin API/download checks
python ui.py --selftest                       # Tkinter GUI startup smoke check

To run the standalone dy-downloader unit tests (must execute inside dy-downloader/ so the root utils/ package does not shadow dy-downloader/utils/):

cd dy-downloader
pip install -r requirements.txt
python -m pytest tests -q

Troubleshooting

  • 0 items downloaded / empty API responses / HTTP 403: Check the startup session validity line (登录态有效期至 ... or Cookie 登录状态已于 ... 过期). If expired, run python cookie_extractor.py (or click 重新登录 in ui.py). Note that after a failed automatic browser refresh, downloader.py waits 6 hours before retrying (tracked in .session_refresh_state.json); run python downloader.py --refresh-session to force an immediate check after logging in.
  • User profile downloads fewer posts than displayed on Douyin: Anonymous sessions only receive the first page of profile posts. Sign in via cookie_extractor.py, or set user_posts_browser_headless: false so you can log in when Playwright opens the profile window.
  • /user/self link fails: Replace https://www.douyin.com/user/self?... with your actual https://www.douyin.com/user/MS4wLjABAAAA... profile URL and set mode: [like] (or post / mix).
  • No frames generated under frames/: Check slice_videos: true and verify ffmpeg -version works or imageio-ffmpeg is installed (pip install imageio-ffmpeg).

Compliance & Credits

This tool is intended for personal learning, research, and authorized content backup only. Respect Douyin's Terms of Service and content creators' copyrights; do not use it for unauthorized redistribution or commercial scraping.

DyFetch builds upon jiji262/douyin-downloader (MIT License) for dy-downloader/, the apiproxy/ layout, and DouYinCommand.py, with downloader.py and ui.py substantially rewritten and extended. dy-downloader/utils/xbogus.py includes Evil0ctal's X-Bogus implementation from Evil0ctal/Douyin_TikTok_Download_API under the Apache License 2.0.

License

Released under the MIT License — see LICENSE. Third-party attributions and license notices are documented in THIRD-PARTY-NOTICES.md and licenses/Apache-2.0.txt.

About

Download Douyin videos, posts, profiles, and collections.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages