JoJo's Bizarre webhook handler

ゴゴゴゴゴゴ ！

Josuke is a tiny CI/deployment tool that reacts on Gogs/Github/Bitbucket webhook's payload.

Josuke is a simple guy, 3 things and he's happy:

• Write a JSON config file
• Run Josuke and feed him your config
• Go to Github/Bitbucket and set webhooks routes as specified in your config

Writing a json config file is required.

Config file path must be given using the -c flag: josuke -c /path/to/config.json.

Example of a classic config.json:

{
    "logLevel": "INFO",
    "host": "127.0.0.1",
    "port": 8082,
    "store": "{directory to store payload, optional)",
    "hook": [
        {
            "name": "gogs",
            "type": "gogs",
            "path": "/josuke/gogs",
            "secret": "7YiuiG8dM1lSh5IzdrVK5XCQcBbRFMvwh5CB4b90"
        },
        {
            "name": "private-gogs",
            "type": "gogs",
            "path": "/josuke/private-gogs",
            "secret": "0061Gki75ieIEWaQ8y8SlGpUhGpx0HEfdF3D61Tz",
            "command": [
                "/home/mkd/Work/josuke/script/hook",
                "%payload_path%",
                "%payload_event%",
                "%payload_hook%"
            ]
        },
        {
            "name": "github",
            "type": "github",
            "path": "/josuke/github",
            "secret": "wd51QvLFIG3VFim5TmltV2xB40YCWwfJmnmxo9pp"
        },
        {
            "name": "bitbucket",
            "type": "bitbucket",
            "path": "/josuke/bitbucket"
        }
    ],
    "deployment":
    [
        {
            "repo": "monkeydioude/donut",
            "proj_dir": "donut",
            "base_dir": "/var/www",
            "branches":
            [
                {
                    "branch" :"master",
                    "actions":
                    [
                        {
                            "action": "push",
                            "commands": [
                                ["echo", "payload written to: ", "%payload_path%"],
                                ["cd", "%base_dir%"],
                                ["git", "clone", "%html_url%"],
                                ["cd", "%proj_dir%"],
                                ["git", "pull", "--rebase"],
                                ["make"]
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}

TLS configuration

Add the cert and key properties inside config's json file, same level as host, port.

{
    "…": "…",
    "port": 8082,
    "cert": "conf/cert.pem",
    "key": "conf/key.pem",
    "…": "…"
}

Generate the default certificate and private key with:

#!/bin/sh
openssl req -x509 -newkey rsa:4096 -nodes \
  -out cert.pem \
  -keyout key.pem -days 365

Keys definition

• logLevel: optional, five levels, from the most verbose to the less verbose: TRACE, DEBUG, INFO, WARN, ERROR. Defaults to INFO.
• host: binds the server to local address. Defaults to localhost.
• port: port Josuke will listen to. Defaults to 8082.
• store: directory, optional. If present, every valid payload is written in this directory with a dynamic name: {hook.name}.{timestamp}.{random string}.json. The local path to this file is available to commands with the placeholder %payload_path%.
• queue_file: file, optional. If present, the job queue is saved in this file and restored at boot. See Job queue. The file is created at boot if missing, its directory must exist and be writable: josuke does not start otherwise.
• jobs_api: optional, enables the HTTP API listing and stopping jobs. See Job queue.
  • token: required, expected in the Authorization: Bearer <token> request header.
  • route: optional, defaults to /jobs.
• hook: array of objects defining SCM hooks for Gogs, GitHub and BitBucket.
• deployment: array of objects defining deployments repository rules Josuke should follow.

Hook Definition

• name : logical name, used in the payload local file name if enabled.
• type: SCM type, currently "gogs", "github" or "bitbucket".
• path: local web path. This path must be specified in the SCM webhook’s parameters.
• secret: signs the payload for Gogs and Github. Optional, but strongly recommended for security purpose. If not set, anybody can fake a payload on your webhook endpoint.
• command: optional command, takes precedence over the deployment commands if set. It is run at each valid request. Only the %payload_path%, %payload_hook% and %payload_event% placeholders are available in this hook scope.

There are four types of hooks:

• bitbucket
• gogs
• github
• webhook

Command samples

Run a shell script /home/me/josuke/script/hook with the payload path and event.

"command": [
	"/home/me/josuke/script/hook",
	"%payload_path%",
	"%payload_event%",
	"%payload_hook%"
]

On Windows, use the shell that comes with Git Bash for instance.

"command": [
	"C:/Users/me/AppData/Local/Programs/Git/bin/sh.exe",
	"C:/Users/me/josuke/script/hook",
	"%payload_path%",
	"%payload_event%",
	"%payload_hook%"
]

Repository rules

The repository rules objects are defined as such:

• repo: name of your repository in the repository universe. No need to specify the whole only the username and repository name is required (ex: monkeydioude/josuke)
• branches: is an array of objects defining the branche behavior towards specified branches.
• base_dir: OPTIONAL Allow you to set what should be a base directory usable at commands definition level (ex: /var/projects/sources)
• proj_dir: OPTIONAL Allow you to set what should be a project directory (or name) usable at commands definition level
• env_file: OPTIONAL env file whose variables are passed to the commands. See Environment variables.
• environments: OPTIONAL array of KEY=VALUE variables passed to the commands. See Environment variables.

branch behaviors objects are defined as such:

• branch: behavior toward a specific branch
• actions: is an array of objects defining the behavior towards specific actions.
• environments: OPTIONAL array of KEY=VALUE variables passed to the commands, overriding the repository ones.

actions objects are defined as such:

• action: is the kind of action sent by the payload, that has been taken toward the source branch (ex: push on a branch, merge a branch with the source branch...)
• commands: is an array of objects defining the series of commands Josuke should trigger for this action
• environments: OPTIONAL array of KEY=VALUE variables passed to the commands, overriding the branch and repository ones.

commands is an array of array of strings that should contain commands to be executed when an action is triggered. 1st index of the array must be the command name. Every following index should be args of the command:

    [
        ["cd", "%base_dir%"],
        ["git", "clone", "%html_url%"],
        ["cd", "%proj_dir%"],
        ["git", "pull", "--rebase"],
        ["make"]
    ]

You can use these 5 placeholders at command level

• %base_dir%: referring to "base_dir" set in config, must be defined by base_dir of each deployment
• %proj_dir%: referring to "proj_dir" set in config, must be defined by proj_dir of each deployment
• %html_url%: retrieved from gogs/github/bitbucket's payload informations, html url of your repo
• %payload_hook%: name of the hook that received the payload (hook[<num>].name in the configuration).
• %payload_path%: path to the payload, available if enabled with store in the configuration. Otherwise, empty.
• %payload_event%: content of the event request header, be it x-gogs-event, x-github-event, x-key-event. It contains the webhook event, push for instance. If the event header is absent, it is empty.

Environment variables

The commands of a deployment get josuke's environment, plus the variables of env_file and environments:

deployment:
  - repo: "monkeydioude/donut"
    env_file: "donut.env"
    environments:
      - STORAGE_DIR=/srv/storage
    base_dir: "/var/www"
    proj_dir: "donut"
    branches:
      - branch: "master"
        environments:
          - STORAGE_DIR=/srv/storage/master
        actions:
          - action: "push"
            environments:
              - STORAGE_DIR=/srv/storage/master/push
            commands:
              - ["cd", "%base_dir%"]
              - ["cd", "%proj_dir%"]
              - ["make", "deploy"]

When a variable is set more than once, the most specific value wins, from the highest priority to the lowest:

1. action environments,
2. branch environments,
3. repository environments,
4. repository env_file,
5. josuke's own environment.

env_file is only available at the repository level. A relative path is relative to the config file directory. The file is read at each run, so changes apply without restarting josuke. It holds one KEY=VALUE per line. Blank lines and lines starting with # are ignored, an export prefix and quotes around the value are removed, and variables are not expanded ($OTHER stays as is).

Josuke does not start if an env_file is missing or invalid, or if an environments entry is not KEY=VALUE. If the env file is missing at run time, the action fails before running any command.

The env file loaded and the names of the variables are logged, never their values. At boot, for each level:

[INFO] monkeydioude/donut env file /etc/josuke/donut.env: DB_URL, STORAGE_DIR
[INFO] monkeydioude/donut environments: STORAGE_DIR
[INFO] monkeydioude/donut@master environments: STORAGE_DIR
[INFO] monkeydioude/donut@master push environments: STORAGE_DIR

And at each run, with the variables merged from all levels:

job#3 [INFO] env file /etc/josuke/donut.env: DB_URL, STORAGE_DIR
job#3 [INFO] environments: STORAGE_DIR

Commands are not run through a shell, so $STORAGE_DIR in a command argument is not expanded: use ["sh", "-c", "ls $STORAGE_DIR"] to do so. The hook command does not get these variables.

Job queue

Each valid webhook request becomes a job and is answered right away with 202 Accepted and the job id (queued job #3), so SCM webhook timeouts are not an issue for long builds. Jobs run one at a time, in the order they were received: builds never overlap. Up to 100 jobs can wait in the queue, further requests are answered with 503 Service Unavailable.

Every log line of a job is prefixed with its id (job#3), and the output of the commands (stdout and stderr) is logged line by line while they run.

With queue_file set, the queue survives restarts: waiting jobs are run at boot. The job that was running when josuke stopped is not run again, as it may be the one restarting josuke. Redeliver its webhook to run it again.

Jobs API

Enabled with jobs_api:

"jobs_api": {
    "token": "q8iVvMcGQ3vXXhAkB7tSHx1uFAfwTKbb"
}

• GET /jobs lists the running job, then the waiting ones:

  curl -H 'Authorization: Bearer <token>' http://localhost:8082/jobs

  [
    {"id": 3, "description": "github push on monkeydioude/donut@master", "status": "running", "queued_at": "…", "started_at": "…"},
    {"id": 4, "description": "github push on monkeydioude/donut@master", "status": "waiting", "queued_at": "…"}
  ]

  status is running, stopping or waiting.
• GET /jobs/logs returns the last 1000 log lines, as written to stderr. Lines are numbered from 1 at each boot, and boot_id changes at each boot:

  curl -H 'Authorization: Bearer <token>' 'http://localhost:8082/jobs/logs?after=41&boot_id=dm14nqdyb9cw'

  {"boot_id": "dm14nqdyb9cw", "last": 42, "lines": [{"seq": 42, "text": "2026/10/10 13:39:07 job#1 [INFO] sh | build line 1"}]}

  after and boot_id are optional: pass the previous last and boot_id to get the new lines only. Every line kept is returned if boot_id is not the current one, as josuke restarted in between.
• POST /jobs/{id}/stop stops a job:

  curl -X POST -H 'Authorization: Bearer <token>' http://localhost:8082/jobs/3/stop

  A waiting job is removed from the queue (200, {"id": 4, "status": "removed"}). A running job has its command stopped, along with the processes it started (202, {"id": 3, "status": "stopping"}): they receive SIGTERM, and are killed if still running 10 seconds later. The remaining commands of the job are not run.

The token is sent in clear over plain HTTP: enable TLS, or keep the API behind a reverse proxy or on localhost.

Tests:

See testdata/.

Functional tests:

Using make ftest will trigger script/functional-test-runner.sh. This script will run every script matching test/functional/test*.sh pattern.

Build and run instructions:

With Golang:

• Install the Go language

Then using Makefile (Unix/Linux/MacOS/WSL on Windows):

• CONF_FILE=/path/to/config/json make go_start

Or with shell startup script (Unix/Linux/MacOS/WSL on Windows):

• CONF_FILE=/path/to/config/json script/run.sh

Or using Golang only (only available option for Windows users not using WSL):

• go install
• josuke -c /path/to/config/json

With Docker

• Install Docker

Then using Makefile (Unix/Linux/MacOS/WSL on Windows):

• CONF_FILE=/path/to/config/json make start

Or with Docker only:

• docker build -f build/Dockerfile -t josuke .
• docker run --network="host" -d -e "CONF_FILE=/path/to/config/json" josuke

Production Docker image

build/Dockerfile.prod builds a small image with the josuke binary, and the tools deployment commands usually need: git, ssh, make, bash, and the docker CLI with compose and buildx. Add yours to the Dockerfile if needed.

docker build -f build/Dockerfile.prod -t josuke .

The deployment commands run inside the container, so it needs:

• the config, at /etc/josuke/config.yml (JSON works too, whatever the extension),
• a volume on /var/lib/josuke for the queue file,
• the host Docker socket, for docker commands to drive the host daemon,
• the deployed projects (base_dir), mounted at the same path as on the host: docker compose sends the bind mount paths of the compose file to the host daemon as they are,
• the SSH keys used by git to fetch the repositories.

The config must listen on all interfaces, the default localhost is only reachable from inside the container:

host: 0.0.0.0
port: 8082
queue_file: /var/lib/josuke/queue.json

Run with Docker Compose:

services:
  josuke:
    image: josuke
    restart: unless-stopped
    ports:
      - "8082:8082"
    environment:
      TZ: Europe/Paris # log timestamps, optional
    volumes:
      - ./josuke.yml:/etc/josuke/config.yml:ro
      - josuke-data:/var/lib/josuke
      - /var/run/docker.sock:/var/run/docker.sock
      - /srv/apps:/srv/apps # base_dir of the deployments
      - /root/.ssh:/root/.ssh:ro

volumes:
  josuke-data:

Or with docker run:

docker run -d --name josuke --restart unless-stopped -p 8082:8082 \
  -v "$PWD/josuke.yml:/etc/josuke/config.yml:ro" \
  -v josuke-data:/var/lib/josuke \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /srv/apps:/srv/apps \
  -v /root/.ssh:/root/.ssh:ro \
  josuke

Logs are read with docker logs -f josuke.

Josuke runs as root in the container: switching users (%user_<name>%) requires it, and access to the Docker socket gives root on the host anyway. Only expose it behind TLS or a reverse proxy.

Healthcheck:

Once Josuke is running, healthcheck HTTP status is available at /healthcheck

Makefile:

• help: display make actions help descriptions
• install (dev only): setup dev env such as git hooks
• stop: stop josuke running docker container
• start: build josuke docker image and run it
• restart: stop + start
• run: run a docker container using already built josuke image
• sr: stop + run
• shell: run a shell (/bin/sh) in a running josuke container
• test: run unit tests inside a container
• ftest: run functional tests
• bb: rebuild josuke binary inside a running container
• logs: read josuke's log file (/var/log/josuke) inside a running container
• offline_logs: read logs of the lastet, running or not, josuke container from host's physical log files (/var/lig/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log)
• attach: attach a tty to a running container. Be advised that detaching the freshly attached tty might require to kill process. ちょっとダメね

Default make rule is start

DORA !