Ghostferry

Ghostferry is a library that enables you to selectively copy data from one mysql instance to another with minimal amount of downtime.

It is inspired by Github's gh-ost, although instead of copying data from and to the same database, Ghostferry copies data from one database to another and has the ability to only partially copy data.

There is an example application called ghostferry-copydb included (under the copydb directory) that demonstrates this library by copying an entire database from one machine to another.

Talk to us on IRC at irc.freenode.net #ghostferry.

• Tutorial and General Documentations: https://shopify.github.io/ghostferry
• Code documentations: https://godoc.org/github.com/Shopify/ghostferry

Overview of How it Works

An overview of Ghostferry's high-level design is expressed in the TLA+ specification, under the tlaplus directory. It may be good to consult with that as it has a concise definition. However, the specification might not be entirely correct as proofs remain elusive.

On a high-level, Ghostferry is broken into several components, enabling it to copy data. This is documented at https://shopify.github.io/ghostferry/main/technicaloverview.html

Documentation

The documentation is written in Markdown under docs/ and can be read directly on GitHub, starting at the Documentation source. The published site is built with Jekyll and the Just the Docs theme; its settings, page titles and navigation order live in docs/_config.yml.

Internal contributors get the gems from dev up, then can run dev docs (live preview) or dev docs-build (build plus link check). Otherwise:

bundle install
bundle exec jekyll build --source docs --destination build/docs
bundle exec htmlproofer build/docs --disable-external --allow-missing-href --no-enforce-https --swap-urls '^/ghostferry/main/:/'
bundle exec jekyll serve --source docs --destination build/docs --host 127.0.0.1 --port 4000

The build writes the site to build/docs/; htmlproofer fails on broken internal links or anchors. The live preview is served at http://127.0.0.1:4000/ghostferry/main/. None of these commands deploy anything.

Development Setup

Installation

For Internal Contributors

dev up

For External Contributors

• Have Docker installed
• Clone the repo
• docker-compose up -d
• nix-shell

Testing

Run all tests

• make test

Run example copydb usage

• make copydb && ghostferry-copydb -verbose examples/copydb/conf.json
• For a more detailed tutorial, see the documentation.

Ruby Integration Tests

Kindly take note of following options:

• DEBUG=1: To see more detailed debug output by Ghostferry live, as opposed to only when the test fails. This is helpful for debugging hanging test.

Examples:

Run all tests

rake test

Run a single file

rake test TEST=test/integration/trivial_test.rb

or

ruby -Itest test/integration/trivial_test.rb

Run a specific test

DEBUG=1 ruby -Itest test/integration/trivial_test.rb -n "TrivialIntegrationTest#test_logged_query_omits_columns"

Releasing new version

Canary

Tag your commit with canary/* and push, i.e.

git tag --sign --message="Initial support for UUIDs as pagination keys" canary/v1.1.2-uuid-pagination-keys-alpha-1
git push origin --tags

This will create the release named by tag.

Production

Final releases are created automatically on merge to main branch, they will end up with release-SHA name.

Remember to update version prior to bigger releases in Makefile along with updating the CHANGELOG.md.