Processing code and reuse examples for GitHub Stars in Context, a dataset
linking 3,671,497 GitHub user accounts to 53,990,808 rows of public, non-fork
repositories they own, with star counts. A row is one repository under one
account. PROVENANCE.md documents how the data were collected and checked.
- Data: https://huggingface.co/datasets/NaiveNeuron/github-stars-in-context (three Parquet tables, 1.57 GiB, CC BY-NC 4.0)
- Code: this repository (MIT)
- Version: data 1.1.0 (tag
v1.1.0), code tagv1.1.1. Codev1.1.1differs fromv1.1.0in documentation and figure fonts only
The dataset card describes the files, the collection and the limits of the data. This repository lets you check the release and reproduce the reuse examples from the paper.
You need CPython 3.12 and about 2 GB of disk. After the download and the package install, no network access or GitHub token is needed.
git clone --branch v1.1.1 --depth 1 https://github.com/NaiveNeuron/github-stars-in-context
cd github-stars-in-context
python3.12 -m venv .venv
.venv/bin/python -m pip install --require-hashes --only-binary=:all: -r requirements-parquet.lock
mkdir -p data
base=https://huggingface.co/datasets/NaiveNeuron/github-stars-in-context/resolve/v1.1.0
for f in users.parquet repositories.parquet portfolios.parquet release.json \
data-dictionary.json data-dictionary.md crosswalk.sql PARQUET-SHA256SUMS; do
curl -fL -o "data/$f" "$base/$f"
done
(cd data && shasum -a 256 -c PARQUET-SHA256SUMS)
make parquet-check PYTHON=.venv/bin/pythonrequirements-parquet.lock installs DuckDB and PyArrow at pinned,
hash-checked versions. The download needs no account or token.
make parquet-check verifies the Parquet checksums, runs the four SQL recipes
in examples/recipes/ with DuckDB and compares 269 values against the saved
results in expected/. It ends with PASS: 269 comparisons. On an Apple M4
Max it takes about 20 seconds and 1.5 GB of memory.
The first four recipes use the 3,615,479 accounts classified as complete from the collector's flags, with 53,987,867 repository rows and 239,997,596 stars. The GH Archive join uses every repository row.
| Recipe | Question | Headline result |
|---|---|---|
01_thresholds.sql |
What does a minimum-star filter keep? | A 10-star filter keeps 2.89% of repository rows and 90.64% of stars, and drops 88.51% of the rows of the owners it keeps |
02_portfolios.sql |
How concentrated are an account's stars? | Among accounts with at least two repositories and at least 10,000 stars, the top repository holds a median 71.7% of them |
02b_rank_overlap.sql |
Do total stars and the h-index select the same accounts? | With ties included, the h-index selects 1,125 accounts and total stars selects 1,000. The sets share 265 |
03_owner_clustering.sql |
How often do selected repositories share an owner? | At 100 stars, 60.07% of selected rows belong to owners with more than one selected repository |
04_gharchive_join.sql |
Do the repository IDs join to GH Archive? | Of 63,567 repositories with events in one hour, 17,382 are in the dataset |
05_owner_grouped_split.sql |
What does splitting by owner change? | In a deterministic 80/20 hash split of repositories with at least 100 stars, 55.70% of test rows have an owner who also owns a training row. Splitting by owner leaves none |
The h-index of an account is the largest h such that h of its repositories have at least h stars each.
make parquet-check covers the first four recipes. The owner-grouped split
and the paper's two figures have their own targets, which also run offline
on the Parquet tables:
make owner-split PYTHON=.venv/bin/python # ends with PASS
.venv/bin/python -m pip install matplotlib==3.11.2
make figures PYTHON=.venv/bin/python # PASS: 99 plotted values match the saved resultsmake figures writes results/figures/r1-thresholds.pdf and
r2-portfolios.pdf, the figures used in the paper. Copies are in
expected/figures/. To get a split you can use, select from the assigned
step of the recipe, which gives every repository row its side.
The GH Archive join needs one 36 MB download from GH Archive:
curl -fL -o data/gharchive-2026-01-03-0.json.gz https://data.gharchive.org/2026-01-03-0.json.gz
.venv/bin/python -c "import duckdb; print(duckdb.sql(open('examples/recipes/04_gharchive_join.sql').read()).fetchall())"
# [(143769, 143767, 63567, 30738, 17382)]The other examples/*.py files are the reference implementations that
produced the saved results. They read a SQLite build of the same tables,
which is not part of the public release because the Parquet files carry the
same rows. expected/ holds their saved outputs. release/PROVIDER-REBUILD.md
documents how the Parquet files were exported from the source database with
scripts/export_release.py.
| Path | Contents |
|---|---|
examples/recipes/ |
DuckDB SQL for the examples |
examples/make_figures.py |
Draws the paper's figures from the Parquet tables |
examples/*.py |
Reference implementations |
scripts/ |
Export, quality audit, validation and comparison code |
expected/ |
Saved aggregate results and figures used as the comparison target |
evidence/comparison/ |
A small capability comparison with ecosyste.ms |
release/ |
Input manifest of the export, recorded environment and rebuild notes |
collector/ |
A later revision of the collector used to produce the source database, with its origin note |
The accounts are a follower-selected set, not a random sample of GitHub. Search cells that could not be split were skipped, and a failed search request could drop a cell, so an unknown number of accounts is missing. Retrieval classes come from the collector's flags and are not an independent check against GitHub. Follower counts are missing for 1,151,092 accounts. See the dataset card for details.
Code in this repository is released under the MIT licence (LICENSE). The
files under evidence/comparison/sources/ are unmodified excerpts from
ecosyste-ms/repos and remain under
AGPL-3.0, as described in evidence/comparison/THIRD-PARTY-NOTICES.md. The data are
released under CC BY-NC 4.0.
A paper describing the dataset is under review. See CITATION.cff.
To request removal of an account from the dataset, write to marek@suppa.sk or open an issue. Please do not post personal documents in a public issue.