Skip to content

About

Processing code and reuse examples for GitHub Stars in Context, a dataset of repository popularity across GitHub accounts

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

GitHub Stars in Context

Processing code and reuse examples for GitHub Stars in Context, a dataset linking 3,671,497 GitHub user accounts to 53,990,808 rows of public, non-fork repositories they own, with star counts. A row is one repository under one account. PROVENANCE.md documents how the data were collected and checked.

The dataset card describes the files, the collection and the limits of the data. This repository lets you check the release and reproduce the reuse examples from the paper.

Reproduce the examples

You need CPython 3.12 and about 2 GB of disk. After the download and the package install, no network access or GitHub token is needed.

git clone --branch v1.1.1 --depth 1 https://github.com/NaiveNeuron/github-stars-in-context
cd github-stars-in-context

python3.12 -m venv .venv
.venv/bin/python -m pip install --require-hashes --only-binary=:all: -r requirements-parquet.lock

mkdir -p data
base=https://huggingface.co/datasets/NaiveNeuron/github-stars-in-context/resolve/v1.1.0
for f in users.parquet repositories.parquet portfolios.parquet release.json \
         data-dictionary.json data-dictionary.md crosswalk.sql PARQUET-SHA256SUMS; do
  curl -fL -o "data/$f" "$base/$f"
done
(cd data && shasum -a 256 -c PARQUET-SHA256SUMS)

make parquet-check PYTHON=.venv/bin/python

requirements-parquet.lock installs DuckDB and PyArrow at pinned, hash-checked versions. The download needs no account or token.

make parquet-check verifies the Parquet checksums, runs the four SQL recipes in examples/recipes/ with DuckDB and compares 269 values against the saved results in expected/. It ends with PASS: 269 comparisons. On an Apple M4 Max it takes about 20 seconds and 1.5 GB of memory.

What the examples compute

The first four recipes use the 3,615,479 accounts classified as complete from the collector's flags, with 53,987,867 repository rows and 239,997,596 stars. The GH Archive join uses every repository row.

Recipe Question Headline result
01_thresholds.sql What does a minimum-star filter keep? A 10-star filter keeps 2.89% of repository rows and 90.64% of stars, and drops 88.51% of the rows of the owners it keeps
02_portfolios.sql How concentrated are an account's stars? Among accounts with at least two repositories and at least 10,000 stars, the top repository holds a median 71.7% of them
02b_rank_overlap.sql Do total stars and the h-index select the same accounts? With ties included, the h-index selects 1,125 accounts and total stars selects 1,000. The sets share 265
03_owner_clustering.sql How often do selected repositories share an owner? At 100 stars, 60.07% of selected rows belong to owners with more than one selected repository
04_gharchive_join.sql Do the repository IDs join to GH Archive? Of 63,567 repositories with events in one hour, 17,382 are in the dataset
05_owner_grouped_split.sql What does splitting by owner change? In a deterministic 80/20 hash split of repositories with at least 100 stars, 55.70% of test rows have an owner who also owns a training row. Splitting by owner leaves none

The h-index of an account is the largest h such that h of its repositories have at least h stars each.

make parquet-check covers the first four recipes. The owner-grouped split and the paper's two figures have their own targets, which also run offline on the Parquet tables:

make owner-split PYTHON=.venv/bin/python   # ends with PASS
.venv/bin/python -m pip install matplotlib==3.11.2
make figures PYTHON=.venv/bin/python       # PASS: 99 plotted values match the saved results

make figures writes results/figures/r1-thresholds.pdf and r2-portfolios.pdf, the figures used in the paper. Copies are in expected/figures/. To get a split you can use, select from the assigned step of the recipe, which gives every repository row its side.

The GH Archive join needs one 36 MB download from GH Archive:

curl -fL -o data/gharchive-2026-01-03-0.json.gz https://data.gharchive.org/2026-01-03-0.json.gz
.venv/bin/python -c "import duckdb; print(duckdb.sql(open('examples/recipes/04_gharchive_join.sql').read()).fetchall())"
# [(143769, 143767, 63567, 30738, 17382)]

The other examples/*.py files are the reference implementations that produced the saved results. They read a SQLite build of the same tables, which is not part of the public release because the Parquet files carry the same rows. expected/ holds their saved outputs. release/PROVIDER-REBUILD.md documents how the Parquet files were exported from the source database with scripts/export_release.py.

Repository layout

Path Contents
examples/recipes/ DuckDB SQL for the examples
examples/make_figures.py Draws the paper's figures from the Parquet tables
examples/*.py Reference implementations
scripts/ Export, quality audit, validation and comparison code
expected/ Saved aggregate results and figures used as the comparison target
evidence/comparison/ A small capability comparison with ecosyste.ms
release/ Input manifest of the export, recorded environment and rebuild notes
collector/ A later revision of the collector used to produce the source database, with its origin note

Limits

The accounts are a follower-selected set, not a random sample of GitHub. Search cells that could not be split were skipped, and a failed search request could drop a cell, so an unknown number of accounts is missing. Retrieval classes come from the collector's flags and are not an independent check against GitHub. Follower counts are missing for 1,151,092 accounts. See the dataset card for details.

Licences

Code in this repository is released under the MIT licence (LICENSE). The files under evidence/comparison/sources/ are unmodified excerpts from ecosyste-ms/repos and remain under AGPL-3.0, as described in evidence/comparison/THIRD-PARTY-NOTICES.md. The data are released under CC BY-NC 4.0.

Citation

A paper describing the dataset is under review. See CITATION.cff.

To request removal of an account from the dataset, write to marek@suppa.sk or open an issue. Please do not post personal documents in a public issue.

About

Processing code and reuse examples for GitHub Stars in Context, a dataset of repository popularity across GitHub accounts

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages