-
Notifications
You must be signed in to change notification settings - Fork 8
fix(links): stop GitHub 503s from filing false broken-link issues #83
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -26,22 +26,45 @@ jobs: | |
| lychee: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
| env: | ||
| # --cache saves passing links to .lycheecache so the re-check below only | ||
| # re-requests the links that failed. lychee never caches a failure. | ||
| LYCHEE_ARGS: "--config lychee.toml --no-progress --cache '**/*.md' '**/*.mdx'" | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Check external links | ||
| id: lychee | ||
| uses: lycheeverse/lychee-action@v2 | ||
| with: | ||
| args: "--config lychee.toml --no-progress '**/*.md' '**/*.mdx'" | ||
| args: ${{ env.LYCHEE_ARGS }} | ||
| fail: false | ||
| format: markdown | ||
| output: ./lychee-report.md | ||
| # Keep the report outside the checkout, or the re-check's '**/*.md' | ||
| # glob reads it as a docs page and lists every failure twice. | ||
| output: ${{ runner.temp }}/lychee-report.md | ||
|
|
||
| - name: Open or update tracking issue on broken links | ||
| # Throttled or briefly unavailable hosts (GitHub in particular) answer | ||
| # live pages with a 503, and lychee reports that without retrying. Only | ||
| # links that still fail after a pause count as broken. | ||
| - name: Wait before re-checking failures | ||
| if: steps.lychee.outputs.exit_code != 0 | ||
| run: sleep 120 | ||
|
|
||
| - name: Re-check failed links | ||
| id: recheck | ||
| if: steps.lychee.outputs.exit_code != 0 | ||
| uses: lycheeverse/lychee-action@v2 | ||
| with: | ||
| args: ${{ env.LYCHEE_ARGS }} | ||
| fail: false | ||
| format: markdown | ||
| output: ${{ runner.temp }}/lychee-report.md | ||
|
|
||
| - name: Open or update tracking issue on broken links | ||
| if: steps.lychee.outputs.exit_code != 0 && steps.recheck.outputs.exit_code != 0 | ||
| uses: peter-evans/create-issue-from-file@v5 | ||
| with: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [nit] If the re-check step itself errors (the action fails to install or crashes) and never sets |
||
| title: 🔗 Broken external links detected | ||
| content-filepath: ./lychee-report.md | ||
| content-filepath: ${{ runner.temp }}/lychee-report.md | ||
| labels: broken-links, automated | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -60,6 +60,16 @@ exclude = [ | |
| # Don't check mailto: links (this is the default; set explicitly for clarity). | ||
| include_mail = false | ||
|
|
||
| # Check GitHub file links (github.com/<owner>/<repo>/blob/<ref>/<path>) against | ||
| # raw.githubusercontent.com, the CDN that serves the same file for scripted | ||
| # downloads. A missing file, ref, or repo still returns 404 there, while | ||
| # github.com's rendered blob pages are the ones that answer the checker with | ||
| # 503s for files that exist. Reports show the raw URL. A blob link must point | ||
| # at a file; the CDN returns 404 for directories, so use /tree/ for those. | ||
| remap = [ | ||
| "^https://github\\.com/([^/]+)/([^/]+)/blob/([^?#]+).* https://raw.githubusercontent.com/$1/$2/$3", | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [nit] The remap looks right: |
||
| ] | ||
|
|
||
| # Per-host throttling. NOTE: TOML table headers capture every key that follows | ||
| # them, so keep all top-level options above this line. | ||
| # | ||
|
|
@@ -69,8 +79,10 @@ include_mail = false | |
| # for pages that are perfectly fine. That produced the false positives in | ||
| # https://github.com/sei-protocol/sei-docs/issues/77. lychee does not retry a | ||
| # 503, and the GITHUB_TOKEN API fallback can only validate bare owner/repo URLs | ||
| # (not blob/tree paths), so the fix is to slow down instead. Measured cost: | ||
| # ~80 GitHub requests at one per second adds about 40 seconds to the weekly run. | ||
| # (not blob/tree paths), so slowing down is what keeps 503s rare. It does not | ||
| # stop them entirely, which is why external-links.yml re-checks every failure | ||
| # before reporting it. Measured cost: with blob links remapped above, ~45 | ||
| # github.com requests at one per second keep the full run under a minute. | ||
| [hosts."github.com"] | ||
| concurrency = 2 | ||
| request_interval = "1s" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nit] If the re-check step fails before it sets its output (for example, the action errors out),
steps.recheck.outputs.exit_codeis''. In an expression,'' != 0evaluates to false, so no issue is filed and the failure goes unnoticed. Consider also checkingsteps.recheck.outcome == 'failure', or usingsteps.recheck.outputs.exit_code != '0', so the step fails closed.