Skip to content

Identify ourselves in document fetches [production] - #498

Merged
duckduckgrayduck merged 3 commits into
masterfrom
identify_ourselves_in_document_fetches
Oct 5, 2026
Merged

duckduckgrayduck merged 3 commits into
masterfrom
identify_ourselves_in_document_fetches

Conversation

@duckduckgrayduck

Copy link
Copy Markdown
Contributor
  • Adds an environment variable where we can configure our user agent when we fetch file by URL instead of it being a generic python-requests user agent which is increasingly being blocked

  • Fixes a latent bug in fetch_file_url that did not retry on 500+ status codes. exc.response is falsy for status codes 400+, so it would never retry on 500. This is because the bool representation of a response with a status code of 400+ is by default False so the rest of the evaluation would become False.

>>> import requests
>>> r = requests.Response()
>>> r.status_code = 503
>>> bool(r)
False
>>> r.status_code = 200
>>> bool(r)
True

I think what this was going for is making sure that exc.response is not None to act as a guard. I added a test to make sure we do retry on 500 codes, which is the spirit of the retry logic there. When I reverted back to if (exc.Response
it failed the test.

  • Adds some tests for this task because there were none to act as a regression guard.

Closes #414

Testing:
I deployed it to staging and confirmed the user agent defined in settings loads and is correctly used when fetch_url is called.

@duckduckgrayduck
duckduckgrayduck merged commit 9399975 into master Oct 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Identify ourselves when fetching files from an outside site

2 participants