r/learnpython • u/Sakuraaa_29 • 1d ago

Best way to handle GitHub API rate limits when scraping repo data in Python?

I'm writing a Python script to fetch PR metadata and repository history from the GitHub REST API to build a dataset.
I'm hitting rate limits fairly quickly once the volume increases. For scripts pulling larger datasets, what's the standard pattern?

2 Upvotes

57% Upvoted

7 comments sorted by

17

u/Moist-Ointments 1d ago

Don't scrape faster than the rate limit.

4

u/Clean_Reaction8168 1d ago

conditional requests with ETag/Last-Modified are your best friend here, they don't count against your limit if the data hasn't changed

also build in exponential backoff, not just a fixed sleep timer. the 403 responses come with a Retry-After header you can parse and respect

3

u/Sakuraaa_29 21h ago

Great call on ETags,if the data hasn't changed, saving the API quota makes a huge difference.

6

u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 1d ago

Why not use the GitHub CLI instead?

2

u/Sakuraaa_29 21h ago

Was not aware of this, thanks will try CLI instead

2

u/jeffrey_f 1d ago

In the response headers, you should find your rate limits. Program your script to stay within this limit.