Building a Reddit Scraper in Go: JSON Listings, Pagination, Rate Limits, and Where It Breaks

TL;DR

Reddit's public .json listings let you scrape a subreddit in about 100 lines of Go. You decode the listing envelope, follow the after cursor, send an honest User-Agent, and back off on HTTP 429. There are two hard limits: a listing stops at roughly 1,000 items, and requests from cloud or datacenter IPs often get a flat 403. For anything in production, use the official OAuth API or a managed endpoint.

Reddit is one of the richest public sources of opinion data on the web. Product teams mine it for feature requests, marketers track brand mentions, and ML engineers build datasets from it. Since the 2023 API pricing changes, "just use the API" has become a harder sell for small projects, so many developers go back to scraping.