How to Scrape a List of Topics from a Subreddit Using Bash


Linux terminal on Ubuntu laptop concept
Fatmawati Achmad Zaenuri/Shutterstock.com

Reddit offers JSON feeds for each subreddit. Here’s how to create a Bash script that downloads and parses a list of posts from any subreddit you like. This is just one thing you can do with Reddit’s JSON feeds.

Installing Curl and JQ

We’re going to use curl to fetch the JSON feed from Reddit and jq to parse the JSON data and extract the fields we want from the results. Install these two dependencies using apt-get on Ubuntu and other Debian-based Linux distributions. On other Linux distributions, use your distribution’s package management tool instead.

sudo apt-get install curl jq

Fetch Some JSON Data from Reddit

Let’s see what the data feed looks like. Use curl to fetch the latest posts from the MildlyInteresting subreddit:

curl -s -A “reddit scraper example” https://www.reddit.com/r/MildlyInteresting.json

Note how the options used before the URL: -s forces curl to run in silent mode so that we don’t see any output, except the data from Reddit’s servers. The next option and the parameter that follows, -A “reddit scraper example” , sets a custom user agent string that helps Reddit identify the service accessing their data. The Reddit API servers apply rate limits based on the user agent string. Setting a custom value will cause Reddit to segment our rate limit away from other callers and reduce the chance that we get an HTTP 429 Rate Limit Exceeded error.

The output should fill up the terminal window and look something like this:

Scrape a subreddit from Bash

There are lots of fields in the output data, but all we’re interested in are Title, Permalink, and URL. You can see an exhaustive list of types and their fields on Reddit’s API documentation page: https://github.com/reddit-archive/reddit/wiki/JSON

Extracting Data from the JSON Output

We want to extract Title, Permalink, and URL, from the output data and save it to a tab-delimited file. We can use text processing tools like sed and grep , but we have another tool at our disposal that understands JSON data structures, called jq . For our first attempt, let’s use it to pretty-print and color-code the output. We’ll use the same call as before, but this time, pipe the output through jq and instruct it to parse and print the JSON data.

curl -s -A “reddit scraper example” https://www.reddit.com/r/MildlyInteresting.json | jq .

Note the period that follows the command. This expression simply parses the input and prints it as-is. The output looks nicely formatted and color-coded:

Extract data from a subreddit's JSON in Bash

Let’s examine the structure of the JSON data we get back from Reddit. The root result is an…

Sasha Harriet

Sasha Harriet

As content editor, I get to do what I love everyday. Tweet, share and promote the best content our tools find on a daily basis.

I have a crazy passion for #music, #celebrity #news & #fashion! I'm always out and about on Twitter.
Sasha Harriet

More from Around the Web

Subscribe To Our Newsletter

Join our mailing list to receive the latest news from our network of site partners.

You have Successfully Subscribed!

Pin It on Pinterest