Scraping public Wordpress data

Finding creative ideas

What we were trying to do

For our existing users, we wanted an easy way to discover the "structure" of their blog. Google Analytics gives sessions for any url, but which url is a page, which is a blog? When where they written, by who,...?

So we looked at how the Wordpress API could help us. Knowing that 60% of websites globally run on Wordpress, it would cover a good percentage of our users. For the others, we would have to scrape the sitemap, and run spiders, and... way more complicated!
 

The surprise discovery

While studying the Wordpress API, we discovered it was fully open... no passwords... no authentication. Very few wordpress users know this, meaning almost no-one implicitly closes it. You can easily get structured data from almost any wordpress website. We figured we could definitely use this for a lead generation campaign!

Try it for yourself with one of the most visited websites in the world: https://techcrunch.com/wp-json/wp/v2/posts.
 

So, What data did we find?

For every Wordpress website out there, we now had access to:
- every blogpost,
- their publish dates,
- their authors,
- their categories,
- their tags,
- their featured images,
- the full text (handy for blogpost length, but also natural language processing and keyword discovery),...
 

How we decided to use it:

After a brainstorm session, we decided to use it in 3 ways:
1. Structured meta data for our existing users.
2. A preview page for our website visitors: enter your domain and you immediately get a demo of what your dashboard could look like.
3. We would send screenshots of that preview page to our prospects in an outbound email campaign. I was sure it would catch their attention!

I will talk about the last two in the next section.
 

Building the Growth Hacks

The preview feature

This was built into our app, because it used the same algorithm as for our existing users: with a given domain, contact the Wordpress API and download the structured data. The only difference was that we didn't store the data, that the download was on-the-fly and that we needed to build a way for anonymous users to have access to the application. But it was maximum 2 days of work.

What we got was our dashboard with the scraped data on the left, and faked traffic & metrics data on the right.

We then added an intercom popup to inform the user about what this preview is exactly.
 

The cold email campaign

1. We first scraped lists of domains/websites of our prospects: through event pages, paid contact databases and builtwith.com. And we wrote the first part of the script that discards the non-wordpress domains.

2. We then found emails of those prospects, by feeding the domains to hunter.io (paid service), and put it in our sheet.

3. The script would then run it's most advanced feature: for every domain, feed it to our preview page: https://app.blogalyzer.net/preview?url=techcrunch.com and take a screenshot once the page is fully loaded. Knowing that we download the data on-the-fly, this wait-for-load-time was not easy to configure. It stored the images in a Google Cloud Bucket and put the link in our sheet.

4. Finally, our script checks for domains that have emails and screenshots and sends emails. I used the sendgrid.com api to do this, as it allows for url-based images and allows for up to 300 emails per day (compared to more conservative services).

That was that!