Scraping google search result with Ruby (For free)

Here is how you can scrape Google search result with Ruby for free, without using any paid service.

We’ll use the same techinique on web scraping with Ruby for beginner

Love to watch a video instead?

Basic setup

We’re using net/http to perform get request and Nokogiri to parse the results

Feel free to change the keyword as you want

require 'net/http'
require 'nokogiri'

keyword = "computer"
url = "https://www.google.com/search?q=" + keyword
response = Net::HTTP.get_response(URI(url))


if response.code != "200"
    puts "Error: #{response.code}"
    exit
end

doc = Nokogiri::HTML(response.body)
print(doc)

Parse only Google search results

Our aim for now is to get only the search results.

Read what other hidden Gems in Google SERP for SEO

Find pattern

To separate the search results with other element, we need to find the pattern. Search, what separate the search results with other content. Either through browser inspect or your print result.

For this case, since Google randomize all the classes name, and differentiate when we browser directly and scraping, we’ll take a look on print result we have.

Int his example, I notice that:

class egMi0 kCrYT are used to wrap the results content.
title inside h3 of this parent
links is wrapped in BNeawe class

Here’s my code to get the title and link from Google search result above

require 'net/http'
require 'nokogiri'

keyword = "vs code"
url = "https://www.google.com/search?q=" + keyword
response = Net::HTTP.get_response(URI(url))


if response.code != "200"
    puts "Error: #{response.code}"
    exit
end

doc = Nokogiri::HTML(response.body)

# egMi0 kCrYT

doc.css('div.egMi0.kCrYT').each do |item|
    title = item.css('h3').text
    link = item.css('.BNeawe').text

    puts title + " || " + link
end

Scrape google search results, the easy way

If you don’t want to do this manually, there are tools out there, that can help you with this.

Here is how you can scrape Google Search results using Python

Why scrape google search results?

People scrape website content for various reasons, including:

Market Research and SEO

Understanding Competitors: By scraping Google search results, companies can understand what strategies competitors are using to rank higher in search results.
Keyword Analysis: SEO professionals might scrape search results to identify keyword opportunities and to develop strategies for ranking higher on specific keywords.

Content Aggregation

Content Discovery: Some platforms aggregate content from various sources to provide a centralized platform for users. They might scrape Google search results to identify new and popular content in their niche.

Academic Research

Data Analysis: Researchers might scrape search results for academic studies and analyses, to identify trends, study algorithm biases, or to gather data for other scholarly endeavors.