From c35ff0cd66a4d5e41312b6088f523368faef7524 Mon Sep 17 00:00:00 2001 From: Josef Strzibny Date: Wed, 21 Oct 2026 13:29:34 +0200 Subject: [PATCH 1/2] Add native Extract API support Add first-class support for the Extract API (engine: extract), which turns any public web page into a structured JSON document and supports scrape (default), map, and crawl modes. New method on SerpApi::Client: - extract(params): runs an extraction; forces engine=extract and fails fast when the required :url parameter is missing. All other parameters (mode, max_pages, max_depth, include_subdomains, include_sitemap, device, prefetch_delay, timeout, include_tags, exclude_tags, include_markdown, include_inline_assets, output) pass through verbatim - the backend remains the source of truth. Includes offline unit specs, live example specs, a runnable demo (demo/demo_extract.rb), and README documentation. --- CHANGELOG.md | 1 + README.md | 42 ++++++++++++- demo/demo_extract.rb | 61 +++++++++++++++++++ lib/serpapi/client.rb | 43 +++++++++++++ .../client/example/example_extract_spec.rb | 27 ++++++++ spec/serpapi/client/extract_api_spec.rb | 39 ++++++++++++ 6 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 demo/demo_extract.rb create mode 100644 spec/serpapi/client/example/example_extract_spec.rb create mode 100644 spec/serpapi/client/extract_api_spec.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 358c3a3..fa01a94 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,5 @@ # Changelog + * [Unreleased] Add native Extract API support via the `extract` method * [Unreleased] Apply the `timeout` option to persistent connections * [Unreleased] Add Image API support * [Unreleased] Add Markdown search and archive output support diff --git a/README.md b/README.md index 6fe7acf..8bc208b 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ client.close - [Asynchronous searches](./demo/demo_async.rb) for submitting non-blocking jobs and retrieving completed results from the Search Archive API. - [Persistent connections and connection pooling](./demo/demo_thread_pool.rb) for reusing HTTP connections across searches. - Search results as Ruby hashes with `search`, token-efficient Markdown with `md`, or raw search-engine HTML with `html`. -- SDK methods for the [Image API](https://serpapi.com/image-api), [Location API](https://serpapi.com/locations-api), [Search Archive API](https://serpapi.com/search-archive-api), and [Account API](https://serpapi.com/account-api). +- SDK methods for the [Extract API](https://serpapi.com/extract-api), [Image API](https://serpapi.com/image-api), [Location API](https://serpapi.com/locations-api), [Search Archive API](https://serpapi.com/search-archive-api), and [Account API](https://serpapi.com/account-api). - Configurable HTTP timeouts and symbolized or string JSON keys. ## Response formats @@ -174,6 +174,46 @@ This code shows a simple solution to batch searches asynchronously into a [queue Here are some examples for some of our most popular APIs. You can find the full list of supported engines and parameters in our [documentation](https://serpapi.com/search-engine-apis). +### Extract API + +Turn any public web page into a fully structured JSON document: page information, meta tags, structured data (Open Graph, Twitter Card, JSON-LD, microdata), headings, inline links, images, media, lists, tables, forms, price offers, and normalized text content. + +```ruby +require 'serpapi' + +client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY']) +results = client.extract(url: 'https://example.com/') +pp results[:page_information] +``` + +The same method exposes two additional modes: + +- `map` performs fast same-origin URL discovery via page links, `robots.txt`, and `sitemap.xml`, and returns a `urls` array. +- `crawl` crawls pages on the same website up to `max_pages` / `max_depth` and returns a fully parsed JSON document for every visited page in a `pages` array. + +```ruby +# discover URLs of a website +results = client.extract(url: 'https://example.com/', mode: 'map') +pp results[:urls] + +# crawl a website and parse every visited page +results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1) +pp results[:pages] +``` + +To extract a page found in the results, pass its URL to `extract` again. Settings such as `device` or `include_tags` are not carried over, so pass them again if needed: + +```ruby +results = client.extract(url: 'https://example.com/') +linked_page = client.extract(url: results[:inline_links].first[:link]) +``` + + * source code: [demo/demo_extract.rb](https://github.com/serpapi/serpapi-ruby/blob/master/demo/demo_extract.rb) + +> **Note:** URLs matching a website already covered by a dedicated SerpApi engine (such as Google, Bing, eBay, or Walmart search pages) are automatically redirected to that engine, and the response follows the matched engine's JSON structure. + +[See documentation](https://serpapi.com/extract-api) + ### Google Shopping Scrape Google Shopping results with product names, prices, ratings, and merchant information. diff --git a/demo/demo_extract.rb b/demo/demo_extract.rb new file mode 100644 index 0000000..44676ab --- /dev/null +++ b/demo/demo_extract.rb @@ -0,0 +1,61 @@ +# This script demonstrates the SerpApi Extract API (engine: extract). +# +# The Extract API turns any public web page into a fully structured JSON document. +# The same endpoint exposes three modes: +# - `scrape` (default): extract a single page into structured JSON +# (page information, meta tags, structured data, headings, inline links, text content, ...) +# - `map`: fast same-origin URL discovery via page links, robots.txt and sitemap.xml +# - `crawl`: crawl pages on the same website up to `max_pages` / `max_depth` +# and return a fully parsed JSON document for every visited page. +# +# To extract a page found in the results (e.g. in inline_links), pass its URL +# to `extract` again. +# +# Ensure that the `SERPAPI_KEY` environment variable is set before running this script. +# +# doc: https://serpapi.com/extract-api + +raise 'SERPAPI_KEY environment variable must be set' if ENV['SERPAPI_KEY'].nil? + +require 'pp' +require 'serpapi' + +# client initialization, engine is set per call by the extract method +client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY']) + +# 1. scrape a single page (default mode) +puts '--- scrape: https://example.com/' +results = client.extract(url: 'https://example.com/') +pp results[:page_information] +pp results[:text_content] + +# 2. map: discover same-origin URLs of a website +puts '--- map: https://books.toscrape.com/' +results = client.extract(url: 'https://books.toscrape.com/', mode: 'map', max_pages: 10) +urls = results[:urls] || [] +puts "discovered #{urls.size} urls" +pp urls.first(5) + +# 3. crawl: parse every visited page up to max_pages / max_depth +puts '--- crawl: https://books.toscrape.com/' +results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1) +pages = results[:pages] || [] +pages.each do |page| + puts "#{page.dig(:page_information, :title)} - #{page.dig(:page_information, :url)}" +end + +# 4. follow a link: extract the first inline link of https://example.com/ +puts '--- follow: the first inline link of https://example.com/' +results = client.extract(url: 'https://example.com/') +link = results[:inline_links]&.first&.dig(:link) +if link + linked_page = client.extract(url: link) + pp linked_page[:page_information] +else + puts 'no inline link found to follow' +end + +# close the underlying persistent connection +client.close +puts 'done' +exit 0 diff --git a/lib/serpapi/client.rb b/lib/serpapi/client.rb index 2667dc7..e0fb700 100644 --- a/lib/serpapi/client.rb +++ b/lib/serpapi/client.rb @@ -136,6 +136,49 @@ def md(params = {}) get('/search', :md, force_output(params, 'md')) end + # Extract structured data from any public web page using the Extract API. + # + # The Extract API turns a URL into a fully structured JSON document: + # page information, meta tags, Open Graph / Twitter Card / Dublin Core, + # JSON-LD, microdata, breadcrumbs, headings, inline links / images / videos, + # scripts, stylesheets, feeds, lists, tables, forms, sections, price offers, + # and normalized text content. + # + # Modes (via the `mode` parameter): + # * `scrape` (default): extract a single page. + # * `map`: fast same-origin URL discovery, returns a `urls` array. + # * `crawl`: crawl the website up to `max_pages` / `max_depth`, returns a `pages` array. + # + # **Example:** + # + # ```ruby + # client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY']) + # + # # scrape a single page + # result = client.extract(url: 'https://example.com/') + # pp result[:page_information] + # + # # crawl a website + # result = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1) + # pp result[:pages] + # ``` + # + # doc: https://serpapi.com/extract-api + # + # @param [Hash] params Extract API parameters, must include :url + # (target absolute http(s) URL). Optional parameters: + # mode, max_pages, max_depth, include_subdomains, + # include_sitemap, device, prefetch_delay, timeout, + # include_tags, exclude_tags, include_markdown, + # include_inline_assets. + # @return [Hash] extraction results formatted as a Hash. + def extract(params = {}) + raise SerpApiError, "params must be hash, not: #{params.class}" unless params.instance_of?(Hash) + raise SerpApiError, 'url parameter is required by the extract API' if params[:url].to_s.empty? && params['url'].to_s.empty? + + search(params.merge(engine: 'extract')) + end + # Get location using Location API # # example: spec/serpapi/location_api_spec.rb diff --git a/spec/serpapi/client/example/example_extract_spec.rb b/spec/serpapi/client/example/example_extract_spec.rb new file mode 100644 index 0000000..6c6b387 --- /dev/null +++ b/spec/serpapi/client/example/example_extract_spec.rb @@ -0,0 +1,27 @@ +require 'spec_helper' + +describe 'example: extract' do + it 'extracts structured data from a web page' do + # initialize the serp api client + client = SerpApi::Client.new(api_key: api_key) + # extract a single page using the serpapi extract engine + results = client.extract(url: 'https://example.com/') + expect(results[:page_information]).not_to be_nil, "No page information found! keys available: #{results.keys}" + + # print the output of the response in formatted JSON + # pp results[:page_information] + # doc: https://serpapi.com/extract-api + end + + it 'maps same-origin urls of a website' do + client = SerpApi::Client.new(api_key: api_key) + results = client.extract(url: 'https://example.com/', mode: 'map') + expect(results[:urls]).not_to be_nil, "No urls found! keys available: #{results.keys}" + end + + it 'crawls a website and returns parsed pages' do + client = SerpApi::Client.new(api_key: api_key) + results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 2, max_depth: 1) + expect(results[:pages]).not_to be_nil, "No pages found! keys available: #{results.keys}" + end +end diff --git a/spec/serpapi/client/extract_api_spec.rb b/spec/serpapi/client/extract_api_spec.rb new file mode 100644 index 0000000..7dfb32e --- /dev/null +++ b/spec/serpapi/client/extract_api_spec.rb @@ -0,0 +1,39 @@ +require 'spec_helper' + +describe 'Extract API' do + let(:client) { SerpApi::Client.new(api_key: 'dummy_secret_key', persistent: false) } + + describe '#extract' do + it 'forces the extract engine and passes the url' do + expect(client).to receive(:search).with(hash_including(engine: 'extract', url: 'https://example.com/')) + client.extract(url: 'https://example.com/') + end + + it 'accepts extra parameters like mode, max_pages and max_depth' do + expect(client).to receive(:search).with( + hash_including( + engine: 'extract', + url: 'https://books.toscrape.com/', + mode: 'crawl', + max_pages: 3, + max_depth: 1 + ) + ) + client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1) + end + + it 'overrides a conflicting engine parameter' do + expect(client).to receive(:search).with(hash_including(engine: 'extract')) + client.extract(url: 'https://example.com/', engine: 'google') + end + + it 'raises an error when url is missing' do + expect { client.extract }.to raise_error(SerpApi::SerpApiError, /url parameter is required/) + expect { client.extract(mode: 'map') }.to raise_error(SerpApi::SerpApiError, /url parameter is required/) + end + + it 'raises an error when params is not a hash' do + expect { client.extract('https://example.com/') }.to raise_error(SerpApi::SerpApiError, /params must be hash/) + end + end +end From b102e628a560a5092c98c3dba8e876246118fbfd Mon Sep 17 00:00:00 2001 From: Josef Strzibny Date: Wed, 21 Oct 2026 13:29:34 +0200 Subject: [PATCH 2/2] Clear string-keyed engine parameters in extract extract forced engine: 'extract' with a symbol key, so a string-keyed 'engine' from the call or the client defaults was sent as well (engine=google&engine=extract). Drop string-keyed engines and nil out any inherited from the defaults. --- lib/serpapi/client.rb | 4 +++- spec/serpapi/client/extract_api_spec.rb | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/lib/serpapi/client.rb b/lib/serpapi/client.rb index e0fb700..b8ae22a 100644 --- a/lib/serpapi/client.rb +++ b/lib/serpapi/client.rb @@ -176,7 +176,9 @@ def extract(params = {}) raise SerpApiError, "params must be hash, not: #{params.class}" unless params.instance_of?(Hash) raise SerpApiError, 'url parameter is required by the extract API' if params[:url].to_s.empty? && params['url'].to_s.empty? - search(params.merge(engine: 'extract')) + # also drop string-keyed engines; 'engine' => nil removes one inherited + # from the client defaults, since query compacts nil values + search(params.reject { |key, _| key.to_s == 'engine' }.merge('engine' => nil, engine: 'extract')) end # Get location using Location API diff --git a/spec/serpapi/client/extract_api_spec.rb b/spec/serpapi/client/extract_api_spec.rb index 7dfb32e..d0618cf 100644 --- a/spec/serpapi/client/extract_api_spec.rb +++ b/spec/serpapi/client/extract_api_spec.rb @@ -3,6 +3,20 @@ describe 'Extract API' do let(:client) { SerpApi::Client.new(api_key: 'dummy_secret_key', persistent: false) } + # Returns the query parameters sent over HTTP, after merging client defaults. + def sent_params + http = double + response = double(status: 200, body: '{}', headers: { 'Content-Type' => 'application/json' }) + sent = nil + allow(HTTP).to receive(:timeout).and_return(http) + allow(http).to receive(:get) do |_url, params:| + sent = params + response + end + yield + sent + end + describe '#extract' do it 'forces the extract engine and passes the url' do expect(client).to receive(:search).with(hash_including(engine: 'extract', url: 'https://example.com/')) @@ -27,6 +41,17 @@ client.extract(url: 'https://example.com/', engine: 'google') end + it 'replaces a string-keyed engine parameter' do + params = sent_params { client.extract('url' => 'https://example.com/', 'engine' => 'google') } + expect(params.select { |key, _| key.to_s == 'engine' }).to eq(engine: 'extract') + end + + it 'replaces a string-keyed engine from the client defaults' do + client = SerpApi::Client.new('engine' => 'google', api_key: 'dummy_secret_key', persistent: false) + params = sent_params { client.extract(url: 'https://example.com/') } + expect(params.select { |key, _| key.to_s == 'engine' }).to eq(engine: 'extract') + end + it 'raises an error when url is missing' do expect { client.extract }.to raise_error(SerpApi::SerpApiError, /url parameter is required/) expect { client.extract(mode: 'map') }.to raise_error(SerpApi::SerpApiError, /url parameter is required/)