API Howto

How the API Works

You need to have programming knowledge in order to use the API. This API is a tool from developers for developers.

There are three different endpoints for the API:

• /crawl - This endpoint allows you to get the HTML from any url. You may use a browser or a plain HTTP requests.
• /serp - This endpoint allows you to scrape several different seach engines such as Google, Bing or Amazon.
• /custom - This endpoint allows you to specify your own crawler logic in a custom Puppeteer class.

All our crawlers scale horizontally. This means that crawling one item takes the same time as crawling 100 items!

API Limitations

The API has a few limitations.

• Crawlers do not use proxies by default. If you want to use proxies, you can purchase your own proxies and specify the proxy option in the API payload. Alternatively, we can find suitable proxies for you.
• We do not keep browsers running all the time. Therefore, after you issue an API call, we start browsers in the backend. This results in a initial time delay of around 1 to 2.5 seconds for the first API invocation. All subsequent API invocations make use of the already running browser and will be significantly faster.

Why should I use your API?

Good quality proxies determine 50% of the success of your crawling/scraping projects. We will not deny this fact. However, it is equally important to make sure that your crawling infrastructure provides the following features:

• Capability to handle crawling concurrency
• No fixed costs for servers that are purchased for the sole purpose of scraping/crawling
• State of the art bot detection evasion by providing credible browser fingerprints
• Advanced human behavior generation to evade bot detection

API Examples

                                    curl https://scrapeulous.com/API \
  -H "Content-Type: application/json" \
  -d '{"API_KEY": "{{YOUR_API_KEY}}",
       "function": "google_scraper.js",
       "region": "ca",
       "items": ["buy french press"]}'
                            

                                    curl https://scrapeulous.com/API \
  -H "Content-Type: application/json" \
  -d '{"API_KEY": "{{YOUR_API_KEY}}",
       "function": "bing_scraper.js",
       "proxies": ["http://11.22.33.44:8080"],
       "items": ["news us election"]}'