FAQ

I want to scrape 100k to 10M Google/Bing/Yandex keywords! Can you do it?

Yes, we can do that. That is our core service. However, the Api doesn't allow such huge volumes, we will make use of our internal distributed backend instead.

For 1M keywords on a specific search engine, we will require around 7 days of time in order to scrape them. Currently, the costs for 1M keywords scraped on the Google Search engine is at ~1000$.

I get an TimeoutError when using the Api

If your Api call returns an error such as

TimeoutError: waiting for selector \"#center_col .g\" failed: timeout 40000ms exceeded

then it's because Google detected the scraping. You can retry the Api request or you can purchase an proxy and pass it to the Api call. A proxified Api call would look like:

                                      {
  "function": "google_scraper.js",
  "items": ["test keyword"],
  "proxies": ["http://username:password@proxy.example.com:8080"]
}
                              

I want to add functionality to a scraper. What should I do?

Some of our crawlers might lack specific functionality that you want to have implemented. You can always ask us to implement it for you and when we consider the functionality to be useful for other customers, we are going to add it.

However, it is much faster if you implement it on your own. All our crawlers are Open Source! Please create a Pull Request on Github and we will review & accept the pull request as soon as possible.

What are the limitations of your Api?

You can crawl up to 100 items per single Api call. For example, you can scrape 100 keywords on the Google Search Engine.

If you have large crawl tasks, we can also process them in our backend. Large is everything that has more than 100.000 (one hundred thousand) items. However, such tasks will take some time. Currently, we manage to crawl one Million Google keywords in roughly 7 days.

What kind of proxies does your SaaS use?

By default, our SaaS merely provides a wide array of public data center IP addresses. However, you can always purchase your own proxies and use them with our infrastructure. Our infrastructure allows you to reduce unnecessary traffic and thus decrease the consumed proxy bandwith & costs.

What proxy providers to you suggest?

For neutrality reasons, we currently don't suggest an specific proxy provider. This might change in the future.