I have few web scrapers written in python to scrape data from ecommerce websites. I want to run these scripts on Azure. What's the best way out? I am looking into Azure functions and Azure Webjobs since I want to have a automated schedules to trigger my scripts. Considering the amount of data to be scraped is huge, will Azure functions will be a costly affair ? And how scalable will be webjobs to get this job done? Are there any other cost effective ways to achieve the task ? (I am New to Azure).
1
-
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.Community– Community Bot2021-09-01 14:44:13 +00:00Commented Sep 1, 2021 at 14:44
Add a comment
|
1 Answer
You can try using the Selenium and HTTP Trigger for web scraping.
For Azure function cost on Consumption plan and Premium plan, you can refer to Azure Functions pricing and Reddit/Azure: Azure hosted Web Scraper, good or bad idea?
You can refer to Run Selenium on azure functions, How to create a Selenium web scraper in Azure Functions, Running webscraping Python file with HTML trigger on Azure and Create a Website Scraper for Azure Functions
2 Comments
Arunav Nag
Hey appreciate the guidance, but I was wondering is there any other way a scraper can be hosted other than using azure functions , so that I can trigger the scripts from a webjob which reads a scheduler table to determine the schedule time and triggers accordingly(no of websites to be scrapped will keep on increasing). I want to avoid azure functions considering the fact that there is a huge data to be scrapped on a monthly basis, so it won't be that cost effective.
jjaguirre394
@ArunavNag were you able to find an alternative? I am also looking for a way to do this in a cost effective way.
