Create a dynamic script to scrape multiple websites.
$250-750 USD
Pagado a la entrega
Create a dynamic script in NodeJS to scrape 15+ sites using configurable json.
Requirement:
-Should be able to scrape lazy loading sites
-Download images
-Go through all pagination pages and scrape the child pages
-If element is a product link "click" that link and scrape that child page
-Get all below elements and retrieve correctly
See below for understanding how to set it up.
//Example configurable object
[
site1: {
url: "[login to view URL]",
parent: {
productList: "div.product-list",
product-link: "a.product-url",
product-name: ".product-info .product-title",
product-image: ".product-images img[src attribute]",
product-price: {
list-price: ".product-price span.old-price",
sale-price: ".product-price [login to view URL]"
}
pagination: "button.load-more-btn",
product-page: {
name: "form#product h1",
price: {
list-price: ".price .old",
sale-price: ".price new"
},
colors: "#options-articles li",
sizes: "#options-variants li",
description: ".description p",
images: {
main: "#images .product-image-box img[src attribute]",
thumbs: "#images .thumbs .product-thumbs li"
}
}
...
}
},
site2: {
Similar to above Site1
...
}
...
]
==================================================================================
//Product Class
Class Product() {
Product(name, link, image, price, colors, sizes, description) {
[login to view URL] = name;
[login to view URL] = image;
[login to view URL] = price;
[login to view URL] = colors;
[login to view URL] = sizes;
[login to view URL] = description;
}
}
===================================================================================
//Main Scraper Class
Class Scraper() {
Scraper(site) {
[login to view URL] = site;
}
getName(attr) {
return name value;
}
getLink(attr) {
return link value;
}
getMainImage(attr) {
return main value;
}
getThumbImages(attr) {
return thumb array;
}
getPrice(attr) {
return Price Object(list, sale);
}
getColor(attr) {
return Colors array;
}
getSizes(attr) {
return Sizes array;
}
getName(attr) {
return desc value;
}
hasPagination() {
if([login to view URL] element exist)
//scrape all pagination pages to the last page
}
downloadImage() {
if([login to view URL] element exist)
//download image
src = getImage();
download(src);
}
getProductInfo(product) {
//Go to product page and scrape using "product-page" attribute from json sample.
return {
name: String,
price {
old: String
new: String
},
images : {
main: String src,
thumbs: [array]
},
sizes: [array],
colors: [array],
description: String
}
}
getListOfProducts() {
//Use [login to view URL] to get listOfProducts
ArrayOfProducts = [];
for( items in listOfProducts ){
//create new product
product = new Product( getProductInfo() );
//add it to list
[login to view URL]( product );
}
}
}
Nº del proyecto: #15659228
Sobre el proyecto
17 freelancers están ofertando un promedio de $430 por este trabajo
Greeting, I have understood your Create a dynamic script to scrape multiple websites. task and can do it with your 100% satisfaction. Please ping me for more discussion. Relevant Skills and Experience I have more th Más
Over 8 +years experience writing almost exclusively web scraping code. I've done it all. I can scrape all LinkedIn profile My languages in order of experience and use is Python, JavaScript, PHP. Python libraries ( sele Más
hello,we are team of experts and can do this project ,so lets discuss about it in detail and then start the project Relevant Skills and Experience Java, Javascript, node.js, Python, Web Scraping Proposed Milestones $ Más
I have gone up through the job description I completely understood your all requirement, So I can easily Create a dynamic script to scrape multiple websites and able to start right now. Relevant Skills and Experience Más
I have briefly read the description on scrap, and I can deliver as per the requirements however I need us to discuss for more clarity on the details, deadline and budget as well. Relevant Skills and Experience python Más
I am an IITK graduate and I have 11 years of experience in software development. I have 100% completion rate and I have finished all the projects with the highest level of customer satisfaction. Relevant Skills and Ex Más
Hi, I have gone through the job description I completely understood your all requirement and I am available with complete expertise in Python, node.js and past experience to meet your job requirement Relevant Skills Más
I'm guessing you want to scrape these data to build a product or price comparison aggregator. I just finished a similar project although for Airline booking service using Nodejs. Relevant Skills and Experience Have b Más
Hey, I'm interested in your project. Please send me a message so that we can discuss more. I can do trial work before hire me sir , will wait for your reply, Thanks, Syed
Been developing with MERN stack for 3+ years and would be able to finish this within 2 days easily. Let me know when to start Relevant Skills and Experience Been developing with MERN stack for 3+ years and would be ab Más
