Web-scraping a big dataset

Este proyecto recibió 22 ofertas de freelancers talentosos con una oferta promedio de €167 EUR.

Obtén cotizaciones gratis para un proyecto como este
Presupuesto de Proyecto
€30 - €250 EUR
Ofertas Totales
Descripción del Proyecto

The task is simple. You have to go to this website [1] and download the publicly available data. Manually it would take forever. That’s why you will have to study the JavaScript that creates the requests. You are expected to write a script which makes POST requests and batch downloads the data.


Currently you can get 24H data for all stations. Using a script create requests for all days one by one (without overwhelming the server). Requests are processed and then the resulting data is dumped on an ftp server [2] with the respective request number. 24H data for all stations should be about 1.5 GB of data. Download them and put them on a server where we can access them in bulk.

Download procedure to be automated:

1) Enter an e-mail address

2) Select data format: Mini-Seed

3) Select time interval: Loop over A single day from 1/1/2009 to 16/10/2016

4)Select stations: Select all stations and all channels

5) Submit request

6) Wait and collect the completed request from [2]

[1] [url removed, login to view]

[2] ftp://[url removed, login to view]

Buscando hacer algo de dinero?

  • Establece tu presupuesto y período de tiempo
  • Describe tu propuesta
  • Consigue pago por tu trabajo

Contrata Freelancers que también oferten en este proyecto

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online