Realtime Hadoop data ingest with Kafka, Storm, HBase (OpenTSDB), and Postgres on AWS

SKILLS REQUIRED This project requires a data engineer with development experience setting up a realtime data ingestion demo using open-source Apache tools Kafka and Storm. It also requires someone that's familiar with setting up a Hadoop environment using the configuration tool Puppet. If you have familiarity with these tools, this will be a fairly easy but well-paying project! BASIC DESCRIPTION I need to ingest realtime time-series data (from an API such as Twitter's) into four different data stores: HDFS, OpenTSDB (a time-series database built on HBase), a PostgreSQL database, and Amazon Web Services S3. First, Kafka will ingest from the API and send the raw messages (data) to both HDFS and Storm. You will need to write a very simple data transformation in a Storm bolt and then another Storm bolt will send the transformed data to those four data stores. Overall, this is a fairly simple development process. Most of the work of the project is just in installing and configuring all of the different software components of this architecture. I will provide an AWS EC2 cluster for it all to run on. You will need to first set up a Puppet environment on top of the AWS cluster. Within that Puppet environment you will need to install, configure, and connect together all the various software components: Hadoop (HDFS), Zookeeper, YARN, Kafka, Storm, OpenTSDB and HBase, PostgreSQL, as well as any associated subcomponents or any other open-source software packages these require to run. I have included a software architecture diagram. PROJECT DELIVERABLES The final goal of the project will be to have a working system that actually ingests data from an external API such as Twitter in realtime and stores that data in these four data stores with low latency. You will need to create a simple demo that continually demonstrates all of this happening using data from the API. You should also be able to walk me through all of the code and you will be required to provide very simple documentation of all the steps you took to set all of this up. In addition, any advice you can provide me along the way on proper cluster setup and sizing, hardware config, or parallelization count (for Storm and Kafka) would also be appreciated! ADDITIONAL INFO Once complete, I will then take this project and add my own code it and expand it. For example, I will be ingesting stock market data (financial time-series) through this platform and writing complex transformations and realtime processing for the data in Storm before it is stored. So the platform you set up should be designed and developed in a way that it can be easily expanded and added to. I can provide much more details to the chosen bidder for this project when they are ready, and will help them get started and assist with everything (such as accessing the AWS cluster) as needed. If you do a good job, I will give you a perfect rating, great feedback, and have a lot of optional paid follow-on work for you to do and make more money.