I need to setup an ELK server, it will:
1. Crawl the web, where, (a) I should be able to define the URLs to start the crawling from, and limit the crawl space (e.g., search just the configured site, search configured site and linked webpages), and (b) Index all metatags in the document head section.
2. Index Twitter streams, where, (a) I should be able to configure the source (@user or #hashtag), (b) Index the tweet and the linked webpage, and (c) perform sentiment analysis on the tweet.
3. Index configured collections from MongoDB databases, where (a) the databases reside behind the firewall on other servers.
4. Build a unified search interface to search all results through the same interface (a) Text Search engine (I prefer a NodeJS engine) and (b) a set of Kibana dashboards to visualize the search results.
Note for task 1: I have evaluated Fess and Nutch crawlers, I think Fess with some custom Elasticsearch templates will work for me for the most parts.
Though I am new here but my team has 4 years of experience into Website Design and Development across all Platforms especially on . Can very well execute this Project and can start immediately.