Python workers, autoscale, SQL Alchemy, database connection pool
$2 USD / hour
Cerrado
Publicado hace más de 5 años
$2 USD / hour
Description of our setup:
We have an API which has a connection pool with 10 concurrent permanent database connections. For the API we use use sqlalchemy, because the database models were written for sqlalchemy. We have different types of python workers, lets call them type A, type B, type C. Each type of worker is processing different things by design.
Description of our problem:
We are using AWS EC2 instance autoscaling feature to scale all types of workers so that we could have for example 6 A, 20 B and 50 C type workers as needed. We are using the basic RDS instance for our MySQL database which I believe has 66 MaxConnectionLimit. The problem is that currently each worker is creating a permanent connection to the DB, which means that at some point we will hit the 66 concurrent connection limit and the app will crash.
The work scope
Have a solution so that the app would not crash when the total number of all types of instances is more than the MaxConnectionLimit. I will not be able to share our source code, therefore you will have to make a simple app that matches our design with some dummy workers, a simple way to launch a new worker so that we can test the case when there are more workers than the MaxConnectionLimit.
If you have read all the description and understand the task, please start your proposal with !SREEHC.