Should we use this framwork? #238
-
|
We are a company receives a lot of IoT data as small json file every day in S3, but we haven't decided what to do with those data, should we use this datalake? The idea way to process those data is trigger s3 notification and kinesis to convert those to parquet, then process those data. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
From my experience, this framework is very heavy on GitOps. If you are going to have multiple teams creating multiple data piplelines per team and multiple environments per team (Dev, Test, Prod ....) this is for you. I think this is very much overkill for just 1 very specific pipeline. You will have to pay for multiple accounts, the administration, security etc. It's really not for a small medium project (or even a single big project). I am looking for alternatives like Airflow. |
Beta Was this translation helpful? Give feedback.
-
|
SDLF 2.x is a bit easier to work with (less repositories to deal with) but yes, this is a fair assessment - deploying SDLF for a single pipeline right now is probably a bit much. Hopefully things will get simpler, we're always open to feedback! |
Beta Was this translation helpful? Give feedback.
From my experience, this framework is very heavy on GitOps. If you are going to have multiple teams creating multiple data piplelines per team and multiple environments per team (Dev, Test, Prod ....) this is for you. I think this is very much overkill for just 1 very specific pipeline. You will have to pay for multiple accounts, the administration, security etc. It's really not for a small medium project (or even a single big project). I am looking for alternatives like Airflow.