Job Description:
We are hiring DevOps Architect who can work for UK based company remotely.
Responsibilities:
Design, Develop and Automate processes in AWS Glue pipelines from S3 with ORC/Parquet/Text Files.
Defining standards, guidelines and access mechanism (naming conventions, IAM Users and Roles, SSO profile)
Define Data Extraction, aggregations using Python, pySpark using relevant libraries
Managing external and managed tables with partitions in S3 and Redshift
Create libraries for user defined functions UDF
Build S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup on AWS.
Must Have
Overall 10+ years of experience on BI/Data Platform
5+ yrs working experience on AWS platform using data services
Working experience in S3, Redshift, Glue and ingestion services like DMS, Appflow, Data Transfer/Data Sync
Create state machines interacting with lamda, glue, clouldwatch, SNS, and even bridge.
Scripting Languages: Python, pySpark
Understanding of cloud watch, SNS and even bridge
Excellent analytical and problem-solving skills.
Experience in CI/CD pipeline (Gitlab). Preferably AWS Code Commit, Code Build, Code Deploy
Good to Have
Exposure to Big Data Ecosystems (Lake Formation, AWS EMR)
Exposure to Alembic
Experience working in Agile, Scrum And this is bare minimum needed
Overall 10+ years of experience on BI/Data Platform with 5+ yrs working experience on AWS platform using data services
Working experience in S3, Redshift, Glue and ingestion services like DMS, Appflow, Data Transfer/Data Sync
Create state machines interacting with lamda, glue, clouldwatch, SNS, and even bridge.
Experience in CI/CD pipeline (Gitlab). Preferably AWS Code Commit, Code Build, Code Deploy