Azure-ADLS, Databricks, Snowflake, Spark, Scala, PySpark, Python, Kafka
Depends on Experience
Work from home available
Cloud Data Engineer Lead
- Programming: Spark, Scala, PySpark, Python, working with JAR, WHL files, packages etc. Should be Expert level, develop as well as review other developers code/programs. Should be expertise in troubleshooting any issues in code, versions, libraries, IDEs, Maven, pom.xml etc.
- Cloud: Azure-ADLS (Gen1 & Gen2), BLOB, Databricks, Databricks CLI, Snowflake - Expert level, as we develop ETL programs across all these platforms.
- Orchestrate all the ETL/jobs thru Apache Airflow
- Streaming/Micro Services: Confluent Kafka, Expert level - Develop streaming jobs using Kafka, SpringBoot, Spark Scala and deploy these in Docker/K8s
- Deployment: GitHub, Jenkins, Docker/K8s, Must have experience working on these as we need to manages GitHub repos, PRs, Version and deploying with jenkins and troubleshooting
- Must lead offshore team, assign, monitor and manage teams work, from work estimations till deployments. Communicate between Clients and offshore teams to resolve blockers/issues.
Good to Have:
- Knowledge on DevOps, to understand the deployment process, troubleshoot any issues in deployments, particularly with Jenkins, Docker, K8s, Helm etc.
- Retail data domain knowledge as we are dealing with a lot of retail data in ETL programs or PowerBI datasets
- Knowledge on PowerBI, creating datasets and loading data into PowerBI
- Knowledge on Data Science models, MLflow