Senior Data Engineer – Azure – Publicis Sapient

Job Description:

As a Senior Data Engineer, your role entails translating client requirements into technical designs and implementing components for data engineering solutions. You will leverage a deep understanding of data integration and big data design principles to create custom solutions or implement packaged solutions. Independently driving design discussions is crucial to ensuring the overall solution’s health.

Your Impact:

• Data Ingestion, Integration, and Transformation
• Data Storage and Computation Frameworks, with a focus on Performance Optimization
• Analytics & Visualizations
• Infrastructure & Cloud Computing
• Data Management Platforms

Responsibilities:

• Develop functionality for data ingestion from multiple heterogeneous sources in batch and real-time.
• Implement functionality for data analytics, search, and aggregation.
• Utilize your skills to build end-to-end data pipelines.

Your Skills & Experience:

• Minimum 4 years of experience in Big Data technologies.
• Hands-on experience with the Hadoop stack, including HDFS, Sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, Hive, Oozie, Airflow, and other necessary components.
• Working knowledge of real-time data pipelines is an added advantage.
• Strong experience in at least one programming language, preferably Java.
• Hands-on working knowledge of NoSQL and MPP data platforms such as HBase, MongoDB, Cassandra, AWS Redshift, Azure SQLDW, and GCP BigQuery.
• Well-versed and working knowledge of data platform-related services on Azure.

Set Yourself Apart With:

• Good knowledge of traditional ETL tools (Informatica, Talend, etc.) and database technologies (Oracle, MySQL, SQL Server, Postgres) with hands-on experience.
• Knowledge of data governance processes (security, lineage, catalog) and tools like Collibra, Alation, etc.
• Understanding of distributed messaging frameworks like ActiveMQ, RabbitMQ, Solace, search & indexing, and Microservices architectures.
• Experience in performance tuning and optimization of data pipelines.
• Cloud data specialty and other related Big Data technology certifications.