Job Description
As a Senior Data Engineer, you will translate client requirements into technical designs and implement components for data engineering solutions. You will leverage your deep understanding of data integration and big data design principles to create custom solutions or implement package solutions. Your role will involve independently driving design discussions to ensure the overall health and performance of the solutions.
Your Impact:
- Data ingestion, integration, and transformation
- Data storage and computation frameworks, performance optimizations
- Analytics and visualizations
- Infrastructure and cloud computing
- Data management platforms
- Building functionality for data ingestion from multiple heterogeneous sources in batch and real-time
- Building functionality for data analytics, search, and aggregation
Your Skills & Experience:
- Minimum 6 years of experience in Big Data technologies
- Hands-on experience with the Hadoop stack: HDFS, Sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, Hive, Oozie, Airflow, and other components required to build end-to-end data pipelines
- Working knowledge of real-time data pipelines is an added advantage
- Strong experience in at least one of the following programming languages: Java, Scala, Python
- Hands-on working knowledge of NoSQL and MPP data platforms such as HBase, MongoDB, Cassandra, AWS Redshift, Azure SQL Data Warehouse, GCP BigQuery, etc.
- Proficient in data platform-related services on Azure and GCP
Set Yourself Apart With:
- Good knowledge of traditional ETL tools (Informatica, Talend, etc.) and database technologies (Oracle, MySQL, SQL Server, Postgres) with hands-on experience
- Knowledge of data governance processes (security, lineage, catalog) and tools like Collibra, Alation, etc.
- Familiarity with distributed messaging frameworks like ActiveMQ, RabbitMQ, Solace, search and indexing, and microservices architectures
- Experience in performance tuning and optimization of data pipelines
- Cloud data specialty and other related Big Data technology certifications
More Information
- Experience 5-10 Years
