Data Engineer/Architect

The Data Architect will play a pivotal role in designing, implementing, and maintaining the data infrastructure for a complex Big Data Platform. This individual will be responsible for ensuring data quality, integrity, and security while optimizing data storage, processing, and retrieval.

Key Responsibilities

• Data Architecture Design: Develop and maintain a comprehensive data architecture that aligns with business objectives and supports the scalability and performance of the Big Data Platform.
• Data Modeling: Create data models that accurately represent the structure and relationships of data entities, ensuring data consistency and integrity.
• Data Integration: Design and implement data integration strategies to ingest data from diverse sources into the Big Data Platform, addressing data quality and transformation requirements.
• Data Quality: Establish data quality standards and implement processes to ensure data accuracy, completeness, and consistency throughout the data lifecycle.
• Data Governance: Develop and enforce data governance policies and procedures to protect sensitive data and maintain data integrity.
• Performance Optimization: Monitor and optimize the performance of the Big Data Platform, identifying bottlenecks and implementing improvements to enhance data processing efficiency.
• Technology Selection: Evaluate and select appropriate data technologies and tools based on business needs and technical requirements.
• Collaboration: Work closely with data engineers, data scientists, and other stakeholders to ensure effective communication and coordination.

Required Technical Skills:

• Big Data Technologies: Proficiency in Hadoop, Spark, Kafka, or other relevant Big Data frameworks.Experience with Airflow is a must.
• Data Warehousing and Data Marts: Experience with data warehousing concepts, ETL processes, and data mart implementation.
• Database Technologies: Strong understanding of relational databases PostgreSQL, NoSQL databases, Relational Databases.
• Data Modeling: Proficiency in data modeling techniques, including ER diagrams and dimensional modeling.
• Data Quality Tools: Familiarity with data quality assessment and profiling tools.
• Programming Languages: Knowledge of Python, SQL, or other programming languages relevant to data processing and analysis.
• Cloud Platforms: Experience with cloud platforms (e.g., AWS, Azure) and cloud-based data services