Senior Data Engineer

🚀 Senior Data Engineer – Mindera

At Mindera, we build technology products that create meaningful business impact. We work with modern cloud technologies and encourage engineering ownership, collaboration, continuous learning, and self-organization.

Our Data Engineering teams build scalable, reliable, and high-performance data platforms that enable analytics, reporting, machine learning, and data-driven decision-making.

Requirements

🧠 What We’re Looking For

Key Responsibilities

• Design and develop scalable ETL/ELT data pipelines using AWS Glue, PySpark, Python, and SQL.
• Develop and maintain distributed data processing applications using Apache Spark.
• Build and orchestrate data workflows using Apache Airflow.
• Develop AWS Glue jobs for batch and incremental data processing.
• Work with Amazon S3 and other AWS services to build cloud-based data solutions.
• Perform complex data transformations, cleansing, validation, and enrichment.
• Write optimized and maintainable SQL queries for large datasets.
• Optimize Spark and PySpark jobs for performance, scalability, and resource utilization.
• Implement data partitioning, caching, broadcast joins, and other Spark optimization techniques.
• Design reliable and fault-tolerant data pipelines with appropriate error handling and retry mechanisms.
• Implement data quality checks and monitoring across data pipelines.
• Troubleshoot production data pipeline failures and performance issues.
• Work with large-scale structured and semi-structured datasets.
• Participate in technical design and architecture discussions.
• Conduct code reviews and ensure adherence to engineering best practices.

Required Skills

Python

• Strong programming experience in Python.
• Experience writing production-quality, modular, and reusable code.
• Knowledge of exception handling, logging, testing, and debugging.
• Familiarity with Python data-processing libraries.

SQL

• Strong SQL skills with experience handling large datasets.
• Complex joins and subqueries.
• CTEs and window functions.
• Aggregations and analytical queries.
• Query optimization and performance tuning.
• Data deduplication and incremental processing logic.

Apache Spark / PySpark

• Strong hands-on experience with Spark and PySpark.
• Understanding of Spark architecture and execution model.
• DataFrames and Spark SQL.
• Transformations and actions.
• Partitioning and repartitioning.
• Shuffle optimization.
• Broadcast joins.
• Caching and persistence.
• Handling data skew.
• Spark job troubleshooting and performance optimization.

AWS Glue

• Hands-on experience developing AWS Glue ETL jobs.
• Experience with Glue Data Catalog and Crawlers.
• Glue Spark jobs.
• DynamicFrames and DataFrames.
• Job bookmarks and incremental processing.
• Partitioned data processing.
• Schema evolution and data quality.
• Integration with Amazon S3 and other AWS services.

Apache Airflow

• Experience designing and developing Airflow DAGs.
• Task dependencies and scheduling.
• Operators and sensors.
• Retries and failure handling.
• Backfills and catchup.
• XCom and task communication.
• Monitoring and troubleshooting DAGs.
• Designing idempotent and reusable workflows.

AWS

• Strong experience with AWS data services.
• Amazon S3.
• AWS Glue.
• IAM.
• CloudWatch.
• Familiarity with services such as Lambda, Athena, Redshift, or EMR is an advantage.

Preferred Skills

• Experience with modern data lake / data warehouse architectures.
• Experience with Parquet and other columnar storage formats.
• Knowledge of data modeling and dimensional modeling.
• Experience with CI/CD and Git-based development workflows.
• Familiarity with Docker and infrastructure-as-code concepts.
• Experience with Agile/Scrum methodologies.
• Experience working with high-volume and large-scale datasets.
• Understanding of data governance, security, and data quality practices.

Senior-Level Expectations

• Independently own data engineering projects from requirements through production.
• Design scalable and maintainable data architectures.
• Make appropriate technology and architecture decisions.
• Identify and resolve performance bottlenecks.
• Troubleshoot complex production issues.
• Establish coding, testing, and data quality standards.
• Mentor other engineers and contribute to technical decision-making.
• Communicate effectively with both technical and non-technical stakeholders.

Benefits

We offer

• Flexible working hours (self-managed)
• Annual bonus, subject to company performance
• Access to Udemy online training and opportunities to learn and grow within the role

We partner with our clients, to understand their products and deliver high-performance, resilient and scalable software systems that create an impact on their users and businesses across the world.

You get to work with a bunch of great people, and the whole team owns the project together.

Our culture reflects our lean and self-organisation attitude.

Our offices are located: Porto, Portugal | Aveiro, Portugal | Coimbra, Portugal | Leicester, UK | San Diego, USA | Chennai, India | Bengaluru, India