Data Engineer

Job Description

Job title: Data Engineer

Location: UK

Duration: 6-month contract Inside IR35

Working Mode: Fully Remote

Vetting: Active SC Clearance

Job Description:

Data Engineer Skills and Technologies Data Engineering:
- Strong understanding of data concepts
- data types, data structures, schemas (both JSON and Spark), schema management etc
- Strong understanding of complex JSON manipulation - Experience working with Data Pipelines using a custom Python/PySpark frameworks
- Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
- Strong understanding of Data Security principles
- data owners, access controls
- row and column level, GDPR etc including experience of handling sensitive datasets
- Strong problem solving and analytical skills, particularly able to demonstrate these intuitively

Requirements

Required Skills:

Languages / Frameworks
- JSON
- YAML
- Python (as a programming language, not just able to write basic scripts. Pydantic experience would be a bonus.)
- SQL
- PySpark
- Delta Lake
- Bash (both CLI usage and scripting)
- Git
- Markdown
- Scala (bonus, not compulsory)
- Azure SQL Server as a HIVE Metastore (bonus) Technologies
- Azure Databricks
- Apache Spark
- Delta Tables
- Data processing with Python
- PowerBI (Integration / Data Ingestion)
- JIRA