Web Data Engineer

Job Description

Position: Web Data Engineer (Data Collection & Automation)

About Our Client:

Our client is a renowned tech company focused on leveraging data to drive insights and decision-making across various industries. They are seeking a skilled Web Data Engineer to develop and automate data collection processes that support their data analytics initiatives. This role is crucial for ensuring that accurate, high-quality data is readily available for analysis.

Key Responsibilities:

• Design, develop, and maintain scalable data collection architectures and workflows.
• Automate data extraction processes from various web sources, APIs, and data feeds.
• Implement web scraping strategies while ensuring compliance with legal and ethical standards.
• Collaborate with data analysts and data scientists to understand data requirements and ensure that the data collected meets their needs.
• Perform data cleansing, transformation, and validation to ensure data integrity and accuracy.
• Monitor and troubleshoot data pipelines, addressing any issues or discrepancies in a timely manner.
• Document processes, workflows, and data sources for visibility and reproducibility.
• Stay updated on industry trends in data collection techniques, tools, and technologies.
• Participate in code reviews and contribute to continuous improvement in data engineering practices.

Requirements

Qualifications & Skills:

• Bachelor’s degree in Computer Science, Data Science, or a related field.
• Proven experience as a Data Engineer, Data Scientist, or in a similar role focused on data collection and automation.
• Strong programming skills in languages such as Python, Java, or Scala.
• Experience with web scraping tools and technologies such as Beautiful Soup, Scrapy, Selenium, or similar frameworks.
• Proficiency in SQL and experience with databases like MySQL, PostgreSQL, or NoSQL solutions.
• Familiarity with cloud platforms (AWS, Google Cloud, or Azure) and data processing tools (Apache Spark, Apache Airflow, etc.).
• Understanding of data governance, data quality, and data management principles.
• Excellent analytical and problem-solving skills, with a keen attention to detail.
• Strong communication skills to collaborate effectively with technical and non-technical teams.

Preferred Qualifications:

• Experience with ETL processes and data integration tools.
• Knowledge of data visualization tools (Tableau, Power BI, etc.) is a plus.
• Familiarity with version control systems such as Git.
• Experience working in Agile development environments.

Benefits

Full Time, Freelance and Contract roles available