Data & Back-End Engineer

Requirements:

• Experience working in agile software development teams: 5+ years;
• Experience with SQL language: 3+ years;
• Python experience: 3+ years;
• Experience working with cloud platforms like Amazon, Azure or Google Cloud Platform: 2+ years;
• Experience working with Docker: 2+ years;
• Experience in software release management: Git, CI/CD;

Responsibilities:

• Implement and maintain different data ingestion/processing pipelines for existing and new projects, extending and improving our existing data pipeline stack built using Airflow, Cloud Dataflow, and Bigquery.
• Handle architectural and design considerations such as performance, scalability, reusability, and flexibility constraints.
• Evaluate the data integrity from each source, finding the way to ingest, curate, and normalize it.
• Document and communicate changes inline code, PR, tickets and release notes with the engineering, product and data team.
• Work jointly with the research team to make prototypes out of proof-of-concepts and operationalize prototype pipelines.
• Work jointly with the product team to understand the needs of the output datasets and find interconnections between projects.
• Work jointly with the front-end and data to make sure the data is being used correctly.
• Implement uptime monitoring for the multiple ingestion pipelines and set up continuous QA metrics on processing pipelines.
• Debug data issues working closely with team members from other areas of the organization (subject matter experts or ML scientists).
• Modify/Fix/Implement new API endpoints owning the full development cycle. Including supporting QA for test automation and product for feature release and documentation.