Senior Data Engineer – Gauteng
The Harambee Software Development team is looking for an exceptional Senior Data Engineer to join their dynamic team based in Johannesburg. The objective of the Senior Data Engineering role is the collection, storing, processing and analysing of large sets of data, with the primary goal of choosing optimal solutions to use for maintenance, implementation and monitoring thereof.
The Senior Data Engineer is also responsible to partner with the Data Engineering Manager on database design and strategy to enable Harambee’s future strategy and technology roadmap. Furthermore, the Senior Data Engineer will be doing all that in an organisation that changes lives and has a culture of teamwork, openness and collaboration. If this prospect excites you, then you may be the person we are looking for.
THE TECHNICAL CHALLENGE
Harambee is building a ground-breaking, scalable, future-fit technology platform that changes lives by helping young first-time work-seekers find jobs and work experiences. Our technology needs to cater for the unique, personalised employment journey of each workseeker – ultimately using the power of big data to help them find their first job.
We run an Agile Development Programme in a DevOps infrastructure and culture. Our cloud-native technology stack is predominantly hosted in Google Cloud Platform with services written in .Net Core and Python. The data landscape includes MYSQL, Big Query and SOLR. Harambee has also developed a Learning Management System and Assessment platform using Open edX. This technology stack is currently being revised to drive a strategy of simplification, business user enablement and data integrity.
Google did a case study on us. READ IT.
1. Planning and strategy:
- Engage with various stakeholders to understand the business and data analysis goals in context of Harambee’s strategy and vision
- Design data solutions for real-time and batch environments
- Document the planning, implementation and operation of the data platform
- Select and integrate any big data tools and frameworks required to enable the business and data analysis goals
- Work closely with the developers and business analysts to design and ensure delivery of data requirements
- Assist in data analysis and data modelling
- Advise on data design and architecture
- Ensure security on data platforms
- Train and facilitate data adoption across Harambee business users
- Prepare analytical data-sets for data mining
- Execute the following database processes:
- Ensure effective infrastructure for processing large volumes of transactional data
- Implement data retention policies
- Monitor performance and advise on necessary infrastructure changes
4. Data analysis:
- Support the Business Intelligence and Data Science teams in ad hoc data analysis to derive
- business insights
- Support the Business Intelligence team on data validation and quality assurance to ensure accurate dashboards and reports
- 8 – 12 years’ experience as a data engineer
- Proven track record of quality delivery
- Comfortable explaining data models, data structures and analytical findings to technical and non-technical audiences
- Willingness to learn
- A passion for knowledge sharing and upskilling other team members
- Data skills (SQL, document stores)
- Large scale ETL (Apache beam or Apache spark, SSIS)
- Cloud experience (Google Cloud Platform, Azure, or AWS)
- R (nice to have)
- Python (nice to have)
- MySQL (nice to have)
- Knowledge of BI and data governance toolsets
- A degree or diploma in Science or Engineering (Mathematics, Statistics, Computer Science or related field)
- Has worked at a company of similar size and complexity to Harambee over the past 24 months
- Importantly, you must fit with the Harambee culture, be passionate about our mission and be an individual who leaves things better than you found them.
- Keen to work for purpose and be inspired by the bigger social problem we want to solve
- Keen to learn and grow
- Comfortable with new systems, designs and processes and willing to help set these up
- Ability to collaborate with technical and non-technical teams