Shaw Daniels Solutions
Data Engineer
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Data Engineer
Remote
About the Role
As a seasoned and flexible Data Engineer, you will play a key role as part of a skilled and multi-disciplinary development team delivering innovative solutions as part of the DX Programme.
You will be responsible for building, maintaining, and securing multiple Azure components primarily focused on Databricks, Synapse Link, and AIS resources that provide the foundation for the wider Data Platform. You will be experienced and comfortable in working with a hybrid mix of on-premises and Azure cloud resources. You will be approaching your work diligently with a security-first mindset and be open to working with DevOps delivery and deployment mechanisms, including CI/CD pipelines and Infrastructure as Code (IaC). You will be able to coach developers in achieving the best security practices, connectivity options, high-performance, and appropriate diagnostic approaches as they access and leverage those resources. You will play a pivotal role in shaping our environment and work with a range of internal and external stakeholders across a wide range of disciplines. Your communication skills via all channels will be first class and you will be able to communicate effectively with a wide range of technical and non-technical people alike.
You will be comfortable and experienced in working with a range of differing data formats, storage, and delivery mechanisms, data retrieval, and manipulation to an advanced level with a focus on core Microsoft technologies including D365 F&O and D365 CE. You will be building pipelines of data to support a wide range of mission and business-critical functions in a highly complex and fast-paced environment.
Additionally, you will be called upon to learn the necessary skills and provide additional support to the team’s Database Administrator (DBA). Likewise, you will coach and mentor the DBA to help provide a wider Azure administrator function split across the two roles. Thus, providing cover to each other when not available and increasing team resilience. It is here you will gain and provide insight into wider Azure administration topics.
Overall Role Objectives
- Provide all administrative and development capabilities including but not limited to building, deploying, networking, maintaining, and securing of a variety of Azure resources primarily relating to Databricks, Synapse Link, AIS, and the data that passes through them. To ensure all resources meet operational SLAs. To build and deploy resources using ClickOps and IaC methodologies as directed.
- As part of Service Delivery, provide support to troubleshooting and resolve issues with a variety of on-premises and Azure resources and their usage in accordance with SLAs.
- Collaborate closely with the team and stakeholders at all levels to design, implement, and monitor a high-performance and diverse data platform, its wider environment, and the components that operate within it.
- Collaborate closely with the team and stakeholders at all levels to design, implement, and monitor a range of monitoring solutions to ensure optimal performance of our data platform, its wider environment, and the components that operate within it.
- Collaborate closely with the team and provide support, advice, and troubleshooting expertise to a diverse group of developers working primarily in the integrations and data engineering spaces.
- Collaborate with the Database Administrator to learn the skills necessary to provide continuity of service covering absences and providing additional resources when called upon.
- Collaborate with stakeholders at all levels to gather requirements, analyse processes, and recommend optimal solutions.
- Fully participate in team planning and work with colleagues to continuously improve the team’s performance.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Tasks/Responsibilities
Technical Excellence
- Extensive experience in Databricks, Synapse Link, and AIS resources
- Build and maintain complex, performant data pipelines between a multitude of endpoints, in a variety of formats as stipulated by specifications
- Administering, securing, and provisioning a range of components and the underlying resources for AIS and data engineering solutions.
- Can support and maintain application or component connectivity, secure end-points, and resolve issues with wider network connectivity.
- Proficient in creating IaC code as required.
- Maintain all Databricks, Synapse Link, and AIS resources within remit ensuring they are secure, functional, and performant to meet all operational SLAs.
- Monitor for and apply and MS updates as required. When necessary, advise wider team on update processes, planning appropriate deployments, testing, and re-configuration as required.
- Maintain, develop, and improve monitoring solutions to be proactive in recognizing issues before they become problems, leveraging a mix of technologies and components including Application Insights and Log Analytics. Become highly proficient in KQL.
- Assist in implementing best practice for information and document management.
- Gather requirements, make recommendations, and estimate effort to complete work.
- Mentor the development team in best practice security principles and aid in the creation of high-performance, highly resilient data engineering and integration solutions.
- Take a proactive role in learning wider Azure SQL administration.
- Assist the DBA as and when necessary. Provide cross-functional support to cover absences and workloads as priorities allow.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Operational Excellence
- Continually demonstrate professionalism across all areas of the role, as well as interaction with the business and third parties.
- Must be self-directed, detail-oriented, and have excellent written & verbal communication skills.
- Actively participate in knowledge sharing across the department.
- Develop a culture of collaboration and business partnering within your own team, the wider IT team, and the business - building and maintaining strong relations across all areas.
- Ensure that appropriate responses are made in a timely fashion to requests/instructions from management, the business, or the SRI Group.
Essential Knowledge, Skills & Experience
Experience/Knowledge
- 5 years’ data engineering, ETL/ELT, and integration development experience with a proven track record in designing and developing complex data pipelines
- 3 years’ development & administration of Azure cloud computing, storage, and integration technologies deployed via ClickOps and IaC methodologies with a particular focus on data engineering via Databricks, Synapse Link, and AIS
- 3 years’ practical experience of networking including DNS, VPN, Private & Service Endpoint configuration, and troubleshooting
- 1 year’s practical experience of incorporating Application Insights and Log Analytics into Azure solutions
Desirable Knowledge, Skills & Experience
Experience/Knowledge
- Azure DevOps
- Exposure and administration of Microsoft systems beyond Azure cloud
- Exposure to multi-platform integration, preferably involving MS Azure
- Hub/Spoke Azure implementation and network architecture
- Knowledge of MS Power Platform technologies particularly Dataverse, PowerApps, and Power BI
- Batch scripting languages
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location