CSHARK (May 2023 - Present)

Senior Data Engineer (August 2024 - Present)

Source integration:
- Onboard data from various APIs using Spark's Python Data Source API.
- Create metadata driven solution to incrementally load data from SQL Server to Delta Lake using CDC.
- Develop both classic and Lakeflow Declarative Pipelines in Databricks.
CI/CD and dev environment:
- Establish complete automated CI pipeline with unit and integration testing, package publishing and deployment using Databricks Asset Bundles in Gitlab CI.
- Design and implement mock services for Python Data Source API integration testing.
- Create modern Python development environment with pre-commit hooks, UV dependency management and Docker containerization.
Performance optimization:
- Optimize Spark Jobs performance for improved cost efficiency.
- Build reusable Python libraries to be shared across projects.

Creation of models to support transactional processing as well as BI analytics.
Data Processing Solutions:
- Development and integration of custom libraries and Spark Jobs for Synapse.
- Creation of parametrized data processing pipelines (ADF).
Data Testing Strategy: Development and implementation of comprehensive data testing strategies.
Automated Testing: Execution of unit, integration, and data validation tests.
CI/CD Pipeline Development: Leading GitLab CI/CD pipeline creation for Azure Synapse resource deployment.
Team Onboarding: Knowledge transfer and training of new team members.