We are looking for a Data Engineer that will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also, be responsible for integrating them with the architecture used across clients.
About Data Masters
Data Masters is a company specialized in the field of Data Science. We are a dedicated and passionate team, extremely motivated to address the challenges of any company and help them become Masters of their Data.
Our team members possess knowledge and expertise in Data Warehousing, Data Engineering, Machine Learning, and Analytical Product Development. We support clients from all over the world, and we would love to welcome new team members that share our passion, mission, and vision.
Job description
Responsibilities
- Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
- Implementing ETL process
- Monitoring performance and advising any necessary infrastructure changes
- Defining data retention policies
Skills and Qualifications
- BSc/BA in Computer Science, Engineering, or relevant field
- Good understanding of distributed computing principles
- Experience with integration of data from multiple data sources
- Knowledge of various ETL techniques and frameworks
- Python 2+ years of experience
- Spark/PySpark experience
- Proficiency in databases and advanced queries (dynamic queries, procedures, etc.)
- Writing complex shell scripts
- Experience using and developing APIs
- Knowledge of cloud platforms (AWS, Azure, Google Cloud)
- Experience with handling big data and scaling up
- Statistics and mathematical knowledge
- Analytical skills
- Data wrangling
- Data processing, ETL, and/or ELT
- Git
