We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will be responsible for designing, implementing, and maintaining our data infrastructure to support our B2B intelligence platform.
Responsibilities
- Design, build, and maintain scalable data pipelines for collecting, processing, and storing large volumes of business data
- Develop ETL processes to integrate data from various sources, including web scraping, APIs, and third-party data providers
- Implement data quality checks and monitoring systems to ensure data accuracy and integrity
- Optimize data storage and retrieval processes for high performance and scalability
- Collaborate with data scientists to implement machine learning models in production environments
- Work with the backend team to design and implement APIs for data access
- Collaborate with data scientists to develop and deploy machine learning models.
- Implement data security and privacy measures to protect sensitive information.
- Stay up to date with the latest big data technologies and best practices
Requirements
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
- 4+ years of experience in data engineering roles
- Strong programming skills in Python, Scala and/or Java
- Expertise in SQL and experience with NoSQL databases (e.g., MongoDB, Cassandra)
- Proficiency with big data technologies such as Apache Spark, Hadoop, and Kafka
- Experience with cloud platforms (AWS, GCP, or Azure) and their data services
- Familiarity with data warehousing concepts and ETL processes
- Experience with data warehousing solutions (e.g., AWS Redshift, Snowflake).
- Knowledge of data modelling, data architecture, and data pipeline design
- Experience with version control systems (e.g., Git) and CI/CD practices
- Excellent problem-solving skills and attention to detail
Preferred Qualifications
- Experience in the B2B data or sales intelligence industry
- Familiarity with web scraping techniques and tools
- Knowledge of data privacy regulations (e.g., GDPR, CCPA)
- Experience with real-time data processing and streaming architectures
