We are looking for an experienced Big Data Engineer that can help us develop set of Integrations between our product, EIQ Platform and various BigData platforms with emphasis on Hadoop and Hadoop eco-system. Integrations go beyond modules for our platform and also include development on top of Big Data platforms as well as all-round Solution Architecture with focus on performance and flexibility. Think about connecting world of Intelligence with waste pools of sensory data along with some cool analytics challenges.
This is a Senior Engineering role which requires strong leadership qualifications, ability to operate independently and occasionally in an consultative manner with customers as well. You MUST be a passionate Big Data visionary since you will be give an opportunity to create our portfolio of Big Data Applications and Integrations.
Our product, EIQ Platform, is a web UI / REST API-based platform built with Python 3 on top of frameworks like Flask, SQLAlchemy, and Celery. It relies on data stores like PostgreSQL and Elasticsearch. The codebase consists of several modules/libraries spread over the application.
As an ideal candidate, you have relevant experience working with various BigData platforms with emphasis on Hadoop, but also are experienced with large, robust, high-performance applications using similar technologies we use; you can responsibly build on and improve them.
- Independently build new Big Data products Integrations and Big Data products related Extensions and resolve issues in an Agile environment.
- Independently build new EIQ Platform Integrations and Integrations related product features and resolve issues in an Agile environment.
- Deliver work with a DevOps approach (you run the code you wrote) and compliant with the team's Definition Of Done.
- Profile Python code, write unit and integration tests, take part in performance testing.
- Perform code reviews and participate in technical design sessions with the rest of Engineering team.
- Closely collaborate with fellow Engineers and Product team members to design and implement product solutions that meet our quality criteria.
- 7+ years’ professional experience.
- 5+ years’ experience building robust applications and/or products using Java or Python.
- 3+ years’ experience building on on top of Hadoop and relevant eco-system.
- Understanding Big Data challenges and techniques in use for building modern data exploratory tooling and integrations.
- Understanding design patterns in use for modern software development using Java and/or Python.
- Appreciation for clean code, thorough testing, and API design.
- Critical and innovative thinking.
- Demonstrable track record working with some of the key Big Data technologies: Hadoop, Hbase, Hive, HDFS, Pig, Impala ...
- Strong experience with ELT, specifically around data wrangling and transformation.
- Demonstrable track record working with the following Python frameworks: Flask, SQLAlchemy, Celery.
- Demonstrable track record working with the following data stores: PostgreSQL, Elasticsearch.
- Demonstrable experience working with the following technologies: Linux, Git, SQL, REST, JSON, XML, HTTP.
Nice to have
- Design data models for complex data structures.
- Complex multi-tier application profiling.
- real-time processing technologies like Storm/Samza/Spark
- Experience with NoSQL databases in general.
- Familiarity with AWS and cloud computing concepts in general.
- Familiarity with DevOps and automation tools like Jenkins, Ansible and Hashicorp suite of tools.