Sr. Data Engineer

Company: The Judge Group
Location: Newark
Posted on: January 12, 2021

Job Description:

Apply Now Posted : November 6, 2020 Job Type : Permanent Category : Information TechnologyDescription Our client is currently seeking a Sr. Data Engineer Data Engineer and BI : IOT Tools - MQTT, Kafka, Apache Spark and data frameLanguages: Java, PythonTechnical/Functional Skills -MUST HAVE SKILLS --- 6 years of experience in Data Engineering and Business Intelligence. --- Proficient in IoT tools such as MQTT, Kafka, Spark --- Proficient with AWS, S3, Redshift --- Experience with Presto and Parquet/ORC --- Proficient with Apache Spark and data frame. --- Experienced in containerization, including Docker and Kubernetes --- Expert in tools such as Apache Spark, Apache Airflow, Presto --- Expert in design and implement reliable, scalable, and performant distributed systems and data pipelines --- Extensive programming and software engineering experience, especially in Java, Python, --- Experience with Columnar database such as Redshift, Vertica Roles & Responsibilities --- Hands-on design and develop streaming and IoT data pipelines. --- Developing streaming pipeline using MQTT, Kafka, Spark Structure Streaming --- Orchestrate and monitor pipelines using Prometheus and Kubernetes --- Deploy and maintain streaming jobs in CI/CD and relevant tools. --- Python scripting for automation and application development --- Design and implement Apache Airflow and other dependency enforcement and scheduling tools. --- Hands-on data modeling and data warehousing --- Deploy solution using AWS, S3, Redshift and Docker/Kubernetes --- Develop storage and retrieval system using Presto and Parquet/ORC

Keywords: The Judge Group, Newark , Sr. Data Engineer, Other , Newark, California