Data Engineer
Job Description
Experience
7 – 9 Years
Job Location
Education
Diploma(Other)
Nationality
Any Arabic National, Any GCC National, Indian, Any Nationality
Gender
Any
Vacancy
1 Vacancy
Job Description
Job Description • Gathering and processing raw data and translating analyses. • Evaluating new data sources for acquisition and integration. • Working directly with the technology and engineering teams to integrate data processing and business objectives. • Create and maintain optimal data pipeline architecture. • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. • Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases. • Strong analytic skills related to working with structured and unstructured datasets. • Build processes supporting data transformation, data structures, metadata, dependency and workload management. • Perform Maintenance and administration activities on DWH and Datalake Platform. • Evaluate and access new platform/solution to fulfil business/technology requirement. Lead E2E POC and do technical evaluation. • Experience supporting and working with cross-functional teams in a dynamic environment. • Analyzes designs, creates and implements Cloudera infrastructures, including access methods, device allocations, validation checks, organization and security. Documents and maintain work-instructions, complete audit required tasks
Desired Candidate Profile
Technical Skills required: • Must have Cloudera Hadoop cluster deployment experience, including but not limited to deploying a Hadoop cluster, maintaining a Hadoop cluster, adding and removing nodes using cluster monitoring tool Cloudera Manager, configuring and upgrading the Cloudera Manager, CDH, CDSW and Kafka etc. • Experience with relational SQL and NoSQL databases, including Teradata, Postgres and Cassandra. • Hands-on experience with data delivery teams to setup new Hadoop users, including but not limited to setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Impala and Impala access for the new users. • Perform installations, configurations, troubleshoot, performance tune, upgrade, backup and recovery of various Bigdata/Hadoop services such as HDFS, Hive, Impala, HBase, Spark, Kerberos. • Must have Experience on Nifi in cluster deployment, including but not limited to maintaining, administrating, perform configuration change, perform upgrade, design & development of new data flows. • Experience with big data tools: Hadoop, Hive, Spark, Kafka, etc. • Strong hands on experience in implementation of Security like Kerberos, Sentry, OS Upgrade and TLS/SSL implementation. • Experience in Cloudera platforms and workloads migration from On-premise to On-premise or On-premise to Cloud (Azure). • Experience with data pipeline and workflow management tools: Nifi, Airflow, etc. • Experience with stream-processing systems: Storm, Spark-Streaming, etc. • Experience with object-oriented/object function scripting languages: Python, Scala, etc. • Experience with Data modeling, create CDM,LDM and PDM using different methodology – Dimensional, 3nf, DataVault etc. • Strong understanding and experience on Teradata CLDM. • Strong Experience with design and development of ETL/ELT solution using Informatica Powercenter and Teradata( SQL, Utilities). • Good understanding of DWH and BI platforms.
