Data Engineer- SC- Contract u ntil June 2022- Remote
We are currently recruiting for an experienced Data Engineer to join a government department
You need an active SC for this role Key Responsibilities
You will development of ETL methods for a range of internal and external data sources converting these from unstructured formats to dynamic tables and views in Hive and Impala;
Supporting the continuing development and transformation of data, making these available to core users in near Real Time;Providing coding support and coaching to a growing team of data engineers sharing best practices and established methods.
Assisting in the development of analytical layers of data from raw HDFS files for use in producing a range of outputs. Key Skills
Extensive proven experience of data engineering and architectural techniques, including data wrangling, data profiling, data preparation, metadata development, and data upload/download;
Proven experience of big data environments, including the Hadoop Stack (Cloudera), including data ingestion, processing, and storage using HDFS, Spark, Hive, and Impala;
Extensive hands-on experience in developing ETL functionality in a cloud or on-premise environment;
Experience of using tools such as python and SQL (in Spark) to profile, query and structure large-volume data;
Proven experience of using Cloud Services particularly in the context of Hadoop;
Experience in developing/utilising programming and query languages eg SQL (Hive Impala specifically), Python (through Spark), Scala.
Capacity to develop code and ETL processes outside of systems, using own equipment where necessary.
- SmartSourcing provides services as an Employment Agency and welcomes applications from all suitably qualified people regardless of age, race, religion, disability, age, gender or sexual orientation.