Lead Data Engineer - R01555482 Job at Brillio, Remote

OGRMemJveUlRM1o3alRUYUlmRExFS0lKT0E9PQ==
  • Brillio
  • Remote

Job Description

Lead Data Engineer/ Sr. Data Engineer (Superstrong in DataBricks, Python, Pyspark,Spark, Advanced SQL, and Delta Lake, ETL, AWS)

Primary Skills
  • Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic + Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front
Job requirements

Lead Data Engineer/ Sr. Data Engineer (Superstrong in DataBricks, Python, Pyspark,Spark, Advanced SQL, and Delta Lake, ETL, AWS)

Remote- (need to work in PST time)

 

  • Design, build, and maintain scalable data pipelines to collect, process, and store from multiple datasets.
  • Optimize data storage solutions for better performance, scalability, and cost-efficiency.
  • Develop and manage ETL/ELT processes to transform data as per schema definitions, apply slicing and dicing, and make it available for downstream jobs and other teams.
  • Collaborate closely with cross-functional teams to understand system and product functionalities, pace up feature development, and capture evolving data requirements .
  • Engage with stakeholders to gather requirements and create curated datasets for downstream consumption and end-user reporting .
  • Automate deployment and CI/CD processes using GitHub workflows , identifying areas to reduce manual, repetitive work.
  • Ensure compliance with data governance policies, privacy regulations, and security protocols .
  • Utilize cloud platforms like AWS and work on Databricks for data processing with S3 Storage .
  • Work with distributed systems and big data technologies such as Python, Pyspark,Spark, Advanced SQL, and Delta Lake .
  • Integrate with SFTP to push data securely from Databricks to remote locations .
  • Analyze and interpret spark query execution plans to fine-tune queries for faster and more efficient processing.
  • Strong problem-solving and troubleshooting skills in large-scale distributed systems .
  • Have couple of projects having work exposure on Analytics and Insights based on Big Data.
  • Exposure on building Data sets on complex Big Data would be an advantage.
  • BE / B Tech in Engineering.
  • Skill set: SQL, Python, AWS, Databricks. Tableau exposure would be optional, good to have.

 

 

  • #L1- AS

Job Tags

Remote job, Full time,

Similar Jobs

Maid Sailors Cleaning Service

Professional Home Cleaner (Seattle, WA) Job at Maid Sailors Cleaning Service

 ...We are looking for an individual with at least 2+ years of professional residential cleaning to join our team as an Independent Contractor: Professional Home Cleaner . Job Responsibilities: Provide general cleaning services in luxury apartments of our clientsin... 

Tremendous

SDR Manager Job at Tremendous

 ...outside investors. Join us before the next international offsite. What you'll bring Were looking for a driven, metrics-focused SDR Manager to elevate our SDR teams performance. This role goes beyond coachingit's a high-impact leadership position for someone who... 

ICEF Public Schools

Manager of Special Education Job at ICEF Public Schools

 ...Position Overview: Manager of Special Education Location: Los Angeles, CA Reports to: School Principal Category: Credentialed...  ...School Principal and receive coaching and support from our Director of Special Education. Key Responsibilities Leadership... 

Veracity Insurance

Email & SMS Marketing Specialist Job at Veracity Insurance

 ...Reporting to the Director of Content, the Email & SMS Marketing Specialist (Remote/Hybrid...  ...of B2B insurance products. You'll work closely with Product Marketing Managers,...  ...to develop SOPs and scalable, repeatable processes and plays~ Analytical mindset with the... 

SS Solutions

Sports-Minded Assistant Manager Job at SS Solutions

 ...Leadership and mentorship Entry-level management training Management and Leadership...  ...mentality to move up into a leadership role Sports-minded and Energetic team players...  ...Communications/Public Speaking Promotions/Events Why Us? We start all our assistant...