Sr. Data Engineer

Role: Sr. Data Engineer
Location: Hybrid – Fort Worth, TX or Tempe, AZ
Duration: 2+ Months
 
Required Skills:

• Advanced Python development (building production grade services, APIs, or data platforms)
• Cloud native development – Azure Databricks
• Strong experience with software design, testing, and CI/CD pipelines

 
Additional Info:
PostgreSQL, Kubernetes, Redis, Git, Github Actions, Maven, Junit, Agile.
 
Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Python.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Develop high-performance data processing solutions using Py Spark and Spark SQL.
• Design and implement data lake architecture leveraging Delta Lake.
• Create and manage data ingestion frameworks from multiple enterprise sources.
• Implement data quality, validation, governance, and monitoring mechanisms.
• Must have hands on experience using Gen AI tools like GitHub CoPilot
• Optimize Databricks jobs, clusters, notebooks, and workflows for performance and cost efficiency.
• Collaborate with business analysts, data scientists, architects, and application teams to deliver enterprise data solutions.
• Support batch and near real-time data processing requirements.
• Participate in architecture discussions, code reviews, and best-practice implementation.
• Ensure secure, scalable, and reliable cloud data solutions aligned with organizational standards.

 
Required Technical Skills
Primary Skills (Must Have):

• Azure Databricks
• Python
• Py Spark
• Delta Lake
• Spark SQL
• Data Engineering
• ETL/ELT Development
• Azure Data Lake Storage (ADLS Gen2)
• SQL
• Azure Platform Skills
• Azure Databricks
• Azure Data Factory (ADF)
• Azure Storage Accounts
• Azure Key Vault
• Azure Monitor
• Azure App Insights
• Azure Networking Concepts

 
Secondary Skills:

• Data Modeling (Star Schema, Snowflake Schema)
• DataWarehousing Concepts
• Performance Tuning and Optimization
• REST API Integration
• NoSQL Databases
• Azure Synapse Analytics

 

At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position's pay range is $40.00/hr – $43.00/hr.