Software Engineer – Batch Processing Platform

Software Engineer – Batch Processing Platform
Duration 12 months
Location - San Francisco California 94107

Batch Processing Platform Team

• The Batch Processing Platform team is responsible for building and maintaining end-to-end big data infrastructure, data warehousing, and business intelligence solutions that power the company’s business operations.
• The team manages critical data infrastructure, including log ingestion, batch and stream processing, big data query engines, and workflow systems. It also develops and maintains data processing and ETL pipelines that support data warehousing and business intelligence.

 
What You’ll Do

• Build and maintain infrastructure platforms and data processing pipelines that support critical business needs.
• Work with cutting-edge big data technologies, including Hadoop, Spark, Airflow, Presto, and Trino.
• Contribute to large-scale data processing and infrastructure solutions capable of handling hundreds of petabytes of data.
• Support data warehousing, business intelligence, and other data-driven business initiatives.

 
Skills and Qualifications
What We’re Looking For

• Industry Experience: 2+ years of experience in big data infrastructure or data warehouse engineering.
• Programming Skills: Strong experience in at least one programming language, such as Java, Scala, or Python.
• Big Data Technologies: Hands-on experience with big data infrastructure and processing technologies, such as Hadoop, Spark, Airflow, Presto, or Trino.
• Data Processing: Experience using big data technologies for large-scale data processing and ETL pipelines.
• Communication Skills: Good verbal and written communication skills.
• Data at Scale: A passion for working with data at hundreds of petabytes scale.
• AI Tools: Understanding of how to use AI tools effectively.
• Research and Publications: Publications at technical conferences are a plus.

 
Education

• Bachelor’s degree in Computer Science or a related field, or equivalent practical experience.

 
  

At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position's pay range is $68.00/hr – $72.00/hr.