Spark Batch Processing Gateway @ Uber

Architecting dynamic configuration services for large-scale Apache Spark ML training and data transformation jobs.
Apache Spark
Golang
Scala
Backend Engineering
Author

Anirudh Prabhakaran

Published

May 1, 2023

Role: Software Engineering Intern
Timeline: May 2023 – July 2023
Tech Stack: Golang, Java, Scala, Apache Spark

Overview

During my internship at Uber, I focused on optimizing the Apache Spark Batch Processing Gateway. This gateway is utilized by internal data and machine learning teams to launch massive-scale data transformation and model training jobs across Uber’s infrastructure.

Key Technical Contributions

  • Dynamic Configuration Service: Architected and deployed a dynamic configuration service utilizing Golang and Scala. This replaced static deployments, drastically reducing configuration rollout latency and streamlining the developer workflow for the Big Data team by eliminating the need for full service redeployments during routine parameter changes.
  • Concurrency & Thread Safety: Solved critical concurrency challenges within the gateway by implementing robust thread-safe logic. This ensured atomic configuration updates, guaranteeing strict consistency for in-flight data jobs while simultaneously serving new requests without interruption.