Databricks and OOP, do they match ?
September 2, 2025
Context #
Databricks and Apache Spark are often used in data engineering, data science, and machine learning workflows. Their APIs are designed around distributed data processing (RDDs, DataFrames, Datasets). The question arises: does Object-Oriented Programming (OOP) fit into this paradigm, or do we need a different style?
Databricks Programming Model vs OOP #
- Spark API: functional and declarative. You express transformations (
map,filter,select) on immutable distributed datasets. - OOP style: encapsulates data + behaviour inside classes, often with mutable state.
Where They Match #
- Encapsulation of business logic: Wrapping Spark transformations inside reusable classes (e.g.,
DataCleaner,FeatureEngineer) helps modularize pipelines. - Abstractions for teams: Teams can expose high-level methods (
.transform(df)) instead of low-level Spark calls. - Testing & reusability: OOP structures allow dependency injection, mock data, and unit testing.
Where They Clash #
- Statefulness: Spark’s lazy evaluation and immutable DataFrames do not align with mutable OOP state.
- Serialization: Classes with methods that capture external state may not serialize well when Spark ships code to executors.
- Functional preference: Many Spark best practices push towards functional patterns (pure functions, stateless transformations).
Note on statefulness: In Learning Spark, Holden Karau makes distinction between stateless and stateful processing and emphazizes it. Stateless transformations are preferred, but spark also provides patterns for stateful processing, particularly in streaming contexts. e.g., updateStateByKey, windowing, watermarking, and event-time state management.