Skip to main content

Spark Connect | Process

Overview​

Spark Connect is a distributed processing service on the Dadosfera platform that lets you run PySpark code remotely. Process large data volumes efficiently, directly from your Notebooks or Data Apps.

Benefits​

  • Distributed Processing: Run operations in parallel across multiple workers.
  • No Setup: An always-available cluster with no resource provisioning required.
  • Native Integration: Works directly with platform Notebooks and Data Apps.
  • Snowflake Compatibility: Process data from your data warehouse efficiently.

When to Use It​

ScenarioRecommendation
Simple transformations on small DataFramesUse pandas directly
Processing millions of recordsUse Spark Connect
Complex aggregations on large volumesUse Spark Connect
ETL with multiple data sourcesUse Spark Connect
Machine Learning at scaleUse Spark Connect

Prerequisites​

Installing Dependencies​

Before connecting to Spark, install the required dependencies:

pip install pyspark==3.5.0 pandas "pyarrow>4" "numpy<2" grpcio grpcio-status protobuf

PySpark Version​

The PySpark client version must match the server version:

pip install pyspark==3.5.0

Basic Connection​

Connecting to the Cluster​

from pyspark.sql import SparkSession

# Conectar ao Spark Connect
spark = SparkSession.builder \
    .remote("sc://spark-connect.spark-jobs.svc.cluster.local:15002") \
    .getOrCreate()

print(f"Conectado! Versão do Spark: {spark.version}")

Closing the Session​

Always close your session when finished:

spark.stop()

Recommended Pattern (Context Manager)​

from contextlib import contextmanager
from pyspark.sql import SparkSession

@contextmanager
def spark_session():
    spark = SparkSession.builder \
        .remote("sc://spark-connect.spark-jobs.svc.cluster.local:15002") \
        .getOrCreate()
    try:
        yield spark
    finally:
        spark.stop()

# Uso
with spark_session() as spark:
    df = spark.range(100)
    df.show()

Limitations​

Spark Connect has some limitations compared with traditional Spark:

  • No RDD API: Only the DataFrame/Dataset API is supported.
  • No SparkContext: Use DataFrame operations.
  • UDFs: Must be self-contained (no external dependencies).
  • Settings: Configuration-level changes, such as cluster settings and packages, cannot be made through the Spark Application because the client and server are isolated.

Next Steps​