Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Migrating to Amazon Athena Apache Spark version 3.5

Amazon Athena for Apache Spark has two release versions: PySpark engine version 3 (the earlier version) and Apache Spark version 3.5 (the current version). This guide helps you migrate from PySpark engine version 3 to Apache Spark version 3.5.

Amazon Athena for Apache Spark release version PySpark engine version 3 reaches end of life on April 1, 2027. After that date, you can no longer create PySpark engine version 3 workgroups, and existing PySpark engine version 3 workgroups can no longer run sessions or calculations. Additionally, there are benefits to migrating to Apache Spark version 3.5. For more information, see Release versions.

This guide walks you through moving your PySpark engine version 3 notebooks to Apache Spark version 3.5 and using Amazon SageMaker Unified Studio Data Agent to assist with migration.

You migrate in two phases:

SageMaker Unified Studio brings data preparation, analytics, and machine learning into a single governed experience. Amazon Apache Spark version 3.5 is the default Spark runtime and is automatically available in SageMaker Unified Studio notebooks. You can migrate Apache Spark (PySpark) notebooks running in Amazon Athena Spark workgroups (PySpark engine version 3) to SageMaker Unified Studio projects. Migrating preserves your work while unlocking the latest Spark capabilities, integrated AI assistance, and project-level governance.

Prerequisites

Before proceeding with the migration, ensure you have:

Timeline

Migrate

Step 1: Find PySpark engine version 3 workgroups

If you have Athena workgroups that use the PySpark engine version 3 release version, you must migrate them.

Step 2: Prepare the target environment

For notebooks, set up SageMaker Unified Studio:

Step 3: Grant the project execution role access to your data

Compare the SageMaker Unified Studio project execution role with your existing Athena Spark workgroup execution role, and add any missing data-access permissions before you migrate.

Step 4: Onboard notebooks to SageMaker Unified Studio

The Athena notebook migration script copies notebooks from your PySpark engine version 3 workgroups into SageMaker Unified Studio projects. For each notebook, it:

Get the script. Download it from the Athena notebook migration script on the GitHub website. The script's README lists the supported Python and AWS SDK versions, the IAM policy the script needs, and all of its options. Check the README before each migration, because the script is updated over time. The following are sample commands:

Check Inventory. List workgroups and notebooks. This pass is read-only.

python3 athena_notebook_migration.py --inventory --region <region>

Dry run (single workgroup). The dry run validates migration scope and destination on a single workgroup without modifying anything.

python3 athena_notebook_migration.py \
  --region <region> --domain-id <domain-id> --project-id <project-id> \
  --workgroups <workgroup-name> --dry-run

Migrate one workgroup at a time. Imports all PySpark engine version 3 notebooks in the supplied workgroup into the SageMaker Unified Studio domain and project.

python3 athena_notebook_migration.py \
  --region <region> --domain-id <domain-id> --project-id <project-id> \
  --workgroups <workgroup-name>

Migrate all workgroups. Imports all PySpark engine version 3 notebooks in every workgroup into the SageMaker Unified Studio domain and project.

python3 athena_notebook_migration.py \
  --region <region> --domain-id <domain-id> --project-id <project-id> \
  --all-workgroups

The script prints a summary and generates a report. The report lists each notebook with its result and new notebook ID, and you can use it to resume an interrupted run. To confirm the migration, open the project in SageMaker Unified Studio and check that the notebooks are imported under the Notebooks section in the project.

Note the following:

Step 5: Update notebook code using SageMaker Unified Studio Data Agent

Migrated notebooks contain your original code. Use the SageMaker Unified Studio Data Agent to update each notebook for Apache Spark version 3.5.

For details, see Using the Data Agent.

Optional steps

The following steps are optional. Complete them if they apply to your migration.

Move applications from calculations to Spark Connect

If your applications or scripts call StartCalculationExecution to submit code blocks, move that code to Spark Connect, because the Calculation APIs aren't supported on Apache Spark version 3.5. Run the same code over Spark Connect instead. You can also connect your own notebook client or IDE this way. For current connection details, see Spark Connect support.

Create a workgroup with the Apache Spark 3.5 engine

If you define workgroups or sessions in the API, CloudFormation, AWS CDK, or Terraform, update them as follows:

aws athena create-work-group \
  --region <region> \
  --name <workgroup-name> \
  --configuration '{
    "EngineVersion": { "SelectedEngineVersion": "Apache Spark version 3.5" },
    "ExecutionRole": "arn:aws:iam::<account-id>:role/<execution-role>",
    "EngineConfiguration": {
      "Classifications": [{
        "Name": "spark-defaults",
        "Properties": { "spark.dynamicAllocation.maxExecutors": "<max-executors>" }
      }]
    }
  }'

For current settings and defaults, see Use Spark properties to specify custom configuration.

Clean up

After cutover is stable:

Troubleshooting

Use the following information to diagnose and fix issues that you might encounter when you migrate notebooks to SageMaker Unified Studio and run them on Apache Spark version 3.5.

IllegalArgumentException: Unsupported format in USING

In SageMaker Unified Studio, CREATE TABLE ... USING DELTA fails with IllegalArgumentException: Unsupported format in USING: DELTA, and a Hudi UPDATE fails with UPDATE TABLE is not supported temporarily, even though the same notebooks run on the Athena console.

This occurs because the SageMaker Unified Studio Spark session registers an Iceberg catalog and SQL extension by default, so non-Iceberg formats have no catalog or extension. You can't reconfigure the session afterward, because the %%configure magic isn't available on Athena Spark compute.

To resolve the error, set the catalog and extension for your table format when the session starts:

For Iceberg (the default):

from sagemaker_studio import sparkutils
spark = sparkutils.init(spark_conf={
    "spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog",
    "spark.sql.catalog.spark_catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",
    "spark.sql.catalog.spark_catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
})

For Delta:

from sagemaker_studio import sparkutils
spark = sparkutils.init(spark_conf={
    "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog",
    "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
})

For Hudi:

from sagemaker_studio import sparkutils
spark = sparkutils.init(spark_conf={
    "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.hudi.catalog.HoodieCatalog",
    "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
    "spark.sql.extensions": "org.apache.spark.sql.hudi.HoodieSparkSessionExtension",
})

To confirm the configuration, check that spark.conf.get("spark.sql.catalog.spark_catalog") and spark.conf.get("spark.sql.extensions") return the Delta or Hudi values. A Hudi UPDATE on an existing table can still be blocked — see HoodieUpgradeDowngradeException: Please upgrade table from version FIVE to 6.

HoodieUpgradeDowngradeException: Please upgrade table from version FIVE to 6

A Hudi write or UPDATE in SageMaker Unified Studio fails with the following error:

org.apache.hudi.exception.HoodieUpgradeDowngradeException:
  Please upgrade table from version FIVE to 6 before upgrading to version EIGHT.

This error occurs when the table was created in Athena for Spark (PySpark engine version 3), which runs Hudi 0.12 and writes on-disk table version 5. If you then write to it from a SageMaker Unified Studio engine, which runs Hudi 1.0 (table version 8), the version mismatch triggers the error. Hudi upgrades a table one version at a time and refuses the v5-to-v8 jump, so you must first upgrade the table to v6. The following table shows the Hudi release for each table version.

To resolve the error, upgrade the table to v6 in the SageMaker Unified Studio session with the upgrade_table stored procedure, then rerun your write. Do the following in the SageMaker Unified Studio session:

To confirm the upgrade, check that hoodie.properties shows hoodie.table.version=8 and hoodie.table.initial.version=6, and that a SELECT returns the updated rows.

Note the following before you upgrade a table:

InvalidRequestException: The EngineConfiguration exceeds the maximum allowed limit of 5000 characters

Running a PySpark cell in SageMaker Unified Studio fails with the following error:

InvalidRequestException: An error occurred (InvalidRequestException) when calling the
StartSession operation: The EngineConfiguration passed in the StartSession request exceeds
the maximum allowed limit of 5000 characters.

This error occurs because SageMaker Unified Studio builds the Spark properties that it sends to the Athena StartSession operation. SageMaker Unified Studio adds about seven properties for every AWS Glue or Amazon S3 Tables catalog in the account. In accounts with many catalogs, the serialized properties exceed the 5,000-character EngineConfiguration limit and StartSession is rejected. To resolve the error, reduce the number of catalogs until the properties fit within the limit:

A session doesn't default to the AWS Glue Data Catalog

When a session starts in SageMaker Unified Studio, a CREATE DATABASE statement (for example, CREATE DATABASE IF NOT EXISTS sparkblogdb) can create the database in an unexpected catalog, such as an Amazon S3 Tables catalog, instead of the AWS Glue Data Catalog. Running spark.sql("SELECT current_catalog()").show() after the statement returns the name of a registered Amazon S3 Tables catalog (for example, v3-test) rather than spark_catalog.

This happens because the session's spark_catalog is an Iceberg SparkSessionCatalog whose AWS Glue settings point at an Amazon S3 Tables catalog rather than the root AWS Glue Data Catalog. In the generated Spark configuration, the catalog's glue.catalog-arn includes an /s3tablescatalog/<name> suffix and its warehouse is an s3tables ARN, so statements resolve to Amazon S3 Tables:

spark.sql.catalog.spark_catalog.glue.catalog-arn = arn:aws:glue:<region>:<account-id>:catalog/s3tablescatalog/<name>
spark.sql.catalog.spark_catalog.warehouse       = arn:aws:s3tables:<region>:<account-id>:bucket/*

To create databases in the root AWS Glue Data Catalog instead, override the catalog configuration at session start with sparkutils.init so that it targets the root catalog:

from sagemaker_studio import sparkutils
spark = sparkutils.init(spark_conf={
    "spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog",
    "spark.sql.catalog.spark_catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",
    "spark.sql.catalog.spark_catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "spark.sql.catalog.spark_catalog.glue.id": "<account-id>",
    "spark.sql.catalog.spark_catalog.warehouse": "s3://<your-bucket>/<prefix>/",
    "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
})

To confirm which catalog and database a session resolves to, run spark.sql("SELECT current_catalog(), current_database()").show().

Appendix: Changes and what to do

Use this list to estimate the work for each workgroup.

Your data doesn't move. AWS Glue Data Catalog tables and Amazon S3 data, including Apache Iceberg, Apache Hudi, and Delta Lake tables, remain where they are. Apache Spark version 3.5 reads them with newer library versions.

For current runtime versions and Regions, see Release versions and Considerations and limitations.

Frequently asked questions