How to Use Databricks Asset Bundles with Python Wheels

Why Asset Bundles Matter on Databricks

When you start building production‑grade notebooks on Databricks, the biggest headache is often keeping code, libraries, and configuration in sync across clusters. Databricks asset bundles were introduced to address exactly that problem: they let you package notebooks, libraries, and other files into a single, version‑controlled unit. Pair that with Python wheels, the standard distribution format for Python packages, and you have a reliable, repeatable deployment pipeline.

Understanding Databricks Asset Bundles

An asset bundle is essentially a zip archive that Databricks can ingest directly. Inside, you’ll find a manifest file (usually manifest.json) that lists each component—whether it’s a notebook, a JAR, a wheel, or a data file. When you upload the bundle, Databricks reads the manifest, creates the appropriate library objects, and makes the notebooks available in the workspace.

Key points to remember:

• Versioning: Each bundle carries its own version identifier, making rollbacks straightforward.
• Isolation: Dependencies defined in the bundle are isolated from the cluster’s global libraries, reducing “dependency hell.”
• Automation‑ready: Bundles can be uploaded via the REST API or the Databricks CLI, which fits nicely into CI/CD pipelines.

Python Wheels: The Basics

Python wheels (.whl files) are binary distributions that contain compiled code, metadata, and any required resources. They’re the preferred format for installing packages with pip because they skip the build step, speeding up deployment on shared clusters.

Creating a wheel is simple if you follow the standard setup.py or pyproject.toml conventions. The resulting file can be uploaded to PyPI, a private repository, or directly bundled into a Databricks asset bundle.

Combining Asset Bundles and Wheels

At first glance, asset bundles and wheels might seem like overlapping solutions, but they serve complementary roles. Wheels handle the Python packaging side—ensuring that your code and its compiled dependencies are ready for installation. Asset bundles, on the other hand, manage the broader context: notebooks, configuration files, and any non‑Python libraries you need.

When you place a wheel inside an asset bundle, Databricks automatically installs it on the target cluster before any notebook runs. This guarantees that the exact version of your custom library is present, no matter which developer or job triggers the execution.

Step‑by‑Step: Packaging and Deploying

1. Build Your Python Wheel

Start with a clean project directory:

• Make sure setup.py or pyproject.toml accurately reflects dependencies.
• Run python -m build (or python setup.py bdist_wheel) to generate the .whl file.
• Test the wheel locally with pip install path/to/your_pkg‑0.1‑py3-none-any.whl.

2. Create the Asset Bundle Manifest

The manifest is a JSON file that tells Databricks what to do with each item. A minimal example looks like this:

{
"version": "1.0.0",
"libraries": [
{
"whl": "dist/your_pkg‑0.1‑py3‑none‑any.whl"
}
],
"notebooks": [
{
"path": "notebooks/etl_job.py",
"destination": "/Workspace/Shared/ETL Job"
}
]
}

Adjust the paths to match your repository layout. You can also add JARs, init scripts, or data files by including the appropriate keys.

3. Assemble the Bundle

Run a simple zip command from the root of your project:

zip -r my_bundle.zip manifest.json dist/ notebooks/

The resulting my_bundle.zip is ready for upload.

4. Upload via CLI or API

Using the Databricks CLI (v0.200+), the upload looks like:

databricks workspace import_dir my_bundle.zip /Shared/AssetBundles/my_bundle.zip --overwrite

Alternatively, the REST endpoint /api/2.0/libraries/bundle accepts a multipart/form‑data payload containing the zip file and a target cluster ID.

5. Attach the Bundle to a Cluster

In the UI, go to your cluster’s “Libraries” tab, click “Install New,” and select “Upload Asset Bundle.” Pick the bundle you just uploaded, and Databricks will handle the rest—installing the wheel, placing notebooks, and restarting the cluster if needed.

Best Practices for Maintaining Bundles

• Semantic Versioning: Treat the bundle version separately from the wheel version. Increment the bundle when you add or remove notebooks, even if the wheel stays the same.
• Pin Dependencies: Inside setup.py, specify exact versions for third‑party libraries to avoid surprises when the wheel is built on a different environment.
• Separate Dev and Prod Bundles: Keep a lightweight “dev” bundle that points to snapshot versions of wheels, and a “prod” bundle that references stable releases.
• Automate Tests: As part of your CI pipeline, spin up a temporary Databricks cluster, install the bundle, and run a smoke test notebook to verify everything loads correctly.

Frequently Asked Questions

What’s the difference between a library and an asset bundle?

A library in Databricks refers to a single artifact—like a JAR or a wheel—installed on a cluster. An asset bundle groups multiple libraries, notebooks, and other files together, providing versioned, atomic deployment.

Can I include native compiled binaries in a wheel?

Yes. Wheels can contain compiled extensions (e.g., .so or .dll) built for the target runtime. Just ensure the compiled binary matches the Databricks runtime’s Python version and operating system.

Do asset bundles work with Delta Live Tables?

They do. When you attach a bundle to a Delta Live Tables pipeline, the wheel is installed before the pipeline’s first run, allowing you to use custom Python transformations inside your DLT notebooks.

How do I handle secret configuration values?

Never hard‑code secrets in the bundle. Use Databricks Secrets or environment variables, and reference them from your notebooks or init scripts after the bundle has been installed.