Dagster - Sky Platform Integration

We offer an integration with Dagster platform to run Dagster operations on the PERIAN Sky Platform. To install the plugin, please run the following command:

pip install dagster-perian
Or this command
poetry add dagster-perian

Our plugin enables you to containerize a specified codebase, generating a Docker image that can be deployed on the PERIAN Sky Platform using the designated @op.

Features

  • Dockerization: Automatically packages your codebase into a Docker container
  • Serverless Execution: Runs your Docker containers on PERIAN’s scalable GPU infrastructure
  • Ease of Use: Simple commands to get your project up and running quickly

Parameters

The following parameters can be used to set the requirements for the PERIAN operations. If any of the requirements are skipped, it is replaced with the cheapest option. At least one requirement value should be set.
  • cores: Number of CPU cores
  • memory: Amount of memory in GB
  • accelerators: Number of accelerators
  • accelerator_type: Type of accelerator (e.g. 'A100'). For a full list of supported accelerators, use the perian CLI list-accelerators command
  • country_code: Country code to run the job in (e.g. 'DE')

Credentials

  • PERIAN credentials:
    • organization: your PERIAN organization name
    • token: your PERIAN auth token
  • Docker registry credentials:
    • docker_registry: your docker repository URL
    • docker_username: your docker registry account username
    • docker_password: your docker registry account password

    (Hint: generate a token) Tokens are more secure and can be generated in Docker Hub settings.

Ops

Our integration introduces the following dagster operations:

  1. 1. containerize_codebase:

    • Packages your code base into a Docker container, including the installation of any necessary dependencies
    • Parameters:
      • path: Directory of the code base
      • dependency_file: Path to a requirements file (optional)
      • python_version: Version of Python to use in the Docker image
      • image_name: Name of the Docker image
      • tag: Tag for the Docker image (default is "latest")
      • docker_username, docker_password, docker_registry: Credentials for Docker Hub
      • command: Command to run in the container
      • parameters: Parameters for the command

  2. 2. create_perian_job:

    • Creates a job on the PERIAN platform using the Docker image built from your codebase
    • Parameters:
      • image_name: Directory of the code base
      • tag: Tag of the Docker image
      • docker_username, docker_password, docker_registry: Credentials for Docker Hub
      • cores, memory, accelerators, accelerator_type, country_code, provider: Configuration for the job instance

  3. 3. get_perian_job:

    • Retrieves the status and logs of a job running on the PERIAN platform
    • input:
      • job_metadata: Metadata containing the job ID

  4. 4. cancel_perian_job:

    • Cancels a running job on the PERIAN platform
    • input:
      • job_metadata: Metadata containing the job ID

Example

Example containerize a code base at a given path and running it on the PERIAN sky platform.

Python code:

from dagster import job
from dagster_perian.resources import perian_job_resource
from dagster_perian.operations import create_perian_job, containerize_codebase


@job(resource_defs={"perian_job_manager": perian_job_resource})
def perian_job_flow():
	containerize_codebase()
	create_perian_job()
config.yaml file:
resources:
	perian_job_manager:
		config:
			api_url: "https://api.perian.cloud"
			organization: "<your-perian-organization-name>"
			token: "<your-perian-token>"

ops:
	create_perian_job:
		config:
			accelerators: 1
			accelerator_type:"A100"
			image_name: "<your-image-name>"
			tag: "<your-image-tag>"
			docker_username: "<your-docker-username>"
			docker_password: "<your-docker-password>"

	containerize_codebase:
		config:
			path: "<your-code-base-to-be-dockerized>"
			image_name: "<your-image-name>"
			tag: "<your-image-tag>"
			python_version: "<python-version-to-be-used>"
			docker_username: "<your-docker-username>"
			docker_password: "<your-docker-password>"
			command: "<running-script-or-module>"
			parameters: "<command-parameters>"