1 of 9

Examples

Hello (cow) World!

A cowsay job

Run Hello, 🐮 World! Job [CLI]

Ensure you have installed all requirements [CLI] Install Run Requirements

Start Docker

Open a Terminal window and run the following command

lilypad run cowsay:v0.0.1 "hello lilypad"

Ensure your user is in the docker group if necessary on your platform

Output:

See the Results

Navigate to the IPFS CID result output in the Results -> https://ipfs.io/ipfs/QmNjJUyFZpSg7HC9akujZ6KHWvJbCEytre3NRSMHzCA6NR

This could take up to a minute to propagate through the IPFS network. Please be patient

Then click on the stdout folder and you should see the job result!

Stable Diffusion (SDXL0.9)

Run a Stable Diffusion Text to Image Job

Overview

Generically, stable diffusion is what happens when you put a couple of drops of dye into a bucket of water. Given time, the dye randomly disperses and eventually settles into a uniform distribution which colours all the water evenly

In computer science, you define rules for your (dye) particles to follow and the medium this takes place in.

Stable Diffusion is a machine learning model used for text-to-image processing (like Dall-E) and based on a diffusion probabilistic model that uses a transformer to generate images from text. There are several open-source stable diffusion models out there (made famous by Stability.ai) and they continue to improve and become even more fully featured - SDXL0.9 is one of the more recently open-sourced models.

[CLI] Running Stable Diffusion SDXL 0.9

Ensure you have installed all requirements [CLI] Install Run Requirements

To run stable diffusion use the SDXL module like so:

lilypad run sdxl:v0.9-lilypad1 "an astronaut riding on a unicorn"

The output will look like this:

Take the ipfs link given in the results and paste it into your browser:

Please be patient! IPFS can take some time to propagate and doesn't always work immediately.

In the /outputs folder, you'll find the image:

Since modules are deterministic, running this command with the same text prompt will produce the same image, since the same seed is also used (the default seed is 0).

See this beginner-friendly article on how seed's work for more info on this

To change the image, you can pass in a different seed number:

lilypad run sdxl:v0.9-lilypad1 '{"prompt": "an astronaut riding on a unicorn", "seed": 9}'

CLI Video

[Smart Contract] Running Stable Diffusion SDXL 0.9

Make sure you have connected to the Lalechuza testnet and funded your wallet with testnet lilETH. See Funding your Wallet from Faucet & Setting up Metamask

To trigger the SDXL0.9 module from a smart contract, firstly you need to create your own client contract to call the module from. In order to receive results back from the Lilypad network, you will also need to 1. Connect to the Lilypad Modicum Contract (and create an instance of it in your own contract using the current address found here) 2. Implement the Modicum Contract receiveJobResults() interface.

// SPDX-License-Identifier: GPLv3
pragma solidity ^0.8.6;

// give it the ModicumContract interface 
// NB: this will be a separate import in future.
interface ModicumContract {
  function runModuleWithDefaultMediators(string calldata name, string calldata params) external payable returns (uint256);
}

contract SDXLCaller {
  address public contractAddress;
  ModicumContract remoteContractInstance;
  
  // See the latest result.
  uint256 public resultJobId;
  string public resultCID;

  // The Modicum contract address is found here: https://github.com/bacalhau-project/lilypad-modicum/blob/main/latest.txt
  // Current: 0x422F325AA109A3038BDCb7B03Dd0331A4aC2cD1a
  constructor(address _modicumContract) {
    require(_modicumContract != address(0), "Contract cannot be zero address");
    contractAddress = _modicumContract;
    //make a connection instance to the remote contract
    remoteContractInstance = ModicumContract(_modicumContract);
  } 

  /*
  * @notice Run the SDXL Module
  * @param prompt The input text prompt to generate the stable diffusion image from
  */
  function runSDXL(string memory prompt) public payable returns (uint256) {
    require(msg.value == 2 ether, "Payment of 2 Ether is required"); //all jobs are currently 2 lilETH
    return remoteContractInstance.runModuleWithDefaultMediators{value: msg.value}("sdxl:v0.9-lilypad1", prompt);
  }
  
  // This must be implemented in order to receive the job results back!
  function receiveJobResults(uint256 _jobID, string calldata _cid) public {
    resultJobId =_jobID;
    resultCID = _cid;
  }

}

NB: You could also add the seed as a parameter to run this.

return remoteContractInstance.runModuleWithDefaultMediators{value: msg.value}("sdxl:v0.9-lilypad1",`` params);

Remix

Try it yourself!

Click this link to open the contract in Remix IDE!

Ensure your MetaMask wallet is set to the Lalechuza testnet and has lilETH testnet funds from the faucet.
Set the remix environment to "Injected Provider - MetaMask" (& ensure MetaMask has the lalechuza chain selected)
Then - Deploy a new contract passing in the Modicum Contract address found here OR - Open the contract at this example address: 0x31e7bF121EaB1C0B081347D8889863362e9ad53A

Call the runSDXL Module, passing in a prompt and sending 2 lilETH in the value field. Your MetaMask wallet should pop up for you to confirm the payment and transaction.

Give it some time and check the resultCID variable. You can then open this result in your browser with https://ipfs.io/ipfs/<resultCID> or ipfs://<resultCID> in IPFS compatible browsers like Brave.

Video

FYI! You can try all examples in one contract. See [Smart Contract] Run "Hello, World!" Job

SDXL Module Code

Find the SDXL module code here. There's also a generic Stable Diffusion module here.

LoRA Fine Tuning

Fine-tuning models from inputs.

Overview of LoRA Fine Tuning

LoRA stands for Low Rank Adaptation and is a mathematical technique to reduce the number of parameters that are trained in a model - so instead of fine-tuning all the weights that constitute the weight matrix of the pre-trained large language model, two smaller matrices that approximate this larger matrix are fine-tuned.

What this means in practice is that instead of needing to build a custom model off all the original input data material (and needed many many GPUs to do so), LoRA means you can fine-tune an existing model (such as SDXL 0.9 Stable Diffusion) to be biased towards a certain result on just one GPU.

For example, an open source Stable Diffusion model can be fine-tuned to produce images in the style of Claude Monet paintings using LoRA. Fun fact: This is how Waterlily.ai trains artist models - look how good the results are even without an up to date Stable Diffusion model like SDXL0.9!

A Claude Monet data set is available in zip form on IPFS here: ipfs://bafybeiglwmvudxxethpi46el3o5m44lrki2sjzgs7whvb6xaz6e65wm7am One of the easiest ways to upload a dataset to IPFS is to use web3.storage.

[CLI] Running LoRA Fine Tuning [coming soon]

Ensure you have installed all requirements [CLI] Install Run Requirements

To run a LoRA fine-tuning job, just provide the training data for the job to the command:

lilypad run lora_training:v0.1.7-lilypad1 '{images_cid: "bafybeiah7ib5mhzlckolwlkwquzf772wl6jdbhtbuvnbuo5arq7pcs4ubm", seed: 3}'

NB: the params above should be yaml eg. {seed: 42, 'images_cid': 'Qm...'} where images_cid contains an images.zip with training images in it.

This will output a result model CID, which can then be used to generate new images in this particular style:

lilypad run lora_inference:v0.1.7-lilypad1 '{lora_cid: <CID result from above>, prompt: "an astronaut riding a unicorn in the style of <s1><s2>", seed: 3}'

[Smart Contract] Running LoRA Fine Tuning [coming soon]

Ensure you have set up your Metamask for Lalechuza Network and have funded your wallet. Setting up Metamask & Funding your Wallet from Faucet

LoRA Module Code

See the code repo here

LLM Inference

A Fast Chat LLM Inference Module for Lilypad

Overview

This LLM Inference Module is a community-contributed module developed at AugmentHack.xyz The repo for this module can be found here.

See the original AugmentHack entry below:

[CLI] Usage

Usage:

lilypad run fastchat:v0.0.1 "paramsStr"

Inputs:

Where "paramsStr" is a question in CID form for the LLM. For example https://ipfs.io/ipfs/QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt is a bare file CID which contains

{
    "template": "You are a friendly chatbot assistant that responds conversationally to users' questions. \n Keep the answers short, unless specifically asked by the user to elaborate on something. \n \n Question: {question} \n \n Answer:",
    "parameters": {"question": "What is a chatbot?"}
}

To use it you would run:

lilypad run fastchat:v0.0.1 QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt

Outputs:

The output will be an IPFS CID, for example running the above input would result in the following link:

https://ipfs.io/ipfs/QmVNXCAfJgER6U7Z5XT8QaAVFPdwmtSFE6c9sUaAx7ttZs

Under link/output/result.json you will see

{"question": "What is a chatbot?", "text": "<pad> A  chatbot  is  a  computer  program  that  can  interact  with  users  in  a  conversational  manner.  It  is  designed  to  answer  questions  and  provide  information  in  a  way  that  is  natural  and  conversational.\n"}

LLM Module Code

Duck DB [coming soon]

Community Member Contribution

Overview

[CLI] Running DuckDB Module

DuckDB Module Code

Filecoin Data Prep

Code not tested.

Overview

The Filecoin Data Prep Module is designed to chunk data into CAR files from an S3 bucket - hence the name, since it prepares the data to be uploaded to Filecoin. The repo for this module can be found here.

[CLI] Usage

Ensure you have installed all requirements [CLI] Install Run Requirements

Run the module

lilypad run filecoin_data_prep:v0.0.1 '{"ipfs_cid": "CID"}'

Execution:

Results:

Full Module

[Advanced] Arbitrary WASM

Run in a deterministic environment

Ensure you have installed all requirements

[Advanced] DIY Module

Contribute your own module to Lilypad

Contributing your own module is currently a non-trivial process. The Lilypad team is aiming to make this DX easier as well as to add tutorials, walkthroughs and approaches to making modules over the next month. In the meantime if you try this and do get stuck - please reach out to us in for help!

Overview

Contributing your own module to use on Lilypad is possible and welcome! Essentially modules on Lilypad currently operate like Job Specifications do - take a look at this page on to see more about how this works.

Requirements

Modules should be either Docker or WASM Images that align to the Bacalhau job specifications (which then have some added metadata that defines the runtime options of the module).

Module Example

Here is an example of the SDXL module in python:

It's a function that given a "string" will return a docker job spec. That string can be whatever you want, JSON, csv, raw LLM prompt etc. In terms of output directories - you can see the SDXL example linked above names the "/outputs" folder and then will use that path in the command inside the Docker container. Any named folders like this that the Docker image can write files into - will be included as part of the results Lilypad gets back out of the job (and will change the result hash)

Using Bacalhau as a testing ground

It's currently advisable to develop your module with first (because it's FAR easier to get setup as a development environment than Lilypad is currently). If you can get a function like the one shown above that will, given a string, write a bacalhau job spec that you can test with the bacalhau CLI - then you have already done 98% of the Lilypad module

To add this to Lilypad, submit a PR which also includes this file .

Contributing Guidelines

The Future of Community Contributed Modules

From Lilypad incentivised testnet [Q4 2023] onwards (and perhaps even earlier), its probable modules contributed by community members will be eligible for a % fee of jobs that run these modules in order to encourage the growth of the module ecosystem on the Lilypad Network.

LoRA Fine Tuning

Fine-tuning models from inputs.

Overview of LoRA Fine Tuning

[CLI] Running LoRA Fine Tuning [coming soon]

Ensure you have installed all requirements [CLI] Install Run Requirements

To run a LoRA fine-tuning job, just provide the training data for the job to the command:

lilypad run lora_training:v0.1.7-lilypad1 '{images_cid: "bafybeiah7ib5mhzlckolwlkwquzf772wl6jdbhtbuvnbuo5arq7pcs4ubm", seed: 3}'

NB: the params above should be yaml eg. {seed: 42, 'images_cid': 'Qm...'} where images_cid contains an images.zip with training images in it.

This will output a result model CID, which can then be used to generate new images in this particular style:

lilypad run lora_inference:v0.1.7-lilypad1 '{lora_cid: <CID result from above>, prompt: "an astronaut riding a unicorn in the style of <s1><s2>", seed: 3}'

[Smart Contract] Running LoRA Fine Tuning [coming soon]

Ensure you have set up your Metamask for Lalechuza Network and have funded your wallet. Setting up Metamask & Funding your Wallet from Faucet

LoRA Module Code

See the code repo here

https://github.com/bacalhau-project/lilypad-modicum/blob/main/src/python/modules/lora.py

import yaml
import textwrap

IMAGE = "quay.io/lukemarsden/lora:v0.0.4"
MODEL_NAME = "runwayml/stable-diffusion-v1-5"

# Don't want to make this variable since the job is fixed price
NUM_IMAGES = 10

def _lora_training(params: str):
    if params.startswith("{"):
        params = yaml.safe_load(params)
    else:
        raise Exception("Please set params to yaml like {seed: 42, 'images_cid': 'Qm...'} where images_cid contains an images.zip with training images in")

    seed = params.get("seed", 42)
    images_cid = params["images_cid"]

    return {
        "APIVersion": "V1beta1",
        "Metadata": {
            "CreatedAt": "0001-01-01T00:00:00Z",
            "Requester": {}
        },
        "Spec": {
            "Deal": {
                "Concurrency": 1
            },
            "Docker": {
                "EnvironmentVariables": [
                    f"RANDOM_SEED={seed}",
                ],
                "Entrypoint": [
                    "bash", "-c", f'(cd /input && unzip images.zip && rm images.zip) && lora_pti --pretrained_model_name_or_path={MODEL_NAME} --instance_data_dir=/input --output_dir=/output --train_text_encoder --resolution=512 --train_batch_size=1 --gradient_accumulation_steps=4 --scale_lr --learning_rate_unet=1e-4 --learning_rate_text=1e-5 --learning_rate_ti=5e-4 --color_jitter --lr_scheduler="linear" --lr_warmup_steps=0 --placeholder_tokens="<s1>|<s2>" --use_template="style" --save_steps=100 --max_train_steps_ti=1000 --max_train_steps_tuning=1000 --perform_inversion=True --clip_ti_decay --weight_decay_ti=0.000 --weight_decay_lora=0.001 --continue_inversion --continue_inversion_lr=1e-4 --device="cuda:0" --lora_rank=1'
                ],
                "Image": IMAGE,
            },
            "Engine": "Docker",
            "Language": {
                "JobContext": {}
            },
            "Network": {
                "Type": "None"
            },
            "PublisherSpec": {
                "Type": "Estuary"
            },
            "Resources": {
                "GPU": "1"
            },
            "Timeout": 1800,
            "Verifier": "Noop",
            "Wasm": {
                "EntryModule": {}
            },
            "inputs": [
                {
                    "CID": images_cid,
                    "Name": "lora_input",
                    "StorageSource": "IPFS",
                    "path": "/input",

                },
            ],
            "outputs": [
                {
                    "Name": "output",
                    "StorageSource": "IPFS",
                    "path": "/output"
                }
            ]
        }
    }

def _lora_inference(params: str):
    if params.startswith("{"):
        params = yaml.safe_load(params)
    else:
        raise Exception("Please set params to yaml like {seed: 42, 'lora_cid': 'Qm...', "+
                        "prompt: 'an astronaut in the style of <s1><s2>'} "+
                        "where lora_cid is the output cid of the above step")

    # TODO add a default we pin
    lora_cid = params["lora_cid"]
    seed = params.get("seed", 42)
    prompt = params.get("prompt", "question mark floating in space")
    finetune_weighting = params.get("finetune_weighting", 0.5)

    return {
        "APIVersion": "V1beta1",
        "Metadata": {
            "CreatedAt": "0001-01-01T00:00:00Z",
            "Requester": {}
        },
        "Spec": {
            "Deal": {
                "Concurrency": 1
            },
            "Docker": {
                "EnvironmentVariables": [
                    f"PROMPT={prompt}",
                    f"RANDOM_SEED={seed}",
                    f"FINETUNE_WEIGHTING={finetune_weighting}",
                    "HF_HUB_OFFLINE=1",
                ],
                "Entrypoint": [
                    'python3',
                    '-c',
                    # dedent
                    textwrap.dedent(f"""
                        from diffusers import StableDiffusionPipeline, EulerAncestralDiscreteScheduler
                        import os
                        import torch
                        from lora_diffusion import tune_lora_scale, patch_pipe

                        model_id = "{MODEL_NAME}"

                        pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to(
                            "cuda"
                        )
                        pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)

                        prompt = os.getenv("PROMPT")
                        seed = int(os.getenv("RANDOM_SEED"))
                        torch.manual_seed(seed)

                        os.system("find /input")
                        patch_pipe(
                            pipe,
                            "/input/output/final_lora.safetensors",
                            patch_text=True,
                            patch_ti=True,
                            patch_unet=True,
                        )

                        coeff = float(os.getenv("FINETUNE_WEIGHTING", 0.5))
                        tune_lora_scale(pipe.unet, coeff)
                        tune_lora_scale(pipe.text_encoder, coeff)

                        image = pipe(prompt, num_inference_steps=50, guidance_scale=7).images[0]
                        image.save(f"/output/image-{{seed}}.jpg")
                        image
                        """)
                ],
                "Image": IMAGE,
            },
            "Engine": "Docker",
            "Language": {
                "JobContext": {}
            },
            "Network": {
                "Type": "None"
            },
            "PublisherSpec": {
                "Type": "Estuary"
            },
            "Resources": {
                "GPU": "1"
            },
            "Timeout": 1800,
            "Verifier": "Noop",
            "Wasm": {
                "EntryModule": {}
            },
            "inputs": [
                {
                    "CID": lora_cid,
                    "Name": "lora_input",
                    "StorageSource": "IPFS",
                    "path": "/input",

                },
            ],
            "outputs": [
                {
                    "Name": "output",
                    "StorageSource": "IPFS",
                    "path": "/output"
                },
            ]
        }
    }

LLM Inference

A Fast Chat LLM Inference Module for Lilypad

Overview

This LLM Inference Module is a community-contributed module developed at AugmentHack.xyz The repo for this module can be found here.

See the original AugmentHack entry below:

[CLI] Usage

Usage:

lilypad run fastchat:v0.0.1 "paramsStr"

Inputs:

Where "paramsStr" is a question in CID form for the LLM. For example https://ipfs.io/ipfs/QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt is a bare file CID which contains

{
    "template": "You are a friendly chatbot assistant that responds conversationally to users' questions. \n Keep the answers short, unless specifically asked by the user to elaborate on something. \n \n Question: {question} \n \n Answer:",
    "parameters": {"question": "What is a chatbot?"}
}

To use it you would run:

lilypad run fastchat:v0.0.1 QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt

Outputs:

The output will be an IPFS CID, for example running the above input would result in the following link:

https://ipfs.io/ipfs/QmVNXCAfJgER6U7Z5XT8QaAVFPdwmtSFE6c9sUaAx7ttZs

Under link/output/result.json you will see

{"question": "What is a chatbot?", "text": "<pad> A  chatbot  is  a  computer  program  that  can  interact  with  users  in  a  conversational  manner.  It  is  designed  to  answer  questions  and  provide  information  in  a  way  that  is  natural  and  conversational.\n"}

Pssst... here's a question on Claude Monet you could try too ;) bafybeihu62yl76fcypidaiz35gq3yjguxawy5zzwadzvlcgpnfkuy2do3i

LLM Module Code

https://github.com/bacalhau-project/lilypad-modicum/blob/main/src/python/modules/fastchat.py


import json

def _fastchat(params: str):
    cmd = ["python3", "main.py"]
    return {
        "APIVersion": "V1beta1",
        "Metadata": {
            "CreatedAt": "0001-01-01T00:00:00Z",
            "Requester": {}
        },
        "Spec": {
            "Deal": {
                "Concurrency": 1
            },
            "Docker": {
                "Entrypoint": cmd,
                "Image": "xqua/carpai-demo-repo:v0.6",
            },
            "Engine": "Docker",
            "Language": {
                "JobContext": {}
            },
            "Network": {
                "Type": "None"
            },
            "PublisherSpec": {
                "Type": "Estuary"
            },
            "Resources": {
                "GPU": "1"
            },
            "Timeout": 1800,
            "Verifier": "Noop",
            "Wasm": {
                "EntryModule": {}
            },
            "inputs": [
                {
                    "CID": params,
                    "Name": "prompt_template_input",
                    "StorageSource": "IPFS",
                    "path": "/prompt_template.json",

                },
            ],
            "outputs": [
                {
                    "Name": "output",
                    "StorageSource": "IPFS",
                    "path": "/output"
                }
            ]
        }
    }

if __name__ == "__main__":
    print(_fastchat('QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt'))

Stable Diffusion (SDXL0.9)

Run a Stable Diffusion Text to Image Job

Overview

In computer science, you define rules for your (dye) particles to follow and the medium this takes place in.

[CLI] Running Stable Diffusion SDXL 0.9

Ensure you have installed all requirements [CLI] Install Run Requirements

To run stable diffusion use the SDXL module like so:

lilypad run sdxl:v0.9-lilypad1 "an astronaut riding on a unicorn"

The output will look like this:

Take the ipfs link given in the results and paste it into your browser:

Please be patient! IPFS can take some time to propagate and doesn't always work immediately.

In the /outputs folder, you'll find the image:

Since modules are deterministic, running this command with the same text prompt will produce the same image, since the same seed is also used (the default seed is 0).

See this beginner-friendly article on how seed's work for more info on this

To change the image, you can pass in a different seed number:

lilypad run sdxl:v0.9-lilypad1 '{"prompt": "an astronaut riding on a unicorn", "seed": 9}'

CLI Video

[Smart Contract] Running Stable Diffusion SDXL 0.9

Make sure you have connected to the Lalechuza testnet and funded your wallet with testnet lilETH. See Funding your Wallet from Faucet & Setting up Metamask

// SPDX-License-Identifier: GPLv3
pragma solidity ^0.8.6;

// give it the ModicumContract interface 
// NB: this will be a separate import in future.
interface ModicumContract {
  function runModuleWithDefaultMediators(string calldata name, string calldata params) external payable returns (uint256);
}

contract SDXLCaller {
  address public contractAddress;
  ModicumContract remoteContractInstance;
  
  // See the latest result.
  uint256 public resultJobId;
  string public resultCID;

  // The Modicum contract address is found here: https://github.com/bacalhau-project/lilypad-modicum/blob/main/latest.txt
  // Current: 0x422F325AA109A3038BDCb7B03Dd0331A4aC2cD1a
  constructor(address _modicumContract) {
    require(_modicumContract != address(0), "Contract cannot be zero address");
    contractAddress = _modicumContract;
    //make a connection instance to the remote contract
    remoteContractInstance = ModicumContract(_modicumContract);
  } 

  /*
  * @notice Run the SDXL Module
  * @param prompt The input text prompt to generate the stable diffusion image from
  */
  function runSDXL(string memory prompt) public payable returns (uint256) {
    require(msg.value == 2 ether, "Payment of 2 Ether is required"); //all jobs are currently 2 lilETH
    return remoteContractInstance.runModuleWithDefaultMediators{value: msg.value}("sdxl:v0.9-lilypad1", prompt);
  }
  
  // This must be implemented in order to receive the job results back!
  function receiveJobResults(uint256 _jobID, string calldata _cid) public {
    resultJobId =_jobID;
    resultCID = _cid;
  }

}

NB: You could also add the seed as a parameter to run this.

return remoteContractInstance.runModuleWithDefaultMediators{value: msg.value}("sdxl:v0.9-lilypad1",`` params);

Remix

Try it yourself!

Click this link to open the contract in Remix IDE!

Ensure your MetaMask wallet is set to the Lalechuza testnet and has lilETH testnet funds from the faucet.
Set the remix environment to "Injected Provider - MetaMask" (& ensure MetaMask has the lalechuza chain selected)
Then - Deploy a new contract passing in the Modicum Contract address found here OR - Open the contract at this example address: 0x31e7bF121EaB1C0B081347D8889863362e9ad53A

Call the runSDXL Module, passing in a prompt and sending 2 lilETH in the value field. Your MetaMask wallet should pop up for you to confirm the payment and transaction.

Give it some time and check the resultCID variable. You can then open this result in your browser with https://ipfs.io/ipfs/<resultCID> or ipfs://<resultCID> in IPFS compatible browsers like Brave.

Video

FYI! You can try all examples in one contract. See [Smart Contract] Run "Hello, World!" Job

SDXL Module Code

Find the SDXL module code here. There's also a generic Stable Diffusion module here.

https://github.com/bacalhau-project/lilypad-modicum/blob/main/src/python/modules/sdxl.py

import yaml

def _sdxl(params: str):
    if params.startswith("{"):
        params = yaml.safe_load(params)
    else:
        prompt = params
        params = {"prompt": prompt, "seed": 0}
    if not isinstance(params, dict):
        raise Exception("Please set params to a dict like {'prompt': 'astronaut riding a horse', 'seed': 42}")
    return {
        "APIVersion": "V1beta1",
        "Metadata": {
            "CreatedAt": "0001-01-01T00:00:00Z",
            "Requester": {}
        },
        "Spec": {
            "Deal": {
                "Concurrency": 1
            },
            "Docker": {
                "Entrypoint": [
                    "bash", "-c",
                    # stderr logging is nondeterministic (includes timing information)
                    "python3 inference.py 2>/dev/null",
                ],
                "Image": "quay.io/lukemarsden/sdxl:v0.9-lilypad1-v2",
                "EnvironmentVariables": [
                    f"PROMPT={params.get('prompt', 'question mark floating in space')}",
                    f"RANDOM_SEED={params.get('seed', 0)}",
                    f"OUTPUT_DIR=/outputs/",
                    "HF_HUB_OFFLINE=1",
                ]
            },
            "Engine": "Docker",
            "Language": {
                "JobContext": {}
            },
            "Network": {
                "Type": "None"
            },
            "PublisherSpec": {
                "Type": "Estuary"
            },
            "Resources": {
                "GPU": "1"
            },
            "Timeout": 1800,
            "Verifier": "Noop",
            "Wasm": {
                "EntryModule": {}
            },
            "outputs": [
                {
                    "Name": "outputs",
                    "StorageSource": "IPFS",
                    "path": "/outputs"
                }
            ]
        }
    }

if __name__ == "__main__":
    print(_sdxl("{prompt: hello, seed: 99}"))
    print(_sdxl("{prompt: 'hello world', seed: 99}"))

https://github.com/bacalhau-project/lilypad-modicum/blob/main/src/python/modules/lora.py

import yaml
import textwrap

IMAGE = "quay.io/lukemarsden/lora:v0.0.4"
MODEL_NAME = "runwayml/stable-diffusion-v1-5"

# Don't want to make this variable since the job is fixed price
NUM_IMAGES = 10

def _lora_training(params: str):
    if params.startswith("{"):
        params = yaml.safe_load(params)
    else:
        raise Exception("Please set params to yaml like {seed: 42, 'images_cid': 'Qm...'} where images_cid contains an images.zip with training images in")

    seed = params.get("seed", 42)
    images_cid = params["images_cid"]

    return {
        "APIVersion": "V1beta1",
        "Metadata": {
            "CreatedAt": "0001-01-01T00:00:00Z",
            "Requester": {}
        },
        "Spec": {
            "Deal": {
                "Concurrency": 1
            },
            "Docker": {
                "EnvironmentVariables": [
                    f"RANDOM_SEED={seed}",
                ],
                "Entrypoint": [
                    "bash", "-c", f'(cd /input && unzip images.zip && rm images.zip) && lora_pti --pretrained_model_name_or_path={MODEL_NAME} --instance_data_dir=/input --output_dir=/output --train_text_encoder --resolution=512 --train_batch_size=1 --gradient_accumulation_steps=4 --scale_lr --learning_rate_unet=1e-4 --learning_rate_text=1e-5 --learning_rate_ti=5e-4 --color_jitter --lr_scheduler="linear" --lr_warmup_steps=0 --placeholder_tokens="<s1>|<s2>" --use_template="style" --save_steps=100 --max_train_steps_ti=1000 --max_train_steps_tuning=1000 --perform_inversion=True --clip_ti_decay --weight_decay_ti=0.000 --weight_decay_lora=0.001 --continue_inversion --continue_inversion_lr=1e-4 --device="cuda:0" --lora_rank=1'
                ],
                "Image": IMAGE,
            },
            "Engine": "Docker",
            "Language": {
                "JobContext": {}
            },
            "Network": {
                "Type": "None"
            },
            "PublisherSpec": {
                "Type": "Estuary"
            },
            "Resources": {
                "GPU": "1"
            },
            "Timeout": 1800,
            "Verifier": "Noop",
            "Wasm": {
                "EntryModule": {}
            },
            "inputs": [
                {
                    "CID": images_cid,
                    "Name": "lora_input",
                    "StorageSource": "IPFS",
                    "path": "/input",

                },
            ],
            "outputs": [
                {
                    "Name": "output",
                    "StorageSource": "IPFS",
                    "path": "/output"
                }
            ]
        }
    }

def _lora_inference(params: str):
    if params.startswith("{"):
        params = yaml.safe_load(params)
    else:
        raise Exception("Please set params to yaml like {seed: 42, 'lora_cid': 'Qm...', "+
                        "prompt: 'an astronaut in the style of <s1><s2>'} "+
                        "where lora_cid is the output cid of the above step")

    # TODO add a default we pin
    lora_cid = params["lora_cid"]
    seed = params.get("seed", 42)
    prompt = params.get("prompt", "question mark floating in space")
    finetune_weighting = params.get("finetune_weighting", 0.5)

    return {
        "APIVersion": "V1beta1",
        "Metadata": {
            "CreatedAt": "0001-01-01T00:00:00Z",
            "Requester": {}
        },
        "Spec": {
            "Deal": {
                "Concurrency": 1
            },
            "Docker": {
                "EnvironmentVariables": [
                    f"PROMPT={prompt}",
                    f"RANDOM_SEED={seed}",
                    f"FINETUNE_WEIGHTING={finetune_weighting}",
                    "HF_HUB_OFFLINE=1",
                ],
                "Entrypoint": [
                    'python3',
                    '-c',
                    # dedent
                    textwrap.dedent(f"""
                        from diffusers import StableDiffusionPipeline, EulerAncestralDiscreteScheduler
                        import os
                        import torch
                        from lora_diffusion import tune_lora_scale, patch_pipe

                        model_id = "{MODEL_NAME}"

                        pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to(
                            "cuda"
                        )
                        pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)

                        prompt = os.getenv("PROMPT")
                        seed = int(os.getenv("RANDOM_SEED"))
                        torch.manual_seed(seed)

                        os.system("find /input")
                        patch_pipe(
                            pipe,
                            "/input/output/final_lora.safetensors",
                            patch_text=True,
                            patch_ti=True,
                            patch_unet=True,
                        )

                        coeff = float(os.getenv("FINETUNE_WEIGHTING", 0.5))
                        tune_lora_scale(pipe.unet, coeff)
                        tune_lora_scale(pipe.text_encoder, coeff)

                        image = pipe(prompt, num_inference_steps=50, guidance_scale=7).images[0]
                        image.save(f"/output/image-{{seed}}.jpg")
                        image
                        """)
                ],
                "Image": IMAGE,
            },
            "Engine": "Docker",
            "Language": {
                "JobContext": {}
            },
            "Network": {
                "Type": "None"
            },
            "PublisherSpec": {
                "Type": "Estuary"
            },
            "Resources": {
                "GPU": "1"
            },
            "Timeout": 1800,
            "Verifier": "Noop",
            "Wasm": {
                "EntryModule": {}
            },
            "inputs": [
                {
                    "CID": lora_cid,
                    "Name": "lora_input",
                    "StorageSource": "IPFS",
                    "path": "/input",

                },
            ],
            "outputs": [
                {
                    "Name": "output",
                    "StorageSource": "IPFS",
                    "path": "/output"
                },
            ]
        }
    }