Stable Diffusion (SDXL0.9 & 1.0)

Run a Stable Diffusion Text to Image Job

Generically, stable diffusion is what happens when you put a couple of drops of dye into a bucket of water. Given time, the dye randomly disperses and eventually settles into a uniform distribution which colours all the water evenly

In computer science, you define rules for your (dye) particles to follow and the medium this takes place in.

Stable Diffusion is a machine learning model used for text-to-image processing (like Dall-E) and based on a diffusion probabilistic model that uses a transformer to generate images from text.

[CLI] Running Stable Diffusion SDXL 0.9 and SDXL 1.0

Ensure you have installed all requirements

Lilypad

Lilypad can run SDXL v0.9 or SDXL v1.0 with the option to add tunables to improve or change the model output.

When running SDXL pipelines in Lilypad, you have the choice between using the Base model or the Refiner model. Each serves a unique purpose in the image generation process:

• Base Model: This is the primary model that generates the initial image based on your input prompt. It focuses on the broad aspects of the image, capturing the main theme and essential elements. The Base model is faster and uses less computational power.

• Refiner Model: This model takes the image from the Base model and enhances it. It refines details, improves textures, and adjusts colors to increase the visual appeal and realism of the image. The Refiner model is used when you need higher quality and more detailed images.

To run SDXL Pipeline in Lilypad, you can use the following commands:

SDXL v0.9

Base:

lilypad run sdxl-pipeline:v0.9-base-lilypad3 -i Prompt="an astronaut floating against a white background"

Refiner:

lilypad run sdxl-pipeline:v0.9-refiner-lilypad3 -i Prompt="an astronaut floating against a white background"

SDXL 1.0

Base:

lilypad run sdxl-pipeline:v1.0-base-lilypad3 -i Prompt="an astronaut floating against a white background"

Refiner:

lilypad run sdxl-pipeline:v1.0-refiner-lilypad3 -i Prompt="an astronaut floating against a white background"

Specifying tunables

If you wish to specify more than one tunable, such as the number of steps, simply add more -i flags. For example, to improve the quality of the image generated add "Steps=x" with x = (up to 200):

lilypad run sdxl-pipeline -i Prompt="an astronaut floating against a white background" -i Steps=69

See the options and tunables section (below) for more information on what tunables are available.

The output will look like this:

SDXL Output

Take the IPFS link given in the results and paste it into your browser:

https://ipfs.io/ipfs/QmVng1jkMxE9ep4k8mYiiCiWaCRiRLvCeo6bJRXirhz1dZ

Please be patient! IPFS can take some time to propagate and doesn't always work immediately.

You could also check the output folder that would have been downloaded at the end of running the job:

open /tmp/lilypad/data/downloaded-files/QmZuE29GJVmenRUh72FQDgkMUT1Zdp967oEJvzjaDwGGVo

Results of SDXL job on Output Directory

In the /outputs folder, you'll find the image:

The Image in the outputs folder

Since modules are deterministic, running this command with the same text prompt will produce the same image, since the same seed is also used (the default seed is 0).

See this beginner-friendly article on how seeds work for more info on this

To change the image, you can pass in a different seed number:

`lilypad run sdxl-pipeline:v0.9-base-lilypad3 -i Prompt='a gigantic lilypad shaped space station' -i Steps=150` 

Output using different seed.

Options and tunables

The following tunables are available. All of them are optional, and have default settings that will be used if you do not provide them.

Name	Description	Default	Available options
Prompt	A text prompt for the model	"question mark floating in space"	Any string
Seed	A seed for the model	42	Any valid non-negative integer
Steps	The number of steps to run the model for	50	Any valid non-negative integer
Scheduler	The scheduler to use for the model	normal	normal, karras, exponential, sgm_uniform, simple, ddim_uniform
Sampler	The sampler to use for the model	euler_ancestral	"euler", "euler_ancestral", "heun", "heunpp2", "dpm_2", "dpm_2_ancestral", "lms", "dpm_fast", "dpm_adaptive", "dpmpp_2s_ancestral", "dpmpp_sde", "dpmpp_sde_gpu", "dpmpp_2m", "dpmpp_2m_sde", "dpmpp_2m_sde_gpu", "dpmpp_3m_sde", "dpmpp_3m_sde_gpu", "ddpm", "lcm"
Size	The output size requested in px	1024	512, 768, 1024, 2048
Batching	How many images to produce	1	1, 2, 4, 8

See the usage sections for the runner of your choice for more information on how to set and use these variables.

Learn more about this Lilypad module on Github.

CLI Video

In this video, Sam demos the use of SDXL in the Lilypad network.

Looking for Smart Contracts?

Try this to start!

https://github.com/bacalhau-project/lilypad/blob/main/docs/smart-contract-jobs.md

## Running jobs from smart contracts

We have deloyed the [LilypadOnChainJobCreator](../hardhat/contracts/LilypadOnChainJobCreator.sol) contract which you can use to trigger running jobs on the lilypad network from other smart contracts.

It works in tandem with the `lilypad jobcreator` on-chain which will watch the on-chain contract and manage jobs on behalf of contracts that submit them.

#### Creating a job

You will need to know the contract address for the on-chain job creator so we can submit transactions to it.

The production controller address is `0x8e136587e3e5266d5244f6aa896E5CAf8E969946` and you can ask it for the address of the on-chain job creator `getJobCreatorAddress()`

Running a job involves 2 phases:

 * calling `approve` on the ERC-20 contract to allow the solver to spend your tokens
 * trigger the job via the on chain job manager

Now we know the address of the on-chain job controller - we can ask it for 3 things:

 * the address of the ERC-20 token contract - `getTokenAddress()`
 * how much the required deposit it - `getRequiredDeposit()`
 * the address of the solver that will handle running the job for us - `getControllerAddress()`

Knowing these 3 things means we can call the standard ERC-20 `approve` to allow the solver to spend our tokens on our behalf.

Now - we can call the `runJob` method of the on chain controller from another contract.  This will cause the job-creator service to kick in and do the following things:

 * check that funds have been approved for the solver
 * transfer those funds to it's wallet
 * run the job on lilypad
 * call the `submitResults` method on the on-chain job creator
 * the on-chain job creator will call the `submitResults` of the original calling contract

The following is an example on-chain smart contract:

```solidity
// SPDX-License-Identifier: Apache-2.0
pragma solidity ^0.8.6;

import "@openzeppelin/contracts/access/Ownable.sol";
import "@openzeppelin/contracts-upgradeable/proxy/utils/Initializable.sol";
import "./ILilypadJobManager.sol";
import "./ILilypadJobClient.sol";

contract ExampleClient is Ownable, Initializable, ILilypadJobClient {

  address private jobManagerAddress;
  ILilypadJobManager private jobManagerContract;

  mapping(uint256 => string) private jobResults;

  event JobCreated(
    uint256 id,
    string message
  );

  event JobCompleted(
    uint256 id,
    string dealId,
    string dataId
  );

  function initialize(address _jobManagerAddress) public initializer {
    setJobManagerAddress(_jobManagerAddress);
  }

  function setJobManagerAddress(address _jobManagerAddress) public onlyOwner {
    require(_jobManagerAddress != address(0), "Job manager address");
    jobManagerAddress = _jobManagerAddress;
    jobManagerContract = ILilypadJobManager(jobManagerAddress);
  }

  function getJobResult(uint256 _jobID) public view returns (string memory) {
    return jobResults[_jobID];
  }

  function runCowsay(
    string memory message
  ) public {
    string[] memory inputs = new string[](1);
    inputs[0] = string(abi.encodePacked("Message=", message));
    uint256 id = jobManagerContract.runJob(
      "cowsay:v0.0.3",
      inputs,
      msg.sender
    );

    emit JobCreated(
      id,
      message
    );
  }

  function submitResults(
    uint256 id,
    string memory dealId,
    string memory dataId
  ) public override {
    jobResults[id] = dataId;
    emit JobCompleted(
      id,
      dealId,
      dataId
    );
  }
}
```

Here is an example of a script that brings all of this together:

```typescript
import bluebird from 'bluebird'
import {
  connectToken,
  connectJobManager,
  connectExampleClient,
  getWallet,
  getAddress,
} from '../utils/web3'
import { ethers } from 'hardhat'

async function main() {
  // it's annoying to not be able to use argv but hardhat complains about it
  const message = process.env.MESSAGE || 'Hello World!'

  const token = await connectToken()
  const manager = await connectJobManager()
  const client = await connectExampleClient()

  const setRequiredDepositTx = await manager
    .connect(getWallet('solver'))
    .setRequiredDeposit(ethers.parseEther("2"))
  await setRequiredDepositTx.wait()

  const requiredDeposit = await manager.getRequiredDeposit()

  console.log(`requiredDeposit: ${Number(requiredDeposit)}`)

  const paytokensTx = await token
    .connect(getWallet('job_creator'))
    .approve(getAddress('solver'), requiredDeposit)
  await paytokensTx.wait()

  console.log(`tokens approved: ${paytokensTx.hash}`)

  const runjobTx = await client
    .connect(getWallet('job_creator'))
    .runCowsay(message)
  const receipt = await runjobTx.wait()
  if(!receipt) throw new Error(`no receipt`)

  console.log(`submitted job: ${runjobTx.hash}`)

  let jobID = 0

  receipt.logs.forEach(log => {
    const logs = client.interface.parseLog(log as any)
    if(!logs) return
    jobID = Number(logs.args[0])
  })

  console.log(`Job ID: ${jobID}`)
  console.log(`Waiting for job to be completed...`)

  let result = ''

  while(!result) {
    result = await client.getJobResult(jobID)
    if(!result) {
      await bluebird.delay(1000)
    }
  }

  console.log(`Job result: ${result}`)
}

main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

```