Stable Diffusion (SDXL0.9)

Run a Stable Diffusion Text to Image Job

Overview

Generically, stable diffusion is what happens when you put a couple of drops of dye into a bucket of water. Given time, the dye randomly disperses and eventually settles into a uniform distribution which colours all the water evenly

In computer science, you define rules for your (dye) particles to follow and the medium this takes place in.

Stable Diffusion is a machine learning model used for text-to-image processing (like Dall-E) and based on a diffusion probabilistic model that uses a transformer to generate images from text. There are several open-source stable diffusion models out there (made famous by Stability.ai) and they continue to improve and become even more fully featured - SDXL0.9 is one of the more recently open-sourced models.

[CLI] Running Stable Diffusion SDXL 0.9

Ensure you have installed all requirements [CLI] Install Run Requirements

To run stable diffusion use the SDXL module like so:

lilypad run sdxl:v0.9-lilypad1 "an astronaut riding on a unicorn"

The output will look like this:

Take the ipfs link given in the results and paste it into your browser:

Please be patient! IPFS can take some time to propagate and doesn't always work immediately.

In the /outputs folder, you'll find the image:

Since modules are deterministic, running this command with the same text prompt will produce the same image, since the same seed is also used (the default seed is 0).

See this beginner-friendly article on how seed's work for more info on this

To change the image, you can pass in a different seed number:

lilypad run sdxl:v0.9-lilypad1 '{"prompt": "an astronaut riding on a unicorn", "seed": 9}'

CLI Video

[Smart Contract] Running Stable Diffusion SDXL 0.9

Make sure you have connected to the Lalechuza testnet and funded your wallet with testnet lilETH. See Funding your Wallet from Faucet & Setting up Metamask

To trigger the SDXL0.9 module from a smart contract, firstly you need to create your own client contract to call the module from. In order to receive results back from the Lilypad network, you will also need to 1. Connect to the Lilypad Modicum Contract (and create an instance of it in your own contract using the current address found here) 2. Implement the Modicum Contract receiveJobResults() interface.

// SPDX-License-Identifier: GPLv3
pragma solidity ^0.8.6;

// give it the ModicumContract interface 
// NB: this will be a separate import in future.
interface ModicumContract {
  function runModuleWithDefaultMediators(string calldata name, string calldata params) external payable returns (uint256);
}

contract SDXLCaller {
  address public contractAddress;
  ModicumContract remoteContractInstance;
  
  // See the latest result.
  uint256 public resultJobId;
  string public resultCID;

  // The Modicum contract address is found here: https://github.com/bacalhau-project/lilypad-modicum/blob/main/latest.txt
  // Current: 0x422F325AA109A3038BDCb7B03Dd0331A4aC2cD1a
  constructor(address _modicumContract) {
    require(_modicumContract != address(0), "Contract cannot be zero address");
    contractAddress = _modicumContract;
    //make a connection instance to the remote contract
    remoteContractInstance = ModicumContract(_modicumContract);
  } 

  /*
  * @notice Run the SDXL Module
  * @param prompt The input text prompt to generate the stable diffusion image from
  */
  function runSDXL(string memory prompt) public payable returns (uint256) {
    require(msg.value == 2 ether, "Payment of 2 Ether is required"); //all jobs are currently 2 lilETH
    return remoteContractInstance.runModuleWithDefaultMediators{value: msg.value}("sdxl:v0.9-lilypad1", prompt);
  }
  
  // This must be implemented in order to receive the job results back!
  function receiveJobResults(uint256 _jobID, string calldata _cid) public {
    resultJobId =_jobID;
    resultCID = _cid;
  }

}

NB: You could also add the seed as a parameter to run this.

return remoteContractInstance.runModuleWithDefaultMediators{value: msg.value}("sdxl:v0.9-lilypad1",`` params);

Remix

Try it yourself!

Click this link to open the contract in Remix IDE!

Ensure your MetaMask wallet is set to the Lalechuza testnet and has lilETH testnet funds from the faucet.
Set the remix environment to "Injected Provider - MetaMask" (& ensure MetaMask has the lalechuza chain selected)
Then - Deploy a new contract passing in the Modicum Contract address found here OR - Open the contract at this example address: 0x31e7bF121EaB1C0B081347D8889863362e9ad53A

Call the runSDXL Module, passing in a prompt and sending 2 lilETH in the value field. Your MetaMask wallet should pop up for you to confirm the payment and transaction.

Give it some time and check the resultCID variable. You can then open this result in your browser with https://ipfs.io/ipfs/<resultCID> or ipfs://<resultCID> in IPFS compatible browsers like Brave.