To interact with Lilypad's API, you'll first need to create an account and obtain an API key for authentication. This key allows you to submit jobs and retrieve results securely.

Sign Up

Visit the Lilypad API Dashboard and sign up using an email address.

Generate API key

Next, log in to the dashboard and click "Create API key".

Now that you've created a new API key you can copy and securely store your API key for usage.

Lilypad allows you to run on-demand compute jobs on the network using the Lilypad API, enabling AI inference and high-performance workloads without managing infrastructure.

Overview

This guide will take you through:

• Obtaining API keys

• Basic usage

Lilypad is powering democratic participation in the AI Innovation Economy by pioneering new avenues for anyone to deploy, distribute and monetize AI models.

As a full-stack, modular AI services platform, Lilypad provides:

• A model marketplace for publishing and monetizing AI workloads

• MLOps tooling for managing and scaling AI pipelines

• A distributed, on-demand GPU network for high-performance inference across agent workflows, ML research and DeSci applications

Lilypad acts as a:

• Demand engine for decentralized compute and storage

• Distribution layer for training networks and agentic frameworks

• Bridge across data, compute and AI ecosystems, enabling straightforward integration

Real-World Impact

In partnership with Dell, Dr. Michael Levin, and Amelie Schroder (Hugging Face's top bioML creator), Lilypad powered a protein folding AI model that discovered a novel heart medication, now progressing through wet-lab validation. We are now working with Dell to build a permissioned, licensable version of the Lilypad Network for enterprise AI adoption.

Run AI Workloads

Lilypad makes it easy to run containerized AI models across a global, decentralized GPU network with no infrastructure setup required. Whether you're working with image generation models like Stable Diffusion and SDXL, text-to-video models, LLMs, or custom AI workloads, Lilypad provides the compute layer to scale inference on demand.

You can:

• Deploy models as Lilypad Job Modules using standard Docker containers

• Trigger jobs via our CLI or inference API, with full support for passing inputs and capturing outputs

• Integrate into ML pipelines or agent workflows, using Lilypad as a backend for large-scale AI tasks

• Leverage the modular architecture to compose your own workflows, fine-tune models, or chain outputs across models

If you're building an app, automating research, or experimenting with new architectures — Lilypad gives you a permissionless, cost-efficient path to run your AI anywhere.

Provide Compute

Join the Lilypad Network as a Resource Provider and earn rewards by contributing your idle GPU or CPU power to run real AI jobs. Whether you're a solo builder with a gaming rig or an organization managing fleets of GPUs, Lilypad lets you monetize underused compute by connecting directly to demand from developers, researchers and decentralized applications.

As a Resource Provider, you’ll:

• Run containerized workloads from users across the network

• Earn rewards for completed jobs

• Retain full control over your hardware with no lock-in or custodial overhead

• Support open innovation by enabling inference for DeSci, agent frameworks, AI art tools, and more

Lilypad’s compute layer is permissionless, modular, and designed to scale. We support consumer GPUs, data center hardware, and containerized environments — including Linux hosts and Docker-native setups.

Get Started Today!

Join the Community & Chat with Us

Once you have your API key, you can start running AI inference jobs using Lilypad's Anura API. Below is a simple "Hello World" example to get started.

Get Available Models

Before running a job, check which models are supported:

Run a Job

Use the API to send a simple chat completion request using one of the available models:

Replace "MODEL_NAME:MODEL_VERSION" with the model you want to use from the previous step.

Retreive Job Results

You can check the status of your job and retrieve outputs using the job ID:

This will return details on whether the job is still processing or has completed.

For more advanced usage, including streaming completions, multi-turn conversations, and additional endpoints, check out the .

Lilypad allows you to run on-demand compute jobs on the network, enabling AI inference and high-performance workloads without managing infrastructure.

AI agents can autonomously process information, make decisions and interact with users or systems. This section explores different use cases for AI agents on Lilypad, providing examples and step-by-step guides on how to build and run them using the network.

Lilypad allows you to run on-demand compute jobs on the network using the Lilypad CLI, enabling AI inference and high-performance workloads without managing infrastructure.

Overview

This guide will take you through:

• the Lilypad CLI.

• .

• , the Lilypad module equivalent to Hello, World!

These instructions provide steps for running the SDXL Turbo Pipeline module on the Lilypad network using Docker and Lilypad CLI. Find the module repo here.

Getting Started

Before running sdxl turbo, make sure you have the Lilypad CLI installed on your machine and your private key environment variable is set. This is necessary for operations within the Lilypad network.

export WEB3_PRIVATE_KEY=<YOUR_PRIVATE_KEY>

Learn more about installing the Lilypad CLI and running a Lilypad job with this video guide.

Run SDXL Turbo

lilypad run github.com/Lilypad-Tech/module-sdxl:d6a89ed92f4e798459b2990340669da00c56c80c -i prompt="your prompt here"

Example:

lilypad run github.com/Lilypad-Tech/module-sdxl:d6a89ed92f4e798459b2990340669da00c56c80c -i prompt="a spaceship parked on a mountain"

Notes

• Ensure you have the necessary permissions and resources to run Docker containers with GPU support.

• The module version (ae17e969cadab1c53d7cabab1927bb403f02fd2a) may be updated. Check for the latest version before running.

• Adjust port mappings and volume mounts as needed for your specific setup.

Output

To view the results in a local directory, navigate to the local folder.

open /tmp/lilypad/data/downloaded-files/<fileID>

Installation

First, you'll need to install the package:

pip install create-lilypad-module

If you've previously installed create-lilypad-module, you should to ensure that you're using the latest version:

pip install --upgrade create-lilypad-module

Now run create-lilypad-module:

create-lilypad-module

The CLI will ask for the name of your project. Alternatively, you can run:

create-lilypad-module project_name
cd project_name

Output

project_name
├── config
│   └── constants.py
├── scripts
│   ├── docker_build.py
│   ├── download_models.py
│   └── run_module.py
├── src
│   └── run_inference.py
├── .dockerignore
├── .env
├── .gitignore
├── Dockerfile
├── lilypad_module.json.tmpl
├── README.md
└── requirements.txt

Overview

The Lilypad Research team published "Validation of GPU Computation in Decentralized, Trustless Networks" diving into the complexities of validating successful compute jobs on GPUs. The investigation then explores verification methods and solutions that Lilypad can implement across the network.

Verifying computational processes in decentralized networks poses a fundamental challenge, particularly for Graphics Processing Unit (GPU) computations. The Lilypad Research team conducted an investigation revealing significant limitations in existing approaches: exact recomputation fails due to computational non-determinism across GPU nodes, Trusted Execution Environments (TEEs) require specialized hardware, and Fully Homomorphic Encryption (FHE) faces prohibitive computational costs.

To address these challenges, this report explores three verification methodologies adapted from adjacent technical domains: model fingerprinting techniques, semantic similarity analysis, and GPU profiling. Through systematic exploration of these approaches, we develop novel probabilistic verification frameworks, including a binary reference model with trusted node verification and a ternary consensus framework that eliminates trust requirements.

These methodologies establish a foundation for ensuring computational integrity across untrusted networks while addressing the inherent challenges of non-deterministic execution in GPU-accelerated workloads.

This section features hands-on tools and resources designed to support developers building with Lilypad. From code editor extensions to social app integrations, it highlights ways to prototype, automate and ship AI jobs more efficiently. Explore how these utilities fit into real workflows and help extend the power of the network.

Need some inspiration? Visit the Helpful Resources page!

There are multiple ways to participate in the Lilypad ecosystem and earn rewards for your contributions.

Developers

Community Members

Resource Providers

In exchange for providing compute power to Lilypad's decentralized compute network, Resource Providers (RPs) accumulate Lilybit_ credits based on their contributions to the network.

RP incentives on Testnet will be provided via the Lilypad RP Beta Program. Any RP can submit info to join the RP beta program, although RPs with more resources will be prioritized in order to run large models on Lilypad Network.

create-lilypad-module is an officially supported package that provides a simple scaffolding for building Lilypad modules. It offers a modern Docker setup with minimal configuration.

Overview

• : install and run create-lilypad-module

• : output and explanation of create-lilypad-module files

• : requirements and explanations of Lilypad module configuration

• : a step-by-step guide on how to create a simple Lilypad module using create-lilypad-module

This is a work-in-progress proof of concept utilizing Extra Labs and Lilypad to create a Lilypad module for running multiple geospatial jobs.

The goal is to provide the end-to-end adapter for users to operate on the point cloud data to generate reconstructed maps.

Extra Labs is revolutionizing the way maps are built by leveraging collaborative acquisition and deep learning to produce affordable, detailed 3D models of cities and territories.

To achieve this proof of concept, the Lilypad module will integrate with Extra Labs' platform, allowing users to submit their geospatial data, whether collected via drones, aircraft, delivery vehicles, or smartphones. This data is processed through advanced algorithms to create detailed 3D reconstructions.

The decentralized nature of the platform ensures that data providers are compensated fairly for their contributions through a blockchain-based reward system. This approach not only democratizes access to 3D mapping technologies but also ensures continuous and up-to-date data acquisition, enhancing the accuracy and detail of the generated maps.

Users should be able to easily interact with the platform, upload their data, and receive high-quality 3D models in return. This process is designed to make advanced geospatial mapping accessible to a wide range of users, from urban planners and architects to developers and hobbyists.

To find out more, please visit the .

The Lilypad Network uses the Arbitrum Sepolia Testnet to settle compute transactions.

This personal RPC endpoint allows Resource Providers (RP) to avoid reliability issues with the RPC endpoints used by Lilypad ensuring rewards can be earned and jobs can be run consistently. RPs running a personal RPC endpoint contribute to the fault tolerance and decentralization of the Lilypad Network! Read more in the Alchemy Arbitrum .

Setup RPC video guide

AI tooling provides the essential infrastructure and frameworks that support the development, deployment and monitoring of AI-powered applications. This section highlights tools that help streamline workflows, enhance visibility and enable customization when building with AI on Lilypad, with examples and guides for integrating them into your projects.

These instructions provide steps for running the Llama2 module on the Lilypad network using Docker and the Lilypad CLI. Find the module repo .

Getting Started

Before running llama2, make sure you have the on your machine and your private key environment variable is set. This is necessary for operations within the Lilypad network.

Learn more about installing the Lilypad CLI and running a Lilypad job with this .

Run Llama2

Run Llama2

Example:

Notes

• Ensure you have the necessary permissions and resources to run Docker containers with GPU support.

• The module version (6d4fd8c07b5f64907bd22624603c2dd54165c215) may be updated. Check for the latest version before running.

• Adjust port mappings and volume mounts as needed for your specific setup.

Output

To view the results in a local directory, navigate to the local folder provided by the job result.

Utility Maximization

A core assumption in much of game theory is that agents are utility-maximizing. That is, agents are completely rational actors, and are able to execute exactly the behavior that maximizes their return, however "return" is defined in the given context.

However, we know that in real life, humans are not completely rational, and are not capable of perfect execution of actions. In that light, how can we look at the game-theoretic approaches in the last section?

Either we can try to account for the irrational behavior of humans, or we can try to emulate the behavior of utility-maximizing agents. While there is a large amount of game-theoretic literature dedicated to the former, we opt for the latter for reasons that will become clear below.

While this problem setting - verifiable computation by way of game theory - is different than many game theoretic settings, we can draw inspiration from commonly used concepts like the revelation principle and strategyproofness. Both strategyproofness and the revelation principle are centered around the idea of incentivizing agents to truthfully report their preferences. Most approaches in the literature rely on analytic methods to determine what rational agents will do by analyzing their payoffs as a function of their preferences, the behaviors of other agents, and the mechanism under analysis. Ultimately, we are also aiming to find (a) mechanism(s) that lead(s) to an equilibrium where all agents choose to not cheat and not collude.

Autonomous Agents

Note that the actual environment of a two-sided marketplace for distributed computation is extremely complicated (e.g. the heterogeneity of hardware, types of computation, latencies and bandwidths, etc.). Any theoretical/analytic approach to the problem that is actually correct should also work in simulation, so we opt for a simulation-driven approach.

The way that we can emulate perfectly rational behavior is by training autonomous agents to act on behalf of their human owners in a utility-maximizing manner. At that point, the challenge is to design the global game to drive the probability of cheating to zero - ideally, to make it be equal to zero - which is no small feat in a trustless and permissionless environment. However, the simplifying assumption that we are in fact operating with utility-maximizing agents conceptually simplifies the problem immensely.

The process begins by creating a digital twin of a two-sided marketplace. In this environment, autonomous agents acting on behalf of client and compute nodes will be trained to maximize returns based on data gathered in simulation. For now, we will avoid maximizing returns by optimizing scheduling, though this is a future topic of interest. We will use techniques primarily from the field of multi-agent reinforcement learning in order to train the agents. The precise methods we will use (e.g. modes of training and execution, homogeneous vs. heterogeneous agents, choice of equilibrium, self-play vs. mixed-play, value-based vs. policy-based learning, etc.) will be determined in the course of building the simulation. See the pre-print by Albrecht, Christianos, and Schäfer for our reference text.

At a minimum, the action space for an autonomous agent representing a compute node should be to cheat or not to cheat, and to collude or not collude within a mediation protocol. The observable environment for nodes on the network should include all data stored on the blockchain - that is, the sequence of deals, results, and mediations - as well as the information in the orderbook. While the orderbook will be off-chain, we model in the digital twin the orderbook acting as a centralized, single source of truth that all agents have access to. In the long-term, nodes will have (potentially non-identical) local information regarding other job and resource offers on the network.

Further work may explore agents forming beliefs about the hardware and strategies of other agents, but that is beyond the scope of the current work.

First Principles Approach

We conclude with two "axioms" upon which we will base our simulations:

1. Every agent attempts to maximize its utility, including cheating and/or colluding if necessary.

2. All other components of the game should lead to a "good" solution, as defined in the problem statement.

Output

After creation, your project should look like this:

project_name
├── config
│   └── constants.py
├── scripts
│   ├── docker_build.py
│   ├── download_models.py
│   └── run_module.py
├── src
│   └── run_inference.py
├── .dockerignore
├── .env
├── .gitignore
├── Dockerfile
├── lilypad_module.json.tmpl
├── README.md
└── requirements.txt

For the module to run, these files must exist with exact filenames:

• src/run_inference.py

  • The Dockerfile ENTRYPOINT.

  • If you change this files name or location, you must also update the ENTRYPOINT in your Dockerfile and lilypad_module.json.tmpl file to match.

• config/constants.py

  • The configuration file that stores the DOCKER_REPO, DOCKER_TAG, MODULE_REPO, and TARGET_COMMIT.

  • If you change this files name or location, you must also update the import statements in scripts/docker_build.py and scripts/run_module.py.

• Dockerfile

  • Required to build your module into a Docker image, and push the image to Docker Hub where it can be accessed by Lilypad Network.

• requirements.txt

  • Used by the Dockerfile to install dependencies required by your module.

  • Technically, this file can be deleted or renamed, but this naming convention is highly recommended as an industry standard best practice.

• lilypad_module.json.tmpl

  • The Lilypad configuration file.

You can delete or rename the other files.

You may create subdirectories inside src. For faster builds and smaller Docker images, only files inside src are copied by Docker. You need to put any files required to run your module inside src, otherwise Docker won’t copy them.

You can create more top-level directories. They will not be included in the final Docker image so you can use them for things like documentation.

If you have Git installed and your project is not part of a larger repository, then a new repository will be initialized resulting in an additional top-level .git directory.

cowsay is a simple, text-based program originally written for Unix-like operating systems that generates ASCII pictures of a cow with a speech bubble containing a specified message.

This module was created as a "Hello World" for the Lilypad Network!

Getting Started

Before running cowsay, make sure you have the Lilypad CLI installed on your machine and your private key environment variable is set. This is necessary for operations within the Lilypad network.

export WEB3_PRIVATE_KEY=<YOUR_PRIVATE_KEY>

Run cowsay

Once you've installed the CLI, run the cowsay command:

lilypad run cowsay:v0.0.4 -i Message="hello, lilypad"  

Output

⠀⠀⠀⠀⠀⠀⣀⣤⣤⢠⣤⣀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⢴⣿⣿⣿⣿⢸⣿⡟⠀⠀⠀⠀⠀    ██╗     ██╗██╗  ██╗   ██╗██████╗  █████╗ ██████╗
⠀⠀⣰⣿⣦⡙⢿⣿⣿⢸⡿⠀⠀⠀⠀⢀⠀    ██║     ██║██║  ╚██╗ ██╔╝██╔══██╗██╔══██╗██╔══██╗
⠀⢰⣿⣿⣿⣿⣦⡙⣿⢸⠁⢀⣠⣴⣾⣿⡆    ██║     ██║██║   ╚████╔╝ ██████╔╝███████║██║  ██║
⠀⣛⣛⣛⣛⣛⣛⣛⠈⠀⣚⣛⣛⣛⣛⣛⣛    ██║     ██║██║    ╚██╔╝  ██╔═══╝ ██╔══██║██║  ██║
⠀⢹⣿⣿⣿⣿⠟⣡⣿⢸⣮⡻⣿⣿⣿⣿⡏    ███████╗██║███████╗██║   ██║     ██║  ██║██████╔╝
⠀⠀⢻⣿⡟⣩⣾⣿⣿⢸⣿⣿⣌⠻⣿⡟⠀    ╚══════╝╚═╝╚══════╝╚═╝   ╚═╝     ╚═╝  ╚═╝╚═════╝ v2.13.0
⠀⠀⠀⠉⢾⣿⣿⣿⣿⢸⣿⣿⣿⡷⠈⠀⠀
⠀⠀⠀⠀⠀⠈⠙⠛⠛⠘⠛⠋⠁⠀ ⠀⠀⠀   Decentralized Compute Network  https://lilypad.tech

🌟  Lilypad submitting job
2025-03-05T12:56:38-06:00 WRN ../runner/work/lilypad/lilypad/cmd/lilypad/utils.go:63 > failed to get GPU info: gpuFillInfo not implemented on darwin
2025-03-05T12:56:38-06:00 INF ../runner/work/lilypad/lilypad/pkg/web3/sdk.go:209 > Connected to arbitrum-sepolia-rpc.publicnode.com
2025-03-05T12:56:38-06:00 INF ../runner/work/lilypad/lilypad/pkg/jobcreator/run.go:27 > Public Address: 0xB86bCAe21AC95BCe7a49C057dC8d911033f8CB7c
Enumerating objects: 42, done.
Counting objects: 100% (22/22), done.
Compressing objects: 100% (4/4), done.
Total 42 (delta 18), reused 19 (delta 18), pack-reused 20 (from 1)
💌  Deal agreed. Running job...
🤔  Results submitted. Awaiting verification...
🤔  Results submitted. Awaiting verification...
✅  Results accepted. Downloading result...
🆔  Data ID: QmP2SQttNC3Hrh2xpY7bNHzV2jHq7MbfLahRC46DVzn5rG

🍂 Lilypad job completed, try 👇
    open /tmp/lilypad/data/downloaded-files/QmQHrsiAuzTLn5VU6jg5LoXBRrAkEVRKiYeJE29w54gg9Q
    cat /tmp/lilypad/data/downloaded-files/QmQHrsiAuzTLn5VU6jg5LoXBRrAkEVRKiYeJE29w54gg9Q/stdout
    cat /tmp/lilypad/data/downloaded-files/QmQHrsiAuzTLn5VU6jg5LoXBRrAkEVRKiYeJE29w54gg9Q/stderr

To view the results in a local directory, navigate to the local folder.

open /tmp/lilypad/data/downloaded-files/QmQHrsiAuzTLn5VU6jg5LoXBRrAkEVRKiYeJE29w54gg9Q

Here, you can view the stdout and stderr as well as the outputs folder for the run:

~ % cat /tmp/lilypad/data/downloaded-files/QmQHrsiAuzTLn5VU6jg5LoXBRrAkEVRKiYeJE29w54gg9Q/stdout
 ________________ 
< hello, lilypad >
 ---------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Problem Setup

The setup is a trustless, permissionless, two-sided marketplace for compute, where clients can purchase compute services from compute nodes. Trustless means that by default, the protocol does not assume that any particular node behaves in a trustworthy manner and that each node should be considered as rationally self-interested (note that this excludes intentionally malicious behavior). Permissionless means that any node can join or leave the network at will.

Matches for compute jobs are made off-chain, with the resulting deals and results recorded on-chain. Both clients and compute nodes need to agree to matches before they become deals, and make deposits to the protocol to enable rewards and punishments. Results are verified using verification-via-replication, and clients can check the results of any job after it has been completed, but before it needs to pay. It does so by calling upon a mediation protocol. The mediation protocol is the ultimate source of truth, and the outcome of the mediation protocol determines how payouts to nodes are made.

The issue of preventing fake results in the presence of a Trusted Third Party (TTP) as a mediator is effectively a solved problem (for example, see the section on prior verification-via-replication protocols, though there is much more literature on this topic). Given the assumption that the mediation protocol is the source of truth, we can treat the mediation protocol as a TTP. Since the fake results problem is basically already solved in this case, the cheating problem reduces down to solving the collusion problem within the mediation protocol. (Note, however, that we will address both cheating and collusion; the framework described here exists to conceptually simplify the problem.)

This is a typical scenario of a Byzantine environment, and we can use well-established approaches to Byzantine Fault Tolerance when invoking mediation. However, most BFT algorithms and cryptographic methods rely on assumptions regarding some fraction of honest nodes. The problem is that rational, utility-maximizing agents may still collude, even in a mediation consortium, in order to converge on incorrect results. On the one hand, we could assume that some fraction of nodes are honest, as is often done. On the other hand, can we do better?

Problem Statement

Task

The task is to find the mechanisms that incentivizes all nodes to behave honestly.

Adversary model

All agents in the protocol are utility-maximizing. This will be elucidated in a subsequent section. Most of the literature has focused on the case where compute nodes are dishonest. However, the client can also behave dishonestly in some manner that maximizes their utility. For example, if the client has some level of control over the choice of mediator, and dishonest nodes have their collateral slashed, the client could collude with the mediator in order to deem a correct result incorrect and get a cut of the honest compute node's collateral.

What is a good solution?

"Good" solutions can take a number of forms:

1. Nodes never have an incentive to be dishonest.

2. Nodes have an incentive to be dishonest that goes to zero as a function of the parameters of the protocol.

3. (1) or (2), but under some simplifying assumptions, such as there being some fraction of honest nodes within every mediation protocol.

A good solution would achieve any of these goals. Alternatively, another valuable outcome of this research would be to discover under what assumptions these goals can or cannot be met, or if the goals are even possible to achieve at all.

Mechanisms for achieving these goals

There are a number of ways that these goals may be achieved. The approach will be to construct a digital twin of the protocol and test a number of different mechanisms in simulation. These mechanisms include variations on mediation protocols, collateralization, staking, taxes and jackpots, and others; see the Mechanisms to Explore section for details.

Guides

Video Tutorials

Helpful Resources

AI Resources

• RAG vs Finetuning

• The promise and challenges of crypto + AI - Vitalik Buterin

• Data on Notable AI Models

• Towards decentralized AI

• Understanding the Intersection of Crypto and AI - Galaxy Digital Research

GPU Info

• How do graphics cards work?

• Nvidia CUDA Toolkit

• Nvidia CUDA in 100 Seconds

Bacalhau Resources

• Bacalhau.org

• Impact of Compute over Data - Juan Benet

• CoD vision & goals - Juan Benet

• Lessons for building Distributed Compute - Juan Benet & Molly Mackinlay

• State of CoD 2023 - David Aronchick

• Containers at the Edge - David Aronchick

The Lilypad ML workbench provides ML researchers, businesses, and more a simple interface for leveraging the Lilypad network.

The ML workbench provides an interface to:

• Run models that are currently available on the Lilypad GPU network

• Add a new model to run on the network as a Lilypad module

• Leverage Jupyter notebooks with Lilypad

• Create multi-module flows

• Fine tune models

• Create an AI agent

In order to run an AI model on the Lilypad network, a docker image must be created for the program using the Lilypad module spec.

AI Inference

Run existing Lilypad modules with a simple text or image prompt. The workbench will output an IPFS CID with the result.

Implement the Lilypad module allowlist and cache the models locally before they are needed! Pin the files to a local IPFS node packaged with the workbench platform.

Finetuning

Import a dataset and use the the "Data" tool to finetune a model for a specific end user. Create a Lilypad module and save the module on IPFS. More on this soon!

Create an AI agent and run on Lilypad

The ML workbench provides a simple interface for creating and training AI agents. The Lilypad team is currently testing a variety of models and fine tuning techniques to determine the optimal path forward for this feature. More on this soon!

Resources

• Source code

The Lilypad Llama3 Chatbot is a conversational agent designed to deliver engaging, real-time AI-powered interactions. Leveraging the Llama3 8B model through the Lilypad API, this chatbot provides context-aware and dynamic responses, making it useful for applications such as automated support, interactive Q&A and virtual assistance.

Features

• AI-Powered Conversations – Utilizes Llama3 8B via the Lilypad Network to generate intelligent responses.

• Lilypad API Connectivity – Simple integration with Lilypad, allowing flexible use of all available models on the Lilypad API.

This chatbot can be extended with:

• Memory & Context Awareness – Store conversation history for more personalized interactions.

• External APIs – Integrate with knowledge bases, search engines, or database lookups.

• Multi-Model AI – Swap Llama3 with other AI models as needed.

How It Works

1. User Input – The chatbot interface captures the user's message.

2. API Request – The message is sent to the Lilypad API, which runs the job on the Llama3 model.

3. Response Generation – Llama3 processes the input, considers context and generates a natural language response.

4. Response Display – The response is rendered in the chatbot interface.

Prerequisites

• Node.js – 18 LTS or newer

Installation

1. Get Anura API key.

2. Clone the repository:

   Copy

   git clone https://github.com/PBillingsby/lilypad-llama3-chatbot.git
   cd lilypad-llama3-chatbot

3. Install dependencies: npm install

4. Add the Lilypad API key to .env: LILYPAD_API_TOKEN=<ANURA_API_KEY>

5. Start the development server: npm run dev

Usage

Access the chatbot at: http://localhost:3000 and enter prompt to start a conversation.

Resources

• Source code

This page overviews the hardware requirements to operate a Lilypad Network node. It's important to note that these requirements continuously evolve as the network grows. If you have questions or suggestions, please join our Discord or open a pull request on the Lilypad documentation repo.

Minimum Hardware Requirements

• Processor: Quad-core x64 Processor or better

• RAM: 32GB (see additional details below)

• Internet: Internet: 250Mbps download, 100Mbps upload (minimum)

• GPU: NVIDIA GPU with a minimum of 8GB VRAM (see additional details below)

• Storage: SSD with at least 500gb of free space

GPU Requirements

Each model operating on Lilypad has specific VRAM (Video Random Access Memory) requirements directly related to the model's complexity and computational demands. For running a Lilypad Resource Provider (RP) with multiple GPUs, a guide using Proxmox can be found here.

• Base Requirement: The simplest model on Lilypad requires a GPU with at least 8GB of VRAM. This is the minimum required to participate in computational tasks on the Lilypad network.

Model-Specific VRAM Requirements

The capability of your GPU to manage multiple or more complex Lilypad jobs is enhanced by the amount of VRAM available:

• Standard Models (SDXL, Ollama): Require at least 8GB of VRAM.

• Advanced Models: Require 14GB+ of VRAM.

Hardware Compatibility

• GPUs with 8GB of VRAM are limited to running models like SDXL, which fit within this specification. Larger GPUs with higher VRAM are required for more demanding models like SDV, which needs at least 14GB of VRAM.

For example:

• A node with a GPU containing 8GB of VRAM can execute Lilypad SDXL module jobs, which require a minimum of 8GB of VRAM.

• Larger capacity GPUs are needed for heavier compute models like SDV, which require at least 14GB of VRAM.

RAM Requirements

Lilypad uses the Resource Provider's GPU to load models, initially requiring the temporary storage of the data in the system's RAM. In a production environment with RP Nodes, it is important to have enough RAM to support the model and the underlying system's operational processes.

Minimum RAM Requirements:

• Base Requirement: A minimum of 16GB of RAM is required, with at least 8GB dedicated to the model and another 8GB allocated for system operations and Lilypad processes.

Additional Considerations

• Wallets for each GPU: You need a separate account for each GPU you want to set up on the network. The wallet you use for your account must have both ETH (to run smart contracts on Ethereum) and Lilypad (LP) tokens in order to receive funds for jobs) on the network.

• Larger Models: Jobs involving more substantial models will demand additional RAM. It's important to note that adequate VRAM alone is insufficient; your system must also have enough RAM to load the model into the GPU successfully. Without sufficient system RAM, the model cannot be loaded into the GPU, regardless of available VRAM.

Apply to our closed Beta Resource Provider program!

Getting Started

Prerequisites:

• A basic understanding of the Lilypad API and Rivet

• Access to an Anura Dashboard account

• Basic knowledge of workflow configuration

Set up Open AI Endpoint in Rivet

1. Download and Install Rivet from https://rivet.ironcladapp.com/

2. Start Rivet App.

3. Navigate to the Settings page. App>Settings

4. Click on the “Open AI” tab.

5. In the Open AI endpoint input field, paste the URL: https://anura-testnet.lilypad.tech/api/v1/chat/completions

Obtain an Anura API Key

1. Log in to your Anura account at https://anura.lilypad.tech/

2. Click Get Started

3. Create a new API key by clicking the “Create API Key” button.

4. Copy the generated API key and paste it into the Open AI API key input field in Rivet.

At Lilypad, we’ve integrated n8n with the Lilypad API to automate dynamic workflows, combining human input, AI generation and multi-platform actions.

Using Lilypad’s OpenAI-compatible endpoints inside n8n lets you:

• 🧠 Source and enrich content: Pull from databases like Notion, Airtable, or Google Sheets and enhance it with AI.

• 🎨 Generate custom outputs: Create AI-written text, summaries, or even modify images dynamically.

• 🔄 Automate publishing: Push results to platforms like Twitter, Discord, Slack, or your own apps.

• 📈 Track and update: Monitor workflow status and feed results back into your systems.

Everything runs automatically inside n8n, with Lilypad API endpoints providing powerful, cost-free AI capabilities at the heart of it.

Imagine building workflows that review customer feedback with AI, generate personalized emails, summarize research papers, trigger alerts from on-chain events, or even create full content campaigns, all without writing a single script. With n8n and Lilypad, AI-driven automation is at your fingertips.

Coming soon:

The Lilypad VS Code Extension allows developers to interact with the Lilypad Anura API within Visual Studio Code. By highlighting code blocks, users can query the available LLMs on Lilypad for explanations, improvements, or suggestions, receiving responses in a formatted webview panel.

Features

• AI-Powered Code Assistance – Select any code in your editor, choose an AI model, and ask Lilypad questions about it

• Formatted Responses – View AI-generated insights in a structured webview panel

• Secure API Configuration – Store your Lilypad API token securely

• Context Menu Integration – Quickly access the extension’s features via the right-click menu

Installation

From the Repository

1. Get Anura API key.

2. Clone the repo: git clone [email protected]:PBillingsby/lilypad-vscode-extension.git

3. Install the extension in VS Code: code --install-extension lilypad-vscode-extension.vsix

4. Add Anura API key to .env: LILYPAD_API_TOKEN=<ANURA_API_KEY>

5. If you make changes to the extension and need to recompile it:

   1. Recompile the code: npm run compile

   2. Repackage the extension: vsce package

   3. Reinstall the updated .vsix file: code --install-extension lilypad-vscode-extension.vsix

Usage

1. Select a block of code in your editor.

2. Right-click and select "Ask Lilypad about this code", or:

   • Open the Command Palette (Ctrl+Shift+P) and select "Ask Lilypad about this code".

3. Choose an AI model to process the query.

4. Enter your question related to the selected code.

5. Wait for Lilypad AI to process your query.

6. View the AI’s response in the webview panel that opens.

Resources

• Source code

Lilypad provides distributed computational services underpinned by the . The network provides infrastructure for use cases like AI inference, ML training, DeSci and more. Lilypad strategically collaborates with decentralized infrastructure networks, such as Filecoin, to formulate a transparent, efficient, and accessible computational ecosystem.

Perform off-chain decentralized compute over data, with on-chain guarantees. Call this functionality directly from a smart contract, CLI, and an easy to use abstraction layer.

The network is actively laying groundwork for multi-chain integration and the deployment of an incentivized testnet.

Lilypad has evolved from earlier versions (v0, v1 & v2), where the network served as a proof of concept for verifiable, decentralized compute directly from smart contracts. These earlier iterations established the groundwork for what is now a robust, scalable platform with expanded features and multichain support.

Bacalhau has been integral to Lilypad since its early versions (v0 and v1), serving as the backbone for verifiable off-chain compute. In these early iterations, Bacalhau was used for Proof of Concept projects, helping users execute decentralized compute jobs from smart contracts.

Find Lilypad on or visit .

Objective and problem statement

Lilypad aims to mitigate the challenges predominantly associated with the accessibility of high-performance computational hardware. At present, numerous barriers impede developers and organizations from smoothly integrating projects that require high-performance computing, such as AI technologies, into their applications.

Unlike conventional centralized systems, where access to powerful compute hardware is restricted and costly, Lilypad endeavors to democratize this access. Through its verifiable, trustless, and decentralized computational network, Lilypad extends unrestricted, global access to computational power. By leveraging decentralized infrastructure networks such as Filecoin, Lilypad is strategically positioned to enhance the efficiency, transparency, and accessibility of high-performance computing hardware.

Applications

Perform off-chain decentralized compute over data, with on-chain guarantees, and to call this functionality directly from a smart contract, CLI, API and an easy to use abstraction layer, opens the door to a multitude of possible applications including:

• Inference AI jobs

• ML training jobs

• Invoking & supporting generic ZK computations

• Cross-chain interoperability complement to bridge protocols

• Federated Learning consensus (with Bacalhau insulated jobs)

• IOT & Sensor Data integrations

• Providing a platform for Digital twins

• Supply chain tracking & analysis

• ETL & data preparation jobs

Key features

Some of the key features of Lilypad include:

1. Verifiable Serverless Decentralized Compute Network: Lilypad is a decentralized compute network that aims to provide global, permissionless access to compute power. The Network orchestrates off-chain compute (a global GPU marketplace) and uses on-chain verification (Arbitrum L2 on Ethereum) to provide guarantees of compute success.

2. Mainstream Web3 Application Support: Lilypad is designed to enable mainstream web3 applications to use its compute network with the , and . It aims to make decentralized AI compute more accessible, efficient, and transparent for developers and users.

3. Open Compute Network: Lilypad is an open compute network allowing users to access and run AI models/other programs in a serverless manner. Module creators and general users can access a curated set of modules or can easily create their own Lilypad module to run an AI model/other program on the network.

4. Multichain Support: The Lilypad Incentivized Testnet on Arbitrum in June 2024 with plans to go multi-chain in the near future. Supporting multiple blockchain networks will increase the scalability and interoperability of the network, allowing users to choose the blockchain that best suits their needs.

5. Incentivized Test Net: The is live! The IncentiveNet program provide users withto participate in running nodes, testing, and improving the network. Learn more by checking out the .

6. Decentralization of Mediators: The team also aims to decentralize the mediators in the network. This means that the decision-making process and governance of the network will be distributed among multiple participants, ensuring a more decentralized and resilient system.

Before you run a Lilypad job, make sure you have and have set in your environment.

Run Cowsay

cowsay is a classic CLI tool that prints messages in a speech bubble from an ASCII cow. You can run it as a job on the Lilypad network.

Run the command:

Wait for the compute to take place and for the results to be published:

View your results:

This guide walks through the steps of setting up a personal RPC endpoint for Arbitrum Sepolia using .

Setup Infura account

on Infura and choose your plan based on how many APIs you need.

Select the “free” tier as the compute units provided should be sufficient to run a Lilypad RP.

Setup RPC endpoint for Arbitrum Sepolia

In the Infura dashboard, a new API key will usually generate automatically. If not, select "Create New API Key". Navigate to "Configure" to setup the API key.

Scroll down the list to the Arbitrum network and ensure the Sepolia testnet box is checked, then save changes.

In the API key dashboard, select "Active Endpoints" and navigate to "WebSockets".

Scroll down the page to find the Arbitrum Sepolia URL. The RPC endpoint for Arbitrum Sepolia is ready to be used with the Lilypad Resource Provider:

Use the new RPC endpoint

Lilypad RPs can use a personal RPC endpoint with a few simple steps. Only Web-socket (WSS) connections are supported.

Docker users

Stop the existing Lilypad Resource Provider (RP) before setting up the new RPC.

Locate the Lilypad RP Docker container using:

Stop the container using the PID:

Use this command to start the lilypad-resource-provider.service with the new RPC:

Check the status of the container:

Ubuntu users

Stop the existing Lilypad RP (if the node is not running, disregard this first step):

Update lilypad-resource-provider.service with the new RPC:

Add following line to [Service] section:

Reboot the node:

If the Lilypad RP was properly as a systemd service, the RP will reboot using the new RPC. Once the reboot is complete, the RP should be running with the updated configuration. To verify your node is back online and running correctly, run the following:

The Lilypad RAG Support Agent is a Retrieval-Augmented Generation (RAG) AI assistant that retrieves relevant information and generates AI-powered responses using the Lilypad Network. It enables automated support and troubleshooting by leveraging vector search and AI-based text generation.

Agent Details

• Context-Aware Responses – Uses all-MiniLM-L6-v2 embeddings to retrieve relevant information.

• AI-Powered Answer Generation – Sends retrieved context and user query to the Lilypad API, which processes it using Llama3 8B.

• Customizable Knowledge Base – Modify the agent’s context source (issues.md) to adapt it for different use cases.

How It Works

1. Embedding Queries with all-MiniLM

   • Converts user queries and stored knowledge into dense vector embeddings for similarity-based retrieval

2. Retrieving Relevant Context

   • Searches a pre-indexed database to find the most relevant information.

3. Generating Responses with Llama3 using the Lilypad API

   • Sends retrieved context and user prompt to the Lilypad API, where Llama3 8B generates a structured response

Expanding to Your Own Support Agent

The Lilypad RAG Support Agent can be adapted to different projects by modifying its retrieval source.

Updating the Knowledge Base (issues.md)

By default, the agent retrieves information from issues.md, a markdown file containing troubleshooting steps.

To customize:

• Open issues.md in the repository.

• Replace or expand the content with relevant support information for your project.

• Format the content clearly to improve retrieval accuracy.

• Restart the agent to index the updated knowledge base.

For more advanced use cases, the agent can be extended to support multiple files or external knowledge sources.

Getting Started

A guide to launch the RAG support agent locally and run inference on the Lilypad Network with the Lilypad Inference API.

Get a Lilypad API Key

Sign up at Anura API and generate an API key.

Clone the Repository

git clone https://github.com/PBillingsby/lilypad-rag-agent.git
cd lilypad-rag-agent

Configure Your API Key

Export your Lilypad API Token as an environment variable:

export LILYPAD_API_TOKEN="your-api-key-here"

To make it persistent, add it to ~/.bashrc or ~/.zshrc.

Setup

Ensure Python 3 is installed, then run:

pip install -r requirements.txt

Usage

After setting up the API key and dependencies, the agent is ready to process queries using Lilypad’s AI-powered retrieval system.

Run the Agent

Execute the agent from the project's root directory:

python3 cli.py

Resources

• Lilypad Docs

• Lilypad Anura API

• Source code

Agent Details

The AI Oncologist serves as a Research agent template that can be used to conduct a wide range of research. The Agent is an intelligent system for analyzing research papers using a multi-agent approach.

The system consists of three main agents:

Paper Relevance Agent

• Searches through PDF documents in the documents directory

• Uses embeddings and cosine similarity for initial filtering

• Verifies relevance using LLM-based analysis

• Returns a list of most relevant paper filenames

Top Paragraphs Agent

• Extracts text from identified papers

• Splits content into manageable chunks

• Scores paragraph relevance using LLM

• Returns top-scoring paragraphs with relevance scores

Text Query Agent

• Analyzes provided text passages

• Generates focused answers to specific queries

• Uses contextual understanding to provide accurate responses

Getting Started

A guide to launch the Research agent locally and run inference on the Lilypad Network with the Lilypad Inference API.

Prerequisites

• Python 3.8+

• OpenAI API key or compatible API (e.g., DeepSeek)

• PDF files

Installation

1. Clone the repository:

git clone https://github.com/mavericb/ai-oncologist.git
cd ai-oncologist

1. Install required packages:

pip install -r requirements.txt

1. Create a .env file in the project root with the following variables:

# OpenAI-Like API configuration
BASE_URL="https://api.deepseek.com"
OPENAI_API_KEY=your_deepseek_api_key
MODEL="deepseek-chat"

ANURA_BASE_URL=https://anura-testnet.lilypad.tech
ANURA_API_KEY=your_anura_api_key
ANURA_MODEL=phi4:14b

# Search configuration
MAX_RESULTS=3
SIMILARITY_THRESHOLD=0.3

• BASE_URL: API endpoint for OpenAI Compatible LLM service (default: "https://api.deepseek.com")

• OPENAI_API_KEY: Your OpenAI Compatible API key (for example: DeepSeek API Docs)

• MODEL: One OpenAI Compatible model to use (default: "deepseek-chat")

• ANURA_BASE_URL: API endpoint for the Anura LLM service (default: "https://anura-testnet.lilypad.tech")

• ANURA_API_KEY: Your Anura API key (get it here: Lilypad Inference API Docs)

• ANURA_MODEL: The Anura model to use (default: "phi4:14b")

• MAX_RESULTS: Maximum number of papers to return (default: 3)

• SIMILARITY_THRESHOLD: Minimum similarity score for document selection (default: 0.3)

Usage

1. Place your PDF research papers in the documents/ directory.

2. Run the main script:

python AIOncologist.py

Resources

• GitHub

Upcoming

View the upcoming events we are attending or hosting.

2025

Past

View the previous events we attended or hosted.

2025

2024

Both Resource Providers (GPU compute nodes) and those looking to run jobs on the Lilypad network need to set up a Metamask account in order to run jobs on Lilypad. The public key of your wallet address is how you are identified on the network, and is how you can look up the transactions you make on the . You need an account for running jobs, and a separate account for each GPU you want to set up on the network.

The wallet you use for your account must have both ETH (to run smart contracts on Ethereum) and Lilypad (LP) tokens in order to pay for jobs (or receive funds for jobs) on the network.

Create a MetaMask wallet

End users of Lilypad can decide which crypto wallet they would like to use. In this guide, we advise using a crypto wallet.

• Install

Connect to the Arbitrum Sepolia Testnet in MetaMask.

The Lilypad Testnet (IncentiveNet) is currently running on the Arbitrum L2 network built on Ethereum.

In order to change to the Arbitrum network in the wallet, open MetaMask and click the network button in the top left of the menu bar:

Then select "Add network":

Next, select "Add a network manually":

Input the required Arbitrum Sepolia Testnet network info, and then "Save":

• Network name: Arbitrum Sepolia

• New RPC URL:

• Chain ID: 421614

• Currency symbol: ETH

• Block explorer URL: (optional)

Import the Testnet LP token

The wallet is now setup and will display an ETH (Arbitrum Sepolia) token balance. In order to also display the LP token balance, the LP token will need to be imported.

Select "Import tokens" from the three dot menu next to the network name:

Select "Custom token" and add the Lilypad token contract address and token symbol. Then "Save".

• Token contract address: 0x0352485f8a3cB6d305875FaC0C40ef01e0C06535

• Token symbol: LP

You should now see both ETH and LP listed in the wallet (initial ETH and LP balances will be 0).

Now you're ready to fund the wallet with testnet LP and ETH tokens!

Funding Your Wallet

Get Testnet Lilypad Tokens (LP) and Arbitrum Sepolia Testnet ETH

tldr: To obtain funds, first ensure the wallet is connected to the Arbitrum Sepolia network. then, collect LP and ETH tokens from these faucets:

• (3rd party faucet list)

Faucet Guide

Get Testnet LP tokens

Find out why you need tokens in the

Follow these steps to successfully claim your Testnet LP tokens:

1. Navigate to the Lilypad Testnet .

2. Authenticate with Discord.

3. Copy your MetaMask wallet address into the input.

4. Click "Request".

Testnet LP tokens will be sent to the wallet address provided to the faucet.

Get Arbitrum Sepolia Testnet ETH

Get Arbitrum Sepolia ETH from this list of . Each faucet is designed differently, so follow the instructions provided.

View tokens

With a balance of both LP and ETH, you're ready to run jobs with the Lilypad CLI!

To update a Lilypad RP, remove any previous versions of Lilypad on the instance and then follow the instructions to setup a Docker RP.

• If the Lilypad install for Linux was previously used on the RP, first . Then follow the .

• If the RP previously used a Docker Lilypad version,, then follow the .

Ensure all processes from Linux install are stopped and removed

This only applies if you had a Linux version installed

1. Stop all systemd processess

2. Disable the systemd services:

3. Delete the service files from the systemd directory.

4. Reload the systemd daemon to apply the changes.

Remove old Docker containers and images

1. If a Docker RP is running stop the system (if the node is not running, disregard this first step)

2. You can check the status of the containers with:

If they are running, stop them with:

Remove the containers:

3. View all Docker images

4. Take note of the IMAGE ID for each lilypad image (resource-provider, bacalhau, and watchtower)

Delete old Docker images that are duplicates for Lilypad (Bacalhau, Lilypad)

Install and Run a new Version of Lilypad

1. Export WEB3_PRIVATE_KEY as an environment variable

2. Use curl to download the docker-compose.yml file from the Lilypad GitHub repository.

3. Start your Lilypad Node

OR start with your own RPC URL

Monitor the Resource Provider

Use the following command to check the status of the resource provider and bacalhau.

Use the following command to view the containers running after starting Docker Compose.

A healthy, updated node should have all containers started, a preflight check, and be adding a resource offer.

The Lilypad CLI wrapper can run locally to create an API endpoint for running jobs on the Lilypad network. This gives developers full control to build a decentralized system running jobs on Lilypad. Github repo can be found .

Build a or AI agent workflow that uses this API endpoint for running jobs! For inspiration, check out JS CLI wrapper + Gradio example. Spin up a Webui with Gradio and use the api with a frontend!

Note: This is a beta tool and would mostly be expected to run locally. When implementing this tool, note that the POST request includes the user's Web3 private key. Stay tuned for a hosted API from Lilypad that will supplement this local CLI Wrapper.

Getting Started

Prerequisites

Installing and Setting up Lilypad Binary

1. Build the Lilypad binary:

Usage

Run node src/index.js to create a local endpoint using the js wrapper with either src/run.js or src/stream.js, then send a post request containing json with your funded WEB3_PRIVATE_KEY key set, see the quick start for more on .

In inputs, each input must be preceded by the -i flag, including tunables. For example: "inputs": "-i Prompt='an astronaut floating against a white background' -i Steps=50"

Note: This tool is for demonstration purposes and can be used to run jobs on Lilypad for free. The tooling will be improved upon in the coming weeks for greater scalability. Use the following post request with the WEB3_PRIVATE_KEY below to run jobs on Lilypad. The wallet/private key below is funded with testnet tokens only and has been setup to simplify the use of this developer tool.

The endpoint can then be tested using curl

This guide walks through the steps of setting up a personal RPC endpoint for Arbitrum Sepolia using .

Setup Alchemy account

and login to the Alchemy dashboard.

Select the “free” tier as the compute units provided should be sufficient to run a Lilypad RP. The free service provides 300 million compute units per month.

Select “skip bonus” or input a credit card with billing info (the card will not be charged unless the compute credits in the free tier are used).

Setup RPC endpoint for Arbitrum Sepolia

In the “Overview” section of the Alchemy dashboard, navigate to “My app” and select “Endpoints”. If an app was not created upon login, create a new one by selecting "Create new app".

By selecting “Endpoints”, the “Networks” tab will open providing an option to configure the Arbitrum API.

• Select “Sepolia”

• Select “Websockets”

The RPC endpoint for Arbitrum Sepolia is ready to be used with the Lilypad Resource Provider:

Metrics for the RPC can be viewed in the “Metrics” tab.

Use the new RPC endpoint

Lilypad RPs can use a personal RPC endpoint with a few simple steps. Only Web-socket (WSS) connections are supported.

Docker users

Stop the existing Lilypad Resource Provider (RP) before setting up the new RPC.

Locate the Lilypad RP Docker container using:

Stop the container using the PID:

Use this command to start the lilypad-resource-provider.service with the new RPC:

Check the status of the container:

Ubuntu users

Stop the existing Lilypad RP (if the node is not running, disregard this first step):

Update lilypad-resource-provider.service with the new RPC:

Add following line to [Service] section:

Reboot the node:

If the Lilypad RP was properly as a systemd service, the RP will reboot using the new RPC. Once the reboot is complete, the RP should be running with the updated configuration. To verify your node is back online and running correctly, run the following:

Prior Verification-via-Replication Protocols

Before explaining our approach, we give a short overview of three prior approaches to verification-via-replication distributed computing protocols: Truebit, Modicum, and Smart Contract Counter-Collusion. We will outline potential improvements, and how our work builds on top of, and differs from, prior work.

Truebit

Truebit is a protocol for outsourcing computation from blockchains, built using smart contracts on top of Ethereum. The original potential use cases were trustless mining pools, trustless bridges, scaling transaction throughput, and, scalable “on-chain” storage. Since its goal is to scale on-chain computation, it aims for global consensus: "Since there exist no trusted parties on Ethereum’s network, by symmetry we must allow any party to be hired to solve any computational task, and similarly anyone should be able to challenge a Solver’s outcome of a computational task. The latter requirement ensures that TrueBit operates by unanimous consensus." (emphasis added)

The components of Truebit are Miners, Task Givers, Solvers, Verifiers, and Judges. In order to incentivize checking results, random errors are forced into computations, with jackpots awarded to those who find them. These jackpots are funded by taxes on computations.

The verification game consists of a series of rounds, where in each round, a smaller and smaller subset of the computation is checked. Eventually, only one instruction is used to determine whether a Solver or Verifier is correct: "In fact, only one instruction line is used in a verification game. There will be a part of the program code where there is a discrepancy between the Solver and the Verifier. The instruction of that discrepancy point is used to verify who is right."

The authors claim that Sybil attacks are mitigated by pairwise Sybil-resistance between the parties of Task Givers, Solvers, and Verifiers, with Judges and Referees, whose roles are played by Miners, assumed to function as intended. Likewise, they claim that attacks to get bogus solutions on-chain by scaring off Verifiers are mitigated by the economics of deposits, taxes, and jackpot rewards. Additionally, a cartel of Solvers who absorb losses until they find a task with a forced error, upon which time they will receive the jackpot, will lose money in the long-term, since the expected Solver deposit per task is higher than the expected jackpot per task. Addressing another attack, the authors claim that an attack involving a flood of Challengers who try to take as much of the jackpot reward resulting from a forced error as possible is mitigated by having the total jackpot reward decrease as the number of Challengers increases.

Challenges

• Does not scale to large/complicated/arbitrary computations

• No formal theorems or proofs, no simulations, many plausible but unsubstantiated claims, especially regarding collusion

• Everything is done on-chain

• This model does not work well with two-sided marketplaces, because

  • It aims for global consensus, where any node is allowed to do the computation, whereas in two-sided marketplaces, clients need to be able to choose which nodes they are paying to do the computation

  • Clients may have time restrictions on their computations, and cannot wait for cases where their computations were injected with forced errors

• No accounting for repeated games

Takeaways

• Taxes and jackpots are a valuable tool to create a global game that affects local outcomes

• Provides a list of potential client attacks

Modicum

The original version of Modicum had five key components: Job Creators (JC), Resource Providers (RP), Solvers (market makers), Mediators (agreed-upon third parties for mediation), and Directories (file systems, which we have replaced with IPFS and Docker registries). Job Creators are clients, the ones who have computations that need to be done and are willing to pay. Resource Providers are those with computational resources that they are willing to rent out for the right price. Solvers are market-makers; they match the offers from JCs and RPs. Mediators are third parties trusted by both JCs and RPs to arbitrate disagreements. The Directories are network storage services available to both JCs and RPs.

Job Creators are only allowed to submit deterministic jobs to the protocol. The Mediator exists to check for non-deterministic tasks submitted by the Job Creator (which can be used by the Job Creator as an attack vector to get free results), and fake results returned by the Resource Provider. The method for determining whether a job is deterministic or not is for the Mediator to run a job n times and check to see whether it receives different answers.

Modicum combines two separate ideas: checking the result from a Resource Provider to see if it is correct, and checking a result from a job submitted by a Job Creator to see if the job was deterministic or not. This implies that there is no capability for a client to simply check whether a result is correct or not, without the possibility of its collateral being slashed.

An alternative to trusting the Mediator (to act as a TTP) by having it run a job n times is having a consortium of n Mediators each run the task a single time. However, this adds the complication of achieving consensus in that consortium.

The issue of the Resource Provider and Mediator colluding to return a fake result is not addressed by this protocol. The authors allow for a Job Creator or Resource Provider to remove a Mediator from their list of trusted Mediators if they no longer trust it. However, that still leaves room to cheat at least once, and ideally this should be disincentivized from the outset.

There is also the issue of collateralization. The Modicum protocol, as well as a number of other protocols, assume (approximate) guesses as to the cost of jobs in advance, so that nodes can deposit the correct amount of collateral. However, doing so is fraught with complications; we provide an alternative approach in the Mechanisms to Explore section.

Challenges

• The Mediator is basically a trusted third party

• Client cannot simply check a result without being slashed, which is a consequence of the client attack model

• No accounting for repeated games

Takeaways

• Potential client attack, though one that can be mitigated by technical means

• The client has benefit of getting correct results, which needs to be accounted for in simulation

• Useful prototype for a two-sided marketplace (the follow-up paper for stream processing applications is also useful)

Smart Contract Counter-Collusion

The authors determine that cryptographic methods for verifiable computation are too expensive for real-world scenarios. For that reason, they rely on verification-via-replication. The scenario is one in which a client simultaneously outsources computation to two clouds, where those two clouds deposit collateral into smart contracts in such a way to create a game between them, where the game incentivizes doing the computation honestly. The central challenge that the authors tackle is the issue of collusion - that is, what if the two clouds collude on an incorrect answer?

In contrast to Modicum, the client is assumed to be honest, and in contrast to Truebit, a trusted third part (TTP) is used to handle disputes.

Three Contracts

The authors use a series of three contracts to counter collusion.

The first game is an induced Prisoner's Dilemma - to avoid the two clouds colluding, one cloud can be rewarded the other cloud's deposit (minus payment to the TTP for resolving the dispute) if the former returned the correct result and the latter did not. Thus, each cloud is better off giving the other cloud fake results while computing the correct result itself. This contract is called the Prisoner's contract. It is analogous to the equilibrium in the classic prisoner's dilemma being defection <> computing honest result and giving other node fake result if offered to collude.

However, the clouds can agree to collude via a smart contract as well. They could do this by both depositing another amount into a contract, where the leader of the collusion must add a bribe (less than its cost of computing) to the contract as well (disregarding the bribe, both clouds deposit the same amount of collateral). The deposit is such that the clouds have more of an incentive to follow the collusion strategy than to deviate from it. This contract is called the Colluder's contract.

In order to counteract the Colluder's contract, a Traitor's contract is used to avoid this scenario by incentivizing the clouds to report the Colluder's contract. The basic concept is that the traitor cloud indeed reports the agreed-upon collusion result to the client in order to avoid the punishment in the Colluder's contract, but also honestly computes and returns the result to the client in order to avoid the punishment of the Prisoner's contract. The client must also put down a deposit in the Traitor's contract. Only the first cloud to report the Colluder's contract gets rewarded. The signing and reporting of the contracts must happen in a particular order in order for this process to work.

The authors prove that these games individually and together lead to a sequential equilibrium (which is stronger than a Nash equilibrium), meaning that it is optimal not only in terms of the whole game, but at every information set (basically the set of options each player has at every turn).

Challenges

• A Colluder's contract can be signed on different chains (or even off-chain). In order to mitigate this, the Traitor's contracts would have to become cross-chain (which is a major technical challenge), not to mention the possibility of cryptographically secure contracts (e.g. MPC contracts) where one of the parties alone would not be able to prove the existence of this contract

• Relies on trusted third party to resolve disputes

• Every task is replicated (that is, two copies of each job are always computed)

• Assumes client is honest

• Assumes amount of collateral known beforehand

• No accounting for repeated games

  • It is well known that in the repeated Prisoner's dilemma, depending on the assumptions, cooperation becomes the equilibrium

Takeaways

• The contracts and the payoffs that they induce offer a valuable toolbox to think about the problem of collusion

• The contracts offer, in a restricted setting, an ironclad way (assuming the proofs are correct) of preventing collusion

Mechanisms to Explore

The following is a list of mechanisms that we are currently considering exploring in order to mitigate attacks. Note that some of these mechanisms clearly would not be able to deter cheating and collusion alone. However, in combination with other mechanisms, they may achieve the goals. In this sense, they should be thought of as modules, optionally composable with each other.

Mediation

Clients call upon a mediation protocol in order to verify the results of a node. There are several variations of the structure of the mediation protocol; the following parameters can be varied:

1. The number of nodes in the mediation protocol.

2. If more than two nodes in the mediation consortium, the consensus threshold that determines which result is the one considered to be correct.

3. How the nodes are chosen.

   • For example, we may want as a baseline the same constraint as in Modicum - that only mediators that both the client and the compute node mutually trust can be used for mediation.

     • Even with this baseline, there is still a question of how to choose the node(s) - it can be random, be determined by an auction, or any other method.

4. Recursive mediation - that is, if there is no consensus in the consortium, do another mediation.

   • There is a large space of possibilities regarding how to execute this.

   • There needs to be a limit to the number of nodes this recursive process can use. For example, the set of potential nodes can be the same as the set of mutually trusted mediators, as described above.

5. Other methods discussed here, such as taxes and jackpots, as well as staking and prediction markets, can be incorporated into the mediation protocol.

Collateralization

There are a number of different types of collateral that need to be included in the protocol.

The client needs to deposit collateral so that the compute node knows that it can be paid. For computations where the cost is known up front, this is simple. However, it becomes complicated for arbitrary compute; the client might not have put up enough collateral initially, so there may have to be a back-and-forth between client and compute node where the latter halts the computation until the former deposits more collateral or simply cancels the computation and pays for the partially computed result. If the client does not pay, then the compute node can invoke a mediation process.

The compute node needs to deposit several types of collateral.

1. Collateral in case they timeout.

   • This is put up as part of the deal agreement - that is, when the deal is posted on-chain.

2. Collateral in case they cheat.

   • The way that the compute node will convey the amount of collateral they will deposit to indicate that they will not cheat is via a collateral multiplier. The compute node commits to a multiple of whatever they will charge the client ahead of time as part of the deal agreement. The actual collateral is put up after the result is computed and sent to the client. This simplifies immensely the task of determining how much collateral to deposit for arbitrary computations.

3. Collateral in case they don't do the computation at the rate they said they would. This is closely related to timeout collateral.

   • Ideally, this is a way of enforcing deadlines on jobs.

   • It is not necessary to make this collateral slashing binary - for example, a late result can be still be rewarded.

   • It is enforceable if, for example, the compute node says that they will do X WASM instructions/time. However, technical limitations may make this unrealistic, and it needs to be tested.

One possible way to overcome collusion is to require super high collateral for some particular nodes against each other that that even discount factors very favorable to collusion would not incentivize collusion, even when accounting for repeated games.

While this is not a part of anti-cheating mechanisms, collateral pooling could lower capital requirements for collateralization. High capital requirements are a second-order concern, but will become a higher priority once robust anti-cheating mechanisms are implemented.

Taxes and Jackpots (inspiration from Truebit)

Taking inspiration from the taxes and jackpots scheme used in Truebit, deals can be taxed, with those taxes going to a jackpot that is then used to reward nodes via some distribution protocol determined by the mediation process. For this, we want to be able to take any fraction of the jackpot(s) and distribute it arbitrarily to arbitrary nodes (perhaps even those not involved in the mediation process).

This is a particularly interesting approach because the taxation + jackpots mechanism inherently create a global game that impacts local outcomes. While it may lead to potential collusion attacks, the tool alone is very useful, especially in conjunction with some other the other methods mentioned here. Modeling it in simulation would also provide the opportunity to test some of the hypotheses in the Truebit paper.

This method may also be useful in creating a robust platform where some clients do not care to check their results. That is, if some clients do not check results in general, it may be difficult to assert that the network is secure. Taxes and jackpots may be a way to address this.

Prediction/Replication Markets

Prediction markets have been well-studied in a variety of different fields. More recently, a type of prediction market called a replication market has been explored in the context of replicability in science. With this inspiration, it may be possible that allowing nodes to make bets regarding the replicability of the computations of nodes may be useful in mitigating cheating. For example, nodes with a low prediction for replicability may act as a signal for that node's reputation and encourage it to behave honestly.

It is possible to overlay this mechanism on top of taxes, allowing nodes to choose where their taxes go in the prediction market.

Additionally, since Automated Market Makers are closely related to prediction markets, we can leverage many DeFi tools in this context.

Staking behind nodes

Allow users to stake behind nodes. This is similar to prediction markets, but with slightly different economics. Like with prediction markets, it may be possible to tax users and then allow them to choose which nodes they stake behind. Overall, this approach is similar to delegated Proof-of-Stake.

Announcing successful cheating

Can a node announcing that it successfully cheated (and thereby receiving a reward) benefit the robustness of the protocol? How much would this node have to be rewarded?

Frequency of checks

How often should a client check results? Clearly it is related to the amount of collateral that the other node deposits, how much they value getting true/false results, reputation, and so on. This is a parameter that the client would need to learn to maximize its own utility.

Reputation

The ledger can maintain a record, for each compute node, of the number of jobs the compute node has completed, the number of times its results were checked, and the number of times those results were replicated successfully. All other nodes (client and compute) could locally run some arbitrary function over these numbers to determine how reputable they find that node.

Storing Inputs/Outputs

Results can only be replicated for as long as the inputs are stored somewhere. The client, compute node, or some other entity can pay for storing the inputs/outputs of jobs. The longer they are stored, the more time there is to check the results, which affects things like collateralization, the frequency of checks, etc.

This is related to, but not totally overlapping with, the amount of time that a node might have to wait before getting paid, which is the same time interval that a client has to check a result. However, checking the result after the node gets paid and receives back its collateral may still be possible, with other penalty schemes (or reward schemes, for example, coming from jackpots).

Anti-Collusion via Obfuscation

Colluding requires the following knowledge in order to enforce the parameters of the collusion.

1. The public keys of the nodes participating in collusion.

2. The results that were posted by those public keys.

3. The payouts to the public keys.

In order to sign a collusion contract to begin with, the public keys must be known. However, in a mediation protocol with enough nodes, it may be possible to obscure (2) and (3) by

1. Having nodes submit results to the mediation protocol in an obscured/anonymous way

2. Have nodes be paid out according to the results of the mediation protocol in an obscured/anonymous way

If these two criteria can be met, then a mediation protocol based on them might be capable of imitating the game-theoretic outcomes seen in the Smart Contract Counter-Collusion paper.

There have been many decades of cryptography and security research focusing on similar problems to these. It may be the case that it is already possible to do this; otherwise, there is a large amount of ongoing research on the topics of privacy-preserving transactions, and much prior work in the flavor of secret-sharing/MPC/Tor/Monero/ZKPs that could enable this.

Configure Your Module

After bootstrapping your module, additional configuration is required to run it.

.env

WEB3_PRIVATE_KEY

🚨 DO NOT SHARE THIS KEY 🚨

The private key for the wallet that will be used to run the job.

This is required to run the module on Lilypad Network.

A new development wallet is highly recommended to use for development. The wallet must have enough LP tokens and Arbitrum Sepolia ETH to fund the job.

config/constants.py

DOCKER_REPO

The Docker Hub repository storing the container image of the module code.

This is required to push the image to Docker Hub and run the module on Lilypad Network.

e.g. "<dockerhub_username>/<dockerhub_image>"

DOCKER_TAG

The specific tag of the DOCKER_REPO containing the module code.

Default: "latest"

MODULE_REPO

The URL for the GitHub repository storing the lilypad_module.json.tmpl file. The visibility of the repository must be public.

The lilypad_module.json.tmpl file points to a DOCKER_REPO and Lilypad runs the module from the image.

e.g. "github.com/<github_username>/<github_repo>"

TARGET_COMMIT

The git branch or commit hash that contains the lilypad_module.json.tmpl file you want to run.

Use git log to easily find commit hashes.

Default: "main"

Available Scripts

Your module will be bootstrapped with some handy scripts to help you download the model(s) for your module, build and push Docker images, and run your module locally or on Lilypad Network. Some additional configuration may be required.

In the project directory, you can run:

python -m scripts.download_models

A basic outline for downloading a model from is provided, but the structure of the script and the methods for downloading a model can differ between models and libraries. It’s important to tailor the process to the specific requirements of the model you're working with.

Most (but not all) models that utilize machine learning use the library, which provides APIs and tools to easily download and train pretrained models.

No matter which model you are using, be sure to thoroughly read the documentation to learn how to properly download and use the model locally.

python -m scripts.docker_build

Builds and optionally publishes a Docker image for the module to use.

For most use cases, this script should be sufficient and won't require any configuration or modification (aside from setting your DOCKER_REPO and DOCKER_TAG).

--push Flag

Running the script with --push passed in pushes the Docker image to Docker Hub.

--no-cache Flag

Running the script with --no-cache passed in builds the Docker image without using the cache. This flag is useful if you need a fresh build to debug caching issues, force system or dependency updates, pull the latest base image, or ensure clean builds in CI/CD pipelines.

python -m scripts.run_module

This script is provided for convenience to speed up development. It is equivalent to running the Lilypad module with the provided input and private key (unless running the module locally, then no private key is required). Depending on how your module works, you may need to change the default behavior of this script.

--local Flag

Running the script with --local passed in runs the Lilypad module Docker image locally instead of on Lilypad's Network.

--demonet Flag

Running the script with --demonet passed in runs the Lilypad module Docker image on Lilypad's Demonet.

lilypad_module.json.tmpl

The default lilypad_module.json.tmpl file is below. Make sure to update the Docker Image to point to your Docker Hub image with the correct tag.

The default lilypad_module.json.tmpl should work for low complexity modules. If your module requires additional resources (such as a GPU) make sure to configure the applicable fields.

Template Fields

• Machine: Specifies the system resources.

• GPUs: Specifies the minimum VRAM required.

• Job: Specifies the job details.

  • APIVersion: Specifies the API version for the job.

  • Metadata: Specifies the metadata for the job.

  • Spec: Contains the detailed job specifications.

    • Deal: Sets the concurrency to 1, ensuring only one job instance runs at a time.

    • Docker: Configures the Docker container for the job

      • WorkingDirectory: Defines the working directory of the Docker image.

      • Entrypoint: Defines the command(s) to be executed in the container as part of its initial startup runtime.

      • EnvironmentVariables: This can be utilised to set env vars for the containers runtime, in the example above we use Go templating to set the INPUT variable dynamically from the CLI.

      • Image: Specifies the image to be used (DOCKERHUB_USERNAME/IMAGE:TAG).

    • Engine: Sets the container runtime (Default: "Docker").

    • Network: Specifies that the container does not require networking (Default: "Type": "None").

    • Outputs: Specifies name and path of the directory that will store module outputs.

    • Resources: Specify additional resources.

    • Timeout: Sets the maximum duration for the job. (Default: 600 [10 minutes]).

Prerequisites

• Linux (Ubuntu 22.04 LTS)

• Nvidia GPU

• (Ubuntu install)

• (NVIDIA Container Toolkit)

Network information and Testnet tokens

The testnet has a base currency of ETH, as well as a utility token called LP. Both are used for running nodes. To add a node to the testnet, follow these steps:

Metamask Configuration

We recommend using MetaMask with custom settings to make things easier. Once you have it installed and setup, here are the settings you need to use:

For a step by step guide on adding the network, please refer to our .

Fund your wallet with ETH and LP

To obtain testnet LP, use the and enter your ETH address.

To obtain testnet ETH, use a third party and enter your ETH address.

The faucet will give you both ETH (to pay for gas) and LP (to stake and pay for jobs).

Setup Arbitrum RPC (Optional)

The Lilypad Network uses the Arbitrum Sepolia Testnet to settle compute transactions. When a transaction is ready to be saved on-chain, Lilypad cycles through a list of public Arbitrum Sepolia RPC endpoints using the endpoint that settles first to save the compute transaction.

Resource Providers have the option to using Alchemy instead of using the default public RPC endpoints.

A personal RPC endpoint helps RPs to avoid reliability issues with the public RPC endpoints used by Lilypad ensuring rewards can be earned and jobs can be run consistently. RPs running a personal RPC endpoint contribute to the fault tolerance and decentralization of the Lilypad Network! Read more in the Alchemy Arbitrum .

Docker Compose Setup

Before we start the Rp with the Docker setup, retrieve the private key from the wallet set up earlier in this guide. For guidance on exporting your private key, refer to . Once the private key has been retrieved, proceed to initialize the Docker containers using the commands provided below.

1. Export Your Private Key

Before starting, export your private key from MetaMask. Follow the official MetaMask guide for instructions on safely exporting your private key.

2. Download the Docker Compose Configuration

Use curl to download the docker-compose.yml file from the Lilypad GitHub repository.

3. Check for Existing Containers

You can check if these containers are running with:

If they are running, stop them with:

If there are still conflicts when trying to running with the docker-compose file, remove the containers:

4. Start the Resource Provider

Start the Lilypad containers using Docker Compose:

To include a custom RPC URL:

5. Monitor Your Node

Use the following command to check the status of the resource provider and bacalhau.

Use the following command to view the containers running after starting Docker Compose.

NOTE: Occasional websocket errors in your RP logs are normal and nothing to worry about. These are simply the result of standard internet fluctuations.

Our solver may also restart periodically, which can temporarily interrupt your connection.

Rest assured that these brief disconnections will not affect your rewards in any way.

Update Lilypad version for Docker RP

When a new version of Lilypad is released, it is critical for Resource Providers to update their installations to ensure compatibility and ability to run Lilypad jobs.

1. If a Docker RP is running stop the system (if the node is not running, disregard this first step)

2. View all Docker images

3. Delete old Docker images that are duplicates for Lilypad (Bacalhau, Lilypad)

4. Follow Docker RP install instructions

Using the Lilypad Docker setup a new RP and run.

View Lilybit_ rewards

To view your Lilybit_ rewards, visit one of the following dashboards and paste your node's public address into the input:

Troubleshooting

are some common troubleshooting techniques when it comes to your resource provider using Docker.

Lilypad is assembling a team of experts across a variety of domains to tackle and build product from the complex R & D required for a decentralised compute network, including verification, privacy, token economics and more.

Core Team

Advisors

Other Partners

A Lilypad module is a Git repository that allows you to perform various tasks using predefined templates and inputs. This guide will walk you through creating a Lilypad module, including defining a JSON template, handling inputs, and following best practices.

For a more in-depth look at building modules, refer to this .

Modules on Lilypad

Below are a few examples of modules you can run on Lilypad. From language models to image generators and fun utilities, the network supports a growing list of AI modules.

To view the full list of available modules on Lilypad, please check out !

Module Structure

Start by creating a Git repository for your Lilypad module. The module's versions will be represented as Git tags. Below is the basic structure of a Lilypad Module.

Prepare Your Model

• Download model files

• Handle all dependencies (requirements.txt)

• Implement input/output through environment variables

• Write outputs to /outputs directory

1. Download the model locally

To use a model offline, you first need to download it and store it in a local directory. This guarantees that your code can load the model without requiring an internet connection. Here's a simple process to achieve this:

1. Install required libraries

2. Use a script to download the model (eg: python download_model.py)

3. Verify that the model files are in your directory

2. Create Run Script (run_model.py for example) that will be used in conjunction with Docker

3. Create a Dockerfile that functions with your run script

4. Build and Publish Image

To make sure your Docker image is compatible with Lilypad, you need to define the architecture explicitly during the build process. This is particularly important if you are building the image on a system like macOS, which uses a different architecture (darwin/arm64) than Lilypad's infrastructure (linux/amd64).

The examples below are for building, tagging and pushing an image to DockerHub, but you can use any platform you prefer for hosting the image.

For Linux: docker buildx build -t <USERNAME>/<MODULE_NAME>:<MODULE_TAG> --push .

For MacOS:

5. Create a lilypad_module.json.tmpl Template

Environment Variables

Format in template:

Usage in CLI:

Formatting your module run command

During development, you will need to use the Git hash to test your module. This allows you to verify that your module functions correctly and produces the expected results.

Below is a working Lilypad module run cmd for reference. (you can use this to run a Lilypad job within the Lilypad CLI):

Test Module before running on Lilypad

Use the following command syntax to run your Module on Lilypad Testnet.

If the job run appears to be stuck after a few minutes (sometimes it takes time for the Module to download to the RP node), cancel the job and try again. Open a ticket in with any issues that persist.

Run Module on Lilypad

Examples

Here are some example Lilypad modules for reference:

• : Lilypad "Hello World" example

• : Text to text

• : Text to image generation

Deprecated examples:

• : An example module for LoRa training tasks.

• : An example module for LoRa inference tasks.

• : An example module related to DuckDB.

These examples can help you understand how to structure your Lilypad modules and follow best practices.

Conclusion

In this guide, we've covered the essential steps to create a Lilypad module, including defining a JSON template, handling inputs, and testing your module. By following these best practices, you can build reliable and reusable modules for Lilypad.

For more information and additional examples, refer to the official Lilypad documentation and the Cowsay example module.

Prerequisites

• Linux (Ubuntu 22.04 LTS)

• Nvidia GPU

• Nvidia drivers

• Docker (Ubuntu install)

• Nvidia Docker drivers

Network information and testnet tokens

The testnet has a base currency of ETH, as well as a utility token called LP. Both are used for running nodes. To add a node to the testnet, follow these steps:

Metamask

We recommend using MetaMask with custom settings to make things easier. Once you have it installed and setup, here are the settings you need to use:

For a step by step guide on adding the network and importing the LP testnet token, please refer to our Setting up MetaMask documentation.

Fund your wallet with ETH and LP

To obtain testnet LP, use the Lilypad faucet and enter your ETH address.

To obtain testnet ETH, use a third party Arbitrum Sepolia testnet faucet and enter your ETH address.

The faucet will give you both ETH (to pay for gas) and LP (to stake and pay for jobs).

Installation

To set up your environment for using Lilypad with GPU support, you need to install several key components. This guide will walk you through installing Docker, the Nvidia Container Toolkit, Bacalhau, and Lilypad. You'll also configure systemd to manage these services efficiently.

Install Docker

Docker is a platform that allows you to automate the deployment of applications inside lightweight, portable containers.

To install Docker Engine, follow the steps specific to your operating system from the official Docker documentation:

• Linux - Docker Engine

Install Nvidia Container Toolkit

To ensure proper operation of your graphics cards and Lilypad, follow these steps to install the Nvidia Toolkit Base Installer: Nvidia Container Toolkit download page

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    
sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit

Configure the container runtime by using the nvidia-ctk command:

 sudo nvidia-ctk runtime configure --runtime=docker --set-as-default

The nvidia-ctk command modifies the /etc/docker/daemon.json file on the host. The file is updated so that Docker can use the NVIDIA Container Runtime.

Restart the Docker daemon:

sudo systemctl restart docker

Install Bacalhau

Bacalhau is a peer-to-peer network of nodes that enables decentralized communication between computers. The network consists of two types of nodes, which can communicate with each other.

To install Bacalhau, run the following in a new terminal window (run each command one by one):

cd /tmp

wget https://github.com/bacalhau-project/bacalhau/releases/download/v1.6.0/bacalhau_v1.6.0_linux_amd64.tar.gz

tar xfv bacalhau_v1.6.0_linux_amd64.tar.gz

sudo mv bacalhau /usr/bin/bacalhau

sudo chown -R $USER /app/data

To check your Bacalhau version use:

bacalhau version

The expected output is:

CLIENT  SERVER  LATEST
v1.6.0  v1.6.0  <latest Bacalhau version>

If the Bacalhau CLIENT version is not v1.6.0, it will need to be replaced. Follow the steps here to uninstall and reinstall Bacalhau.

Install Lilypad

The installation process for the Lilypad CLI involves several automated steps to configure it for your specific system. Initially, the setup script identifies your computer's architecture and operating system to ensure compatibility. It will then download the latest production build of the Lilypad CLI directly from the official GitHub repository using curl and wget.

Once the CLI tool is downloaded, the script sets the necessary permissions to make the executable file runnable. It then moves the executable to a standard location in your system's path to allow it to be run from any terminal window.

Via official released binaries

# Detect your machine's architecture and set it as $OSARCH
OSARCH=$(uname -m | awk '{if ($0 ~ /arm64|aarch64/) print "arm64"; else if ($0 ~ /x86_64|amd64/) print "amd64"; else print "unsupported_arch"}') && export OSARCH
# Detect your operating system and set it as $OSNAME
OSNAME=$(uname -s | awk '{if ($1 == "Darwin") print "darwin"; else if ($1 == "Linux") print "linux"; else print "unsupported_os"}') && export OSNAME;
# Remove existing lilypad installation if it exists
sudo rm -f /usr/local/bin/lilypad
# Download the latest production build
curl https://api.github.com/repos/lilypad-tech/lilypad/releases/latest | grep "browser_download_url.*lilypad-$OSNAME-$OSARCH-gpu" | cut -d : -f 2,3 | tr -d \" | wget -i - -O lilypad
# Make Lilypad executable and install it
chmod +x lilypad
sudo mv lilypad /usr/local/bin/lilypad

To verify the installation, run lilypad in the terminal to display the version and a list of available commands, indicating that Lilypad CLI is ready to use.

Write env file

You will need to create an environment directory for your node and add an environment file that contains your node's private key.

To do this, run the following in your terminal:

sudo mkdir -p /app/lilypad
sudo touch /app/lilypad/resource-provider-gpu.env

Next, add your node's private key into /app/lilypad/resource-provider-gpu.env:

WEB3_PRIVATE_KEY=<YOUR_PRIVATE_KEY> (the private key from a NEW MetaMask wallet FOR THE COMPUTE NODE)

This is the key where you will get paid in LP tokens for jobs run on the network.

Setup Arbitrum RPC

The Lilypad Network uses the Arbitrum Sepolia Testnet to settle compute transactions. When a transaction is ready to be saved on-chain, Lilypad cycles through a list of public Arbitrum Sepolia RPC endpoints using the endpoint that settles first to save the compute transaction.

Resource Providers have the option to setup their own Arbitrum RPC endpoint using Alchemy instead of using the default public RPC endpoints.

A personal RPC endpoint helps RPs to avoid reliability issues with the public RPC endpoints used by Lilypad ensuring rewards can be earned and jobs can be run consistently. RPs running a personal RPC endpoint contribute to the fault tolerance and decentralization of the Lilypad Network! Read more in the Alchemy Arbitrum docs.

Install systemd unit for Bacalhau

systemd is a system and service manager for Linux operating systems. systemd operates as a central point of control for various aspects of system management, offering features like parallelization of service startup, dependency-based service management, process supervision, and more.

To install systemd, open /etc/systemd/system/bacalhau.service in your preferred editor:

sudo vim /etc/systemd/system/bacalhau.service

[Unit]
Description=Lilypad V2 Bacalhau
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Environment="LOG_TYPE=json"
Environment="LOG_LEVEL=debug"
Environment="HOME=/app/lilypad"
Environment="BACALHAU_SERVE_IPFS_PATH=/app/data/ipfs"
Restart=always
RestartSec=5s
ExecStart=/usr/bin/bacalhau serve --orchestrator --compute

[Install]
WantedBy=multi-user.target  

Install systemd unit for GPU provider

Open /etc/systemd/system/lilypad-resource-provider.service in your preferred editor.

[Unit]
Description=Lilypad V2 Resource Provider GPU
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Environment="LOG_TYPE=json"
Environment="LOG_LEVEL=debug"
Environment="HOME=/app/lilypad"
Environment="OFFER_GPU=1"
EnvironmentFile=/app/lilypad/resource-provider-gpu.env
Restart=always
RestartSec=5s
ExecStart=/usr/local/bin/lilypad resource-provider 

[Install]
WantedBy=multi-user.target

Reload systemd's units/daemons (you will need to do this again if you ever change the systemd unit files that we wrote, above)

sudo systemctl daemon-reload

Start Lilypad node

Start systemd units:

sudo systemctl enable bacalhau
sudo systemctl enable lilypad-resource-provider
sudo systemctl start bacalhau
sudo systemctl start lilypad-resource-provider

Now that your services have been installed and enabled, check the status of Bacalhau to ensure it is running correctly on your node:

sudo systemctl status bacalhau

View node status

To check if the node is running use the following command:

sudo systemctl status lilypad-resource-provider

This will give a live output from the Lilypad node. The logs will show the node running and accepting jobs on the network.

Run the following command to get more status info from your node:

sudo journalctl -u lilypad-resource-provider.service -f

To restart your resource provider run:

sudo systemctl restart lilypad-resource-provider

Support for Lilypad RPs

For complex issues, bug reports, or feature requests, open a discussion in the Lilypad-Tech Github organization discussion board.

• Navigate to the discussion board, select "New Discussion", choose "rp-issues", and fill out the template.

• Without a discussion opened, our team will not be able to support the problem.

For quick questions or minor issues, use the Lilypad Discord #i-need-help channel and provide the following info.

• Description (including Lilypad version running on your node)

• Hardware Info (including Linux/Windows version)

• Related blockchain/ETH addresses of transaction hashes

• Output Logs - sudo systemctl status lilypad-resource-provider

• Related links/urls

• Screenshots

Update Lilypad version

When a new version of Lilypad is released, it is important for resource providers to update their installations to ensure compatibility and access to the latest features and improvements.

Please note that using sudo rm -rf is very powerful and can be dangerous if not used carefully.

1. If the Lilypad RP is running, stop the system (if the node is not running, disregard this first step):

sudo systemctl stop bacalhau
sudo systemctl stop lilypad-resource-provider

1. Remove the Lilypad executable by running:

sudo rm -rf /usr/local/bin/lilypad

1. Reinstall Lilypad with the latest version

2. Start your resource provider by running:

sudo systemctl start bacalhau
sudo systemctl start lilypad-resource-provider

Disconnecting a node

To disconnect your node from Lilypad you will need to do a few things to completely offboard.

First, stop the node:

sudo systemctl stop bacalhau
sudo systemctl stop lilypad-resource-provider

Next, you must remove the .service files related to Lilypad and Bacalhau. These files are typically stored in /etc/systemd/system/. To remove them, run the following command:

sudo rm -rf /etc/systemd/system/lilypad-resource-provider.service /etc/systemd/system/bacalhau.service

Next we notify the systemd manager to reload its configuration by running:

sudo systemctl daemon-reload

Then, remove the environment file for the Lilypad resource provider. This file is usually stored in /app/lilypad/. To remove it, run:

sudo rm -rf /app/lilypad/resource-provider-gpu.env

Finally, if you followed the installation instructions from the Lilypad documentation and moved the executable to /usr/local/bin/lilypad, it can be removed from there. If the executable is stored in a different directory on your machine, navigate to that directory and remove it from there. To remove the executable, run:

sudo rm -rf /usr/local/bin/lilypad

To remove Bacalhau, run:

sudo rm -rf /usr/bin/bacalhau

As every system is different, these instructions may vary. If you have any issues, please reach out to the team in the Lilypad Discord for help!

View Lilybit_ rewards

To view your Lilybit_ rewards, visit one of the following dashboards and paste your node's public address into the input:

• Grafana dashboard

• Lilypad leaderboard

Security

If you want to allowlist only certain modules (e.g. Stable Diffusion modules), to control exactly what code runs on specific nodes (which can be audited to ensure that they are secure and will have no negative impact on the nodes), set an environment variable OFFER_MODULES in the GPU provider to a comma separated list of module names, e.g. sdxl:v0.9-lilypad1,stable-diffusion:v0.0.1.

Visit the Lilypad GitHub for a full list of available modules.

Run a node video guide

Getting Started

This guide will walk you through creating a basic sentiment analysis module using and (which will be referred to as Distilbert from now on). We will be referring back to the Hugging Face page throughout this guide, so it's best to keep it open and accessible.

Input:

Output:

If you prefer to follow along with a video guide, you can view our live workshop below! 👇

Prerequisites

To build and run a module on Lilypad Network, you'll need to have the , and on your machine, as well as and accounts.

For this guide, we'll be using which requires and uses .

Downloading The Model

The first thing you'll need for your module is a local model to use.

A basic outline for downloading a model from is provided in scripts/download_models.py. The structure of the script and the methods for downloading a model can differ between models and libraries. It’s important to tailor the process to the specific requirements of the model you're working with.

You can get started by attempting to run the download_models.py script.

Since the script hasn't been properly configured yet, it will return an error and point you to the file.

Open scripts/download_models.py and you will see some TODO comments with instructions. Let's go through them each in order. You can remove each TODO comment after completing the task.

First we have a reminder to update our requirements.txt file, which is used by the Dockerfile to install the module's dependencies. In the next line is a commented out import statement.

To find the dependencies that our model requires, we can refer back to Distilbert's Hugging Face page and click on the "Use this model" dropdown, where you will see the library as an option. Click it.

You should see a handy modal explaining how to use the model with the Transformers library. For most models, you'd want to use this. However, Distilbert has a specific tokenizer and model class. Close the modal and scroll to the section of the model card. We're going to use this instead.

For now, let's look at the top 2 lines of the provided code block:

Notice that torch is also being used. Copy the transformers import statement and paste it over the existing import statement in our download_models.py file.

Now open requirements.txt:

These are 2 of the most common libraries when working with models. Similar to the import statement in the download_models.py file, they are provided by default for convenience, but commented out because although they are common, not every model will use them.

Since this model happens to use both of these libraries, we can uncomment both lines and close the file after saving.

Return to the download_models.py file, and look for the next TODO comment.

If we take a look at the , we can use the copy button next to the name of the module to get the MODULE_IDENTIFIER. Paste that in as the value.

For our use case, it should look like this:

You're almost ready to download the model. All you need to do now is replace the following 2 lines after the TODO comment:

Instead of using AutoTokenizer and AutoModelForSequenceClassification, replace those with the DistilBertTokenizer and DistilBertForSequenceClassification we imported.

The script is now configured! Try running the command again.

The models directory should now appear in your project. 🎉

Building Your Model

Now for the fun part, it's time to start using the model!

This time we'll get started by running the run_module script.

You should see an error with some instructions.

1. Implement Job Module

Let's tackle the run_inference.py script first. This is where your modules primary logic and functionality should live. There is a TODO comment near the top of the file.

We've already updated the requirements.txt file, so we can skip that step. Go ahead and uncomment the import statements and replace the transformers line with the DistilBertTokenizer and DistilBertForSequenceClassification.

We should refer back to the "How to Get Started With the Model" section of Distilbert's model card to figure out how to use the model.

Let's implement this into our run_inference script. Scroll down to the main() function and you'll see another TODO comment.

Same as before, uncomment and replace AutoTokenizer with DistilBertTokenizer and AutoModelForSeq2SeqLM with DistilBertForSequenceClassification. This is now functionally identical to the first 2 lines of code from Distilbert's example.

Below that, the tokenizer and model are passed into the run_job() function. Let's scroll back up and take a look at the function. This is where we'll want to implement the rest of the code from Distilbert's example. The inputs are already functionally identical, so let's adjust the output.

From the Distilbert model card, copy all of the code below the inputs variable declaration, and paste it over the output variable declaration in your modules code.

All we need to do from here is set the output to the last line we pasted.

That's everything we'll need for the modules source code!

2. Delete Code Block

We still need to finish step 2 that the error in the console gave us earlier. Open the run_module.py script.

Find the TODO comment and delete the code block underneath.