LLM Inference

A Fast Chat LLM Inference Module for Lilypad

Overview

This LLM Inference Module is a community-contributed module developed at AugmentHack.xyz The repo for this module can be found here.

See the original AugmentHack entry below:

CarpAIDevpost

[CLI] Usage

Usage:

lilypad run fastchat:v0.0.1 "paramsStr"

Inputs:

Where "paramsStr" is a question in CID form for the LLM. For example https://ipfs.io/ipfs/QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt is a bare file CID which contains

{
    "template": "You are a friendly chatbot assistant that responds conversationally to users' questions. \n Keep the answers short, unless specifically asked by the user to elaborate on something. \n \n Question: {question} \n \n Answer:",
    "parameters": {"question": "What is a chatbot?"}
}

To use it you would run:

lilypad run fastchat:v0.0.1 QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt

Outputs:

The output will be an IPFS CID, for example running the above input would result in the following link:

https://ipfs.io/ipfs/QmVNXCAfJgER6U7Z5XT8QaAVFPdwmtSFE6c9sUaAx7ttZs

Under link/output/result.json you will see

{"question": "What is a chatbot?", "text": "<pad> A  chatbot  is  a  computer  program  that  can  interact  with  users  in  a  conversational  manner.  It  is  designed  to  answer  questions  and  provide  information  in  a  way  that  is  natural  and  conversational.\n"}

Pssst... here's a question on Claude Monet you could try too ;) bafybeihu62yl76fcypidaiz35gq3yjguxawy5zzwadzvlcgpnfkuy2do3i

bafybeihu62yl76fcypidaiz35gq3yjguxawy5zzwadzvlcgpnfkuy2do3i

LLM Module Code

https://github.com/bacalhau-project/lilypad-modicum/blob/main/src/python/modules/fastchat.py

import json

def _fastchat(params: str):
    cmd = ["python3", "main.py"]
    return {
        "APIVersion": "V1beta1",
        "Metadata": {
            "CreatedAt": "0001-01-01T00:00:00Z",
            "Requester": {}
        },
        "Spec": {
            "Deal": {
                "Concurrency": 1
            },
            "Docker": {
                "Entrypoint": cmd,
                "Image": "xqua/carpai-demo-repo:v0.6",
            },
            "Engine": "Docker",
            "Language": {
                "JobContext": {}
            },
            "Network": {
                "Type": "None"
            },
            "PublisherSpec": {
                "Type": "Estuary"
            },
            "Resources": {
                "GPU": "1"
            },
            "Timeout": 1800,
            "Verifier": "Noop",
            "Wasm": {
                "EntryModule": {}
            },
            "inputs": [
                {
                    "CID": params,
                    "Name": "prompt_template_input",
                    "StorageSource": "IPFS",
                    "path": "/prompt_template.json",

                },
            ],
            "outputs": [
                {
                    "Name": "output",
                    "StorageSource": "IPFS",
                    "path": "/output"
                }
            ]
        }
    }

if __name__ == "__main__":
    print(_fastchat('QmcPjQwVcJiFge3yNjVL2NoZsTQ3GBpXAZe21S2Ncg16Gt'))