[LLM world record] Testing the waters

Part I: warming up to our attempt to the largest distributed LLM inference with consumer-grade devices

Oct 31, 2024

Previously, on Kalavai…

LLM world record attempt
Join our discord community to be part of it.

Edit (2/11/2024): we renamed clusters to pools to better reflect what kalavai does. All references have been changed, including CLI commands.

After the release of our public cluster feature, we are ready to get things going! Before we are ready to attempt to deploy +800B parameters models, we need to battle test Kalavai.

We’ve created a public pool on our platform that users can join. We are going to keep it up for 1 week (from the 31st of October 2024 to the 7th of November 2024). We ask you to join it and keep your device online as much as possible.

How to join the pool

Follow these steps to join the cluster with a compatible device.

Install the kalavai CLI on your local device
Create a free account on our platform
Get the joining key from the platform
Join the pool with the joining key
[Optional] Subscribe to this substack

1. Install the `kalavai CLI`

Linux

On your terminal, run the following:

curl -sfL https://raw.githubusercontent.com/kalavai-net/kalavai-client/main/assets/install_client.sh | bash -

Windows

On a PowerShell with admin permissions (run as admin):

wsl --install -d Ubuntu-24.04

Make sure to enable systemd by editing (or creating if required) a file /etc/wsl.conf to contain the following:

[boot]

systemd=true

Restart the WSL instance by exiting and logging back in:

exit

wsl --shutdown

wsl -d Ubuntu-24.04

Inside WSL, install Kalavai:

curl -sfL https://raw.githubusercontent.com/kalavai-net/kalavai-client/main/assets/install_client.sh | bash -

2. Create a free Kalavai account

To be able to join public clusters you need an account in Kalavai. Go to our website and register one.

3. Get the joining key

On our website, go to Public Pools and find the pool named llm-record-test and click Join.

You will get the joining key that you can use in the kalavai CLI to join.

4. Join the pool

Run the following command in the device you installed kalavai CLI, substituting <token> with the one you got in the previous step:

kalavai pool join <token>

If all has gone well, you should see a success message:

You are connected to llm-record-test

That’s it! Now sit back and relax, we’ll do the heavy lifting.

[Optional] Subscribe to this substack

Now what?

First off, huge thanks for your support. We mean it.

There will be a lot of back and forth whilst we test things! Join our discord channel to be part of the conversation!

FAQs

What do I do if I have problems when running the above?

If you encounter any issues when running the above, please tell us on our discord channel. Or if you feel brave, you can report an issue on our open source repository.

What will you do with my device?

As we get ready to do the largest distributed LLM deployment, we need to battle test Kalavai, and this means we’ll be deploying smaller models and run some diagnostics on the network. We will share all the deployment details in our discord channel, and users are encouraged to peek, at any time, on what’s being run on the cluster by running:

// Get all running jobs (with id)
kalavai job list
// Get more details on a particular job
kalavai job manifest <job_id>
// Get logs on the job
kalavai job logs <job_id>

Do I have to have my PC running 24/7?

Whilst we ask you to share for as much as you can, we understand you need your computer! The more you share, the better tests we can do. But you can pause at any time with:

// Stop sharing 

kalavai pool pause

// Start sharing again

kalavai pool resume

What do I get in return?

Our eternal gratitude: your name will be listed in our world record publication.

A place in history: how cool is it to be part of a history’s first??

Community kudos: bragging rights within our community, and karma points in Kalavai, which will let you create your own public clusters and tap into community devices!

How long will this pool be up?

For this test, we expect the pool to be up and running 1 week between the 31st of October 2024 and the 7th of November 2024. Any updates will be made in our discord channel.