Kalavai welcomes BitTorrent-style LLMs

Petals comes to Kalavai

Dec 23, 2024

We are on a quest to transform access to AI infrastructure. And part of our strategy is to recognise those who came before us that pursued the same goal. We believe in the community and want to lift up and empower any project that is aligned with our mission.

One of them is Petals, a BitTorrent-style LLM library to Generate text with distributed Large Language Models and fine‑tune them for your own tasks. They have been doing a great job at enabling LLM deployment across consumer-grade GPUs, and currently they support Llama 3.1, Mixtral, Falcon and BLOOM architectures.

Today we are thrilled to announce the release of a public Kalavai pool dedicated to host Petals workers.

Join our discord community for updates and to help shape the future of collaborative computing.

What you get

Eternal kudos from the community

Usage of all of the models, directly from Petals or our Kalavai endpoint

Credits on Kalavai, which can be used in all of our future computing pools.

How to join

1. Install Kalavai

Create a free Kalavai account. Then, follow the instructions here to install the client app in the machine you want to use as worker. These are the requirements for workers:

1+ NVIDIA GPU
2+ CPUs
4GB+ RAM
Free space 4x available VRAM (for an 8GB VRAM GPU, you'll need ~32GB free space in your disk)

2. Authenticate kalavai

Run the following command, using your account credentials:

$ kalavai login

[10:33:16] Kalavai account details. If you don't have an account, create one at https://platform.kalavai.net                                                                 
User email: <your email>
Password: <your password>

[10:33:25] <email> logged in successfully

3. Join the pool

Clicking Join will reveal the information on how to make your worker join the pool. Run the command displayed in your local machine;

$ kalavai pool join <token>

[16:28:14] Token format is correct
           Joining private network

[16:28:24] Scanning for valid IPs...
           Using 100.10.0.8 address for worker
           Connecting to PETALS @ 100.10.0.9 (this may take a few minutes)...
[16:29:41] Worskpace created
           You are connected to PETALS

Go nuts!

Kalavai's pool connects directly to the public swarm on Petals, which means we can use their health check UI to see how much we are contributing and what models are ready to use.

Models with at least one copy of each shard (a green dot in each column) are ready to be used. If not, wait for more workers to join in.

Check out how to get started doing inference with the models. The inference endpoint will always be displayed on the Community pools page.

What’s next

We are starting with single model support Mixtral 8x22B, but as more users join and our capacity increases we will include most of the models supported by Petals

Have enough hardware on your own and are concern about privacy and security? Coming soon, support for private swarms.