Petals creates a free distributed network to run text-generating AI

BigScience, a collaborative project backed by startup Hugging Face that aims to make text-generating AI widely available, is developing a system called Petals that can run AI like ChatGPT by collecting resources from people on the internet. With Petals, whose code was released last month, volunteers can donate their hardware power to handle some of the text generation workload and collaborate to complete larger tasks, similar to the [email protected] and other forms of distributed computing.

“Petals is an ongoing collaboration of researchers from Hugging Face, Yandex Research, and the University of Washington,” Alexander Borzunov, Petals lead developer and research engineer at Yandex, said in an email interview with TechCrunch. “Unlike APIs, which tend to be less flexible, Petals is completely open source, allowing researchers to integrate next-generation text and system customization methods not yet available in APIs, or access internal system states to modify their properties to to investigate.”

Open source but not free

For all its shortcomings, a text-generating AI like ChatGPT can be quite useful, at least when viral demonstrations on social media are a reference. ChatGPT and its relatives promise to automate some of the mundane work that typically slows programmers, writers, and even data scientists, by generating human-like code, text, and formulas at scale.

But they are expensive to maintain. after a i estimatedChatGPT costs the OpenAI developer $100,000 per day, which equates to $3 million per month.

The costs associated with running advanced text-generating AI have relegated them to startups and AI labs with significant financial backing. It’s no coincidence that companies offering some of the most powerful text generation system technologies, including AI21 Labs, Cohere, and the aforementioned OpenAI, have raised hundreds of millions of dollars in VC capital.

But Petals, in theory, democratizes things. Inspired by Borzunov’s previous work focusing on training AI systems over the internet, Petals aims to dramatically reduce the operating costs of text-generating AI.

“Petals is a first step toward enabling continuous and truly collaborative improvement of machine learning models,” Colin Raffel, faculty researcher at Hugging Face, told TechCrunch via email. “This … marks an ongoing shift from large models primarily confined to supercomputers to something more universally accessible.”

Raffel pointed to the kind of gold rush that happened last year in the open source text generation community. Thanks to the efforts of volunteers and the generosity of the tech giants’ research labs, the kind of advanced text-generating AI that was once beyond the reach of small developers was suddenly available, trained and ready to use.

BigScience introduced Bloom, a language model similar in many ways to OpenAI’s GPT-3 (ChatGPT’s predecessor), while Meta opened up a relatively powerful AI system called OPT. Meanwhile, Microsoft and Nvidia have teamed up to deliver one of the best voice systems ever developed, MT-NLG.

But all of these systems require powerful hardware to run. For example, running Bloom on a local machine requires a GPU purchase of hundreds to thousands of dollars. Enter the Petals network, which Borzunov says will be powerful enough to run and optimize AI systems for chatbots and other “interactive” applications once it reaches sufficient capacity. To use Petals, users install an open source library and visit a website with instructions on how to connect to the Petals network. Once connected, they can generate Bloom text that runs on Petals or create a Petals server for computers to contribute to the network.

The more servers, the more robust the network. If a server goes down, Petals automatically tries to find a replacement. While servers go offline after about 1.5 seconds of inactivity to conserve resources, Borzunov says Petals is smart enough to resume sessions quickly, resulting in only a slight delay for end users.

In my tests, generating text with Petals took from a few seconds for simple instructions (e.g., “Translate the word ‘cat’ into Spanish”) to over 20 seconds for more complex requests (e.g., “Write an essay in the style of Diderot on the nature of the universe”). One prompt (“explain the meaning of life”) lasted about three minutes, but to be fair, I instructed the system to respond with a more verbal response (about 75 words) than the previous prompts.

This is considerably slower than ChatGPT, but also free. While ChatGPT costs nothing today, there is no guarantee that it will in the future.

Borzunov has not revealed how large the Petals network currently is, other than that “several” users with “different powerful GPUs” have joined since its launch in early December. The goal is to eventually introduce a rewards system to encourage people to donate their computers; Donors receive “Bloom Points” that they can spend on “higher priority or security guarantees” or potentially redeem them for other rewards, Borzunov said.

Limitations of Distributed Computing

Petals promises to provide a cheap, if not completely free, alternative to paid text generation services from providers like OpenAI. But the main technical problems have not yet been solved.

Most troubling are the security issues. The GitHub page for the Petals project notes that because of the way Petals works, it is possible for servers to extract input text, including text marked as private, and maliciously include and modify it. This could mean sharing sensitive data, such as names and phone numbers, with other users on the network, or modifying the generated code to intentionally break it.

Petals also does not address any of the shortcomings inherent in today’s leading text generation systems, such as: B. the tendency to generate toxic and biased text (see the Limitations section on Flower Post in the Hugging Face repository) . In an email interview, Max Ryabinin, senior researcher at Yandex Research, made it clear that Petals is for research and academic use, at least for now.

“Petals transmit intermediate data … over the public network, so we ask that they are not used for sensitive data, because other colleagues can (theoretically) retrieve it from intermediate representations,” Ryabinin said. “We suggest that people who want to use Petals for sensitive data create their own private swarm, hosted by organizations and people they trust and authorized to process that data. For example, several startups and small labs can collaborate and set up a private swarm to protect their data from others, while reaping the benefits of using Petals at the same time.”

As with any distributed system, Petals can be exploited by end users, whether attackers attempting to generate toxic text (e.g., hate speech) or developers using resource-intensive applications. Raffel admits that Petals will inevitably “face some problems” initially. But he believes the mission – to lower the threshold for running text-generating systems – will be worth the initial bumps along the way.

“Given the recent success of many community-led machine learning efforts, we feel it is important to continue developing these tools, and we hope that Petals will inspire other decentralized deep learning projects,” said Raffel.