AI is ruining everything, right? The economy, air quality, and computing as a whole have all felt the impact of the exponential growth in data centers for machine learning. However, there’s one little hardware AI project that proves bigger isn’t always better, and even shows a possible way out of the current memory crisis.
It’s called CrankGPT by Squeez Labs (via Hackaday) and the gist of it is simple: Have a little microcomputer run a tiny local model for AI voice assistance, then stick it in a box and power the whole thing with a hand crank. No massive power station, no endless racks of GPUs, no DRAM-destroying demands.
The computer in question is powered by a standard 8 GB Raspberry Pi and pretty much nothing else. It handles the voice recognition node, the local LLM (large language model), and the text-to-speech stuff. CrankGPT’s creators built their own edge voice agent to process the complete algorithm (i.e. voice input > LLM stage > text-to-voice output).
There’s a brief demo of CrankGPT in action at the bottom of the webpage for the project, and it seems to work pretty well. Of course, there are strict limitations as to what it can do, as the Raspberry Pi 5 isn’t exactly designed to be an inference powerhouse. It also takes roughly 30 seconds of cranking for the system to boot and be ready for any input, too.
What interests me most about CrankGPT is the fact that, as a proof of concept, it shows that edge AI has a clear future ahead of it. Being entirely offline and with a local LLM, it’s unmatched for privacy, but I reckon there’s something more significant here. The hardware required for this is extremely light: just a little processor, 8 GB of LPDDR4X, and a small SD card to host the OS and required data.
Video credit: Squeez Labs
AI training is always going to be done via hulking data centers, but ChatGPT shows that you don’t need the same for small-scale inference. If a hand-powered box can do it, then so can a basic laptop, phone, or even a watch. This hardware already exists on a vast scale across the world; all that’s needed are the right AI models and agents to make it all work as intended.
Should inference truly head off in that direction, it could significantly lessen the rampant demand for the kind of DRAM and NAND flash used in massive AI machines, and thus help bring an end to the current memory crisis.
With hundreds of billions of dollars invested in AI training and inference, though, there’s not much impetus for the industry to scale things right back and target the hardware that we already have. But wholesale change rarely happens overnight; all that’s needed is for someone to show the way forward, and that’s what CrankGPT has done.