daemon
Thought a bit more lately about how to organize the low-level software responsible for recruiting and orchestrating diverse worker devices, such as old smartphones, tablets, laptops, desktops, and servers. This software infrastructure layer that sits below the actual user-facing apps has a few components. First, there’s some kind of daemon shipped to the worker devices to get them to participate in the cluster and listen for jobs. Second, there’s the firmware flashed onto the microcontrollers (MCU) responsible for emulating various USB devices. Third, there’s some process on the microprocessor (MPU) that’s responsible for talking to the previous two and managing them.
Let’s zoom in on the first item, the payload shipped to the worker devices. Its purpose is to tell the MPU about what sorts of resources are available locally, and be able to take on various kinds of processing job on the worker device it’s hosted on. This roughly corresponds to a Kubelet in the world of Kubernetes, but there’s no way we’ll touch that bloated unwieldy ecosystem to link up devices revived from a junk drawer. I was thinking maybe we could write a minimal worker daemon in Rust, such that we can then compile it into a lean binary for any relevant worker architecture.
To allow the worker daemon to natively tap into as much hardware as possible, we can wrap the shared logic into a minimal platform-specific wrapper. On Android smartphones and tablets, this could mean an “app” with no UI, which can nevertheless grant our daemon access to the GPU, wireless, among others. On POSIX-capable systems like laptops, desktops, and servers, this might mean a systemd service wrapping the binary. The worker daemons might, in turn, launch battle-tested tools as subprocesses, such as llama.cpp and ripgrep, as a way to harness local hardware resources using existing libraries. The worker daemon could preserve the underlying tools’ standard APIs, such as the OpenAI-compatible endpoints for LLM inference, or it could abstract these into uniform jobs and resources.
crustacean
I went through the rustlings project again over a couple days, as my knowledge of the Rust language was, well, rusty. It’s a fantastic learning resource, consisting of just under 100 tiny exercises. In each exercise, you get a file with certain compiler or runtime errors, and you need to fix those issues before moving on to the next puzzle. There are a few chunkier quizes after larger groups of chapters, and each exercise also has a hint that tends to link to the relevant documentation.
Going through it again was so much fun, probably the most fun I had “at work” in a while, thanks to the self-contained nature of the puzzles, the progress bar nudging you to complete all challenges, and the smooth increase in difficulty. At the end, you get a visit from a crab in your terminal, bringing you more obligatory puns.
llms
The experience above helped me realize how much I missed actual programming rather than manipulating systems by chatting with an intermediary LLM. I had a similar experience a few months ago, when I was shocked when I realized that I really enjoyed creating an actual slide deck, but mostly because of the direct interaction with the object of work. Extensive mediation makes you feel numb, while direct and reactive interfaces support states of flow.
Beyond the personal satisfaction angle, I also grew more skeptical lately about the long-term maintainability of projects developed with heavy LLM involvement. Actually sitting with the problem, even if tedious or mechanical at times, helps you incrementally build valuable context. And if you aim to be involved with maintaining a project, you might as well understand it deeply. What I can’t build, I don’t understand. And why would I take the time to use a system you haven’t taken the time to develop.
Anyway, I wanted to pull together those few strings related to using LLMs into a coherent policy for this project, partly inspired by Rust’s own. It’s okay to use LLMs to study new topics and debug systems you’ve built, but no synthetic content should end up in the final product. It’s okay to use LLMs to prototype and experiment with different ways of implementing functionality, but the end result should be completely hand-written, someone should be responsible for it. Also, even if education and experimentation are allowed uses of LLMs, the used models must be open weights. This is meant to prefigure the practice of actually relying on local compute.
firmware
Here’s an example of a way in which I plan to use open weights LLMs the moment the development boards I ordered arrive. The firmware we’ll need to flash on the MCUs connected to the worker devices via USB will need to be very good at controlling the physical silicon blocks (PHYs) dedicated to USB communication. There are a few promising libraries and stacks that might be able to grant us control over the USB PHYs found in the incredibly cost-effective MCUs manufactured by WCH.
First, there are the examples shipped by WCH itself, but they appear to be little more than flaky demos. Second, there’s CherryUSB and lwIP, libraries grown out of Asian hardware communities with strong support for WCH chips, but they might be quite low-level. Third, there’s the option to use those two libraries as part of the real-time OS called RT-thread, but it might add bloat and overhead. Fourth, there’s the option to use the Rust library embassy, which would consolidate the codebase, but has only experimental support for WCH PHYs.
I’d like to have LLMs have a go at implementing dummy firmware behaviors using these various stacks, such as getting the MCU to act as a network switch across its USB ports, or emulating a dummy storage device when connected. And I’d study whether it actually worked, the number of lines of code each approach required, and the general shape of the resulting implementation. I wouldn’t use those rapid prototypes as part of the final firmware that ships with the product. Instead, I’d only throw LLMs at the problem to help triage the available options going forward based on concrete findings. And probably with a Flash or Air model variant out of the Chinese labs. Can’t wait for the development boards to arrive.
e-waste
I also had to drop by an e-waste processor this week, and I’m including it here as it’s reasonably related. We picked up a batch of servers to use it in our pilot course that will take place in three schools this coming academic year. Each of them is a veritable mystery box, you don’t know whether you’ll get one packed with a dozen RAM sticks or whether even the fans will be missing. It’s also satisfying to learn where they came from, in our case it was mostly from a bank and a utility company.
It’s interesting how some of them come preloaded with those hardware testing procedure that go beyond a basic POST. You get checks and result reports per component, down to individual fans. When recruiting worker devices in little cloud, we should probably test capabilities when trying to detect them in a similar way.
