adb
Smartphones and tablets together are the largest category of e-waste by computing resources, and most of them run Android. If we are to aggregate junk electronics into usable clusters, we need to be able to deeply control these pieces of hardware. Thankfully, the Android Debug Bridge (ADB) is a protocol that allows us to do virtually anything with an Android device attached via USB. You just need to enable USB debugging, then you can access a bunch of options and resources.
I played around with reading specs and configs off of my Pixel 8, and you can really get down to the manufacturer of the RAM die or the current mobile carrier. You can also toggle settings, transfer files, run all kinds of binaries in a shell, even emulate taps, typing, and physical button presses.
ADB really is a Swiss army knife for controlling Android devices. This is great, because smartphones and tablets are otherwise quite fragmented in terms of hardware, making it difficult to find the drivers to build a custom ROM into which you could boot. On Android devices, we should work with the vendor binaries shipped with the original OS if we want to have a chance at tapping into mobile GPUs effectively, for instance.
target
The next thing I tried was building the default “hello world” Rust project for an Android target. This was surprisingly painless. You just need to have the Android NDK available, tell Rust to prepare for compiling for an Android target, and install Cargo NDK as glue between the two. After this initial setup, we got an output binary.
We can “push” the file to the connected device via ADB, then drop in a shell where we can execute it. And there it is, we’re greeted by code that’s running natively on the Android device. It’s not doing anything useful, yes, but it validates that we can simply target Android with Rust code. Actually seeing this pan out is good news, because it’s the Android environment that felt more exotic compared to the standard x86 / amd64 computers.
agentless
This binary we transferred to and ran on the smartphone could have been our worker daemon! It only had to expose some kind of API around underlying tools like llama.cpp or ripgrep that could be consumed by the orchestrator. But then I thought, wait, what if the orchestrator can just run things directly on worker devices through shells? Do we really need to mantain a hand-rolled daemon to abstract away from underlying resources and capabilities?
I seriously considered these, intrigued by the prospect of drastically simplifying our software infrastructure. We’d still need the orchestrator to manage workers, and we’d still need to write firmware to get microcontrollers (MCU) to emulate USB devices. But we could otherwise be looking at an opportunity to remove a whole part of the software burden. Apparently Ansible is also “agentless” as a DevOps tool, in that it only runs ephemeral code on the machines in a cluster, rather than standing a long-lived daemon the way Kubernetes does. So it’d not be completely unheard of.
However, removing the worker daemon as a whole and having the orchestrator talk to worker shells directly would naturally also have some downsides. The main one is the messiness of the interface exposed to the orchestrator, command-line calls and output parsing instead of some normalized /jobs/ paths in an API. The messiness goes beyond diverse invocation arguments, all the way to how to keep track of service status across the cluster. There might also be some overhead on processing, but opting for “scp” / “adb push” instead of transferring files through standard shell output would make it competitive for bulk transfers.
fpga
After coming up with the idea to remove worker daemons in favor of an agentless orchestrator, I was feeling on a roll when it comes to simplifying the design. Accordingly, I thought I’d also push further on the idea that we need the MCU firmware.
My previous best guess for how to achieve a flexible USB array was to connect USB-enabled MCUs to the ports, and then network the MCUs via the orchestrating microprocessor (MPU). WCH has various dirt-cheap USB-enabled MCUs, but simultaneously optimizing for cost and high-speed capabilities felt like overfitting to very specific chip offerings. For instance, the WCH CH32V407 has two USB high-speed buses and Ethernet for <2$ at 1K volume, so maybe we could use one every two ports only, and network them via on-board Ethernet as the interconnect.
There’s no chip I could find that’s available today, integrates more than a couple dual-role USB buses, and is Linux-capable such that we could tap into that rich repository of USB-related drivers. So, what if... we instead instantiated the silicon blocks necessary to have Linux drive lots of USB buses inside an FPGA? This stands for field programmable gate array, and refers to a type of chip that can effectively be dynamically rewired to implement different kinds of physical functionality. Libraries such as LiteX allow you to effectively come up with the specs you’d want on a system-on-a-chip and configure an FPGA to become that chip. Maybe you pick a certain processor with this many cores, some PCIe or SATA controllers, or whatever other specs you want, as long as it fits on the FPGA.
balance
What I’m sayings here is that maybe we could effectively fold many of our key components into one FPGA, including the MPU, MCUs, any hubs / switches, and so on. We could instantiate USB buses directly as parts of our ideal system-on-a-chip that we’re otherwise having trouble finding for our very specific application. This would remove a significant part of our system’s complexity, but it would also add complexity in other places, so let’s look at the balance sheet.
First, effectively folding our MCUs into the FPGA would eliminate the need to write custom firmware for them as a separate software component. Ideally, the Linux instance running on the FPGA would simply see many available hardware buses and operate them natively. We might still need a fixed-function chip per port if we want high-speed, though. Second, consolidating the MPU, MCUs, as well as any miscellaneous hubs and switches into one single part would vastly simplify the layout and routing of the actual circuit. Third, it would drastically simplify our bill of materials, as our board mostly reduces to a chunky FPGA fanning out to a bunch of USB ports. Sure, we’d still have some power assembly, a separate flash chip, and a few other one-off niceties, but we’d cut down on most hyper-specific parts.
Switching to FPGA also adds complexity elsewhere, primarily in the actual configuration of what we want from it, or the “gateware.” We need to instantiate a processor and various USB peripherals. That said, FPGAs turns hardware specs into software, and software is trivial to share. There’s a growing collection of open source blocks to throw into your design, as exemplified by LiteX. To be honest, I’d also not be mad about getting the opportunity to play around more with FPGAs, especially because the resulting configuration code can be converted into files that can essentially be sent to fabs to get custom chips taped out. I’m pretty sure we won’t get anywhere close to that level in this project, but it’d still be interesting to get familiar with this corner of the industry. Looking at the bottom line of our balance sheet, I’m overall pretty intrigued by the FPGA approach to little cloud’s design here, and I feel like these first weeks are well characterized by the following quote. It takes time to simplify.
“I have only made this letter longer because I have not had the time to make it shorter.” — Blaise Pascal