Last week, I covered the power schematic. This week, we’ve also got a data schematic, split in fewer but larger sections. First up, we need to configure the System-on-Chip. In general, I opted for doing chip configuration without any additional board flash memory that must be written in advance. Instead, we go with strap resistors as much as possible across all chips. However, we need some place to store the actual operating system that the F133-A System-on-Chip in particular boots from. It’ll be the same storage from which any binaries are shipped to worker devices. We have a basic SD card in a slot wired to the SoC, which is cheaper in small volumes than actually assembling flash on little cloud’s board. Beyond that we mostly only have a crystal for the clocking the SoC.

devices

For connecting to workers that typically act as USB devices, such as smartphones and tablets, we connect the SoC as the upstream on a CH338X hub, which connects downstream to ports. We’ve added some CH412K chips which bring ESD protection tailored for USB differential pairs. We use strap resistors to ask the hub to behave in certain ways, such as to match the polarity of the pins it uses to command the per-port power switches on the power schematic, or for reporting that it’s self-powered (rather than powered from upstream), which allows it to meaningfully charge downstream workers. It gets a clock, too. Plus, I was thinking of having one LED per port driven by the SoC instead of relying on the various LED outputs on hubs and switches that vary in semantics. You can see the LEDs in the top-left, on a subunit of the SoC symbol. I split the SoC into 4 subunits after all, to make the data schematic sane.

hosts

This was undoubtedly the final boss of the individual circuits across both schematic pages. It took a day on its own, but was arguably also the most satisfying to look at as it was taking shape. The multiple layers of repeated patterns feel almost artistic.

The idea of communicating with hosts since a few weeks ago was to rely on a constellation of fixed-function chips. Ports going to workers that act as USB hosts, such as laptops, desktops, and servers, feed into a per-port CH334P ($0.2 @ 1K) as the respective upstreams. As a downstream on each hub, we have a per-port RTL8152B USB-to-Ethernet NIC that feeds into a unified RTL8367S Ethernet switch. This grants these workers a baseline of network connectivity. But in addition, the SoC’s only device-capable USB bus feeds into a CH448U analog switch that can connect as downstream to one of the CH334P hubs, enabling further specialized device profiles during recruitment, such as a flash drive, keyboard, video output, etc.

This was the approach after a minor course-correction triggered by a scare I had. Previously, instead of a CH334P per port, I was thinking of a CH442E 2:1 analog switch that could select either fixed network via a NIC or flexible USB device profiles via the SoC’s bus. But in that case, I would’ve had two switches in series, one from the port and one on the SoC’s bus, which might have degraded speeds from USB high-speed (480 Mbps) to full-speed (12 Mbps), yikes. So I thought maybe we could replace these with the cheapest possible hub ICs, smaller and dumber than the CH338X. This costs a whopping 0.1$ more per port! In return, it solves the signal integrity concern, allows simultaneous port connections to its NIC and the SoC, and saves the SoC two GPIO pins per port which we used to toggle the previous switch.

Also, by default, the SoC’s device-capable bus is wired to a dedicated debug port, through which it will be programmed and tested during manufacturing. We’ll see if we can manage without any UART or JTAG shenanigans.

impedance

After the “final draft” for the power and data schematics, I started working on the actual PCB layout. The schematic focuses on how parts are meant to be connected functionally, whereas in the layout you answer questions about where to actually place parts, and you worry about the physics of their connections. For the physical buildup, I entered the specs of the standard JLCPCB buildup, the right thicknesses of layers, etc. So far, so good.

What I wrestled with more was the controlled impedance part. Various data protocols, such as our USB high-speed, 100 Mbps Ethernet, or Gigabit RGMII, require the traces through which they’re connected to be tuned in width and gaps to hit certain target impedances, such as 90 Ohm differential for USB high-speed. I was overthinking for a bit whether it’s worth doing anything else other than one rough 100 / 50, differential / single-ended profile throughout. Even Phil from the advanced digital hardware design lumps everything on his board on a single 100 / 50 profile, and usually I feel like he’s overzealous with his worries (do hardware designers really need to be worried about the fiberglass weave!?). That said, I was paranoid about the 100 Ohm impedance already being slim on tolerance for USB’s target, and that’s before taking into account the manufacturer’s own ±10% wobble. So I went for peace of mind and settled on just doing separate 100 / 90 / 50 impedance profiles.

In his KiCAD course, Phil defines trace thicknesses and then thinks about when to use which during routing. I found what seems like a more maintainable approach using “net classes” and assignments of net labels to such classes. Basically, I defined a net class for each of the three impedance profiles, and added wildcard rules to assign any net whose label contains “Z90” to the 90 Ohm differential impedance profile. The required trace widths and gaps I got from JUCPCB’s calculator. Note that impedance is impacted by distances to nearby plane(s), and because I settled on the internal L3 layer for my main high-speed routing, impedance profiles are only valid there. Though I also configured the default to be 50 Ohm single-ended on the outer L1 layer, because why would we stray too far from target impedances even below critical trace lengths?

parts

The two schimatic pages contain a few hundred symbols between them. If we want the circuit board manufactured, we need to assign each of these actual product numbers from a real distributor, such that the manufacturer will know where to place what concrete part. With LCSC and JLCPCB, the distributor and manufacturer is the same conglomerate, but we still need LCSC codes for each part, down to the last resistor. The actual parts of choice obviously dictate the footprint, and so it makes sense to do this bookkeeping before layout and routing. Because why would we intentionally have to redo it later when we find out we had different footprints to work with. It’s true that we knew the important parts for a few weeks now, which microprocessor and switch and such. But now’s the time to concretize the bulk caps and inductors and such.

Now’s also the time to do a few cost shaving passes. Because why would we redo the layout after a later swapped part. So I did a few passes of trading excessive part specs and ratings for a few dollars of at 1K volume, while still staying within safe ranges. Just made the margins a bit tighter while effectively keeping functionality constant.

It feels like many weeks have been leading up to this. We’ve got what feels like half of the actual schematic done, the half focused on power as opposed to data. These power connections will allow little cloud to power its own systems, as well as supply power to downstream USB-chargeable worker devices such as smartphones and tablets. It was incredibly satisfying to actually lay this out neatly, section by section.

The power’s journey starts on the upstream USB charging port. It connects to any USB-C charger annotated PD25W or more, which is roughly the wattage it will draw to cover both internal systems and downstream chargees. First up, power reaches the CH224A power delivery controller, whose purpose here is to ask the charger for a default contract of +9V. It’s the first standard USB-PD voltage above +5V, and we want to stay above +5V to be able to derive every rail with uniform step-down converters. The CH224A will later be controllable by the F133-A System-on-Chip once that wakes up. When the PD controller manages to negotiate its +9V default, it signals via its Power Good pin to the first buck converter that it can wake up.

sequence

The first two rails are ordered in accordance with the SoC’s required sequencing. The third and last is for miscellaneous other chips. The whole sequence is +3V3, +0V95, +5V. The SoC can derive its own +1V8, which it needs for the 64MB of DRAM packaged right inside. Each buck’s Power Good signal feeds into the next one’s Enable pin. The signal between the first and second gets a tiny delay to conform to the SoC’s sequencing needs. The final PG triggers the main power LED and lifts Reset on the SoC. The SY8388A bucks themselves have been picked based on their 8A output, comfortably above the ~4A drawn from the +5V one by the downstream workers. Plus they have the critical PG pins to enable chaining. And dirt-cheap at around $0.3@1K, lovely.

decoupling

This power schematic page got a chunk of the SoC and the on-board Ethernet switch, even if they’re data heavy. I was torn between placing them on the power or data page, because both aspects are involved for them. But apparently you can split a schematic symbol into multiple units that represent a single physical package in the actual PCB layout. So, I split off everything power related and put it on this power schematic. Each of the two chips needs an army of decoupling capacitors on their different power input pins. They’re generally consolidated here to avoid too many lines on the BoM during assembly, but I’ll probably do further consolidation passes later on. No split ground. The 5 Realtek NICs for connecting to laptops or desktops got the tame treatment.

downstream

The remaining non-trivial part of the board from a power point of view is the CH338X hub through which some downstream worker devices are charged. Again, I split the hub into a power and a data symbol, to keep things clean. Power doesn’t directly flow through the hub, but the hub has control over per-port fixed-function CH217A power switches. We need per-port control, to for instance refuse charging a particular device whose battery is not healthy enough to run. Plus this allows us to control battery level cycling per worker while keeping off 0% and 100% battery levels. The switches also block workers from drawing more than the 5V@0.5A USB Type-A standard.

review

Because it was my first time working with many of these patterns, such as buck chains and PD control, I repeatedly asked LLMs for design reviews, in the hope of screening for meaningful issues. Even if I set myself a usage standard that boils down to “AI is fine for learning & prototyping, but open weights only,” I found it hard to trust models not on the frontier for feedback on topics I’m not already familiar with. Accordingly, I found myself reaching out for whatever public model was best at the time, such as Opus 5.5, to come up with issues for each subcircuit. Not sure how to navigate this, maybe open weights only for prototyping, but not learning? It is appealing to reach out to the most advanced on-demand tutor sometimes.

I digged deeper into approaches to logistics and fulfillment following last week’s entry. First, I realized that PCBWay runs sponsorships for non-profit and open source projects, so I took some time to apply. It’d be awesome if I could get the prototyping costs partly covered until I reach a version worth applying to Crowdsupply with. From that point on, I could potentially rely on the same manufacturing pipeline to fulfill runs through Crowdsupply’s Mouser parent. Though TME and Olimex in particular also seem like relevant and inexpensive distributors, especially in an EU context.

Also interesting is that companies like PCBWay and Elecrow can integrate manufacturing together with kitting, packaging, and direct to consumer shipping. But if you sell through their storefronts, it’s still your product and you retain the EU compliance hurdles. Maybe PCBWay or Elecrow for direct shipping outside the EU, combined with TME or Olimex to handle EU hurdles, even if they of course add their own further margin. We’ll see how the PCBWay prototyping experience goes, I hope it’d work out to scale through them without other dependencies.

order

So I did a final pass on the enclosure model before before sending it off to PCBWay. Switched to 10x10x10mm threaded cubes from 10x10x6mm pancakes. Moved the PCB to the top side of its dedicated pocket, getting us silkscreen markings around the ports. That top layer will mostly be GND with some information LEDs at most, nothing high-speed. No need to extend the PCB all the way to the other side, as well. If some power bricks are placed in the storage drawer, they can be behind the PCB, which feels better for thermals, too. Plus minor hole size adjustments and cut-out experiments for the actual Type A and C ports.

I started adding in the plates as different parts. The total price wasn’t looking great at one piece volume. However, they shouldn’t be treated as different parts, there are no bends and so it’s just a big flat piece, even if non-contiguous. Moved everything to a shared 2mm Aluminum thickness to allow same-sheet manufacturing, and arranged the different plates into a 2D drawing in Onshape. First time I did this, feels really powerful for communicating assembly instructions and what’s inside, like those Ikea manuals. So I sent the DWG output of the drawing instead. The automated quote didn’t work right away on my file, maybe I haven’t followed some conventions initially.

Someone responded and flagged that they’re separate parts and so should be sent separately. I was thinking, fair, they’re not a contiguous piece, so I edited it to get an actual single piece with thin tabs between plates, and I’d snap them apart at home. But nope, this didn’t go through either, and the price per part was 3-4x compared to JLCCNC. So even if I managed to get an initial PCBWay sponsorship in credits, the JLCCNC version was still cheaper after applying the credits. They seem to add a significant premium for the fact that everything’s under one roof, and they need to cover the aggressive sponsorships somehow. So if I want the rock-bottom pricing of JLCCNC and JLCPCB, I have yet to find a vendor to solve kitting, packaging, labeling, etc.

stackup

In the section above, I mentioned that the PCB moved to the top of its pocket, being partly exposed. This implicitly impacts stackup, because that exposed layer can’t be a full signal one anymore. Beyond that, during the lesson on stackup and buildup from Phil’s Lab, I realized I was too naive when thinking of going for SIG/GND/PWR/SIG in one of the first build logs. For good EMC performance, we need every SIG and PWR layers to be near a GND one. High-speed signals should ideally be flanked by two contiguous planes even, so routed on internal layers.

If we go to a 6-layer PCB, we could follow these guidelines by doing SIG/GND/SIG/GND/PWR/GND, with a handful of low-speed signals on the exposed outer GND layer to populate the silkscreen. Going to 8-layers on a similar pattern would buy us another internal SIG, not sure if it’d be worth it. We’ll see during routing.

pdn

The power distribution network feels like the most intimidating part of the system, to be frank. For routing data signals it’s mostly making sure to use controlled impedance, spacing out traces, and keeping them short. For power, we need to derive 4 power rails just for the on-board components. In contrast to the ZettBrett board discussed in the Phil’s Lab course, we also want to charge downstream USB devices, which make up the bulk of the load.

To keep it manageable for this version, I looked for various corners to cut. At some point I thought it’d be cool for laptops to provide emergency power to little cloud’s on-board systems, but merging that with the standard upstream power seemed tricky. So, gone. Maybe a battery slot to effectively add an internal UPS for little cloud’s essential systems? Detecting it and caring for it adds further complexity, gone. Most USB-C chargers have ports that go up to 15/20/25W, maybe we could have two power input ports to access more power with standard parts? Hard to merge the sources, gone. Maybe we could vary the configuration pins on downstream Type-C ports to advertise 5V@0.5/1.5/3A to charged devices, depending on the power budget. Type-C ports aren’t allowed to be “hot” with power by default, we’d need voltage sensing to detect a sink via an ADC or comparator. So, gone. “Hardware is only as hard as you make it.”

Here’s what we’re left with. One upstream port that needs around 25-30W PD. No backfeed from laptops. Downstream smartphones and tablets only get a standard 5V@0.5A via Type-A ports, totaling 15W peak. Bonus: lowest possible wear on the worker device batteries. We have power rails for internal housekeeping, plus a separate one for ports, even if also 5V. Just like in the course, an identical wide-input buck converter arrangement per rail, stepping down from some default contract at 9/12/15V negotiated by the upstream PD controller. I picked a Silergy buck, like in the course, but rated for 5A output to cover 6*0.5A ports comfortably. Also picked an WCH power switch to use per charging port, plus we need a different hub variant hub with per-port power enable pins after all. We’ll do TVS diodes and fuses later, as refinements.

This week has been about creating symbols in KiCAD for each relevant chip we’ll be using on little cloud’s circuit board. Turns out it’s hard to find well-organized ready-made symbols for the Chinese parts we’ve selected due to cost-effectiveness. Though nowadays we can learn about these parts from their Chinese datasheets using LLMs, so it’s pretty doable. While I did the schematic symbols, the packaging formats are pretty standard and so KiCAD does have most relevant footprints and 3D models baked into the standard library, such as MSOP-10 or QSOP-24 for the USB switches we’ll be using for host workers (see previous update).

I was dreading the mechanical task of annotating, for instance, 24 pins on the CH448U switch, but it turned out to give me a proper chance to get familiar with the parts I’ll be using, which I don’t regret. Some gotchas to be aware of are active-low pins and bit ordering for the selector.

By the way, the reason I’m looking at both the CH442E 2:1 switch and the CH448U 8:1 switch ($0.1@1K) is because of the following. If we only did 2:1 switching to toggle a port to use the microprocessor’s (MPU) USB device bus, we’d end up with an unterminated stub going to the switch of each port that doesn’t connect to that. At USB high-speed frequencies, physics would complain through reflections and capacitance. Instead, we also place one 8:1 switch on the device bus to “confirm” the port connecting to it and ensure the connection there remains cleanly point-to-point.

eth

Moving on to the Ethernet specific parts, they were a bit weirder. The fact that they mix different high-speed protocols that they need to partially process in silicon makes them require odd power routing. For instance, the the USB-to-Ethernet RTL8152B nominally only requires a +5V power input into the chip. However, it also produces other specific voltages on other pins, as power outputs. But this isn’t a convertor or anything, these power outputs shouldn’t be used by other chips. But then, why route them out to the chip’s pins, you might wonder. Well, because we need to do some filtering and decoupling outside the package, to protect the sensitive internal circuitry from the noise as much as possible, and then route that back in on a different pin. Kinda weird.

Also, because our MPU only exposes an Ethernet MAC interface, while previous $0.8@1K choice of Ethernet switch only exposes PHY interfaces, we need to upgrade to a switch that also exposes at least one MAC one. It turns out that if you want at least 5x PHYs (for connecting to the USB-to-Ethernet chips) and 1x MAC on the switch, then the cheapest version that supports Gigabit speeds costs the same as the one that supports 100 Mbps only, at ~$1.5@1K. Maybe the 100 Mbps part is scarcer or something. Anyway, it means we effectively get a 10x upgrade on the interconnect between MPU and Ethernet switch, because it costs the same. The downstream port connections are still capped at 100 Mbps, but the MPU-switch link was more of a bottleneck. Any USB host and device workers would cross that link to communicate.

Or would they? Here’s an idea of how to increase the bandwidth between workers at no additional hardware cost. We already said we might use a worker device’s WiFi radio to connect to the internet. Well, we can also get them to connect to each other with low latency as an add-on to the wired connection that also handles charging and recruitment. This can double or triple the aggregate throughput between them, a feature especially relevant for inference workloads with bits of the LLM split on different workers.

The switch symbol itself was quite a beast compared to the previous ones, at around 128 pins. A third or so of which were power related, I guess this is the actual status quo with mixed-signal chips. The Gigabit MAC interface will hook into the MPU’s, and the Gigabit PHYs on the right will hook into the USB-to-Ethernet chips, though only half the channels because the latter are 100 Mbps only. I didn’t expect a 1.5$ fixed-function chip like this to have so many bells and whistles, an integrated microcontroller which we won’t touch, options to attach separate configuration memory which we won’t use, etc.

usb

Then came the USB power delivery controller we’ll be using to negotiate with an upstream charger to get the power we need for smartphones and tablets. Quite tiny compared to the 128-pin switch, but packed with features. It can be set to default to a 5V contract in order to power the MPU, which then comes back and asks it to negotiate more via I2C. It also has support for legacy fast-charging protocols, to be able to talk with chargers that predate USB PD. Why not.

The 7:1 USB hub resembles the earlier 8:1 analog switch that can work with USB signals. It has some more LEDs, supports configuration memory which we won’t use, and has a few other bells and whistles. Still very manageable compared to the 128-pin switch and upcoming MPU.

mpu

Then came the centerpiece, the Allwinner MPU. It doesn’t have more pins than the Ethernet switch, but they’re more diverse. I had to also add in an exposed pad in the footprint, because the KiCAD library lacked such a variant. Apparently different manufacturers size the exposed pad differently, so maybe that’s the reason. Anyway, the MPU has it all, a bunch of really handy interfaces you can toggle the pins to inherit programmatically. Half the work on this symbol was pruning away the functionality such that we only get the features we need instead of half a dozen options per pin listed in the label. Ran my symbol and the datasheet through LLMs to spot typos and mismatches, which there were quite a few of. Hopefully the symbols are stable enough to use conveniently in the schematic.

non-technical

Been also looking into ways of minimizing little cloud’s price beyond the physical parts that go into it, namely in terms of legal and logistics. From further research into relevant regulations for selling across the EU, we’ll need to align with the Electromagnetic Compatibility Directive (EMC) for Electromagnetic Interference (EMI) and Electrostatic Discharge (ESD) concerns. We’ll need to align with Restrictions of Hazardous Substances (RoHS) for self-evident reasons. General Product Safety Regulations, too. Thing is, for our Energy Source 1 (<75V DC) and Power Source 2 (<100W) class, and in a consumer setting, it tentatively seems like we can achieve compliance by simply ensuring the design adheres to standards, documenting it, testing it internally, and then just self-declaring the results. No authorized lab or notified body with eye-watering quotes, seemingly.

But there’s a thornier compliance issue: WEEE, or e-waste management. Thing is, little cloud might reduce e-waste by helping households breathe new life into old electronics. But, it would itself be an electronic product, that can only be sold to consumers in the EU through settings that involve registration with local institutions responsible for e-waste. Annoyingly, these are not really harmonized or integrated beyond the national level, making it such that you require a legal entity in each member state where you want to sell directly.

Now, the catch is that if you sell to businesses in other member states, such as local distributors of maker products, they take on the responsibility for local WEEE concerns. They’re properly registered as a seller of electronics, etc. They also handle returns, replacements, and buffer local storage for cheaper and faster deliveries. Of course, they also take their cut. But you could sell to multiple distributors in the same region to drive lower markups on that leg of the product’s trip. Let’s say the marginal cost of a little cloud at 1K volume is 30€, it might be possible to barely get it on various EU shelves at around 40€, before VAT and shipping. Not great, not terrible.

There’s also another approach I found, namely that of “co-creation” with a “design partner.” Basically, you bring the idea and a working design, and they take it on and manufacture, sell, and ship it. They give you a royalty fee for the IP. Now, sure you might get the best margins if you sold directly or through distributors. But then, this is not what we’re here for. Co-creation could potentially be interesting in that it lowers the underlying cost of goods quite a lot, because the design partner has huge quantities of relevant parts already. Some programs like this focus on integrating new modules in an ecosystem, which isn’t too interesting here. But if I find a program where they handle the logistics and put pressure on the final price, I might be down for that.

Let’s recap the quest for a satisfactory approach to little cloud’s hardware. Early on, my best guess was that we’d go with some very specific microcontrollers manufactured by WCH, using them to flexibly drive USB ports. But that felt complex and overfit to WCH’s offering. Over the past couple weeks, my best guess was that we’d fold everything we need into an FPGA as soft cores, including a processor, USB buses, and so on. But that irresistible appeal of total consolidation quickly died with a reality check of trying to synthesize all that on a budget FPGA. So where do we go next?

Here’s my intuition after a few weeks of turning this problem over on a bunch of sides. The promise of USB lies in its universality. It can enable a flash drive holding an OS, it can support networking between devices, it can be used to control smartphones, and so on. But perhaps we jumped to the wrong conclusion when we mandated that every single port needs to be able to do everything at anytime.

The following nuance is growing on me. Recruiting devices is much messier than orchestrating them, but only happens occasionally. You talk differently via USB to a smartphone than to a desktop to recruit them as worker devices into a cluster. But orchestration could be made to only require a reliable network once the worker has been recruited.

split

Because recruitment generally only happens once per device, and only one device is being recruited at a given time, the whole recruitment hardware can boil down to a single dual-role port or two single-role ports. Want to recruit an Android smartphone or tablet? Have little cloud present itself as the host and drive it via ADB. Want to recruit a laptop, desktop, or server? Have little cloud present itself as a composite of a mass storage device holding a minimal OS and a HID keyboard device to handle the boot or install. At the end, just make sure you’ve left a privileged shell for little cloud to reach.

Because orchestration is constant but generally only requires reliable networking, the orchestration hardware can boil down to making sure there are many network-capable ports, either device-only or per-type. The device-only option is to only use fixed-function USB-to-Ethernet chips (~0.6$ @ 1K), and connect them all plus the microprocessor to one switch (~2$ @ 1K). The downside here is that smartphones and tablets lacking dual-role power (manufactured prior to ~2017) wouldn’t be able to be charged on the same cable. This is because these chips present themselves as USB devices, and only on Type-C with full power delivery (PD) support can the two sides swap power roles.

The other orchestration option would be to have one USB host bus combined with a standard hub chip for smartphones and tablets, and the Ethernet switch for laptops, desktops, and servers. More backwards-compatible, but less uniform. Yet what I like about this option is that it lacks microcontrollers and FPGAs entirely. Just a microprocessor and a few fixed-function chips that just need to be placed properly. One firmware only, and no gateware.

block

So I got the Advanced Digital Hardware Design course from Phil’s Lab that I mentioned in a recent update, primarily to use as an outline when working on little cloud’s board. The first thing to follow along with was a high-level block diagram of the system architecture, so I made a slightly cleaner version that draws from various rough sketches from my notebook, with part numbers, costs at 1K, used interfaces, etc. It reflects this third approach described in the previous sections, where we split hosts and devices, and use neat fixed-function chips on the host side to cheaply aggregate them.

I notice that while the “data plane” is pretty well-specified, I’m still somewhat confused as to how to approach power. We’ll get to that while following along a lesson focused on this, but I can already see there’ll be a few different subsystems related to it. We’ll need to negotiate a variable USB PD profile from upstream and convert that to 5V efficiently, we’ll need a way to power sequence the System-on-Chip and give it a few different voltage rails, and we’ll need to power the USB devices in a way that preserves battery life and minimizes heat. We’ll get there eventually.

Also, I realized that our dozen or so different USB high-speed and Ethernet 100mbps connections might be a source of electromagnetic interference (EMI) issues, which gets tested during product safety certifications. We’ll really need to nail grounding and shielding and filtering, and so I’ll probably walk back on the standoff sandwich approach to the enclosure and go with a fully enclosed one instead, see update #5. And I might fold the fan grills back into the enclosure, although with a simpler layout, to ensure we can ground the 120mm fan holes, too.

maximalist

I did another iteration on the enclosure, feeling a bit paranoid about the product safety certifications we’ll need to nail early to avoid extra spins. There’s the EMI aspect, where we essentially want to shield the high-frequency circuitry with the enclosure as much as possible, ideally with no large openings. Then, there’s the fire safety stuff, which to really make sure we cross it, we’d again ideally enclose “potential ignition sources” inside metal. Our enclosure here is also critical for airflow, so maybe we could have a plenum to distribute it uniformly across devices? Though not sure how important the latter is.

The result of optimizing for comfortably passing certifications in this iteration led to a somewhat overengineered design. The PCB ended up the front panel, but with all surface-mounted components on the inside face. The airflow grills resemble the original (expensive and dense) hole matrix design due to seeking EMI shielding. The sides have matching tabs and slots to aid assembly, which is the first time I tried something like this. Critical flaw: the ports facing upwards would be obstructed by the worker devices occasionally.

Similar to how we’ve relaxed constraints on the hardware schematic side of things over time, I think the enclosure could also get the same treatment. With this overengineered design in mind, I’m thinking we could handle PCB enveloping (for EMI and safety), separately from airflow to worker devices. And so maybe we could use simple wire mesh grills for fan outlets, as well as plates on standoffs, even if not ideal for EMI. And that because we could separately sandwich the PCB tightly between sheet metal plates in its own area, outside the airflow path entirely. The search continues.

cubes

Here’s another iteration, following the idea of decoupling the PCB shielding from airflow paths. We basically structure the solution around a few different floors. Worker devices sit on the roof. On the floor below, we have a chamber dedicated to ventilation, as well as power brick storage. The back side is missing, to allow air intake and access to storage. On the floor below, we have the actual PCB, enclosed on all sides. A tiny panel turned off in the render would be placed flush against the ports. Also, the sides in this flat-only iteration can extend upwards to create handles and downwards to create feet to avoid resting on tiny screws and scratching the table or below.

I also tried incorporating these neat fasteners, called threaded cubes. They’re basically female-female standoffs but 3D instead of 1D. They’re pretty handy for interlocking flat plates on multiple sides without countless flanges. Even compared to 2D 90° brackets, these allow us to use fewer parts, as some of them connect on even 4x sides. They come in cute colors and are dirt cheap in volume, maybe it’s a way to add an accent color to an otherwise spartan choice of parts.

Of course, once actually modeling it based on some rough sketches, I realized we could potentially make some improvements. If we limit one side of the PCB to at most low-speed signals like LEDs, like in the previous sections, we could skip the metal sheet that currently separates the two internal floors, as well as 4x threaded cubes. One of the PCB plates becomes the metal sheet roughly, but with the ports oriented correctly this time, and with a better structure around it. However, the laptop power bricks that might reasonably be placed in the storage areas might need to be isolated themselves from the PCB, as they have mains AC inside, transformers and switching, and so maybe the extra sheet isn’t too bad for different reasons.

This week, I wanted to get into the weeds of working with an FPGA, following last week’s push for hardware consolidation. I had a second-hand Tang Nano 20K laying around, which is a development board built around a Gowin FPGA with about 20K look-up tables that define the internal circuit. From some initial napkin calculations, packing a Linux-capable processor, a bunch of USB buses, and a few other hardware blocks would require us to go for a 20-25K FPGA on the final product. It’s unlikely we’ll end up going with Gowin FPGAs after I saw the costs on LCSC, compared for instance to Lattice ones. That said, it’s the right caliber, so we’ll use it to experiment until I sleep on the whole FPGA approach enough to warrant a board closer to what we’ll put in the final product.

So, onto getting the FPGA to blink an LED. We need two input files, a Verilog one defining the logical behavior of the circuit, and one for generally mapping variables to specific pins on the physical chip. The Verilog in the picture below does the following. We’re defining a tiny block of circuitry that has one input and one output “wire.” We’re then effectively defining an integer over 24 bits. Then, on every rising edge of the clock signal that we tapped into, we increment that integer. Finally, we take the most significant bit of the integer and wire it to output. Basically, we’re implementing a clock divider to slow down the native 27kMhz clock by 2^24 or so, and use the resulting ~2Hz tempo to flip the LED state.

The tooling we need to “synthesize” this circuit on the FPGA is quite involved. First, I used “Yosys” to turn the Verilog into what I understand to be abstract representations of how look-up tables (LUTs) and registers need to be wired to implement the specified behavior. Then, I used “Nextpnr” (place-and-route) to assign specific blocks on this specific Gowin FPGA to those roles. Then, I used “gowin_pack” to build the previously generated JSONs into a binary the FPGA understands. Finally, I used “openFPGALoader” to flash that binary over USB to the actual board. Granted, the reason we’re dealing with four different tools here is that they’re the open source, reverse engineered alternative to Gowin’s integrated yet proprietary tooling. It boils down to four legible commands that can be scripted, could have been worse.

litex

Next day after the Verilog blink test, I decided to actually give LiteX a go. This is a library of open source cores that allows you to turn an FPGA into a System-on-Chip with exactly your specs. It has a tutorial with a few labs, so I mostly went through that first. “Mostly” because the tutorial was based on a different development board which had more outputs and sensors. I did a binary clock with the 6 LEDs on the Tang Nano 20K instead of displaying the time as digits. I also got to implement pulse width modulation, the kind of peripheral a microcontroller would use to dim an LED with a square wave, here shown as a simulated waveform.

Getting a Linux-capable processor baked into the Gowin FPGA on the Tang Nano was trickier, because Linux doesn’t really fit into the 8MB of integrated RAM on the chip. So I might need to already switch to a development board that’s closer to the intended use case rather than what I had laying around. I did manage to flash a dummy shell on one of these soft cores, without loading any actual OS underneath. Apparently, LiteX also automatically builds a BIOS for your SoC definition, equipped with a debugging shell over serial. And it only takes a couple dozen lines of LiteX to define, build, and flash a SoC that exposes such a shell. Pretty neat.

While I was binging anything I could find about FPGAs on YouTube, I realized that Phil’s Lab has a course focused on designing a board that happens to incorporate an FPGA, high-speed USB, memory routing, external flash, etc. Phil’s previous course was the one I followed to learn how to use KiCAD in the first place. So I thought maybe I could take this other one too, but instead of reproducing the demo board he builds throughout the materials, I could just use the course outline as a guiding structure and apply the same principles to little cloud directly. I contacted Phil to ask whether it’d be fine to submit this other board at the end of the course for the certificate and all, and he seemed fine with it as long as it’s of a “similar complexity.” It’d be really cool to get an actual expert review on the little cloud hardware before submitting that first version to manufacturing.

bye

I had a first scare when I realized that, wait, the mere fact that there are open source USB soft cores out there doesn’t mean Linux would automatically be able to drive them with native drivers. In fact, I could only find a pair of host and device cores with native Linux compatibility, but they were only for USB full-speed (12 Mbps), not high-speed (480 Mbps). I’d really like to match the widespread high-speed found on would-be worker devices, to avoid little cloud itself being the bottleneck.

I did find the LUNA project, which brings open source USB high-speed device cores that can be controlled through software. Even if not the native Linux interfaces outright, it can do networking, mass storage, keyboard input, etc. For USB high-speed host cores, I did find guh, which again enables some use cases but falls short of full Linux compatibility. One option is to have little cloud usually emulate a USB high-speed device to everyone, including smartphones and tablets. And only use a Linux-compliant full-speed host for the mobile worker bring-up, such as for running ADB. It is possible to do both high-speed host and device from Linux, as exemplified by the proprietary IP cores that boast such capabilities, but the open source stuff is currently limited to either full-speed Linux-compliant or high-speed hand-rolled.

But a worse scare here was when I tried to get an actual empirical estimate of how many LUTs would be required for what we need here. I had GLM 5.3 Flash experiment with defining and synthesizing a few SoC variants that incorporate the specs we need. Well, the initial calculations were too optimistic, especially if you account for some way of moving traffic between USB buses themselves and from an SD card. For doing this efficiently, you either need a DMA controller per bus that copy data to RAM without processor involvement, or you need some routing between these in the actual gateware. Anyway, when I tried going from napkin calculations to having an LLM actually experiment with synthesizing a few configurations, it became pretty clear that the few FPGAs that are not crazy expensive wouldn’t have enough capacity to hold all of what we’d need them to. The search continues.

cnc

This same week, I wanted to try getting a hang of the CNC router at a local makerspace, as mentioned in entry #5. The idea was that maybe I could iterate on the enclosure faster and cheaper than working with a CNC shop. So I signed up for an intro session, during which we actually got all the way to CNC-ing a slightly tweaked file export from that same Onshape project. And we cut it out of... a glorious slab of 18mm scrap plywood that we grabbed from a discard pile. Quite a bit chunkier than the 1mm aluminum I had in mind, it feels like the volume of a whole enclosure rather than a plate.

I realized early on during the intro session to what extent the modern software of products like Bambu Lab’s A1 have spoiled me. After you export a DWG/DXF out of your CAD of choice, you first need to process it using “CAM” software, which is the equivalent of slicers in 3D printing, only way more manual and clunky (at least the AutoDesk one we worked with). Then we put it on a stick and loaded it on an ancient piece of software that controlled the actual CNC through motor drivers, which is the equivalent of Klipper in 3D printing. The CNC itself lacked vacuum suction for holding down the material, hence the countless screw marks on the working area. The spindle was powered separately, and relied on a DIY water-cooling loop. Overall it felt too janky, so I don’t think I’ll go ahead with using it for real.

fdm

What I did try using afterwards after all was a standard 3D printer. This is mostly only relevant for the simplified flat plate “enclosure” approach described in update #5. Because of the reduced size, it fits in the build volume of an A1. Yes, it’s not actual sheet metal, but it’s enough to get a feel of the proportions.

After an initial print, I realized that the support holes for the worker devices are a bit sparse, the two rows are a bit too far from each other, and the margins make it a bit finicky to use nuts with the standoffs. A second iteration felt like a step in the right direction, though worker supports are still finicky to move around.

Smartphones and tablets together are the largest category of e-waste by computing resources, and most of them run Android. If we are to aggregate junk electronics into usable clusters, we need to be able to deeply control these pieces of hardware. Thankfully, the Android Debug Bridge (ADB) is a protocol that allows us to do virtually anything with an Android device attached via USB. You just need to enable USB debugging, then you can access a bunch of options and resources.

I played around with reading specs and configs off of my Pixel 8, and you can really get down to the manufacturer of the RAM die or the current mobile carrier. You can also toggle settings, transfer files, run all kinds of binaries in a shell, even emulate taps, typing, and physical button presses.

ADB really is a Swiss army knife for controlling Android devices. This is great, because smartphones and tablets are otherwise quite fragmented in terms of hardware, making it difficult to find the drivers to build a custom ROM into which you could boot. On Android devices, we should work with the vendor binaries shipped with the original OS if we want to have a chance at tapping into mobile GPUs effectively, for instance.

target

The next thing I tried was building the default “hello world” Rust project for an Android target. This was surprisingly painless. You just need to have the Android NDK available, tell Rust to prepare for compiling for an Android target, and install Cargo NDK as glue between the two. After this initial setup, we got an output binary.

We can “push” the file to the connected device via ADB, then drop in a shell where we can execute it. And there it is, we’re greeted by code that’s running natively on the Android device. It’s not doing anything useful, yes, but it validates that we can simply target Android with Rust code. Actually seeing this pan out is good news, because it’s the Android environment that felt more exotic compared to the standard x86 / amd64 computers.

agentless

This binary we transferred to and ran on the smartphone could have been our worker daemon! It only had to expose some kind of API around underlying tools like llama.cpp or ripgrep that could be consumed by the orchestrator. But then I thought, wait, what if the orchestrator can just run things directly on worker devices through shells? Do we really need to mantain a hand-rolled daemon to abstract away from underlying resources and capabilities?

I seriously considered these, intrigued by the prospect of drastically simplifying our software infrastructure. We’d still need the orchestrator to manage workers, and we’d still need to write firmware to get microcontrollers (MCU) to emulate USB devices. But we could otherwise be looking at an opportunity to remove a whole part of the software burden. Apparently Ansible is also “agentless” as a DevOps tool, in that it only runs ephemeral code on the machines in a cluster, rather than standing a long-lived daemon the way Kubernetes does. So it’d not be completely unheard of.

However, removing the worker daemon as a whole and having the orchestrator talk to worker shells directly would naturally also have some downsides. The main one is the messiness of the interface exposed to the orchestrator, command-line calls and output parsing instead of some normalized /jobs/ paths in an API. The messiness goes beyond diverse invocation arguments, all the way to how to keep track of service status across the cluster. There might also be some overhead on processing, but opting for “scp” / “adb push” instead of transferring files through standard shell output would make it competitive for bulk transfers.

fpga

After coming up with the idea to remove worker daemons in favor of an agentless orchestrator, I was feeling on a roll when it comes to simplifying the design. Accordingly, I thought I’d also push further on the idea that we need the MCU firmware.

My previous best guess for how to achieve a flexible USB array was to connect USB-enabled MCUs to the ports, and then network the MCUs via the orchestrating microprocessor (MPU). WCH has various dirt-cheap USB-enabled MCUs, but simultaneously optimizing for cost and high-speed capabilities felt like overfitting to very specific chip offerings. For instance, the WCH CH32V407 has two USB high-speed buses and Ethernet for <2$ at 1K volume, so maybe we could use one every two ports only, and network them via on-board Ethernet as the interconnect.

There’s no chip I could find that’s available today, integrates more than a couple dual-role USB buses, and is Linux-capable such that we could tap into that rich repository of USB-related drivers. So, what if... we instead instantiated the silicon blocks necessary to have Linux drive lots of USB buses inside an FPGA? This stands for field programmable gate array, and refers to a type of chip that can effectively be dynamically rewired to implement different kinds of physical functionality. Libraries such as LiteX allow you to effectively come up with the specs you’d want on a system-on-a-chip and configure an FPGA to become that chip. Maybe you pick a certain processor with this many cores, some PCIe or SATA controllers, or whatever other specs you want, as long as it fits on the FPGA.

balance

What I’m sayings here is that maybe we could effectively fold many of our key components into one FPGA, including the MPU, MCUs, any hubs / switches, and so on. We could instantiate USB buses directly as parts of our ideal system-on-a-chip that we’re otherwise having trouble finding for our very specific application. This would remove a significant part of our system’s complexity, but it would also add complexity in other places, so let’s look at the balance sheet.

First, effectively folding our MCUs into the FPGA would eliminate the need to write custom firmware for them as a separate software component. Ideally, the Linux instance running on the FPGA would simply see many available hardware buses and operate them natively. We might still need a fixed-function chip per port if we want high-speed, though. Second, consolidating the MPU, MCUs, as well as any miscellaneous hubs and switches into one single part would vastly simplify the layout and routing of the actual circuit. Third, it would drastically simplify our bill of materials, as our board mostly reduces to a chunky FPGA fanning out to a bunch of USB ports. Sure, we’d still have some power assembly, a separate flash chip, and a few other one-off niceties, but we’d cut down on most hyper-specific parts.

Switching to FPGA also adds complexity elsewhere, primarily in the actual configuration of what we want from it, or the “gateware.” We need to instantiate a processor and various USB peripherals. That said, FPGAs turns hardware specs into software, and software is trivial to share. There’s a growing collection of open source blocks to throw into your design, as exemplified by LiteX. To be honest, I’d also not be mad about getting the opportunity to play around more with FPGAs, especially because the resulting configuration code can be converted into files that can essentially be sent to fabs to get custom chips taped out. I’m pretty sure we won’t get anywhere close to that level in this project, but it’d still be interesting to get familiar with this corner of the industry. Looking at the bottom line of our balance sheet, I’m overall pretty intrigued by the FPGA approach to little cloud’s design here, and I feel like these first weeks are well characterized by the following quote. It takes time to simplify.

“I have only made this letter longer because I have not had the time to make it shorter.” — Blaise Pascal

Thought a bit more lately about how to organize the low-level software responsible for recruiting and orchestrating diverse worker devices, such as old smartphones, tablets, laptops, desktops, and servers. This software infrastructure layer that sits below the actual user-facing apps has a few components. First, there’s some kind of daemon shipped to the worker devices to get them to participate in the cluster and listen for jobs. Second, there’s the firmware flashed onto the microcontrollers (MCU) responsible for emulating various USB devices. Third, there’s some process on the microprocessor (MPU) that’s responsible for talking to the previous two and managing them.

Let’s zoom in on the first item, the payload shipped to the worker devices. Its purpose is to tell the MPU about what sorts of resources are available locally, and be able to take on various kinds of processing job on the worker device it’s hosted on. This roughly corresponds to a Kubelet in the world of Kubernetes, but there’s no way we’ll touch that bloated unwieldy ecosystem to link up devices revived from a junk drawer. I was thinking maybe we could write a minimal worker daemon in Rust, such that we can then compile it into a lean binary for any relevant worker architecture.

To allow the worker daemon to natively tap into as much hardware as possible, we can wrap the shared logic into a minimal platform-specific wrapper. On Android smartphones and tablets, this could mean an “app” with no UI, which can nevertheless grant our daemon access to the GPU, wireless, among others. On POSIX-capable systems like laptops, desktops, and servers, this might mean a systemd service wrapping the binary. The worker daemons might, in turn, launch battle-tested tools as subprocesses, such as llama.cpp and ripgrep, as a way to harness local hardware resources using existing libraries. The worker daemon could preserve the underlying tools’ standard APIs, such as the OpenAI-compatible endpoints for LLM inference, or it could abstract these into uniform jobs and resources.

crustacean

I went through the rustlings project again over a couple days, as my knowledge of the Rust language was, well, rusty. It’s a fantastic learning resource, consisting of just under 100 tiny exercises. In each exercise, you get a file with certain compiler or runtime errors, and you need to fix those issues before moving on to the next puzzle. There are a few chunkier quizes after larger groups of chapters, and each exercise also has a hint that tends to link to the relevant documentation.

Going through it again was so much fun, probably the most fun I had “at work” in a while, thanks to the self-contained nature of the puzzles, the progress bar nudging you to complete all challenges, and the smooth increase in difficulty. At the end, you get a visit from a crab in your terminal, bringing you more obligatory puns.

llms

The experience above helped me realize how much I missed actual programming rather than manipulating systems by chatting with an intermediary LLM. I had a similar experience a few months ago, when I was shocked when I realized that I really enjoyed creating an actual slide deck, but mostly because of the direct interaction with the object of work. Extensive mediation makes you feel numb, while direct and reactive interfaces support states of flow.

Beyond the personal satisfaction angle, I also grew more skeptical lately about the long-term maintainability of projects developed with heavy LLM involvement. Actually sitting with the problem, even if tedious or mechanical at times, helps you incrementally build valuable context. And if you aim to be involved with maintaining a project, you might as well understand it deeply. What I can’t build, I don’t understand. And why would I take the time to use a system you haven’t taken the time to develop.

Anyway, I wanted to pull together those few strings related to using LLMs into a coherent policy for this project, partly inspired by Rust’s own. It’s okay to use LLMs to study new topics and debug systems you’ve built, but no synthetic content should end up in the final product. It’s okay to use LLMs to prototype and experiment with different ways of implementing functionality, but the end result should be completely hand-written, someone should be responsible for it. Also, even if education and experimentation are allowed uses of LLMs, the used models must be open weights. This is meant to prefigure the practice of actually relying on local compute.

firmware

Here’s an example of a way in which I plan to use open weights LLMs the moment the development boards I ordered arrive. The firmware we’ll need to flash on the MCUs connected to the worker devices via USB will need to be very good at controlling the physical silicon blocks (PHYs) dedicated to USB communication. There are a few promising libraries and stacks that might be able to grant us control over the USB PHYs found in the incredibly cost-effective MCUs manufactured by WCH.

First, there are the examples shipped by WCH itself, but they appear to be little more than flaky demos. Second, there’s CherryUSB and lwIP, libraries grown out of Asian hardware communities with strong support for WCH chips, but they might be quite low-level. Third, there’s the option to use those two libraries as part of the real-time OS called RT-thread, but it might add bloat and overhead. Fourth, there’s the option to use the Rust library embassy, which would consolidate the codebase, but has only experimental support for WCH PHYs.

I’d like to have LLMs have a go at implementing dummy firmware behaviors using these various stacks, such as getting the MCU to act as a network switch across its USB ports, or emulating a dummy storage device when connected. And I’d study whether it actually worked, the number of lines of code each approach required, and the general shape of the resulting implementation. I wouldn’t use those rapid prototypes as part of the final firmware that ships with the product. Instead, I’d only throw LLMs at the problem to help triage the available options going forward based on concrete findings. And probably with a Flash or Air model variant out of the Chinese labs. Can’t wait for the development boards to arrive.

e-waste

I also had to drop by an e-waste processor this week, and I’m including it here as it’s reasonably related. We picked up a batch of servers to use it in our pilot course that will take place in three schools this coming academic year. Each of them is a veritable mystery box, you don’t know whether you’ll get one packed with a dozen RAM sticks or whether even the fans will be missing. It’s also satisfying to learn where they came from, in our case it was mostly from a bank and a utility company.

It’s interesting how some of them come preloaded with those hardware testing procedure that go beyond a basic POST. You get checks and result reports per component, down to individual fans. When recruiting worker devices in little cloud, we should probably test capabilities when trying to detect them in a similar way.

Having run out of ideas for simplifying the enclosure (for now), I started digging a bit deeper into the actual protocols through which little cloud’s microprocessor (MPU) could communicate with the individual port microcontrollers (MCUs). The initial idea from a couple weeks back was that the MCUs could act as software-defined USB controllers driven by the MPU, such that they could present themselves as anything from a memory stick to a network card to the worker devices.

However, I’m not sure anymore about SPI as the protocol of choice for having the MPU communicate with the port MCUs. There are a few downsides. First, SPI would require the MPU to explicitly “chip-select” and query every MCU regularly, eroding bandwidth and increasing complexity. Second, we’d need to route 5 or so pins out of each MCU. Third, the USB driver ecosystem doesn’t natively overlap with SPI all too much. I did get to order an evaluation board predicated on this approach before growing skeptical about it, though.

ethernet

Another option for the MPU-MCU interconnect would be Ethernet. In this arrangement, the MPU and MCUs would connect their Ethernet pins to a dedicated on-board switch chip. 100 Mbps seems quite common as a choice of Ethernet speed on MCUs that implement it, which is higher than typical SPI speeds. Plus, it natively supports many-to-many communication, unlike SPI where only the MPU could initiate exchanges as master.

There is also USB/IP, which appears to enable the MCU USB controllers to appear as local, native ports to the MPU. It works by sending USB packets over a network, in our case a network that lives entirely on a PCB. Though USB/IP appears to only handle host roles natively, not so much device roles. We’d also incur the Ethernet switch on the BOM, and Ethernet on MCUs carries a premium. Could we do better?

usb

What if we used the same stack and tooling to connect the MPU with the port MCUs as we do when connecting MCUs to worker devices? The MPU could be the USB host that connects to many MCU devices. Crucially, MCUs would need to have two independent USB buses, a device-only one oriented towards the MPU, and a dual-role one oriented towards the worker devices. An interesting precedent here are those PC-to-PC cables that present themselves as a device to both USB hosts and shuttle data through.

If we could reduce the interconnect protocol to USB, we’d get the benefit of only having to use one stack for both interconnect on the PCB and communication with worker devices. CherryUSB appears to be a very versatile USB library, with strong support for the cheap Chinese MCUs. USB would also only require us to mostly route 2 data pins only out of an MCU, easier than 5. Most MPUs already have dedicated USB silicon, which is not the case for Ethernet. The MCUs might be able to only lightly edit USB packets shuttling through.

We would still need to use a USB hub between the MCUs and the MPU, and two USB buses on an MCU is not too common. But these parts are below 2$ still. I also realized we definitely need high-speed USB at least for having laptops or desktops potentially boot a live OS off of little cloud. Full-speed only would take forever to load the OS in memory. USB hub concurrency is also much more mature than via SPI.

hub

If we have a USB hub IC standing between the MCUs and the MPU, we could also route some of the hub buses out directly to ports. This could enable ports that can only connect to phone or tablet workers, not laptops or desktops, but for a very low cost. If MCUs with two high-speed USB buses grow into a meaningful cost center on the bill of materials, we could potentially offset this by only having some ports with full dual-role data capabilities. I’m not a fan of the broken symmetry across ports, although it could match the real-life e-waste distribution.

switches

Another option I considered briefly that I wanted to mention in the logs was that of analog switches on the ports. Need to emulate a host? Connect the port to a hub IC. Need to emulate a storage device? Connect it to a read-only flash on the board. This would mean lowering the emulation logic down to the hardware level. Though I figured it would be unwieldy to manage and iterate on, besides disabling the option to emulate multiple devices on the same port. I’d rather stick to two-sided high-speed MCUs, potentially occupying only part of a USB hub to the MPU. Driven entirely by the MPU software.

I took the general ideas for little cloud’s form factor from log entry #2 and implemented them in Onshape. The construction is pretty basic. Because we get lots of symmetry in a 1U rack-mountable box, we start by sketching out a quarter of both the inner and outer shells. We then flange from each towards the opposite plate.

Then, create mounting holes on the short and long sides. Also, we can remove material from the bends themselves, following the “fold it yourself” idea from a few weeks back.

Then, mirror both inner and outer shells across the two base axes to get the actual clamshell box. Finally, create a through-all hole and pattern it linearly across the same axes to implement the modular mounting interface idea.

simpler

After seeing it take shape and bouncing ideas off of others, I decided to push for modifications related to the grill, supports, and size. When it comes to the grill, I realized that sheet metal costs are partially a function of the perimeter length of all cuts. A grill with a gazillion holes takes a toolhead more to cut out, keeping it busy instead of fulfilling other jobs. Also, even if punctured by a lot of holes, the grill’s remaining material might still pose resistance to the fans, reducing the volume of air they manage to move around worker devices per unit time. So we’ll go for a basic large hole instead.

When it comes to the supports, I realized that it would be simpler to get hold of them through the same manufacturing process as the enclosure itself. We might as well just build them out of sheet metal while we’re at it, consolidating our supply chain. I played around with a few possible geometries, some requiring more material and bends than others, some requiring additional screws for mounting. I quite like the simplest one, two bends and no screws required. Instead, it seems to allow worker devices to securely rest on one side because of its bottom geometry and how it interacts with the enclosure. Move it around in your head.

When it comes to size, I realized we can maintain server rack compatibility while making the enclosure smaller. A smaller enclosure appears more manageable for end-users, more in line with household e-waste statistics, and of course it would cost a bit less. We can then recover the rack-mountable property through a separate face plate that attaches to both the enclosure and the server rack, but only if needed. For lack of a more principled constraint, I then opted for the general size of “fits two standard fans side-by-side” for the main enclosure.

quote

I was curious to run the current 3D models through JLCCNC for an instant quote at 1K volume. If we go for 1mm aluminum for the shells, we get a bit over 3$ per shell. Opting to laser-engrave the outer shell as a finish gets us instead to 5$. That finish seems cheaper, more durable, and more recyclable than layering silkscreen paint on the enclosure, and it would be neat to be able to make the device explain itself through labels and visual cues. The supports would be a bit over 0.5$ per piece, perhaps these could be stainless steel. The cheapest delivery option is in the same ballpark as the shells, but it would take forever to arrive. I signed up for a course on operating the CNCs at a local makerspace to be able to prototype faster. I’ll share how that goes in a future update.

simplest

Then I thought, what if the case would just be the bare PCB and a top cover on standoff spacers? It seems there’s no hard regulatory requirement for a box to explicitly be enclosed on all sides. The PCB would match the footprint of a flat metal sheet sitting above it on 4-6 standoffs. The sheet would retain the fan cut-outs and support mounts. Though, if we use standoffs, we might as well use them as supports, too.

I’d rather have the whole arrangement be incredibly cheap than feel premium, given it’s intended for reviving e-waste. Running the only metal plate through JLCCNC, we get an instant quote of about 1.5$ at 1K volume for no finish. The 4-layer PCB with the same outer dimensions is a bit pricier at around 4$, but I think it would still be cheaper than a smaller PCB combined with an extra metal plate on the bottom and more stand-offs involved. Same rack-mountable face plate adaptation as above, but hugging the top and bottom sides with some tiny flanges instead.

Let’s think through the structure of the printed circuit board (PCB) that will act as the physical core of little cloud. We likely need an array of USB ports through which diverse worker devices will be recruited into a unified cluster. USB is very versatile, as it can support internet tethering, storage media, debug shells, keyboard input, video output, and many others. To decouple hardware requirements from the exact software methods we’ll end up using for handling various classes of worker devices, we can keep our options open and simply mandate that every port should be able to present itself as anything the USB standard supports.

USB is an asymmetric protocol that requires an explicit host and device. Most laptops, and virtually all desktops and servers, are hard-coded to only act as hosts. Old smartphones and tablets are hard-coded to only act as devices, while modern versions can act as both. The chips behind standard USB hubs are typically hard-coded to connect many USB devices to one USB host. This means we can’t really get away with just using one hub chip to back all our ports, because then little cloud wouldn’t be able to easily connect to many laptops, desktops, and servers over USB.

What we could do is have one cheap “dual-role data” microcontroller (MCU) per port, whose role would only be to present itself as anything that’s needed and act as a simple bridge to some beefier integrated circuit (IC) that manages them all. Typical USB connections span “full speed” (12 Mbps), “high speed” (480 Mbps), and “super speed” (5 Gbps). The upstream link from the port MCU to its beefier manager might be SPI (1 to 100 Mbps), on-board Ethernet (100 or 1000 Mbps), or actually one of the same USB variants above. I think we could go for high speed on the port and SPI on the manager link initially.

That said, I did manage to find a very narrow niche of MCUs that manage to pack insanely many USB buses on the same chip. WCH’s CH645 for instance, whose product page isn’t even available in English at the moment, boasts 8 independent buses, 4 of which can effectively be dual-role data, with the other 4 host-only. With a niche MCU like this, we’d get extreme design consolidation and cost reduction, at the cost of a more brittle supply chain. Would it be worth it? Not sure, we might regret going for this if supply suddenly becomes unavailable. But only needing one dirt-cheap MCU per product instead of one per port is appealing for sure.

power

Not only is USB versatile in terms of what kind of data is being transferred, but it is also versatile in terms of what power it can carry. Older USB can do 5V@0.5A (2.5W) all the way to 5V@1.5A (7.5W). On a Type-C connector with a proper power delivery (PD) controller behind it, two devices can negotiate dynamically up to 20V@3A (60W), and even higher if the cable itself is marked electronically.

In addition, modern USB can flip power direction relative to data roles, such that the device can power the host. It can also negotiate an emergency contract where power flips if the wall-plugged device loses its power while a battery-enabled one carries on. As with data, let’s reduce coupling between hardware and software constraints and push for ports that are able to simply do whatever is needed when it comes to power.

However, we can cut a corner here and limit ourselves to 5V only across ports. This is partly to avoid having to add DC-DC converters and further support circuitry on each separate port that would’ve allowed it to independently climb up to 20V. After all, we need to charge phones and tablets as slowly as possible anyway, in order to avoid heat buildup and maintain battery life. We need to be able to dynamically charge them a bit faster than they drain, though, so hard-coding 5V@0.5A wouldn’t cut it. The 5V simplification largely does lock us out of being able to power laptops through vendor-specific adapters, though I think the vast majority of people just use the original power-brick cables. We’re not touching AC in this version, so anything other than a phone or tablet needs to separately plug into its own power outlet.

manager

The manager chip needs to be a bit beefier than the port-facing MCU(s). One option is to go with a tiny Linux-capable microprocessor (MPU) with co-packaged RAM to avoid having to route (and buy) dedicated RAM chips. Allwinner MPUs seem decent at around 4$ at 1K volume via Chinese distributor LCSC, depending on the exact model. They have 64-256 MB of RAM, >1GHz processing, and all kinds of handy peripherals. Our bill of materials (BOM) could swallow ~4$ given it’d be a single MPU per board. Compare that to per-port MCUs that plummet under ~1$ really fast at higher volumes.

We could also just go with a beefier MCU for the manager. There are ecosystems like Zephyr OS, backed by the Linux Foundation, which provide mature networking stacks and other handy utilities designed to fit in the KBs of RAM of an MCU. Going for an MCU brain would also consolidate the software toolchain, and might constantly force the removal of non-essential features to keep things lean. I’m not sure if it’d be beefy enough to orchestrate and network the worker devices, though. Plus I think targeting an MCU with an Yggdrasil build would be extremely difficult, even though WireGuard has been ported to MCUs successfully as another VPN technology.

auxiliary

Besides the port MCUs and the manager MPU, the top signal layer will likely need to be populated with a few other things as well. A few gigabytes of persistent storage in some flash chip would be handy for storing prebuilt images or binaries for pushing on worker devices during recruitment. We’d also have the miniPCIe slot gated by a pin multiplexer for accommodating future long-range radio, as foreshadowed in earlier updates. We could also throw in an RJ45 port, integrated into a MagJack to avoid worrying about magnetics ourselves, as an internet uplink for when we lack worker devices equiped with radios of their own, such as some desktops and most servers.

We also need some components to handle the incoming power responsible for little cloud’s own chips, as well as for any phone and tablet workers it powers. We settled on having laptops, desktops, and servers get their own power separately, though maybe we could at least integrate some hidden tray for storing the laptop power-bricks. Little cloud should be able to negotiate as much power as it needs via PD from its upstream USB-C power supply, such that it can charge phones and tablets a hair faster than they drain. It can locally negotiate above 5V, but we should clean that up and convert it to 5V as soon as possible.

stackup

It might drastically simplify routing to then dedicate an entire internal PCB layer to our unified 5V, turning it into our power plane. Then, from anywhere on the top signal layer, we can use several vias and a copper pour to tap into 5V directly. Similarly, a ground plane would also be pretty handy for routing, and also useful for shielding during certification tests. We mainly get the bottom signal layer for crossing below other intersecting traces, maybe also with some extra writing on that bottom silkscreen. We’d try to use co-packaged RAM and avoid ultra-dense footprints in order to get away with just two signal layers and simpler signal routing. This gets us better hand-solderability and repairability as a bonus.

That would leave us with four PCB layers in total. Even if a bit more expensive than bare 2-layer boards, I think it’d vastly simplify the design, especially given the non-trivial currents we’d be moving around. These would have otherwise required chunky traces all over the place if it weren’t for a dedicated power plane. Plugging some numbers into JLCPCB’s online price calculator, we find that a 45 x 5 cm 4-layer board with default settings is around 2.5$ at 1K volume, compared to 1.7$ for a 2-layer one. I’ll take the easier power routing, especially given the MCUs and MPU would probably be at least 10$ together.

I’ve recently been thinking about a tension inherent to little cloud and how to resolve it. On one hand, it’s important for it to be convenient to set up and use, such that anyone could turn their unused electronics into platforms they can use and rely on. On the other hand, the spirit of the project is in large part about resilience: hardware you own, not corporate data centers; mesh networks, not internet service providers; solar panels, not the energy grid.

Here’s my current best guess as to how to get both the convenience of an integrated solution and the resilience of grassroots infrastructure. There could be two ways for someone to access the platforms hosted on their own little cloud or those of their friends. On one hand, consider a seamless option that can be configured in under a minute, but against which companies and governments still hold the leverage of a kill switch. On the other hand, consider a resilient option that takes ten minutes to initially set up, but which then runs entirely independent from mainstream internet infrastructure.

Crucially, it would be possible to upgrade your setup to resilient mode later, such as during an internet blackout or under a compute surveillance regime, based on instructions bundled with the device itself. The mere presence of a credible option to switch away from classic infrastructure might be a partial deterrent against overreaching authorities.

seamless

For the seamless path, I’m thinking of something along the following vignette. Your little cloud has picked a random identifier during its initial setup, and you can now use your phone to access it by heading over to the matching subdomain of littlecloud.ngo. Because your phone has been previously bound to your little cloud’s account using passkeys, you can then sign in without using a password or magic link. That’s it.

On this seamless path, however, we rely on: third-party DNS resolvers to recognize the domain, third-party certificate authorities for the HTTPS certificate, a centralized gateway into the little network, as well as the tons of third-party plumbing between these components. But this allows us to simply visit our URL from any device and just sign in, all without having to remember a password.

Unbeknownst to disinterested users, the random identifier of their little cloud simply encodes its Yggdrasil address. Yggdrasil is similar to a VPN like Tailscale, in that it allows nodes across different local networks to talk to each other. However, unlike Tailscale, it has no centralized coordination server. Who gives out names to each node then? They pick their own random names. Who tells nodes what other nodes are on the network? They gossip with each other automatically to discover peers. The subdomain namespace in the vignette above simply acts as a convenient gateway into the Yggdrasil network, analogous to Tailscale Funnel or Cloudflare Tunnels.

resilient

Okay, so you can head over to a URL and sign into your little cloud. But you’re still at the mercy of various corporate and government actors when trying to remotely access your own tiny cluster or that of your friends. That doesn’t feel quite right. Fortunately, it only takes ten minutes to make your connection truly resilient, and you’d be guided step-by-step along the process.

First, install Yggdrasil to be able to bypass the gateway and connect directly to little clouds. Second, register a private certificate authority in your device settings, to be able to recognize certificates burned into the little clouds directly. Third, configure your device to use your little cloud’s own DNS resolver to be able to reach the same domains as before. Assume further that you have a wired, WiFi, or long-distance radio connection to a little cloud, and that it has been equipped with long-distance radio to reach your own one over the mesh network.

With such a setup, you’d be able to sign into your little cloud or those of your friends in the same exact way you were doing before. Only that in this advanced mode, you wouldn’t encounter any of the traditional third-party middlemen of mainstream internet. And it’d be okay to ignore this more involved process until the situation actually calls for it. Until then, just go to your URL and press a button to sign in.

binding

What about the initial sign up before being able to sign into your account? One option would be to allow the owners of a little cloud to configure it such that anyone who can reach it can create an account. I’d personally not be too fond of this setup, but people should be free to decide what they’re willing to do with the devices they own.

Another option would be to take advantage of the inherent access controls inherent to the physical device. If it’s running inside a home, for instance, physical access is already restricted to a trusted circle of family and friends. So what if you had to initially be present near little cloud for getting access to it? Inherit the culture-specific access controls implicit to different types of spaces directly.

Maybe to bind your client device to little cloud, you initially need a wired connection to one of its many ports. For adding new devices, maybe you either connect your already-paired and not-yet-paired devices at the same time. Or scan a code between them. Either way, identity should remain federated to avoid having to “sign in with” a centralized identity provider.

setup

Before you can sign into your account, and before you can create an account, you need to first turn your little cloud on and configure it in the first place. This boils down to connecting it to the network and adding your old electronics as worker devices.

There are a few options I considered for a setup wizard. Maybe you turn a worker device with a screen into a control panel. But that requires you to have already recruited it and to have such a screen-enabled worker device. Maybe you hook it up to a monitor or TV to double as a media system. But then you might also need a keyboard, which gets awkward. I’d like to avoid dedicated WiFi or Bluetooth for the first version, which rules out access points and smart-home-like apps as setup pathways. On-device screens and buttons might be frustratingly tiny on a 1U-tall rack-mountable box.

The more promising option is inspired by reMarkable’s tablet ecosystem. It has this interesting web interface that you can access from any browser once you have a wired connection to the tablet. People could use their existing daily-driver devices to configure little cloud by simply connecting to it via a single USB cable and visiting a URL. There, you could handle these initial device setup issues, such as adding worker devices or using their existing radios to connect to WiFi or mobile data.

These first few log entries will continue to break down the concept of a device that turns your old devices into a personal cloud.

One option I’ve considered for the form factor of “little cloud” is that of a network-attached storage unit, or NAS. These typically house a bunch of HDDs or SSDs in neat little bays. The idea here was that you could slot in your old smartphones, tablets, and laptops into dedicated bays. The upside being that it would wrap the mess of different devices into a visually coherent system of matching containers.

However, there are quite a few downsides. First, each worker device would require its own bay, increasing the manufacturing costs of a product that is explicitly about reviving e-waste. Second, because smartphones, tablets, and laptops would require bays of different sizes, it would add friction to the process of integrating a patchwork of worker devices. Third, I don’t think wrapping each device into its own container is too useful for airflow and cooling.

stations

Another possible form factor would be the general category of charging stations, docks, and stands. Many of these are already trying to accommodate a range of consumer devices with different form factors, if you think about it. They also tend to have many ports, such that they can physically connect to each and every device, though newer ones lean into wireless charging.

In contrast to the bay systems, these charging stations are quite efficient in terms of material. Each device barely gets a flap to rest on. Because of that, they’re also not too prescriptive about exactly what kind of device you’re allowed to connect where.

When it comes to cooling, I’m not too sure. This would be a situation where the components that dissipate heat (i.e., the worker devices) would be somewhat outside of our device’s main enclosure, merely resting on top of it. Contrast this to desktop cases or rack-mounted servers, where the hot components are often in the middle of an air tunnel whose geometry has been carefully tuned. Exposing junk electronics right on top might also create some amount of visual clutter, though maybe at least cables could run down the back side.

racks

In the previous log entry, we emphasized the overlap in responsibility between little cloud and traditional enterprise equipment, such as network switches. Rack-mountable equipment has been optimized to allow for connectivity between densely packed devices. Accordingly, they tend to boast dozens of ports on their front and back panels, always accompanied by clear labels and grouped using visual cues.

It would be nice if little cloud adopted the general form of a charging station, but whose base housing happened to also be compatible with rack-mounted standards. This would allow it to be used at home by placing small consumer e-waste on top of it, but it would also allow it to be used in actual racks filled with enterprise e-waste.

The latter would make this product slot right into a related initiative I’m involved with, where we’re redirecting data center e-waste into high schools and universities for students to run their own local platforms on them. We’d get a unified plug-and-play device for reviving e-waste ranging from smartphone all the way to server caliber. That level of consolidation would be pretty sweet.

process

3D printing the enclosure won’t cut it, both because it doesn’t scale favorably with volume, but also because we might be dealing with quite a bit of power inside this enclosure. Typical filament isn’t really well-suited for this, and this would turn into a real liability during product safety certification. Injection molding is how plastic enclosures scale instead, but the upfront cost for a production run is prohibitive.

Sheet metal is quite interesting here, it’s a bit like folding cardboard but instead with aluminum or steel. Excellent thermal conductivity. Cheap to prototype, but also scales favorably with higher volumes. If our geometry boils down to a basic box with maybe some flaps and legs, then manufacturability wouldn’t be an issue. We don’t really need the ability of 3D printing to create intricate internal structures.

I found some fascinating precedents from Teenage Engineering that exploit the similarity between sheet metal and folding cardboard. Their computer-1 desktop case ships with metal sheets that they’ve cut with a CNC, but that they haven’t yet actually bent. The end-user is able to bend the sheets and screw them together, probably creating some degree of Ikea effect in the process. They did the same with their pocket operator modular series, they ship the flat sheets and call on the end-user for the final assembly. Such a DIY option might simplify manufacturing and delivery.

modularity

For computer-1, the top handles mount to a grill pattern drilled into the metal sheet. I think it was just a neat way for them to mount the handles without digging further holes outside the regular grid. However, think about what such geometry could enable in our case. You could move the flaps for worker devices around freely, depending on their unique shapes and sizes. You could also take them off entirely if doing a rack-mounted setup with servers only.

If mirrored on the bottom side, the hole matrix could also allow the legs to be adjusted. This could make little cloud sit well on either a flat surface or an old desktop that has been recruited into the cluster, as well. Internally, you could also mount stand-offs against the same grill to hold circuit boards, fans, and other components. Not to speak of the original purpose of the computer-1 grill, which was probably airflow.

Pulling these threads back together, we get to what looks like a somewhat large charging station built out of sheet metal, whose base enclosure happens to be rack-mountable. Its top and bottom sides are perforated to allow for flexible mounting options, while its front and back sides have clear labels that organize its many ports. Finally, it might be available in the form of a cheaper DIY kit that you assemble yourself.

This newsletter is about the journey of bringing to life a physical product from scratch, from concept all the way to manufacturing.

The product we’ll be building along the way is a device that allows you to take your existing, old electronics and instantly turn them into your own little cloud.

Think about all the smartphones, tablets, laptops, and maybe even desktops that are sitting in your junk drawer and simply collecting dust. What if you could connect these devices you already own into a cluster of tiny computers that can host all kinds of platforms for you? What if you didn’t have to be a sysadmin professional to do it, and instead you had a plug-and-play experience? Maybe you could get these to handle cloud storage, music streaming, or even language models, all without relying on subscriptions to Big Tech. That’s the pitch, that’s what we’re building.

background

If you’ve got a bunch of old devices stuffed in a drawer somewhere, you’re not alone. Surveys estimate that around four out of five US residents hoard unused smartphones. For tablets, the figure is around two out of three. Similar for laptops. That’s a lot of devices.

Even when electronics are recycled at authorized e-waste processors, less than 2% actually get reused in the EU. Most of them are reduced to raw metals, even if the majority of devices are still functional.

You can also ask language models to deep research the total processing power of hoarded devices and e-waste, and compare that to Big Tech data centers. It turns out that the sum total of consumer junk and e-waste lands comfortably above the largest data center, and around the total amount of compute across flagship AI data centers. But it is much more scattered and diverse, and so difficult to coordinate.

Now, you might be skeptical that your smartphone can compare to a GPU. Let’s use a concrete example, take my Pixel 8 from a few years back and an Nvidia GTX 1070 I pulled from a desktop. Both of them can have around 8 GB of GPU-available memory. The Pixel’s GPU has around 30x more half-precision throughput, while at the same time drawing 30x less power. However, the Nvidia card has 5x higher bandwidth between its memory and the actual cores, and much better cooling. It’s fair to say they’re at least trading blows, and both can run tiny language models alright.

features

Let’s get back to the product we’ll be building. To recap, “little cloud” will be a device that can physically connect to a bunch of other devices with wildly different form factors and computational resources.

On a hardware level, this new device will be responsible for handling the power, network, control, and cooling of the tiny computers attached to it. It effectively needs to bundle together the roles of several traditional pieces of enterprise equipment: a power distribution unit, a network switch, a KVM switch, and a coolant distribution unit.

On a software level, it will be responsible for automatically “recruiting” the diverse worker devices into a unified cluster, across which it can then orchestrate workloads to ensure users gain access to the platforms they want. It effectively needs to handle the job of traditional on-premises tooling used to define and run bare metal fleets, such as Talos and Kubernetes.

We’ll partially rule out some features to preserve focus for an initial version. First, we’ll leave an expansion slot for an eventual long-distance radio module, such that people will be able to connect their tiny personal clusters into a “little network” without relying on their internet service providers. Second, we’ll avoid adding connectors specific to solar power, but we’ll make sure portable panels with USB-C outputs can still be used to help people partially power their little clouds without relying on the grid.

development

We want to build towards a collection of files that can then be sent directly to manufacturing houses. These standardized files taken together specify the entire product, from the enclosure through the printed circuit board to the firmware, and allow the product to be manufactured reproducibly across fabs. This is known as “digital fabrication,” and allows us to avoid having to rely on any one particular manufacturing business.

We’ll use Onshape for modeling the enclosure and KiCAD for the printed circuit board, mostly because they’re the only CAD tools I know for these respective jobs. Here’s a picture of an open frame rig for which I used Onshape to model the mounts for the computer parts, it was pretty okay to work with. We’ll use Rust for low-level software, as it seems to have a decent path to native hardware on mobile devices, and because it’s the only language I used to get language models to run on consumer devices. It was GPT-2 back then, simpler times.

The design files would be open. The litmus test for openness here is whether someone has all the files they need get their own device built directly with a turnkey manufacturer. However, economies of scale help drive down costs significantly at higher volumes, and so people should be able to participate in those savings. We’ll probably aim for a first real production run through a platform like Crowdsupply, which helps open hardware projects with crowdfunding and fulfillment logistics.

format

I’m pretty excited about building this in public, both for the accountability pressure and as a way to meet people who might be interested in this little device. I’ll try sharing weekly build logs in here, first time publishing on Substack. We might occasionally visit places like makerspaces, e-waste processors, manufacturing houses, or repair workshops to mix things up.

If you’d like to follow along the full journey of bringing a product to life, including different kinds of engineering that go into it, consider subscribing! If you’re curious about this actual product and want to steer its development, use the comments to share feedback and suggestions. Either way, looking forward to seeing you around!