Designing an Ethernet Switch ASIC

It seems I have just built the world’s first open source switch ASIC and I am getting it back from the fab mid november.

Here is its repository :

Buckle up and welcome to the tale of more madness.

Where are the open source networking ASICs?#

Recent FCC decisions have put the central importance of networking equipment top of mind, and in doing so, have also put the total absence of any entirely open source networking equipment hardware at the top of my mind.

Although Open Source Silicon is in its infancy we are currently seeing a number of projects being designed, tested, and for the most ambitious ones, even taped-out with some proven silicon already in the wild.

That said, the vast majority of the most ambitious projects are predominantly RISC-V SoCs.

Surprisingly, there has been much less interest in building open hardware for networking equipment.

Well .. unsurprisingly actually … networking equipment is far less “sexy” than CPUs and has thus received much less attention from the open source community. On the other hand, this means there’s a huge untaped boulevard of projects open to anyone with more time than common sense to build open source networking equipment chips!

Because, on the other side of the great silicon divide, the world of open source networking equipment is actually quite rich, both in terms of its flourishing software ecosystem and the open PCB/electronics ecosystem, with a flurry of fully featured open source routers available. Yet, due to the current ecosystem’s limitations, under the hood, these are all still all running closed-sourced, blackbox proprietary chips for the central compute and routing tasks.

So what would it take to build a router chip?

It’s a very ambitious project, since a router involves a complex SoC needing both a powerful enough CPU to run the networking stack alongside specialized networking hardware, and analog frontends for the wired and wireless connections. Since building a full router outright is much too ambitious of a greenfield project for a single person to ever hope to pull off, let’s begin with a more approachable first step: building a switch. After all, although most people only have one router, you can have a number of smaller switches, and we also don’t have any open source chips for those.

Oh and, I am not talking about building an FPGA switch, oh no, I am talking about building a switch ASIC, taping it out and then proving the silicon works.

Switch Flavors#

Before we start figuring out what kind of switch we want to build, let me give you a quick overview of the different flavors of switches out there. Readers deeply familiar with networking equipment can skip this part.

Glossary#

Physical Layer#

Ethernet supports multiple physical layers, outlined in the 802.3 IEEE spec. Each clause in the spec defines the underlying medium characteristics for carrying the Ethernet protocol over the medium at a given bandwidth.

Both a 10 Gb fiber Ethernet link and a 10 Mb coaxial cable can carry Ethernet packets and, to the layers above the physical layer, the packets will look exactly the same.

Where they will differ is at the physical layer and, outlined in the spec is precisely how, with which timing and with what encoding data will be transmitted over the medium.

This ensures that devices made by different manufacturers agree on what “talking Ethernet” looks like, making them interoperable.

So one of the first questions I must answer is which physical layer my switch should support. This will define how much traffic I must route, how fast I should do so and how much bandwidth I can carry.

Managed vs Unmanaged#

The second question is: what type of switch do I want to build?

There are two big families of switches: managed and unmanaged. As the name implies, managed switches can be configured and managed from the outside. This allows for much smarter routing, such as supporting different VLANs.

While unmanaged switches are essentially unconfigurable pieces of networking equipment that you plug into your network and just work (until power surge do you part).

Cut-through vs Store-and-Forward#

The last switch characteristic is cut-through versus store-and-forward.

Cut-through switches start forwarding a packet while the packet is still arriving, leading to much lower networking latency, whereas store-and-forward switches wait for the entire packet to arrive before checking that no corruption has occurred, and then only forward the packet if that check passes.

Where issues arise is when a packet is corrupted, a cut-through switch might still forward it, propagating corrupted packets through the network since forwarding began before the FCS (frame check sequence) could be checked.

Designing against constraints (again)#

As usual with my ASIC design work, what I build is shaped as much by what I want to build as it is by the constraints of the silicon.

Pins#

My first major constraint is the pins: not only in their amount but also in their maximum bandwidth.

Since I will be taping this first generation chip out over the Tiny Tapeout shuttle chip, using purely digital tiles, I’m constrained by the limits of these pins.

In total this will afford me 24 pins: 8 input, 8 output, and 8 bi-directional (configurable to be either inputs or output pins) GPIO pins, rated to run reliably at 50 MHz on both inbound and outbound data.

The absence of any analog front end makes building anything IEEE-physical-layer-compliant directly a challenge, but there’s a way around this: using an external PHY (physical layer) chip and interfacing with it over the standardized RMII bus. Though this consumes 7 bits per Ethernet interface it makes my 50Mbps pins capable of sending and receiving over 100Mbps Ethernet (100BASE-TX).

| Index | Input | Output | Bidirectional | |---|---|---|---| | 0 | ui[0] | uo[0] | uio[0] | | 1 | ui[1] | uo[1] | uio[1] | | 2 | ui[2] | uo[2] | uio[2] | | 3 | ui[3] | uo[3] | uio[3] | | 4 | ui[4] | uo[4] | uio[4] | | 5 | ui[5] | uo[5] | uio[5] | | 6 | ui[6] | uo[6] | uio[6] | | 7 | ui[7] | uo[7] | uio[7] |

I will be targeting the widely available Microchip LAN8720A/LAN8720AI (which will be the only mention of AI in this article) PHY chip for this interface. I will be going more in-depth as to how this ASIC will be interfacing with this PHY later.

Area#

My second major constraint is my limited die area.

I am paying for all this out of pocket afterall, and more area means a higher manufacturing cost. Now I am not trying to accumulate a pool of gold or anything, it’s just that, in compliance with Maslow’s hierarchy of needs, waffles rank higher than area, and higher manufacturing cost means less waffles. So just like any semi-conductor company, I am incentivised to keep my area budget under control.

Because store-and-forward requires the switch to store the entire packet before forwarding it, and because ethernet frames can reach upwards of +1.5k Bytes (and +9k for jumbo frames), they require massive amounts of storage. A workaround for that would be to store the packet to some off-chip memory but I don’t have the pin budget to afford that right now, so on-chip memory is my only option.

The problem is, on-chip memory consumes a lot of area given the large area footprint of SRAM and the even larger area per stored bit footprint of flip-flops.

Now if we were talking of a few bytes this could be negotiable, but for multiple 1.5k Bytes packets this is a dealbreaker, thus ruling store-and-forward out.

Assuming I was using Tim Edward’s excellent OCD 256x8 SRAM IP, a single 256 Byte SRAM occupies 301.3 x 224.93 um (using W orientation in this implementation) and consumes just by itself 1/3 of the floorplan. If we want to store even a single full packet we would need 6 of such instances, and since we have 3 ports, we would then need to replicate that 3 times, so 18 instances in total.