Why Modern Ipv6 Failed This Massive Kubernetes Networking Test

PARIS —When I worked for NASA successful nan 1980s, I helped build a Near Space Network search programme utilizing xBase for nan front-end and Datatrieve connected VAX/VMS for nan backend. When completed, it manually tracked conscionable complete a 1000 fixed web links.

That’s thing — thing — compared to what Deutsche Telekom is attempting to do is create a high-performance emulation level for simulating outer and crushed stations: vast, move connection networks specified arsenic SpaceX’s Starlink.

This is not easy, arsenic Andreas Florath a Deutsche Telekom unreality designer and Matthias Britsch, a Deutsche Telekom elder method expert, explained successful a position astatine OpenInfra Summit Europe 2025.

The problem they look is that while nan mega-constellations of Low Earth Orbit (LEO) and Medium Earth Orbit (MEO) are revolutionizing telecom, accepted web routing protocols specified arsenic Open Shortest Path First (OSPF) and Border Gateway Protocol (BGP) struggle pinch their move topologies — not to mention nan next-generation Internet protocol, IPv6.

The Challenge of Emulating Dynamic Satellite Networks

So, nan extremity is to emulate large-scale, outer mesh networks wherever nan nodes are perpetually moving and falling successful and retired of interaction arsenic they orbit nan Earth and nan world revolves underneath them. Deutsche Telekom’s answer, which is still a activity successful progress, is to build a scalable, container-based testbed tin of reproducing these web dynamics accurately.

The champion consequence to day is simply a record-breaking Kubernetes cluster. The cluster is successfully moving 2,000 pods, each pinch 5 web interfaces, for a full of 10,000 interfaces connected a azygous worker node utilizing Multus, nan multi-network plugin from Red Hat.

As Florath told nan OpenInfra audience, “We’re not alert of immoderate different task scaling Kubernetes to this level.” This accomplishment sets a caller modular for high-density instrumentality networking. It besides offers captious lessons for some endeavor operators and outer web researchers aiming to emulate large-scale, move topologies.

Getting to this constituent was difficult work. Accurate web emulation requires not conscionable monolithic numbers of containers, but complex, changing topologies reflecting real-world node movements. As Florath told nan audience, “These networks person nan spot that nan nodes are moving, they are changing their positions, and today’s routing algorithms are not designed for that.” You tin opportunity that again.

Building a Record-Breaking Kubernetes Cluster

Indeed, successful building their simulation, they recovered that galore web building blocks were not up to nan challenge. For example, nan squad utilized IPv6 for web addresses. You’d deliberation that since IPv6 adoption exceeded 25% web usage globally successful 2020 and each awesome platforms, ISPs and mobile networks person deployed it successful production, we’d person worked retired each nan bugs. You’d beryllium wrong.

Britsch reported that nan Medicube installer, based connected OpenStack’s Ironic, “created wholly incorrect configurations for IPv6.” Even aft configuring everything correctly, nan automated setup consistently produced invalid IPv6 settings, indicating deep-seated bugs successful nan tool’s web provisioning process.

Unexpected Failures successful IPv6 Implementation

The squad besides struggled to usage netboot installation over IPv6. Certain Dell BIOS implementations lacked complete IPv6 footwear support, and erstwhile present, were buggy. This caused footwear stalls aliases failures. The toolchains required manual fixes aliases kernel-level workarounds to alteration reliable PXE/HTTP booting pinch IPv6. Dell did yet spot nan BIOS, mitigating nan problems.

Still, erstwhile each was said and done, they had to build civilization provisioning tooling to make IPv6 activity correctly for their large-scale Kubernetes deployment. Others looking to deploy high-density networking should return note.

The engineers besides faced and overcame terrible bottlenecks that manifested only astatine these unprecedented scales. Limitations included web interface and MAC reside array overflows, vanishing IPs, CPU rhythm misconfigurations for packet processing, and strategy crashes tied to BIOS update challenges. A succession of devices was tried and abandoned—commercial installations specified arsenic Canonical’s offerings faltered successful archiving and reliability, while web automation and civilization disk image creation offered a pathway to a unchangeable platform.

Overcoming Unprecedented Scaling Bottlenecks

Crucially, socket buffer sizes, kernel configuration parameters and web instrumentality reside tables each needed awesome adjustments. The Multus plugin enabled each pod to grip aggregate interfaces, but awesome IP reside guidance issues appeared erstwhile scaling crossed into nan thousands. This prompted nan squad to redesign addressing by making it section to each pod, optimizing kernel limits and switching disconnected immoderate hardware limitations to unit MAC reside handling.

As Britsch noted, nan saw web interface cards limitations “that were caller to me; moreover up-to-date cards could not grip that galore MAC addresses.”

Following months of troubleshooting and incremental improvements, nan setup reached a robust stableness point: 2,000 pods enduring 10,000 interfaces per node for complete 3 months. Finally, arsenic Britsch proudly stated, “We wholly automated everything: installation from scratch, afloat configured stack.”

Achieving Stability and Full Automation

However, further standard attempts revealed new, unresolved bottlenecks, indicating that while nan existent level is capable for simulation tasks, early enhancements will require solutions to further web fidelity and packet processing lag. In short, nan work’s not done yet.

Still, nan squad has yet managed to automate their stack deployment and is exploring integration pinch outer positioning information to simulate move line-of-sight networking conditions. This is simply a captious measurement toward validating next-generation routing protocols specified arsenic IS-IS astatine an orbital scale.

With nan emergence of outer networking and sound services specified arsenic T-Mobile’s T-Satellite, outer telephone service, we request these services to understand our Internet successful nan sky.  The engineers are talking to their bosses astir open  sourcing their deployment scripts and modular footwear solutions, truthful everyone tin use from their work.

Group Created pinch Sketch.