it is completely bonkers that the internet works at all.
a brief summary of the internet: computers connected to the internet are assigned an IP address through systems out of scope for this summary. those IP addresses belong to networks, which are groups of contiguous addresses. those networks are controlled by systems that operate independently of each other (or perhaps "autonomously"). we call them "autonomous systems", and we give them numbers. by virtue of sending emails and money to the right people, i am one of these, and my number is 64241.
the autonomous systems know what their own networks are, but they don't inherently know what anyone else's are. the other systems have to tell them. because there are a lot of networks, this is not something you can do by hand, so there is a protocol for that called BGP:
The Border Gateway Protocol was sketched out in 1989 by engineers on the back of "three ketchup-stained napkins", and is still known as the three-napkin protocol.[3]
the idea is simple: you set up a connection between two autonomous systems for the purpose of telling each other about your networks, and then you route traffic to those networks via that connection. in a lot of cases, this means running an actual physical cable from your servers to someone else's servers at a data center. there's only room for so many cables, so this gets expensive fast.
some recent innovations have made hobbyist networks (like mine!) possible. the main one is Internet Exchange Points, or IXPs. instead of setting up a connection between just two systems, you create a network of interconnected autonomous systems that can all talk to each other; by joining an IXP, you can now talk to dozens or possibly hundreds of networks over BGP with a single network cable. all you have to do is send an email to another network administrator and ask to set up peering over the IXP you're already both part of.
another innovation is the route server, which helps do away with even having to work up the confidence to send another sysadmin an email. if you advertise your networks to the route server, it'll advertise it to everyone else who is connected to it. up in seattle, you can directly talk to a decent chunk of the internet just by doing this.
you still can't talk to the whole internet just by connecting to an IXP; at some point you're going to want to pay someone to get the rest of the internet (and let the rest of the internet reach you). this is known as "IP transit", and is usually billed based on usage. this creates a monetary incentive to run as much traffic as possible over free peering arrangements, very often at IXPs.
and yet, sometimes, that monetary incentive isn't enough:
- although AS 64241 peers with some very large companies — e.g. amazon (16509), facebook (32934), microsoft (8075), apple (714) — some networks (e.g. fastly, 54113) prefer not to directly peer with small networks despite it just being a reconfiguration when you share an IXP. (they do advertise on the route servers, though, so traffic is still direct between the two of us.)
- google (15169) has completely unhinged peering requirements and no real NOC support, which is a reflection of everything else google does. they require outdated MD5 signatures using a shared secret; they won't peer with you over IPv6 if your router responds to full-MTU pings with fragmented packets (explicitly permitted by RFCs) because fragmented ICMP is dropped at their firewalls; they require you log into a proprietary portal every six months to click some buttons to verify your contact information. we de-peered google months ago because they don't behave like any other network.
- and, recently: we notified cloudflare (13335) about a month ago that we terminated peering with them. i've been perpetually irritated at how cloudflare acts about the internet for years, but defending hosting hate speech (and doubling and tripling down) was enough for me to yank them out of bgpd.conf (and also configure my IXP's route server's equivalent of "don't talk to me or my router ever again").
and it's not just me out there de-peering networks because i feel like it. far larger networks have done this for far less.
anyways i don't really have a conclusion here. i just wanted to talk about how weirdly... social? maybe even political? a fundamental core part of the internet's infrastructure is. some networks just plain don't talk to each other and you have to take another route.
