i was a very early employee at one of the first two commercial US ISPs, UUNET.
all of us were using cisco routers. there was a bug where at about 24-26 hours, the BGP session in them crashed, causing waves of BGP convergence issues, affecting all the tier 1 providers.
the solution was to reboot carefully at about 22 hours. we waited desperately for cisco to get us a patch.
we were at a usenix conference in the terminal room at about 11pm when we got an email that they'd isolated the bug and thought they had a fix. at this point, the senior router folks for UUNET, MCI, and Sprint sat in the terminal room, waiting.
about 1:30am we got email that tony thought he'd fix it. we asked how much testing he'd been able to do.
tony: "it booted in the lab"
us: "we'll take it!"
then, all 6 of us downloaded the code and proceeded to reboot about 70% of the internet backbone to start using the new code, all of UUNET, MCI, & Sprint.
fortunately, in those days, cisco let their senior engineers talk directly to customers and they got things fixed fast.
this never made the newspapers at the time. i can't even imagine any ISP these days allowing engineers to reboot their entire backbone from a terminal room with code that "booted in the lab".
kind of miss those old days.
@paul_ipv6 @blogdiva @Viss
> the solution was to reboot carefully at about 22 hours
Early Internet: The Battlestar Galatica Era
oh, we played fast and loose. ;)
cisco didn't have a rackmount solution for AGS+ or early 7000s that worked for center mount relay/telco racks. we were going into a sprint facility behind the san jose airport and it was only 18 months after the '89 quake. we were told that all gear must be firmly mounted in the racks and that shelf mounting wasn't acceptable or sufficient.
so, we went to the local hardware store, got webbing strap clamps and xmas present "wrapped" our ciscos with webbing straps to the rack shelves. problem solved.
@paul_ipv6 @inthehands @Viss you don't even find a lot of those hardware stores anymore. and don't get me started about losing Radio Shack
i'm not sure how i'd do pop installs in the bay area any more. no weird stuff. no frys. greybars doesn't have everything. ma & pa hardware stores are gone. back then, i also had access to the DEC lab on university to build cables, etc. all gone.
techbros made sure with smart phones they'd capture generations completely indebted to their tech fiefdoms. with billions ignorant as to was it ROOT can't worry about them finding out what they have in their pockets is like the computers they put on their desks or laps. and now those techbros are getting rid of all those developers and engineers that built said fiefdoms, thanks to the shiny new AI. we really need to go back to the days of hacking everything.
@inthehands @paul_ipv6 @Viss for reals, LMAO