UPDATE 3: Added another news source. Swamped by homework so tweets will have to wait, but other than those I think I’ve got all the online info on the outage linked from here.

UPDATE 2: Added another couple of news sources, corrected one of my usual typos…some how I forgot to add a “not” into the paragraph about Wal-Mart taking credit card transactions. Sorry for the confusion. Also modified information based on an article from the Fredericksburg Standard Radio-Post. The link below takes you to the subscriber-only version of the article; the non-subscriber snippet just makes the paper look bad (very little info).

UPDATE: Added some more news articles, (still) working on collecting all tweets about the incident, revised some information for accuracy (originally wasn’t sure about CricKet and Pocket, but they too were taken down by the outage…DSL was also apparently iffy in Kerrville).

Granted, I wasn’t in the area when it happened, but since I have plenty of family and friends down there I’m certainly going to talk about it!

Kerrville Daily Times Articles
March 24th Front Page (JPEG image)
Photo
Editorial
Note about landlines/safety
Info about 911 operations
Main article (covers cellular, credit cards, banks) Reprint in iStockAnalyst
Businesses adapt (anyone remember credit card imprint machines?)
Humorous recap by one Daily Times reporter (not a newspaper article)

MySanAntonio
911 article
Original article

Articles from other outlets
KENS5 KVUE WOAI WOAI VIdeo KSAT12 Boerne Star Burnet Bulletin Fredericksburg Standard Radio-Post Blanco County News TMCNet

Rather scary stuff for the folks affected. The outage, by my calculations, lasted for a full seven hours, and knocked out cellular communications completely in the Hill Country. Verizon cells? Down. Sprint? Down. AT&T? Dead. T-Mobile? Not 100% sure, but probably gone. HCTC phone service? Nada. Pocket and CricKet were dead in the water as well.

Oddly enough, the Fredericksburg Standard Radio-Post cites Five Star Wireless as not being down during the incident, for local or long distance service. This on one hand makes sense, as Five Star doesn’t necessarily use phone-company T1 lines to power their cell sites like other providers do. However I tried calling a Five Star connected friend yesterday during the outage and couldn’t get through. I’m going to keep poking around for more info on what happened with that provider.

On the landline front, you could call to your own exchange but no further. This was the case for both Verizon and Windstream…and, as remarked before, HCTC. Time Warner Cable digital voice users apparently had the reverse issues; able to call out but unable to call folks in town…just like everyone else in the world. Sprint’s IP-based long distance also stayed online…it’s what powers Time Warner Cable’s digital voice service (at least for the moment…see this article) and I suppose runs on TWC’s network in the Hill Country.

Internet was more of a mixed bag. Time Warner Cable’s fiber route is different from that of the AT&T cable that was cut (near Waring), so their service, including service for the City of Fredericksburg, continued unabated.

Windstream DSL also continued without issue in Fredericksburg; my mom’s workplace uses them and was able to send an e-mail out about the incident. The phone lines on which the DSL was served couldn’t call beyond Fredericksburg, but internet connectivity did just fine. Oddly enough, Kerrville DSL didn’t weather the outage with as much gusto, according to reports by the Kerrville Daily Times.

Verizon DSL was flat-out down. Bee Creek Communications, riding on that AT&T fiber for their backbone, was out as well. On the flip side, Hill Country Telephone Cooperative’s dual internet links (one to the north, one to the south) didn’t miss a beat. As such, Kerrville’s school system was also unaffected; they use HCTC.

The impact of the outage went a bit beyond calling from point A to point B. Gas stations and Fredericksburg’s Wal-Mart could not take credit card transactions. I’m guessing Wal-Mart has a T1 internet connection, and their T1 was down along with everyone else’s (Sprint and AT&T’s cellular backhauls, for example). The gas stations were probably using plain old telephone lines to process plastic. For some businesses, old-style credit card imprint machines were dragged out and used to process transactions, but for many businesses those machines just don’t exist, relegating purchases to cash or check only.

Last but not least, some banks stopped early for the day (the outage started around 3 p.m. Central Time), as they also tend to use T1 connections to get online. Other banks just processed transactions manually, following procedures similar to blackout protocol.

This just goes to show how much the Hill Country’s communications infrastructure relies on a single provider. Pretty scary if you ask me. It also shows that voice over IP, combined with a real redundant network, is just a good idea; with VoIP it doesn’t matter if you’re running on cable, DSL or mobile broadband, as long as you’ve got access to the Internet. It’s fortunate that the service came up when it did, that TWC, HCTC internet and Windstream DSL customers were apparently not affected, and that emergency services were quickly restored in the area. However running all communications through a single AT&T fiber link (as opposed to a ring, or at least some sort of redundant route!) is just plain bad policy.

Comments/thoughts/additions? Post ‘em!

Tweets compiled about the event (more coming):

Windstream – @bmodisett @kerrvillegossip yes there is an outage; currently no ETA ; fiber cut by another carrier^MC
HCTC – Another good reason to be on Hill Country’s Ethernet backhaul service! Listen up wireless carriers! Can you spell “Diverse Routes”. We can! Call me!!! http://bit.ly/bpPHZU
KerrvilleGossip - Phone service down,, Windstream to blame…. working on the story, looks like HCTC is about to be the BIG DOG! #kerrville (retweeted by bmodisett)\
KerrvilleGossip - Cell service is starting to come back! GO KERRVILLE!

bmodisett - what is up with #Kerrville and the cell phones, it was like they pulled the plug for everyone here.
MrAdkins - Any 3G outages going on out there in twitterland?
tsampson - Anyone in Kerrville experiencing cell phone/land phone outages? Please at reply for potential story on this crazyness.
bmodisett - @tsampson i guess windstream finally had it and are leaving town.
tsampson- @shueytexas Working theory is that Aliens have landed, sending out a electromagnetic pulse… or something to that effect.
tsampson - @mradkins Also, have you heard anything from any carriers? I went over to Windstream and Verizon. No cause yet.
mradkins - @tsampson Just in from Sprint: there is an AT&T Outage affecting both cell and landlines west of Boerne on I-10 and north to Dallas.
mradkins - @tsampson But not sure if this is true. One carrier will blame another. WS won’t say they have a problem even if you drop by.
tsampson - So it’s Apocalypse Cell in the Hill Country today. Read all about it in the Daily Times’ team coverage tomorrow.

KENS5 - Communications cut in Texas hill country, services impaired http://bit.ly/b0QhIz
KVUE - Communications cut in Texas Hill Country, services impaired: It’s a terrible Tuesday for some people living in the… http://bit.ly/bDnGdc
News4WOAI - Major communication outage in Hill Country area: News 4 WOAI has confirmed that a fiber line was cut somewhere bet… http://bit.ly/awvd1X (retweeted by barbaramagana)
KABBFOX29 - Cut Fiber Line Causes Large Outage in Hill Country: A contractor working on roadside drainage causes a major…
http://tinyurl.com/yh3fku5

phlezk - DUDE WTF? phones (cell AND LANDLINE) in the Hill Country (my home!) are DOWN?! HOW THE–?!
phlezk - News 4 WOAI confirmed a Verizon fiber line was cut somewhere between Fredericksburg & Comfort, causing outages throughout the Hill Country.
that1collegeguy - @phlezk Dude, the entire cell network is like…non-existent at the moment!
that1collegeguy - NEWS FLASH: Texas Hill Country economy collapsing rapidly due to cell & landline networks crashing. LifeAlert reports record “pushes”!
BobGambert - Anybody up in the Hill Country? Talk to me about the major comm outage.
bbridges620 - AT&T says it is Verizon’s fault phone are down in the Hill Country, Verizon says it’s AT&T. What is up with Red Vs Blue everywhere? (retweeted by Brandy_Williams)
jfstx - all verizon communications down in Hill Country, no cell, home service , sprint is limping along on 1xrtt
priskillya - the entire hill country was cell phone and land lineless today. i also had no internet. that’s ok cause i had to move. it was just a pain
katiewright - cell service is finally FINALLY back up after about seven hours of being down in the entire Hill Country area.
2dogexpedition - phones in hill country down someone cut a fiber optic
metroane - this fn blows…….major phone, cell, internet services out since 1pm in hill country, can’t upload mag to press……

TaxesNews4u – San Antonio News: Widespread phone outage in Hill Country: Kerr, Gillespie counties are among the areas affected http://bit.ly/aW1t1y
SanAntonioCP - San Antonio: Major communication outage in Hill Country area http://bit.ly/dw8pUi
tweetsa - Widespread phone outage in Hill Country http://bit.ly/ayMHTA
salocalnews - Major communication outage in Hill Country area – http://fx.ly/a/16_78e8750bdc
salocalnews - Widespread phone outage in Hill Country – http://fx.ly/a/16_ad169f14d2
MsTechAngel - Widespread phone outage in Hill Country: By Zeke MacCormack – Express-News Damage to an AT&T fiber-optic line caus… http://bit.ly/cwMjyH
JeffVaughn - Breaking: Parts of Hill Country experiencing phone outage, including #Kerville: http://tinyurl.com/ydvwm2m
wrstech - Major communication outage in Hill Country area http://shar.es/mgfnZ
wxProBrian - Red vs Blue! RT @ksatnews: Now, Verizon tells us it’s an AT&T Fiber line that was cut – causing phone/cell/net outages in Hill Country.
GSMTalks - GSMtalks :: Service returns after widespread phone outage in Hill Country http://bit.ly/bml2pm
SanAntonioLocks - Phone Lines Out In Hill Country: A severed AT&T fiber line has cut cell, land and Internet connections in the Hill… http://bit.ly/b8Q9cj

So naturally I pulled out my HTC Touch Pro, which has WiFi, to see what network they were running on. After registering for an account on the hotspot provider (Less Networks, never heard of them) I checked whatismyip.com to see who I was on. Sure enough, the connection there was GVTC, registered as “Guadalupe Valley Telephone”. Right before we pulled out of the place I opened up my laptop, logged into the WiFi network that way and did some tests. Here’s what I got for a traceroute:

traceroute to softlayer.com (66.228.118.53), 64 hops max, 52 byte packets
1  192.168.169.1 (192.168.169.1)  194.115 ms  181.762 ms  77.970 ms
2  96-8-192-3.block0.gvtc.com (96.8.192.3)  94.851 ms  107.971 ms  146.974 ms
3  12.88.219.169 (12.88.219.169)  47.516 ms  218.308 ms  149.762 ms
4  cr1.santx.ip.att.net (12.123.154.10)  81.998 ms  58.617 ms  64.528 ms
5  cr2.dlstx.ip.att.net (12.122.30.130)  71.363 ms  55.947 ms  70.918 ms
6  gar6.dlstx.ip.att.net (12.122.100.65)  52.436 ms  58.138 ms  56.583 ms
7  12.90.228.14 (12.90.228.14)  52.971 ms  48.888 ms  80.834 ms
8  border3.tge4-1-bbnet2.ext1.dal.pnap.net (216.52.191.83)  52.188 ms  51.494 ms  52.917 ms
9  te2-1.cer03.dal01.dallas-datacenter.com (216.52.189.30)  55.883 ms  54.237 ms  55.133 ms
10  po3.dar01.dal01.dallas-datacenter.com (66.228.118.209)  53.084 ms  84.979 ms  47.910 ms
11  po1.slr01.dal01.dallas-datacenter.com (66.228.118.138)  66.371 ms  54.266 ms  57.160 ms

I have a feeling that the WiFi conection was flakey, otherwise I would have gotten a better traceroute.

Speed-test-wise I got the following:

I’m guessing the line was provisioned at 8 Mbps down and 1 Mbps up, but I can’t be sure.

The IP address, for what it’s worth, was 96.8.236.103.

All in all, not a bad hotspot, and the traceroute is nice and short between where I was and AT&T’s network in San Antonio, which GVTC appears to use exclusively to connect to the internet nowadays. So someone using an AT&T aircard and GVTC FTTH will find themselves with similar traceroutes within just a few hops, at least around here.

I guess I’ll leave it at that…if I talk any more about GVTC I’ll start getting envious of their 20/3 and 40/10 FTTH service, which I have no chance of getting at my place. Can’t even get DSL there :/

So the big question in my mind is, how would I design the perfect set top box, from both a consumer and a provider point of view? Here are some specs I came up with:

1. nVidia ION chipset. Everyone says the thing can decode 1080p video with no problem. Paired with a decent operating system and a halfway-decent processor (like the low-power Atom 330) you’ve got an STB that not only performs well, but also can be used for a variety of things and isn’t going to run up the power bill.
2. CableCard support, multiple streams. Granted, this is a given these days, but why not make things clear? RF video (via GPON) is what I’m assuming as the last mile medium; IPTV would work a bit differently of course.
3. Easy-access 12.5mm height 2.5″ hard disk bay to turn pa plain set top box into a DVR. Pop in a bare SATA drive and pay a one-time upgrade fee, then start recording shows. Or pay a slightly larger fee plus the price of a handpicked drive and a tech will do all of that for you. Bottom line: there should be no hardware difference between a regular set top box and an HD one, aside from the presence of a hard drive, or lack thereof. If you’re wondering why a 2.5″ drive, it’s because the drive is smaller and takes less power. If people want to store gobs of content they can add an external drive. There might come a time when all video content is cached at the edge of the network (networked DVRs) but until then, let people have as many bits in their box as they possibly can install.
4. Flash storage for non-video, namely the operating system and program guide. You want that stuff to load quickly, so why commit it to spinning metal when you can get enough flash to hold it for under $20?
5. HD on every STB so you can say that there’s no extra equipment required. Plus, having a single model of box makes things a whole lot easier to keep track of.
6. 2 USB ports. Aside from easy testing of devices, this would allow people to view media on their STB by plugging in a memory card reader or something equally quirky-but-cool. More importantly, people would be able to connect a whopping 2TB drive to their STB if they wanted and record shows onto it. The more shows they have recorded, the longer they’ll stick around to watch ‘em, and the longer they’ll pay for service. Also, you can hook…
7. …an external Blu-ray player to one of those USB ports. Make one that fits in with the decor of the typical media center and you’ve got yet another feature that makes that STB “sticky”. Why not? The onboard processor can handle it.
8. Whole-home functionality. No need having multiple DVRs recording the same thing when you can just pull between them. That way one DVR does all the work and the rest sit there with less stuff to break.
9. TiVo as the operating system, with no TiVo monthly fees. If you’re a service provider, you can probably work with TiVo on lowering the “price for life” subscription per box. Also, why use any other piece of software than the best? People say “I TiVoed that” just like they say “I googled that” so why not take advantage of the latent mindshare?
10. Farm out on-demand as much as possible, with an “you’re on our awesome box” commission of a few percent. Nobody wants to spend $8 on a cable-provided movie on demand. Though on the other hand some sports stuff has to be done that way, at which point just download the on-demand stuff over the internet rather than via RF.

The above list might make for an expensive” dumb” set top box, but then again it might now. I’m pretty sure that, when purchased in bulk, all the above components would total less than $200, so if you charge $8 per month per box with the first one free you’re making your money back before the boxes are obsolete. Plus the fact that you’re using a single box across your network will make things a lot easier to deal with for customer service. I’m pretty sure any provider who instituted all of the above would have people running from competitors, even if the service did cost a few bucks more per month. Then again, I’ve never had pay-TV in my life :p