Yannblog

UPDATE 4: Added a Kerrville Daily Times link for a new article on how 911 is handled. Lots of good information there.

UPDATE 3: Added another news source. Swamped by homework so tweets will have to wait, but other than those I think I’ve got all the online info on the outage linked from here.

UPDATE 2: Added another couple of news sources, corrected one of my usual typos…some how I forgot to add a “not” into the paragraph about Wal-Mart taking credit card transactions. Sorry for the confusion. Also modified information based on an article from the Fredericksburg Standard Radio-Post. The link below takes you to the subscriber-only version of the article; the non-subscriber snippet just makes the paper look bad (very little info).

UPDATE: Added some more news articles, (still) working on collecting all tweets about the incident, revised some information for accuracy (originally wasn’t sure about CricKet and Pocket, but they too were taken down by the outage…DSL was also apparently iffy in Kerrville).

Granted, I wasn’t in the area when it happened, but since I have plenty of family and friends down there I’m certainly going to talk about it!

Kerrville Daily Times Articles
March 24th Front Page (JPEG image)
Photo
Editorial
Note about landlines/safety
Info about 911 operations
Main article (covers cellular, credit cards, banks) Reprint in iStockAnalyst
Businesses adapt (anyone remember credit card imprint machines?)
Humorous recap by one Daily Times reporter (not a newspaper article)

MySanAntonio
911 article
Original article

Articles from other outlets
KENS5 KVUE WOAI WOAI VIdeo KSAT12 Boerne Star Burnet Bulletin Fredericksburg Standard Radio-Post Blanco County News TMCNet

Rather scary stuff for the folks affected. The outage, by my calculations, lasted for a full seven hours, and knocked out cellular communications completely in the Hill Country. Verizon cells? Down. Sprint? Down. AT&T? Dead. T-Mobile? Not 100% sure, but probably gone. HCTC phone service? Nada. Pocket and CricKet were dead in the water as well.

Oddly enough, the Fredericksburg Standard Radio-Post cites Five Star Wireless as not being down during the incident, for local or long distance service. This on one hand makes sense, as Five Star doesn’t necessarily use phone-company T1 lines to power their cell sites like other providers do. However I tried calling a Five Star connected friend yesterday during the outage and couldn’t get through. I’m going to keep poking around for more info on what happened with that provider.

On the landline front, you could call to your own exchange but no further. This was the case for both Verizon and Windstream…and, as remarked before, HCTC. Time Warner Cable digital voice users apparently had the reverse issues; able to call out but unable to call folks in town…just like everyone else in the world. Sprint’s IP-based long distance also stayed online…it’s what powers Time Warner Cable’s digital voice service (at least for the moment…see this article) and I suppose runs on TWC’s network in the Hill Country.

Internet was more of a mixed bag. Time Warner Cable’s fiber route is different from that of the AT&T cable that was cut (near Waring), so their service, including service for the City of Fredericksburg, continued unabated.

Windstream DSL also continued without issue in Fredericksburg; my mom’s workplace uses them and was able to send an e-mail out about the incident. The phone lines on which the DSL was served couldn’t call beyond Fredericksburg, but internet connectivity did just fine. Oddly enough, Kerrville DSL didn’t weather the outage with as much gusto, according to reports by the Kerrville Daily Times.

Verizon DSL was flat-out down. Bee Creek Communications, riding on that AT&T fiber for their backbone, was out as well. On the flip side, Hill Country Telephone Cooperative’s dual internet links (one to the north, one to the south) didn’t miss a beat. As such, Kerrville’s school system was also unaffected; they use HCTC.

The impact of the outage went a bit beyond calling from point A to point B. Gas stations and Fredericksburg’s Wal-Mart could not take credit card transactions. I’m guessing Wal-Mart has a T1 internet connection, and their T1 was down along with everyone else’s (Sprint and AT&T’s cellular backhauls, for example). The gas stations were probably using plain old telephone lines to process plastic. For some businesses, old-style credit card imprint machines were dragged out and used to process transactions, but for many businesses those machines just don’t exist, relegating purchases to cash or check only.

Last but not least, some banks stopped early for the day (the outage started around 3 p.m. Central Time), as they also tend to use T1 connections to get online. Other banks just processed transactions manually, following procedures similar to blackout protocol.

This just goes to show how much the Hill Country’s communications infrastructure relies on a single provider. Pretty scary if you ask me. It also shows that voice over IP, combined with a real redundant network, is just a good idea; with VoIP it doesn’t matter if you’re running on cable, DSL or mobile broadband, as long as you’ve got access to the Internet. It’s fortunate that the service came up when it did, that TWC, HCTC internet and Windstream DSL customers were apparently not affected, and that emergency services were quickly restored in the area. However running all communications through a single AT&T fiber link (as opposed to a ring, or at least some sort of redundant route!) is just plain bad policy.

Comments/thoughts/additions? Post ’em!

Tweets compiled about the event (more coming):

Windstream – @bmodisett @kerrvillegossip yes there is an outage; currently no ETA ; fiber cut by another carrier^MC
HCTC – Another good reason to be on Hill Country’s Ethernet backhaul service! Listen up wireless carriers! Can you spell “Diverse Routes”. We can! Call me!!! http://bit.ly/bpPHZU
KerrvilleGossip – Phone service down,, Windstream to blame…. working on the story, looks like HCTC is about to be the BIG DOG! #kerrville (retweeted by bmodisett)\
KerrvilleGossip – Cell service is starting to come back! GO KERRVILLE!

bmodisett – what is up with #Kerrville and the cell phones, it was like they pulled the plug for everyone here.
MrAdkins – Any 3G outages going on out there in twitterland?
tsampson – Anyone in Kerrville experiencing cell phone/land phone outages? Please at reply for potential story on this crazyness.
bmodisett – @tsampson i guess windstream finally had it and are leaving town.
tsampson– @shueytexas Working theory is that Aliens have landed, sending out a electromagnetic pulse… or something to that effect.
tsampson – @mradkins Also, have you heard anything from any carriers? I went over to Windstream and Verizon. No cause yet.
mradkins – @tsampson Just in from Sprint: there is an AT&T Outage affecting both cell and landlines west of Boerne on I-10 and north to Dallas.
mradkins – @tsampson But not sure if this is true. One carrier will blame another. WS won’t say they have a problem even if you drop by.
tsampson – So it’s Apocalypse Cell in the Hill Country today. Read all about it in the Daily Times’ team coverage tomorrow.

KENS5 – Communications cut in Texas hill country, services impaired http://bit.ly/b0QhIz
KVUE – Communications cut in Texas Hill Country, services impaired: It’s a terrible Tuesday for some people living in the… http://bit.ly/bDnGdc
News4WOAI – Major communication outage in Hill Country area: News 4 WOAI has confirmed that a fiber line was cut somewhere bet… http://bit.ly/awvd1X (retweeted by barbaramagana)
KABBFOX29 – Cut Fiber Line Causes Large Outage in Hill Country: A contractor working on roadside drainage causes a major…
http://tinyurl.com/yh3fku5

phlezk – DUDE WTF? phones (cell AND LANDLINE) in the Hill Country (my home!) are DOWN?! HOW THE–?!
phlezk – News 4 WOAI confirmed a Verizon fiber line was cut somewhere between Fredericksburg & Comfort, causing outages throughout the Hill Country.
that1collegeguy – @phlezk Dude, the entire cell network is like…non-existent at the moment!
that1collegeguy – NEWS FLASH: Texas Hill Country economy collapsing rapidly due to cell & landline networks crashing. LifeAlert reports record “pushes”!
BobGambert – Anybody up in the Hill Country? Talk to me about the major comm outage.
bbridges620 – AT&T says it is Verizon’s fault phone are down in the Hill Country, Verizon says it’s AT&T. What is up with Red Vs Blue everywhere? (retweeted by Brandy_Williams)
jfstx – all verizon communications down in Hill Country, no cell, home service , sprint is limping along on 1xrtt
priskillya – the entire hill country was cell phone and land lineless today. i also had no internet. that’s ok cause i had to move. it was just a pain
katiewright – cell service is finally FINALLY back up after about seven hours of being down in the entire Hill Country area.
2dogexpedition – phones in hill country down someone cut a fiber optic
metroane – this fn blows…….major phone, cell, internet services out since 1pm in hill country, can’t upload mag to press……

TaxesNews4u – San Antonio News: Widespread phone outage in Hill Country: Kerr, Gillespie counties are among the areas affected http://bit.ly/aW1t1y
SanAntonioCP – San Antonio: Major communication outage in Hill Country area http://bit.ly/dw8pUi
tweetsa – Widespread phone outage in Hill Country http://bit.ly/ayMHTA
salocalnews – Major communication outage in Hill Country area – http://fx.ly/a/16_78e8750bdc
salocalnews – Widespread phone outage in Hill Country – http://fx.ly/a/16_ad169f14d2
MsTechAngel – Widespread phone outage in Hill Country: By Zeke MacCormack – Express-News Damage to an AT&T fiber-optic line caus… http://bit.ly/cwMjyH
JeffVaughn – Breaking: Parts of Hill Country experiencing phone outage, including #Kerville: http://tinyurl.com/ydvwm2m
wrstech – Major communication outage in Hill Country area http://shar.es/mgfnZ
wxProBrian – Red vs Blue! RT @ksatnews: Now, Verizon tells us it’s an AT&T Fiber line that was cut – causing phone/cell/net outages in Hill Country.
GSMTalks – GSMtalks :: Service returns after widespread phone outage in Hill Country http://bit.ly/bml2pm
SanAntonioLocks – Phone Lines Out In Hill Country: A severed AT&T fiber line has cut cell, land and Internet connections in the Hill… http://bit.ly/b8Q9cj