UPDATE 3: Added another news source. Swamped by homework so tweets will have to wait, but other than those I think I’ve got all the online info on the outage linked from here.

UPDATE 2: Added another couple of news sources, corrected one of my usual typos…some how I forgot to add a “not” into the paragraph about Wal-Mart taking credit card transactions. Sorry for the confusion. Also modified information based on an article from the Fredericksburg Standard Radio-Post. The link below takes you to the subscriber-only version of the article; the non-subscriber snippet just makes the paper look bad (very little info).

UPDATE: Added some more news articles, (still) working on collecting all tweets about the incident, revised some information for accuracy (originally wasn’t sure about CricKet and Pocket, but they too were taken down by the outage…DSL was also apparently iffy in Kerrville).

Granted, I wasn’t in the area when it happened, but since I have plenty of family and friends down there I’m certainly going to talk about it!

Kerrville Daily Times Articles
March 24th Front Page (JPEG image)
Photo
Editorial
Note about landlines/safety
Info about 911 operations
Main article (covers cellular, credit cards, banks) Reprint in iStockAnalyst
Businesses adapt (anyone remember credit card imprint machines?)
Humorous recap by one Daily Times reporter (not a newspaper article)

MySanAntonio
911 article
Original article

Articles from other outlets
KENS5 KVUE WOAI WOAI VIdeo KSAT12 Boerne Star Burnet Bulletin Fredericksburg Standard Radio-Post Blanco County News TMCNet

Rather scary stuff for the folks affected. The outage, by my calculations, lasted for a full seven hours, and knocked out cellular communications completely in the Hill Country. Verizon cells? Down. Sprint? Down. AT&T? Dead. T-Mobile? Not 100% sure, but probably gone. HCTC phone service? Nada. Pocket and CricKet were dead in the water as well.

Oddly enough, the Fredericksburg Standard Radio-Post cites Five Star Wireless as not being down during the incident, for local or long distance service. This on one hand makes sense, as Five Star doesn’t necessarily use phone-company T1 lines to power their cell sites like other providers do. However I tried calling a Five Star connected friend yesterday during the outage and couldn’t get through. I’m going to keep poking around for more info on what happened with that provider.

On the landline front, you could call to your own exchange but no further. This was the case for both Verizon and Windstream…and, as remarked before, HCTC. Time Warner Cable digital voice users apparently had the reverse issues; able to call out but unable to call folks in town…just like everyone else in the world. Sprint’s IP-based long distance also stayed online…it’s what powers Time Warner Cable’s digital voice service (at least for the moment…see this article) and I suppose runs on TWC’s network in the Hill Country.

Internet was more of a mixed bag. Time Warner Cable’s fiber route is different from that of the AT&T cable that was cut (near Waring), so their service, including service for the City of Fredericksburg, continued unabated.

Windstream DSL also continued without issue in Fredericksburg; my mom’s workplace uses them and was able to send an e-mail out about the incident. The phone lines on which the DSL was served couldn’t call beyond Fredericksburg, but internet connectivity did just fine. Oddly enough, Kerrville DSL didn’t weather the outage with as much gusto, according to reports by the Kerrville Daily Times.

Verizon DSL was flat-out down. Bee Creek Communications, riding on that AT&T fiber for their backbone, was out as well. On the flip side, Hill Country Telephone Cooperative’s dual internet links (one to the north, one to the south) didn’t miss a beat. As such, Kerrville’s school system was also unaffected; they use HCTC.

The impact of the outage went a bit beyond calling from point A to point B. Gas stations and Fredericksburg’s Wal-Mart could not take credit card transactions. I’m guessing Wal-Mart has a T1 internet connection, and their T1 was down along with everyone else’s (Sprint and AT&T’s cellular backhauls, for example). The gas stations were probably using plain old telephone lines to process plastic. For some businesses, old-style credit card imprint machines were dragged out and used to process transactions, but for many businesses those machines just don’t exist, relegating purchases to cash or check only.

Last but not least, some banks stopped early for the day (the outage started around 3 p.m. Central Time), as they also tend to use T1 connections to get online. Other banks just processed transactions manually, following procedures similar to blackout protocol.

This just goes to show how much the Hill Country’s communications infrastructure relies on a single provider. Pretty scary if you ask me. It also shows that voice over IP, combined with a real redundant network, is just a good idea; with VoIP it doesn’t matter if you’re running on cable, DSL or mobile broadband, as long as you’ve got access to the Internet. It’s fortunate that the service came up when it did, that TWC, HCTC internet and Windstream DSL customers were apparently not affected, and that emergency services were quickly restored in the area. However running all communications through a single AT&T fiber link (as opposed to a ring, or at least some sort of redundant route!) is just plain bad policy.

Comments/thoughts/additions? Post ‘em!

Tweets compiled about the event (more coming):

Windstream – @bmodisett @kerrvillegossip yes there is an outage; currently no ETA ; fiber cut by another carrier^MC
HCTC – Another good reason to be on Hill Country’s Ethernet backhaul service! Listen up wireless carriers! Can you spell “Diverse Routes”. We can! Call me!!! http://bit.ly/bpPHZU
KerrvilleGossip - Phone service down,, Windstream to blame…. working on the story, looks like HCTC is about to be the BIG DOG! #kerrville (retweeted by bmodisett)\
KerrvilleGossip - Cell service is starting to come back! GO KERRVILLE!

bmodisett - what is up with #Kerrville and the cell phones, it was like they pulled the plug for everyone here.
MrAdkins - Any 3G outages going on out there in twitterland?
tsampson - Anyone in Kerrville experiencing cell phone/land phone outages? Please at reply for potential story on this crazyness.
bmodisett - @tsampson i guess windstream finally had it and are leaving town.
tsampson- @shueytexas Working theory is that Aliens have landed, sending out a electromagnetic pulse… or something to that effect.
tsampson - @mradkins Also, have you heard anything from any carriers? I went over to Windstream and Verizon. No cause yet.
mradkins - @tsampson Just in from Sprint: there is an AT&T Outage affecting both cell and landlines west of Boerne on I-10 and north to Dallas.
mradkins - @tsampson But not sure if this is true. One carrier will blame another. WS won’t say they have a problem even if you drop by.
tsampson - So it’s Apocalypse Cell in the Hill Country today. Read all about it in the Daily Times’ team coverage tomorrow.

KENS5 - Communications cut in Texas hill country, services impaired http://bit.ly/b0QhIz
KVUE - Communications cut in Texas Hill Country, services impaired: It’s a terrible Tuesday for some people living in the… http://bit.ly/bDnGdc
News4WOAI - Major communication outage in Hill Country area: News 4 WOAI has confirmed that a fiber line was cut somewhere bet… http://bit.ly/awvd1X (retweeted by barbaramagana)
KABBFOX29 - Cut Fiber Line Causes Large Outage in Hill Country: A contractor working on roadside drainage causes a major…
http://tinyurl.com/yh3fku5

phlezk - DUDE WTF? phones (cell AND LANDLINE) in the Hill Country (my home!) are DOWN?! HOW THE–?!
phlezk - News 4 WOAI confirmed a Verizon fiber line was cut somewhere between Fredericksburg & Comfort, causing outages throughout the Hill Country.
that1collegeguy - @phlezk Dude, the entire cell network is like…non-existent at the moment!
that1collegeguy - NEWS FLASH: Texas Hill Country economy collapsing rapidly due to cell & landline networks crashing. LifeAlert reports record “pushes”!
BobGambert - Anybody up in the Hill Country? Talk to me about the major comm outage.
bbridges620 - AT&T says it is Verizon’s fault phone are down in the Hill Country, Verizon says it’s AT&T. What is up with Red Vs Blue everywhere? (retweeted by Brandy_Williams)
jfstx - all verizon communications down in Hill Country, no cell, home service , sprint is limping along on 1xrtt
priskillya - the entire hill country was cell phone and land lineless today. i also had no internet. that’s ok cause i had to move. it was just a pain
katiewright - cell service is finally FINALLY back up after about seven hours of being down in the entire Hill Country area.
2dogexpedition - phones in hill country down someone cut a fiber optic
metroane - this fn blows…….major phone, cell, internet services out since 1pm in hill country, can’t upload mag to press……

TaxesNews4u – San Antonio News: Widespread phone outage in Hill Country: Kerr, Gillespie counties are among the areas affected http://bit.ly/aW1t1y
SanAntonioCP - San Antonio: Major communication outage in Hill Country area http://bit.ly/dw8pUi
tweetsa - Widespread phone outage in Hill Country http://bit.ly/ayMHTA
salocalnews - Major communication outage in Hill Country area – http://fx.ly/a/16_78e8750bdc
salocalnews - Widespread phone outage in Hill Country – http://fx.ly/a/16_ad169f14d2
MsTechAngel - Widespread phone outage in Hill Country: By Zeke MacCormack – Express-News Damage to an AT&T fiber-optic line caus… http://bit.ly/cwMjyH
JeffVaughn - Breaking: Parts of Hill Country experiencing phone outage, including #Kerville: http://tinyurl.com/ydvwm2m
wrstech - Major communication outage in Hill Country area http://shar.es/mgfnZ
wxProBrian - Red vs Blue! RT @ksatnews: Now, Verizon tells us it’s an AT&T Fiber line that was cut – causing phone/cell/net outages in Hill Country.
GSMTalks - GSMtalks :: Service returns after widespread phone outage in Hill Country http://bit.ly/bml2pm
SanAntonioLocks - Phone Lines Out In Hill Country: A severed AT&T fiber line has cut cell, land and Internet connections in the Hill… http://bit.ly/b8Q9cj

Traceroute to Google:

traceroute to google.com (74.125.159.104), 64 hops max, 52 byte packets
1  64.sub-66-174-185.myvzw.com (66.174.185.64)  82.040 ms  71.681 ms  140.969 ms
2  127.sub-66-174-184.myvzw.com (66.174.184.127)  85.990 ms  88.710 ms  93.889 ms
3  146.sub-66-174-44.myvzw.com (66.174.44.146)  94.923 ms  99.735 ms  92.011 ms
4  235.sub-66-174-45.myvzw.com (66.174.45.235)  120.975 ms  88.777 ms  86.127 ms
5  194.sub-66-174-45.myvzw.com (66.174.45.194)  86.937 ms  99.731 ms  92.958 ms
6  98.sub-66-174-44.myvzw.com (66.174.44.98)  89.816 ms  92.702 ms  91.968 ms
7  6.sub-69-83-96.myvzw.com (69.83.96.6)  88.964 ms  90.749 ms  101.982 ms
8  3.sub-69-83-96.myvzw.com (69.83.96.3)  91.973 ms  89.745 ms  92.877 ms
9  253.sub-69-83-97.myvzw.com (69.83.97.253)  86.946 ms  87.701 ms  86.078 ms
10  gigabitethernet3-0-0.gw1.dfw7.alter.net (65.208.8.17)  93.040 ms  86.646 ms  86.916 ms
11  0.ge-2-1-0.xl3.dfw7.alter.net (152.63.103.6)  99.833 ms  102.809 ms  89.928 ms
12  0.ge-6-0-0.br2.dfw13.alter.net (152.63.96.181)  89.939 ms
0.ge-4-0-0.br2.dfw13.alter.net (152.63.96.45)  91.661 ms
0.ge-6-0-0.br2.dfw13.alter.net (152.63.96.181)  89.622 ms
13  te-5-1-0.edge2.dallas3.level3.net (4.68.111.121)  99.987 ms  116.649 ms
te-5-2-0.edge2.dallas3.level3.net (4.68.111.173)  90.885 ms
14  ae-83-80.ebr3.dallas1.level3.net (4.69.145.180)  92.662 ms
ae-73-70.ebr3.dallas1.level3.net (4.69.145.116)  92.622 ms
ae-63-60.ebr3.dallas1.level3.net (4.69.145.52)  97.740 ms
15  ae-7.ebr3.atlanta2.level3.net (4.69.134.22)  115.793 ms  137.608 ms  130.824 ms
16  ae-11-51.car1.atlanta1.level3.net (4.68.103.2)  120.070 ms  117.856 ms  116.900 ms
17  google-inc.car1.atlanta1.level3.net (4.78.209.194)  133.861 ms  118.900 ms  117.002 ms
18  209.85.254.241 (209.85.254.241)  122.982 ms  116.700 ms  119.880 ms
19  209.85.254.2 (209.85.254.2)  113.964 ms  120.700 ms
209.85.254.6 (209.85.254.6)  137.883 ms
20  yi-in-f104.1e100.net (74.125.159.104)  109.804 ms  129.913 ms  131.328 ms

Traceroute to SoftLayer (specifically, the server that hosts this website):

traceroute to echo.mddhosting.com (74.86.48.34), 64 hops max, 52 byte packets
1  64.sub-66-174-185.myvzw.com (66.174.185.64)  73.485 ms  72.917 ms  73.973 ms
2  127.sub-66-174-184.myvzw.com (66.174.184.127)  91.918 ms  94.745 ms  184.212 ms
3  146.sub-66-174-44.myvzw.com (66.174.44.146)  100.537 ms  99.876 ms  95.165 ms
4  235.sub-66-174-45.myvzw.com (66.174.45.235)  94.993 ms  100.098 ms  98.174 ms
5  194.sub-66-174-45.myvzw.com (66.174.45.194)  98.010 ms  99.438 ms  93.713 ms
6  98.sub-66-174-44.myvzw.com (66.174.44.98)  94.994 ms  89.077 ms  85.875 ms
7  6.sub-69-83-96.myvzw.com (69.83.96.6)  100.961 ms  105.189 ms  88.124 ms
8  3.sub-69-83-96.myvzw.com (69.83.96.3)  87.943 ms  96.379 ms  97.863 ms
9  253.sub-69-83-97.myvzw.com (69.83.97.253)  98.888 ms  68.155 ms  87.944 ms
10  gigabitethernet3-0-0.gw1.dfw7.alter.net (65.208.8.17)  86.948 ms  69.442 ms  96.147 ms
11  0.ge-2-1-0.xl3.dfw7.alter.net (152.63.103.6)  112.539 ms  89.027 ms  85.911 ms
12  0.so-6-0-0.gw3.dfw13.alter.net (152.63.100.73)  85.015 ms  91.726 ms  86.006 ms
13  internapgige1-gw.customer.alter.net (65.208.15.230)  113.416 ms  96.205 ms  98.848 ms
14  border3.tge3-1-bbnet1.ext1.dal.pnap.net (216.52.191.22)  125.890 ms  95.198 ms  74.901 ms
15  te2-1.cer03.dal01.dallas-datacenter.com (216.52.189.30)  93.058 ms  94.118 ms  95.968 ms
16  po3.dar02.dal01.dallas-datacenter.com (66.228.118.211)  99.913 ms  100.785 ms  107.881 ms
17  po2.fcr02.dal01.dallas-datacenter.com (66.228.118.182)  111.010 ms  92.603 ms  121.966 ms
18  echo.supportedns.com (74.86.48.34)  97.877 ms  96.688 ms  153.805 ms

So pings are a bit lower, on average, than either AT&T or Sprint. They’re also rather predictable, unlike those of CricKet. Then again, I have full EvDO signal sitting at the dining room table with the USB727′s antenna extended. For what it’s worth, this traceroute is pretty close, after the MyVZW hops, to what I see on the family’s brand-spanking-new DSL connection, though that can be expected since both services are from the same company.

Here’s a speedtest…

Well gosh, at least at my location Verizon is nearly as fast as AT&T on the downstream, respectable on the upstream and a good bit more enjoyable to use than AT&T. Heck, the EvDO connection soundly beats the new Verizon DSL, though the DSL connection doesn’t cost $60 per month, nor does it have a ridiculously low data transfer cap.

That said, I’m not the only one on the tower…

Still respectable, just not quite as nice.

Now for a ping test…

Right in line with what I’ve seen on Sprint EvDO, except with a little less latency.

The bottom line: Verizon knows how to run a solid mobile broadband network. So does Sprint. Whether one serves you better than the other depends on who has the big pipes to the towers in your area, and who has the towers in your area period.

Well, and there’s the question of whether you want a 10GB cap for $60 or unlimited service for $70, both via Millenicom of course…unless you really, really want to sign a contract in exchange for a free EvDO modem.

Which provider would I choose at this point, if I had to pick up mobile broadband again? Tough decision; I own an AT&T aircard and don’t want to sign a contract, so the other options have an added expensve for me. With that in mind, I might end up going with AT&T for a month or two, but Sprint if I needed a home replacement connection. If I didn’t need to use the connection as my only internet and didn’t have an AT&T card already, I’d likely choose Verizon, with Sprint as a very close second.

At this point though, for someone new to mobile broadband, there’s one company I wouldn’t recommend anyone to go with, at least for my area: AT&T.

1. My mom saw a Verizon truck in the county fair parade today (I didn’t go; saw no need and was tired). The truck has “High Speed Internet coming soon!” or similar painted on te side. This is DSL, not FiOS (Verizon always refers to their fiber offering by its trademarked name). However from what I hear Verizon’s DSL release is just a ploy to drive up the sale cost of their system here; Fredericksburg isn’t a large metropolitan area, and there are too mayn bridged taps and other phone system anomaies here for DSL to reach beyond a small part of town. At least that’s what the scuttlebutt says. My prediction: Verizon offers some flavor of DSL to folks who can currently get Windstream DSL, then sells the system to whoever will take it, using the “we have DSL in our central office” argument to jack the price up slightly above what the system is worth.
2. Bee Creek Communications, the local our-hands-are-tied-and-if-anyone-competes-with-us-we-lose wireless ISP, was in the parade today as well. They also have a third-of-a-page ad in the paper advertising free installation to new customers until September 15th. A few years ago an installation with similar equipment would have run you around $220. The problem here is that their network is already over capacity in some areas, they’re spread too thinly in terms of manpower and their speeds are the stuff of yesteryear, barely competitive with satellite internet (though their monthly transfer caps are higher). The service also requires a two-year contract with a $125 early termination fee. If you want a one-year agreement service will run you another $10 per month. In short, don’t buy.
3. Clearwire is launching WiMAX service in San Antonio and Austin on September 1st. Maybe they’ll bring their service here. Would put Bee Creek right out of business, since Clear has an uncapped-download, one-megabit-upload (or is that uncapped too?) tier for a mere $45 per month. They even have cheaper tiers to compete with your garden-variety DSL options. They even have business tiers that will fill the gap between Bee Creek’s woefully slow packages and Time Warner Cable’s woefully expensive ones. In a nutshell, something for everyone as long as you’re within range of one of their towers. I’ll be keeping close track of what happens here. For what it’s worth, I told Bee Creek about Clear’s possible imminent launch nearby, and I wouldn’t be surprised if their larger-than-usual print ad was a direct result of this.
4. Hill Country Telephone Cooperative has run fiber around the city of Fredericksburg and plans to get CLEC (competitive local exchange carrier) operations started before the end of the year, or something close to that anyway. They may just focus on high-end business operations (like they’re doing in nearby Kerrville, where they recently connected the city’s independent school district) however they’re a residential provider in their home area, so they know how to deal with “normal” customers. Plus, they have a lot of options for deploying service; in addition to their new fiber-optic network they have an AWS (1700 MHz cellular) license for this area, a 3.65 GHz “light license” which they’re using for WiMAX service in the nearby town of Mason, and maybe (if the price is right) Verizon’s old copper network, though I’m not sure why they’d want it. I heard HCTC offered $50 million for the network a few years back, and Verizon wanted $80 million fot it. Knowing that the network wasn’t worth that much, HCTC walked.

In short, it will be interesting to see what happens around here in the relatively near future with regard to internet service. Right now Time Warner Cable is the only provider who will give you more than a 2 Mbps connection (okay, HughesNet will give you 5 Mbps but they don’t count due to obscenely low caps and ridiculously high monthly fees). They’re also the only provider around here with customer-available fiber; Verizon apparently doesn’t have any. Sounds like that may change soon though. I’ll tell all I know (or am allowed to tell, whichever is less) as I find out more.