UPDATE 3: Added another news source. Swamped by homework so tweets will have to wait, but other than those I think I’ve got all the online info on the outage linked from here.

UPDATE 2: Added another couple of news sources, corrected one of my usual typos…some how I forgot to add a “not” into the paragraph about Wal-Mart taking credit card transactions. Sorry for the confusion. Also modified information based on an article from the Fredericksburg Standard Radio-Post. The link below takes you to the subscriber-only version of the article; the non-subscriber snippet just makes the paper look bad (very little info).

UPDATE: Added some more news articles, (still) working on collecting all tweets about the incident, revised some information for accuracy (originally wasn’t sure about CricKet and Pocket, but they too were taken down by the outage…DSL was also apparently iffy in Kerrville).

Granted, I wasn’t in the area when it happened, but since I have plenty of family and friends down there I’m certainly going to talk about it!

Kerrville Daily Times Articles
March 24th Front Page (JPEG image)
Photo
Editorial
Note about landlines/safety
Info about 911 operations
Main article (covers cellular, credit cards, banks) Reprint in iStockAnalyst
Businesses adapt (anyone remember credit card imprint machines?)
Humorous recap by one Daily Times reporter (not a newspaper article)

MySanAntonio
911 article
Original article

Articles from other outlets
KENS5 KVUE WOAI WOAI VIdeo KSAT12 Boerne Star Burnet Bulletin Fredericksburg Standard Radio-Post Blanco County News TMCNet

Rather scary stuff for the folks affected. The outage, by my calculations, lasted for a full seven hours, and knocked out cellular communications completely in the Hill Country. Verizon cells? Down. Sprint? Down. AT&T? Dead. T-Mobile? Not 100% sure, but probably gone. HCTC phone service? Nada. Pocket and CricKet were dead in the water as well.

Oddly enough, the Fredericksburg Standard Radio-Post cites Five Star Wireless as not being down during the incident, for local or long distance service. This on one hand makes sense, as Five Star doesn’t necessarily use phone-company T1 lines to power their cell sites like other providers do. However I tried calling a Five Star connected friend yesterday during the outage and couldn’t get through. I’m going to keep poking around for more info on what happened with that provider.

On the landline front, you could call to your own exchange but no further. This was the case for both Verizon and Windstream…and, as remarked before, HCTC. Time Warner Cable digital voice users apparently had the reverse issues; able to call out but unable to call folks in town…just like everyone else in the world. Sprint’s IP-based long distance also stayed online…it’s what powers Time Warner Cable’s digital voice service (at least for the moment…see this article) and I suppose runs on TWC’s network in the Hill Country.

Internet was more of a mixed bag. Time Warner Cable’s fiber route is different from that of the AT&T cable that was cut (near Waring), so their service, including service for the City of Fredericksburg, continued unabated.

Windstream DSL also continued without issue in Fredericksburg; my mom’s workplace uses them and was able to send an e-mail out about the incident. The phone lines on which the DSL was served couldn’t call beyond Fredericksburg, but internet connectivity did just fine. Oddly enough, Kerrville DSL didn’t weather the outage with as much gusto, according to reports by the Kerrville Daily Times.

Verizon DSL was flat-out down. Bee Creek Communications, riding on that AT&T fiber for their backbone, was out as well. On the flip side, Hill Country Telephone Cooperative’s dual internet links (one to the north, one to the south) didn’t miss a beat. As such, Kerrville’s school system was also unaffected; they use HCTC.

The impact of the outage went a bit beyond calling from point A to point B. Gas stations and Fredericksburg’s Wal-Mart could not take credit card transactions. I’m guessing Wal-Mart has a T1 internet connection, and their T1 was down along with everyone else’s (Sprint and AT&T’s cellular backhauls, for example). The gas stations were probably using plain old telephone lines to process plastic. For some businesses, old-style credit card imprint machines were dragged out and used to process transactions, but for many businesses those machines just don’t exist, relegating purchases to cash or check only.

Last but not least, some banks stopped early for the day (the outage started around 3 p.m. Central Time), as they also tend to use T1 connections to get online. Other banks just processed transactions manually, following procedures similar to blackout protocol.

This just goes to show how much the Hill Country’s communications infrastructure relies on a single provider. Pretty scary if you ask me. It also shows that voice over IP, combined with a real redundant network, is just a good idea; with VoIP it doesn’t matter if you’re running on cable, DSL or mobile broadband, as long as you’ve got access to the Internet. It’s fortunate that the service came up when it did, that TWC, HCTC internet and Windstream DSL customers were apparently not affected, and that emergency services were quickly restored in the area. However running all communications through a single AT&T fiber link (as opposed to a ring, or at least some sort of redundant route!) is just plain bad policy.

Comments/thoughts/additions? Post ‘em!

Tweets compiled about the event (more coming):

Windstream – @bmodisett @kerrvillegossip yes there is an outage; currently no ETA ; fiber cut by another carrier^MC
HCTC – Another good reason to be on Hill Country’s Ethernet backhaul service! Listen up wireless carriers! Can you spell “Diverse Routes”. We can! Call me!!! http://bit.ly/bpPHZU
KerrvilleGossip - Phone service down,, Windstream to blame…. working on the story, looks like HCTC is about to be the BIG DOG! #kerrville (retweeted by bmodisett)\
KerrvilleGossip - Cell service is starting to come back! GO KERRVILLE!

bmodisett - what is up with #Kerrville and the cell phones, it was like they pulled the plug for everyone here.
MrAdkins - Any 3G outages going on out there in twitterland?
tsampson - Anyone in Kerrville experiencing cell phone/land phone outages? Please at reply for potential story on this crazyness.
bmodisett - @tsampson i guess windstream finally had it and are leaving town.
tsampson- @shueytexas Working theory is that Aliens have landed, sending out a electromagnetic pulse… or something to that effect.
tsampson - @mradkins Also, have you heard anything from any carriers? I went over to Windstream and Verizon. No cause yet.
mradkins - @tsampson Just in from Sprint: there is an AT&T Outage affecting both cell and landlines west of Boerne on I-10 and north to Dallas.
mradkins - @tsampson But not sure if this is true. One carrier will blame another. WS won’t say they have a problem even if you drop by.
tsampson - So it’s Apocalypse Cell in the Hill Country today. Read all about it in the Daily Times’ team coverage tomorrow.

KENS5 - Communications cut in Texas hill country, services impaired http://bit.ly/b0QhIz
KVUE - Communications cut in Texas Hill Country, services impaired: It’s a terrible Tuesday for some people living in the… http://bit.ly/bDnGdc
News4WOAI - Major communication outage in Hill Country area: News 4 WOAI has confirmed that a fiber line was cut somewhere bet… http://bit.ly/awvd1X (retweeted by barbaramagana)
KABBFOX29 - Cut Fiber Line Causes Large Outage in Hill Country: A contractor working on roadside drainage causes a major…
http://tinyurl.com/yh3fku5

phlezk - DUDE WTF? phones (cell AND LANDLINE) in the Hill Country (my home!) are DOWN?! HOW THE–?!
phlezk - News 4 WOAI confirmed a Verizon fiber line was cut somewhere between Fredericksburg & Comfort, causing outages throughout the Hill Country.
that1collegeguy - @phlezk Dude, the entire cell network is like…non-existent at the moment!
that1collegeguy - NEWS FLASH: Texas Hill Country economy collapsing rapidly due to cell & landline networks crashing. LifeAlert reports record “pushes”!
BobGambert - Anybody up in the Hill Country? Talk to me about the major comm outage.
bbridges620 - AT&T says it is Verizon’s fault phone are down in the Hill Country, Verizon says it’s AT&T. What is up with Red Vs Blue everywhere? (retweeted by Brandy_Williams)
jfstx - all verizon communications down in Hill Country, no cell, home service , sprint is limping along on 1xrtt
priskillya - the entire hill country was cell phone and land lineless today. i also had no internet. that’s ok cause i had to move. it was just a pain
katiewright - cell service is finally FINALLY back up after about seven hours of being down in the entire Hill Country area.
2dogexpedition - phones in hill country down someone cut a fiber optic
metroane - this fn blows…….major phone, cell, internet services out since 1pm in hill country, can’t upload mag to press……

TaxesNews4u – San Antonio News: Widespread phone outage in Hill Country: Kerr, Gillespie counties are among the areas affected http://bit.ly/aW1t1y
SanAntonioCP - San Antonio: Major communication outage in Hill Country area http://bit.ly/dw8pUi
tweetsa - Widespread phone outage in Hill Country http://bit.ly/ayMHTA
salocalnews - Major communication outage in Hill Country area – http://fx.ly/a/16_78e8750bdc
salocalnews - Widespread phone outage in Hill Country – http://fx.ly/a/16_ad169f14d2
MsTechAngel - Widespread phone outage in Hill Country: By Zeke MacCormack – Express-News Damage to an AT&T fiber-optic line caus… http://bit.ly/cwMjyH
JeffVaughn - Breaking: Parts of Hill Country experiencing phone outage, including #Kerville: http://tinyurl.com/ydvwm2m
wrstech - Major communication outage in Hill Country area http://shar.es/mgfnZ
wxProBrian - Red vs Blue! RT @ksatnews: Now, Verizon tells us it’s an AT&T Fiber line that was cut – causing phone/cell/net outages in Hill Country.
GSMTalks - GSMtalks :: Service returns after widespread phone outage in Hill Country http://bit.ly/bml2pm
SanAntonioLocks - Phone Lines Out In Hill Country: A severed AT&T fiber line has cut cell, land and Internet connections in the Hill… http://bit.ly/b8Q9cj

1. SIP VoIP, over 3G and WiFi, with Bluetooth and in the background
I’ve successfully run phone calls over AT&T’s HSPA network, so assuming the processor on the iPad is “all that” this shouldn’t be difficult. If I can get a SIP client that connects to a Bluetooth headset and allows other applications “up front” without killing voice quality then I’ve got a very versatile system for nformation gathering while “on the phone.” This is particularly useful when you see my other requests below:

2. Full-fledged RDP, SSH and VNC clients…and either GoToMyPC or LogMeIn if at all possible
The iPad can be an excellent remote access tool; its screen has enough resolution to run any modern OS without having to scroll around. Add in the keyboard accessory and you’ve got a lovely little remote terminal that works anywhere AT&T has service (I’m talking about the 3G version here). Or any other non-AWS GSM or 3G carrier for that matter; the device is unlocked. SSH would be a godsend for doing command line heavy lifting, though you’d need a host to SSH to in this case (not a big deal for me). Even nicer, all of the above will likely be available for the iPad as App Store apps; no jailbreaking required.

3. A high-quality PDF viewer
I want a PDF viewer that can search and copy text, preferably at least as good as Preview. Yes, Preview can be outdone with PDFs…it’s not the most compatible piece of PDF reader in the world unfortunately.

4. File download capability, preferably with an FTP/SFTP client as well as your standard HTTP access
The iPad has an accessory port, and that accessory port already allows for an SD card reader and a USB port. If you could mount that USB port or SD reader as a drive letter (possible with the OS, but probably will need to be hacked) then there’s a TON of stuff you could do with the iPad, though things start to get kludgy once you realize that you can’t hook up a keyboard and a hard disk/SD card simultaneously. Hmm, maybe the keyboard dock could have a pass-through dock connector…or have an application that would use internal memory as a cache for editing files so you could unplug the drive, edit the files with the keyboard attached, then plug back in and write the files. Heck, wile I’m at it, why not let shares on AFP and SMB be mountable to the device?

5. WiFi printing
This will likely be implemented in short order as an application, though hopefully it will be available throughout the operating system. THat way I can go to a website, grab a PDF of some sort (documentation, textbook, whatever) and send it to a networked printer. No muss, no fuss.

6. Network diagnostics, both standard and wireless (including 3G)
I’m not just talking ping/traceroute/speedtest, though those would be nice. I’m talking about field-test-level stats on wireless and 3G. Why 3G? Well, for one AT&T isn’t the only 3G carrier out there, and if network engineers on smaller carriers can troubleshoot their cellular deployments on a cheap-ish, small device then maybe there will be more field testing going on. On the WiFi side, an iStumbler-esque application would be lovely, complete with a signal strength history graph, with geotagging of signals if at all possible (though this would take a toll on battery life). Heck, add the geolocation to the 3G field test if at all possible and just for kicks port WireShark to the iPad and you’ve got one powerful network analysis device in a pound-and-a-half package.

7. Voice recorder, preferably with backgrounding
I’m guessing this will be included when the iPad ships, but it would be lovely to be able to take the iPad to a lecture or an interview and use it to record the conversation. Even cooler would be an application that would combine a text editor with the voice recorder; in “review” mode you could seek through a recording by going to a place in the text notes, and the other way around.

8. OneNote-like application
Capacitative styli are available. Make an application that allows for text notes, handwriteen notes/diagrams and even voice notes all together. Great for school and for college newspaper writers like myself

9. Access to a bash shell on-device
I’m sure this will only be available when the device is jailbroken, but it would definitely be useful. A full shell, complete with nano for text editing etc., would make the iPad a lightweight BSD workstation, and who doesn’t like lightweight *nix workstations?

Sure, I’m reaching a bit for the above things, but the hardware is all there; you just have to either open up the software or make an application. There are a few subsystems that would be nice to have,a dn they might even be available inthe upcoming iPhone OS 3.2. In all honesty, I think Apple is going to have to make this device rather hackable for enough people to buy it; they’re blazing the trail for a new category of device and if they shun the “Woz box” mentality they’ll be selling very few of these things once the fanboys and their contituents finish buying (which should take about 45 days).

So naturally I pulled out my HTC Touch Pro, which has WiFi, to see what network they were running on. After registering for an account on the hotspot provider (Less Networks, never heard of them) I checked whatismyip.com to see who I was on. Sure enough, the connection there was GVTC, registered as “Guadalupe Valley Telephone”. Right before we pulled out of the place I opened up my laptop, logged into the WiFi network that way and did some tests. Here’s what I got for a traceroute:

traceroute to softlayer.com (66.228.118.53), 64 hops max, 52 byte packets
1  192.168.169.1 (192.168.169.1)  194.115 ms  181.762 ms  77.970 ms
2  96-8-192-3.block0.gvtc.com (96.8.192.3)  94.851 ms  107.971 ms  146.974 ms
3  12.88.219.169 (12.88.219.169)  47.516 ms  218.308 ms  149.762 ms
4  cr1.santx.ip.att.net (12.123.154.10)  81.998 ms  58.617 ms  64.528 ms
5  cr2.dlstx.ip.att.net (12.122.30.130)  71.363 ms  55.947 ms  70.918 ms
6  gar6.dlstx.ip.att.net (12.122.100.65)  52.436 ms  58.138 ms  56.583 ms
7  12.90.228.14 (12.90.228.14)  52.971 ms  48.888 ms  80.834 ms
8  border3.tge4-1-bbnet2.ext1.dal.pnap.net (216.52.191.83)  52.188 ms  51.494 ms  52.917 ms
9  te2-1.cer03.dal01.dallas-datacenter.com (216.52.189.30)  55.883 ms  54.237 ms  55.133 ms
10  po3.dar01.dal01.dallas-datacenter.com (66.228.118.209)  53.084 ms  84.979 ms  47.910 ms
11  po1.slr01.dal01.dallas-datacenter.com (66.228.118.138)  66.371 ms  54.266 ms  57.160 ms

I have a feeling that the WiFi conection was flakey, otherwise I would have gotten a better traceroute.

Speed-test-wise I got the following:

I’m guessing the line was provisioned at 8 Mbps down and 1 Mbps up, but I can’t be sure.

The IP address, for what it’s worth, was 96.8.236.103.

All in all, not a bad hotspot, and the traceroute is nice and short between where I was and AT&T’s network in San Antonio, which GVTC appears to use exclusively to connect to the internet nowadays. So someone using an AT&T aircard and GVTC FTTH will find themselves with similar traceroutes within just a few hops, at least around here.

I guess I’ll leave it at that…if I talk any more about GVTC I’ll start getting envious of their 20/3 and 40/10 FTTH service, which I have no chance of getting at my place. Can’t even get DSL there :/

One problem arose when I got home: my aircard kept switching to EDGE during the middle of a data session. Not a good thing for a $60 + taxes and fees data plan. So I needed to force 3G on the card. The problem there was that only the Windows Sierra and AT&T clients allow forced 3G. So (not knowing that settings are stored on the modem and thus once I set 3G-only on my Windows computer it would be 3G-only on my Mac) I scrounged around and found this page. I followed those instructions and was able to make sure my aircard was 3G-only.

If you don’t want to read through a (relatively short) page on how to kick your aircard into high gear, you’re in luck. Here’s what you’d do on a Mac to force-3G a Sierra aircard. Note that this procedure was performed on OS X 10.6.1 and with the Apple “native” driver installed (a tiny download from here). This is also with the latest modem firmware, which I had to use the Sierra driver (vs. the native one) to upgrade to.

1. Go to Terminal (Applications > Utilities)
2. Type in screen /dev/tty.wwanCMD 115200 and hit enter.
3. Type in AT!band? and hit enter.
4. If you see the number 02, you’re already locked into 3G-only mode. Otherwise, type in AT!band=02 and hit enter to set the mode. If you wanted to go 2G-only, the code would be 04 instead of 02. AUtoselect is 00.

Hope this helps! Now I’m cooking along with 2-3 bars of signal (-87 dBm right now) and pulling down these speeds:

Though connection quality is quite a bit more variable than Sprint’s (slower, particularly on uploads) connection at my place…

In comparison, SPrint usually pulls down 1.3 Mbps down, 300-500 kbps up and gets a B on connection quality.

Preliminary Notes

Verizon doesn’t use SIM cards. Neither does Sprint. While I think it’d be awesome to see a dual-radio network-agnostic iPhone (like the Blackberry Storm and some new HTC units) I don’t foresee that happening. As for a dual CDMA radio setup that would “bond” two networks together for fast downloads, not gonna happen. The engineering and billing logistics just don’t work out. The best that’ll realistically happen is either an LTE iPhone (I doubt it’ll come out for WiMAX unfortunately) or *possibly* an iPhone “world phone” with CDMA and GSM/HSPA a la the Blackberry Storm if Verizon gets the new exclusive.

Now that that’s off my, and Mike Rundle’s, chest…

Q: Do you think Apple would be more, or less, successful if they adopted a more open strategy (i.e. allowing other MP3 players in iTunes)?

A: At worst, they’d be just as successful. They might be forced to compete a little more on features, especially on the Mac side (where there’s no Windows Media Player and thus no simple way to sync music with a non-iTunes-supported player). However the influx of people using third-party devices on iTunes would make up for any people who might pick a $30 other-brand MP3 player over Apple’s own. Not like those people would buy an iPod in the first place…

On other fronts, they’d probably indeed be more popular by taking a more open stance on some projects (ahem, iPhone). At least blame the carriers for crippling your device rather than sitting down and letting AT&T pass the buck to you on why Google Voice has to be a (perfectly functional) web app rather than an on-phone application.

Back to the music side of things. I’ve actually owned both an iTunes-compatible non-iPod MP3 player (Creative MuVo Mix, came free with an eMusic subscription, Audible compatible) and an MP3 player (Creative Zen Vision M) and online service (Yahoo Music Unlimited To Go, now Crap…er…Rhapsody) that didn’t work so great on OS X, ostensibly through no fault of Apple’s own. Of course, if Microsoft has clandestinely offered Apple the ability to use PlaysForSure in OS X (I used VMWare + Windows XP to use the software) and Apple refused, the openness ball is back in Apple’s court. Sure, the Zen works in Mass Storage mode in OS X, but the ideal situation is in a Windows/Windows Media Player 10-or-higher environment.

That said, Apple has ported iTunes to Windows. Then again, it was an obvious business move. Who’s going to pay $1000 up front for a computer to use a $200 MP3 player, even if it is the best thing since sliced bread? Apple has shown that they have at least some brains regarding accepted standards in other avenues too; the iPhone supports Exchange now and Snow Leopard will support it when that OS comes out.

Q: Do you think Apple should face serious antitrust action?

A: For the iPod/iTunes semi-lockin, absolutely not. For the long iPhone exclusivity contract (longer than any popular phone I know of) and annoying yearly-on-a-two-year-contract upgrade schedule, yes. While product cycles in the smartphone market are about a year, Apple’s rather slow feature creep is a bit ludicrous, punishing anyone who bought the iPhone 3G by offering a rather minor upgrade halfway through people’s contracts. Granted, people can choose to buy (or not buy) the iPhone, but with the exclusivity contract thrown in, things get even dicier.

My proposition: exclusivity contracts for phones must expire after two years for a given model, or three months prior to that model’s discontinuation, whichever comes first. I won’t go so far as to say that GSM-only phones must be made available on CDMA-only networks and vice versa, but the exclusivity agreement on the iPhone has given AT&T more customers than they know what to do with (or at least more than their engineering department knows how to cope with on the 3G network side of things). Yet AT&T keeps selling iPhones, and the market failure exacerbates itself. With the exclusivity idea I suggested above, the first-gen iPhone would be available on T-Mobile, and the iPhone 3G would be available in a few more months, if not already.

Q: Do you think Apple’s dexterity and competence forgive their bad behavior?

A: Nope.

So, what do you think?

Starting off with the wireless division, we see the iPhone getting a few new laurels to rest on. It has snagged the top spot for most sold US wireless phone this quarter from the Motorola Razr, and also appears to be 2X the reliability of BlackBerries and nearly 3X the reliability of Treos. AT&T has also opened up free AT&T WiFi access to iPhone and other smartphone users. This is particularly cool since the company also bought up hotspot operator Wayport right around ten minutes ago.

Last but not least, 1.1 million customers and a lot of spectrum, towers and other such infrsatructure got traded for a cool $944 million in AT&T’s buyout of Centennial Communications, yet another regional GSM carrier to fall under AT&T’s sway.

UPDATE: Looks like AT&T will be allowing iPhone tethering…for $30 on top of your regular data plan. The transfer limit is 5 GB with this option, and if you’re thinking that the total data cost is equal to that of a dedicated modem, you’re abso-freaking-lutely correct. Ripoff? Yes, but I suspect people will bite. Though Sprint’s network is generally faster, and tethering is only $15 on top of a data plan, which is also $30 on top of a voice plan but also includes everything from navigation to text messaging.

Speaking of WiFi, you get that free with most AT&T DSL/U-Verse plans. Which may be capped. The monthly transfer caps are as follows:

$10-$20, 768K, 20GB (DSL only)
$25-$32, 1.5M, 40GB
$30-$36, 3M, 60GB
$35-$43, 6M, 80GB
$55, 10M, 150GB (U-Verse only)

The caps are a bit biased speed-wise toward the lower-end tiers, but of course cost-per-GB-wise toward the higher tiers. Overages will be $1 per GB. The caps are squarely between Time Warner Cable’s proposed 5-40 GB limits and Comcast’s 250 GB cap. What’s funny about this is that AT&T hs plenty of backbone infrastructure, especially with U-Verse, to allow everyone to download as much as they want. Of coursse, business-class users won’t see these caps, but users in the Reno, NV test market already are, or they’re being grandfathered in at 150GB per month. What’s unnerving is the result if you only qualify for a lower-speed DSL connection due to distance from the CO: low caps, high overages.

Speaking of U-Verse, AT&T has introduced an 18 Mbps tier (1.5 Mbps upload) for $10 more than the 10/1.5 Mbps one, or $65 per month, though people have reported that they’ve gotten better deals by calling AT&T to upgrade. The caveat: the U-Verse service has to share bandwidth with any TV that’s running, and the shared pipe is only 25 Mbit/s wide downstream and 2 Mbit/s upstream. This “profile” is due to the fact that VDSL (the tech upon which U-Verse is based) drops off quickly speed-wise as distance from the VRAD (fiber-fed DSL terminal) increases. The result: if you’re watching TV, you get 14-15 Mbps internet speeds instead of 18 Mbps, though TV quality doesn’t degrade while the internet is being used. No word on caps on this tier; let’s hope they’re to the tune of 250GB…competitors are sure to deploy DOCSIS 3 in response to this new U-Verse tier, and Comcast‘s caps are well above AT&T’s proposed ones.

Last but not least, AT&T is trying to create a video search engine (!?!) in partnership with a startup by the name of Divvio. Once can’t but think that this new service, dubbed Videocrawler, will be a nice tool for chewing up your gigabytes once caps are in place. Why I oughta…