As I am sure you knew to begin with, and have since been reminded, there are a thousand different ways to do this, and no one solution will be right for every installation/client. Depending on what is most important (incoming calls from clients vs intra-company calls among developers, vs outbound 911, etc) and what sort of budget is available, all sorts of different solutions will arise.
What the proper response is also depends on the location of the PBX (solutions for hosted sites will be different than for on-site machines) and the geographic location of the PBX. For example, the likelihood that FIOS and Comcast would both go down at the same time in a major metropolitan area is far different than the likelihood of that happening in a more rural area. For example: we once had a car hit a telephone pole a mile down the road and lost power, phones, AND internet: that would never happen in an urban environment (barring major disasters).
Our solution for the central office is a machine that is located on-site (so the bulk of our employees do not count on internet for their phones). Almost all calls come in on a PRI. If this circuit is down (PBX crashes, extended power failure, provider problems), it fails over to an ITSP. If the ITSP cannot reach the PBX, it fails over to my cell phone. I don’t like to rely on an ITSP for our primary voice traffic, as it is only as good as our internet connection, which is simply not as good as a nice, old-fashioned, expensive T1/PRI. Having said that, I can justify the cost based on the amount of voice traffic we see, plus the fact that we are, quite simply, OUT OF BUSINESS if our phones are down, more so than other companies might be.
Branch offices have analog phones on their desks, in addition to their SIP phones. This is because the internet/IPSec connection to the central FreePBX machine just isn’t reliable enough for my liking. I have often considered putting a PBX in each office and routing the analog lines through them via a second line appearance on the SIP phones, but the idea of administering a dozen more PBXs has put me off of that solution for now.
More than anything, I would counsel that you sit down with each client and “have the talk” about redundancy. Explain what could go wrong, what would happen if it did, what the options are to avoid it, and how much it costs. From there, find out what their priorities are and help them craft a solution. As a bonus, by involving them it will be harder for them to blame you when the corner they chose to cut comes back to bite them.
Good luck, and let us know what you find,
PS: As for “Why don’t I use POTS for all of my lines?” Well, the answer is additional hardware cost, plus ongoing charges from the phone company. On the other hand, there is nothing wrong with a FreePBX box that has 1-15+ analog lines connected via a SIP gateway, PCI card, or Xorcom Astribank. That’s not the right solution for everyone, but it might be perfect for some.