4.0 What sorts of returns can I expect making as an ISP?

4.1 Introduction: The world is changing.

For quite a while, you could start an ISP on $ 10k worth of equipment and a smile. Karl Denninger's MCS-NET (mcs.com) did that and became a sizable presence in Chicago very quickly.

Congratulate him. It's not so easy anymore, especially in areas with present competition. You probably have your best shot if existing ISPs have dreadful reputations. In fact, I decided to start my provider after noticing Netcom's bad service; Netcom is the main provider I know of for Southern California. (I am based in Van Nuys, California, which is in the San Fernando Valley. I believe myself to be the only full-access provider native to that area, although there are a couple of BBSs that offer various forms of access to newsgroups).

Bryant Durrell was kind enough to drop me a line commenting on my anti-Netcom bias. This is really quite fair; he's right, because I have direct experience with his system and not his competition. In my experience, NO Internet provider that I know of, large or small, has been run without significant user complaints or even anger. This is not to say that you or I might not do better someday; it's a cautionary remark to note that this is a far more complex business than it looks. I may badmouth Netcom a bit in this FAQ, and in a sense this is not unreasonable because it's the elephant of our industry. Pro Netcom forces might note that my Netcom account is still active; I use it to read newsgroups I don't get, and to test telnet to my system and other things.

My thanks to Draper Kauffman for providing the initial inspiration for this section.

4.2 How do things pencil out? Some reasonably hard numbers

[NOTE: This is a brand new section of the FAQ, written by yours truly with minimal input from the outside. I'd greatly appreciate comments or corrections].

One of the most controversial aspects of Internet provider lore is how much load you can put on your system and its Internet connection before things become intolerable to your customers.

To run a high-quality, conservative service, the consensus seems to be the following:

28.8K SLIP: You can run three phone lines and get reasonable, but not great, service for all of them. This is what I'm doing at present, and it does work well, although the lag in telnet connections (caused by high latancy on the phone lines) is bothersome to many. CSLIP (compressed SLIP) seems to produce a dramatic improvement over SLIP.

56k: You can run up to eight phone lines and get satisfactory service. However, see our new section "What about a 56k line?" for some details and some evidence from 56k defenders.

T1: Now we're motoring! A T1 line can support up to 27 times more lines than a 56k - thus, around 200 phone lines.

Here are some rough rate calculations, together with the provider that supplied them:

-------------  ---------  ------------  ---------   ----------  -------------

Provider       Service    Monthly Cost  Line Cost   Total Cost  Cost/Incoming

Cinenet 28.8 SLIP US$ 125.00 20.00 145.00 48.00 Internetworks 56k US$ 350.00 100.00 450.00 56.00 Net-99 T1 US$ 1,999.00 500.00* 2,499.00 12.00 ------------- --------- ------------ --------- ---------- -------------

* This number is off the top of my head; quotes I've seen for T1 connections range from a shade under $ 400/month to well over $ 1,000.00.

Net-99 has recently announced a discount price of $ 1,699, but it requires a three-year contract period. I'm not sure if a three MONTH contract period isn't too risky for this business! :-)

Net-99's connection allows full resale, including SLIP. Internetworks' allows shell account resale but no SLIP or other IP sale. Cinenet is unlimited, but resale of a 28.8kbps connection, even as SLIP, doesn't make a great deal of sense. Internetworks would charge $ 500/month for a full resellable connection. Net-99's fully-resellable 56k is $ 999/month. Their non-resellable T1 is $ 999/month, so you can see that resale is supposed to bring a substantial burden to the backbone provider.

Cinenet is my present provider for SLIP; Internetworks is a provider I've been considering for 56k, and Net-99 is the provider I would most likely use for T1 if I got that far. (Internetworks does not offer T1 in my area; Net-99's 56k charge is a rather stiff $ 999/month).

The "Cost/Incoming" column is the cost of the connection per each incoming dialup line. Each line should be able to accomodate about 10 subscribers, using the most conservative calculations.

It's easy to say that the T1 is by far the most appealing option here, unless you don't have investors able to pay $ 2,500/month until you're properly settled.

Let's dream a little and assume we could run each one of these options at full capacity. The numbers run like this:

Connection Phone Rate/L Cost/ Cost/ Gross Connection Cost/Line Lines Commercial Line User Income/L Total

----------  ---------  -----  ------------  -----  -----  -------  --------

28.8 SLIP       48.00    003  30 (bus)      78.00  7.80    100.00    300.00
56k             56.00    008  30 (bus)      86.00  8.60    100.00    800.00
T1              12.00    200  30 (bus)      42.00  4.20    150.00  30000.00
-------------  ---------  ------------  ---------   ----------  ---------

/-------------------------------------------------------------------\

A note to our European and Asian friends:  Rates for you are much |
higher.  In fact, they're over TEN TIMES more than this chart!    |
I believe Net-99 is planning international expansion; you might   |
want to contact them if you're interested.                        |
\-------------------------------------------------------------------/

Here are some sample European rates, kindly contributed by Hans Michalec :

FYI the recent rates for the data lines in Austria (via EBONE). Rates are in ECU (approx. =1$)/year. So, a 64 kbps line does approx. 33.000 $ a year - LUCKY AMERICA!

speed/kbps: 9.6 19.2 64 128 192 256 512 1024 1536 2048 3072 4096

commercial cost/kEcu: 14 20 33 60 91 118 206 292 376 442 694 832 *)

academic cost/kEcu: 8 11 15 27 41 53 92 122 153 174 275 320

*) commercial members whith their own US line get a reduction of 3 kEcu per 64 kbps access.

(end of quote)

The cost per user assumes 10 users per line, the standard. Gross income per line assumes rates of $ 10/month for 56k or below, going up to 15/month for T1 service. Running the numbers shows extremely low gross profit for 56k or 28.8 SLIP connections using the standard measures.

Of course the T1 user can also be a SLIP account, which commands about a $ 20/month or more fee. So you can see how much better a T1 is than a 56k; you can not only sell more accounts; you can sell more expensive ones. To make matters worse for the 56k connection, the standard of 10 users per phone line is effective for about 16 phone lines and up. For fewer lines, you may be talking about 7-8 users per phone line, because the concentrations of users tend to clump more for a small number of people.

Louis Epstein [lepslog@j51.com] reminds us that in some areas, a fractional T1 can be a good investment. Where he is, for example, he can get a 384k connection for $ 1,600, while a full T1 would cost him $ 2,200 a month. I had talked to people about fractional T1 service before, and my impression had always been that it was not that good a deal compared to a full T1. This example doesn't exactly refute this; for only 50% more a month, you get four times the bandwidth. But if you're in a tight budget, and don't have use right now, it sounds like an option well worth checking out. In the case of his provider, they will upgrade to the higher speeds for just the price difference between the fractional and full T1. So when you're ready, it's easy to switch. This is a big black mark against the 56k, whose service needs to be completely changed and re-installed when you convert over to T1.

28.8K SLIP deserves mention as a viable option for the start-up provider, just to get its personnel familiar with the various pitfalls of running a system connected to the Internet. I've learned a great deal with my 28.8, and recommend it very highly to anyone considering a start in this business. As you can see by the table above, it's no moneymaker, but it's not a dramatic money loser, either. Anyone with a more or less decent daytime job could maintain a 28.8 SLIP provider for as long as their interest continued.

The main problem with a 28.8 SLIP is its main advantage: You do not use a router or a CSU/DSU. It's good, because you don't need to cough up the cash for them. But you're not going to get to know one, either, and that can be a problem when you upgrade to a better connection. Another disadvantage is that you can't distribute the load between machines; SLIP connections are pretty well confined to a single unit. (I am having this problem now; expiration of news is taking a long time on my system, so I would benefit greatly from having a news server. Alas, I can't hook one up because I'm just connecting a single machine and not a network).

The 56k option looks like an extraordinarily unhappy compromise. If we subtract basic expenses of $ 86/line from the gross income of $ 100/line, we get only $ 14/line in gross profit, or $ 112/month for an 8-line system. Obviously, nobody's going anywhere on a 56k connection unless they either charge a lot more than the going rate as I know it, or bend the rules to the breaking point. However, it certainly is a way to get to know your router and CSU/DSU without spending massive amounts on them.

Finally, if you are lucky enough to be able to afford a T1 connection to the Internet, and have a successful marketing plan, the T1 option is very clearly an excellent viable business. If you could fill up all 200 lines a T1 is capable of serving at a rate of $ 15/month, you'd have 2,000 users and $ 30,000/month to dispose of. You'd probably need to hire a few employees at that level - it couldn't be done properly with just one person - but you'd be able to afford them. At this level, you could also resell 56k network connections to companies that needed their own net presence. Joe McDonald says that this is a surprisingly easy thing to do, and should be considered in any projections involving T1 lines.

Tony Sanders suggests fractional T1 as a good way to 'ease in' to the real thing. "You pay full T1 line charges but the total cost is quite a bit less and it makes for a very easy upgrade path." This seems to depend a great deal on both the provider and the telephone company you have to deal with. When I was talking to ISI Network Associates and Pacific Bell, I found that ISI charged the same for Frac-T1 and T1, and Pacific Bell only charged about 20% less for the smallest increment of frac-T1 (256k).

T1 people can also sell SLIP connections reliably, which are generally significantly more expensive (in the $ 20/month and up category). SLIP is generally a high-bandwidth eating operation, so normally you won't want to sell this form of connection on a 56k or lower line. However, note well: Joe McDonald has successfully sold and operated SLIP connections on a 64k Frac-T1; he says both he (when he calls in from home) and his customers are happy with the service.

Craig Warner ; http://www.ceram.com/sells an Internet Starter Kit for a bit over $ 20,000 (see the section on Sun hardware, below). He says that this should support 1000 users before upgrades; projected break-even point is around 700 users. "One thing to keep in mind is that business accounts can generate a great deal more over individual accounts - we paid about $ 200 a month - with only a dial-up UUCP connection until recently. We now have a 56k Frame Relay connection which costs about $ 400 per month in total fees." The aforementioned Joe McDonald charges $ 200/month for permanent 28.8k SLIP; he already has one such customer after only two months of operation.

4.3 What if I oversell my connection?

Probably the most tempting option for the 28.8K or 56k provider who'd like to get some decent profits out of his system is to oversell the connection - that is, to exceed the recommendations listed in the previous section.

The argument is seductive: Many people are cheap. They'd rather have an inexpensive connection than one that worked perfectly. I (the start-up provider) am just one person, and I can't provide a perfect system in any case; I just don't have the capital for a T1 or a 24-hour staff. Could I play the ISP game anyway, by just selling an overstressed connection for less money than other providers?

The main problem with this game is that it's too easy to play. If you offer cheap service at cheap prices, there are bound to be people with more resources than you who can offer cheaper service at even cheaper rates. This is Karl Denninger of MCS.COM's comment: There will always be someone who can undercut you. Quality service, though, is very hard to come by in the Internet provider business. Many very large providers, such as Netcom, are highly vunerable to complaints about terrible service. If you can capture some of their customers by offering excellent service - even for more money - you probably have a very good shot at gaining market share, even over very large companies. So you may want to at least consider the high road, not the low.

Craig Warner adds: "As a case study, our provider failed to upgrade his hardware to keep up with growth - they lost over 500 accounts in a month."

Michael Krause is an excellent example of a provider who's managed to succeed despite the handicap of a slow link to the Internet. His system runs 8 phone lines on a 28.8K link. What makes this possible is that only a few services - most notably FTP and downloading images through WWW (which shell or BBS account people can't do anyway) require a major proportion of the data pipe. Mudders, Telnet and IRC users are low users of the system capacity. Usually he sees a maximum of two simutaneous FTP sessions, which can be handled by the system. He points out that, as long as his customers don't have experience with faster providers, the speed is not a tremendous issue with them. He suggests that the peak number of users on a 56k would probably be between 75 and 100. A couple of other people have mentioned 40 as a "reasonable" maximum number of simutaneous connections on a 56k with light FTP traffic. Note that heavy use of FTP and WWW may change this dramatically in time.

The aforementioned Craig Warner suggests: "Intelligent local caching is another way to survive with lower bandwidth. A 3-5GB cache on a server could cut bandwidth requirements significantly."

From personal experience, I know that users will stick around after just about any disaster, as long as access is free, and it's understood beforehand that the system is experimental. I knew that my system would be unreliable due to the new software I was writing, and my inexperience as a system administrator. As a result, I started by running it at no charge, with the understanding that lost mail, connection problems and such would be accepted as typical experimental system pains. So far, the overwhelming majority of my users are very loyal, but that may change when I start charging even a nominal amount of money. In short, giving service with problems at a low price may be a reasonable strategy, but unfortunately there's a big difference between "low" and "free" in most people's minds.

A couple of my users have told me that they don't want to pay for the system as long as telnet connections are so slow, so even my strategy may not pay off. This may, however, be due in part to problems with runaway processes I've had on my system, which have decreased performance for all users. Since then, however, I've solved my runaway problems and gotten a CSLIP connection, which is much faster, and I seem to have silenced the doubters.

News and mail reading and writing, of course, use virtually no bandwidth at all, and you could run a news/mail only system on a very slow SLIP connection. However, the value of this to the general public is questionable. Still, if this is the primary interest of your users, they'll probably be pleased with virtually any bandwidth level. Note, however, that even a 28.8k SLIP doesn't seem to be enough for a full newsfeed through INN, at least through my present provider. (Things have improved dramatically since I started using CSLIP with them, but how much I'm not sure at this time).

Many people get Internet access through their work or school, but those institutions normally censor the content provided. For example, it's pretty tough to find alt.sex.bondage on a corporate machine, unless you're its manager. This opens up a surprisingly large market of people who have access already, but want to telnet in to check out the "forbidden" topics and sites out of their school or employer's wary eye. This is likely to grow with the recent decision at a major university (I think it was CMU) to halt access to sexual newsgroups. This might make a system with an excellent news connection and nothing else a viable site. It's also an excellent way of competing with the Freenets, which are cheap but heavily censored. Many providers, including MCS (Karl Denninger's provider), offer cheap telnet in accounts for that reason. Because the users are just using a small portion of your network connection and no phone lines, you can charge them a lot less for access and still make money.

I have noticed that, as long as FTP works, the slow speed doesn't bother me much. What does bother me is delays in character echo when I type. This seems to indicate that, if your system hardware is fast enough, you may be able to deliver service that's perceptively better than Netcom's even with quite a sluggish link.

My conclusion to all this is that many ISPs can get away with bending the rules for a while, at least until the competition gets a T1. Then, all that careful business planning and development may go to naught. As Tony Sanders summarizes the situation, "It's kind of like a race for the T1 connection :-)"

4.4 New Information on the Viability of a 56k line

Several people have written recently that I have significantly understated the potential of a 56k line. Apparently many people are running between 10 and 20 simutaneous users off a 56k with considerable success and happy customers.

The reason for the discrepency has to do with the usual use people put to the line. In a traditional Internet provider, a large percentage of users were engaging in FTP sessions; the minimum allowable bandwidth was based on a large percentage of users continuously FTPing stuff. Now, most people who were formerly FTPing are using the Web. Fortunately for providers, this means that most of the time they are just looking at documents, instead of sucking them up and departing. As a result, people stay on your system longer, but use less of your bit pipe.

Christopher X Candreva writes:

"I'm currently running off a 56kb: Full news feed, 20 dial-in lines. It's just getting tight. Large FTPs are slow, but PPP people are still limited by their 28.8 modems. The only people who complain are shell FTPers.

"And I'm upgrading to T1 by the end of the month."

Jacob Westfall has a similar experience.

"I run a small ISP with SLIP/PPP users. We have 20 incoming lines and get 150M of news a day. Typical transfers for the 14.4 users is 1-2k/s. Most of them use WWW and speeds on those lines are pretty good. Most pages load in a couple of seconds, the longest page to load I have seen was 1 min. The only complaints about speed I have ever had were first time Internet users who have just finished using a BBS and complain that they aren't getting the same transfer rates. Some of my users were signed on with larger Internet providers who have full T1s and they say the speed with my company is the same as with their previous provider. The main issue is not overcrowding the line. 20 lines on this 56k is as far as we are going. Interesting note: Our provider has a T1 and says he is only using about 256k of it. What most people have to realize and this is sysadmins included is that IP is not constant traffic. Watch the lights on a modem transferring IP sometime and you get the drift. 80-90% of the time the lights are idle while customers are using the web. What you have to look for in a Frame Relay 56k which is what we have, is that the provider you get it from has a large enough link into the cloud. Our provider had 128k into the cloud and just doubled that to 256k. Our speeds have almost doubled in terms of NNTP transfers from them and overall user speeds have jumped about 20 percent."

One way you might be able to stretch a 56k is to use 14.4kbps modems instead 0f 28.8. Most people still have 14.4kbps, and they are likely to be quite happy with service off a 56k line. 28.8 might be somewhat more probematic, at least for all your lines.

Note, however, that many of those people are in places such as Canada where T1 lines are prohibitively expensive, and 56k service costs almost as much as T1 here.

Still, even in the US it would appear that you can get a 56k line to work as a provider, as long as your service offers a stripped down newsfeed. A full newsfeed of around 450 megabytes a day is definitely not going to fit well within a 56k. Many 56k providers solve this problem by getting a Pagesat newsfeed (around $ 500-800 for equipment plus $ 30/month); this significantly decreases the burden on the connection. More information on Pagesat is available elsewhere in this FAQ.

Note that if your newsfeed is slow, you may have the equivalent of a partial newsfeed without realizing it. INN will not use all of the data pipe for transferring news, at least not in my experience. As a result, my system, running off a 28.8k SLIP, probably isn't getting anywhere near a full feed, even of the groups (alt, rec and misc) I'm getting. This may make your 56k or lower connection appear to work better than it actually does.

Despite this apparent good news, everyone I've seen is going to a T1 or trying to do so. Once you hit that 20 user barrier, you're almost certainly going to need one, and need it yesterday.

If you can afford a T1, get one by all means. But if you're on a shoestring, and have some way you can compete with other providers in your area, a 56k just might do the job. If nothing else, it'll tell you if there's demand for your service before you take the plunge and start spending the big money.

4.5 Draper Kauffman's 56k Connection Sudden Death Scenerio

This message is both so scary and so dramatic that I can't resist keeping it in the FAQ, even though I've summarized many of the financial arguments above. The message is that if you have a 56k connection, you have at most a $ 500 gross profit, even with a $ 20/month rate. And $ 20/month is unsustainable in today's market. Here's what that $ 500 would cost you, if you could even get it at all:

From Draper Kauffman :

How hard do you have to work to get that? To begin with, if you are growing fast you've recruited a lot of new users recently. They'll have a million questions and requests. You're also doing all the accounting, putting out accurate monthly invoices for 100 erratic users, depositing 70-80 checks, and deciding what to do with the no pays and late pays. Meanwhile, you're trying keep your system up and your 8 bargain modems working, and deciding what to do when mail runs 2 days late, or your newsfeed stops coming, or whichever of the normal Net-crises hits you that day.

All told, you probably put in 50-70 hours per week, maybe a lot more. You make less than a ditchdigger, and you aren't getting paid for the use of your equipment, so you haven't got any source of cash for upgrading or expanding your system.

But you're still adding customers! And here comes the crunch that threatens almost every low-capital ISP these days: too many users asking too many questions, not enough bandwidth, people bitching about how the system slows to a crawl every time you get your newsfeed, programs crashing because the user disk is full, and so on.

Too many problems are allowed to fester and turn into flames, and suddenly you just can't cope. There aren't enough hours in the day or dollars in the bank.

Angry customers quit and bad-mouth the system and new people stop coming. If you keep your rates at a moderate level you won't have enough volume to cover your costs. If you drop your rates to bargain levels, you keep more customers, but you're losing money on every one, so that's no help. You're bleeding cash, and pretty soon some unforeseen expense will put you out of business. You probably won't even know it until your check to the telco bounces, since you haven't had time to do the bookkeeping for months.

That's the nightmare. Even if you can raise more capital at that point and try again, you have to fight the bad rep of having run a shoddy operation. And bad word of mouth hurts you more on the Net than in almost any other business.

What causes this scenario? Here are ten good reasons:

1. Having insufficient capital.

2. Underestimating the time and routine expenses involved in a startup.

3. Overestimating the owner's knowledge, abilities, and stamina--the heroic programmer complex.

4. Grossly underestimating the manhours needed to get the system up and keep it running smoothly under load.

5. Starting with inferior services: slow connection, inadequate disk space, skimpy software (gopher, lynx, etc.), slow or partial newsfeed, and/or running too many functions (news/mail/users/etc.) on one CPU and drive.

6. Charging too low a price (as a result of points 2 through 5).

7. Using cheap hardware and no backups.

8. Starting with a system with insufficient capacity to produce enough profit to finance continued growth.

9. Allowing growth to exceed the sustainable system capacity.

10. Sloppy and inadequate accounting--it's easy to get behind, and usually fatal.

(I would add inadequate marketing to the list, except that that can easily be a blessing. One highly successful marketing effort could add 2-300 users in a week, completely overloading the system.)

Here's Draper's summary of the present environment:

Although there is market to market variation, today's ISP startup faces a significantly more difficult challenge than those that started in the last few years. Quality expectations are up and user fees are down. The result is that low budget/low volume/low quality/low cost systems can no longer expect to find a profitable initial niche in most markets. Without new capital or a steady stream of profits, they have no way to increase quality or capacity

Increasingly, a new service in a competitive area needs to offer a full range of services, good user support, and a fast, reliable, and accessible system in order to charge a premium price. Success will require larger amounts of capital or inventive ways to overcome the numerous barriers to low-volume profitability.

4.6 So, what can we do about this?

These are my personal suggestions as FAQ maintainer on what to do if you're a low-budget IP startup and somewhat intimidated by all this:

(1) Create some innovative services. For example, I'm coming up with a unique program to make it easier to access the net. I'm also developing a fresh new Pipeline-like GUI system. A more viable example for less technically oriented people would be to spend a few hours a week net surfing and reading net oriented publications like Wired. Then, you can announce the "newsgroup of the week", "URL of the week" and "Telnet Site of the Week". It would also not be a bad idea to put out a monthly newsletter that contained that information and told people to be sure to come online for that and similar events.

I think if you make your system a valuable resource to show people what they can do on the net, you'll build loyalty that will pay off when the crunch comes. And you don't have to do that just through direct personal communications; inexpensive media such as a simple informational menu and newsletters will do fine.

(2) Try to find a backup source of capital. From what we've seen in previous sections, it's highly doubtful that 56k is a viable solution, especially if you need to share your returns with investors. As a result, it's pretty much T1 or nothing. Costs for this seem to vary dramatically depending on your market, as I've covered previously.

(3) Don't quit your day job. A 28.8k system will work fine for you to test your ideas until you can get financing to get closer to the big time.

Next section: The Big-Time Competition: Should you worry?