Office of
Information
Technologies

As part of the IT Architecture Initiative, the Office of Information Technologies (OIT) is producing a series of papers outlining directions in information technology architecture.

In the spirit of RFCs, the papers are intended to facilitate understanding of and open dialogue about information technology trends at Cornell, with the ultimate goal of improving the utilization and interoperability of information technology services throughout Cornell.   

---

pdf version

Network Service Billing Strategies at Cornell

Prepared by R. David Vernon

SYNOPSIS

This document outlines the Cornell network design and related costs to provide context for an exploration of network cost recovery strategies. It includes:

• Network elements overview.

• Network resource consumption relative to the elements outlined.

• Client impact on network resources as a function of application type.

• Cost modeling alternatives.

• Closing thoughts and observations.

---

Network Elements Overview

In order to explore and develop funding models for network services at Cornell and have them broadly supported by the patron base there must be a common understanding of the elements that drive the cost of network services. For the IP data network at Cornell there are many elements that contribute to the cost of the services. In addition, there is a great variety in the impact that client applications have on the network.

In order to model costs it is helpful to view the network as being comprised of three integrated network services, each with its own cost ramifications. These network services are:

• Local Area Network services (LAN)
• Campus Area Network services (CAN)
• Wide Area Network services (WAN)

Local Area Network:

Though there is a fair amount of variation in how people define "LANs," for the purposes of this paper LAN services are defined as all client equipment interconnected by "switching" hardware. Switches are low price devices with limited "intelligence" that operate at very high speeds to interconnect devices within a single ethernet broadcast domain (Broadcasts from one host are seen by every other host in a given "broadcast domain."). Though it is beyond the scope of this paper to go into great detail about network hardware, the nature of the network data traffic movement "switches" perform is fairly finite when compared to network "routers." Therefore switches are much less expensive to build per port and per aggregate network capacity provided. Switch cost is also driven down by their "commodity" nature. They are a product sold in massive volumes by multiple hardware vendors. Most building intraconnections of hardware at Cornell are supported by low cost, very high speed (up to 100 Mbs), "commodity" switch network hardware. Often people define switches as "level 2" devices and routers as "level 3" devices. These "levels" are references to the OSI model for network services at large (Seven layer model for networking protocols and distributed applications developed by the International Standards Organization (ISO)).

In brief review, LAN's are comprised of "level 2," low cost, high speed, switched interconnected clients. Patrons of network services can think of switched LAN's as "cheap," "dumb," and "FAST" when compared to the network cost associated with level 3 devices often utilized in CAN and WAN network provisioning.

Campus Area Network

The Campus Area Network at Cornell is comprised of level 3 "router" hardware interconnected by very high speed data links that in turn interconnect campus switched LAN's. The cost per "port" on routers is orders of magnitude more expensive than the cost per port on network switched devices. This high cost is a function of lower product demand and the additional data manipulation routers perform. These advanced router functions require powerful processing engines to assure fast interconnection - thus driving up the price of these core devices.

While there is some variation in the number of level 3 devices deemed critical in Campus Area Network design, the use of routers in the Campus Area Network to interconnect LANs instead of low cost switches is driven by the administrative and operational control routers enable. In addition to others, these controls include improved security and limiting the ability of users within one LAN broadcast domain from stealing the IP numbers of a user in another domain.

In brief review, Campus Area Networks at Cornell are comprised of "level 3," interconnected, high cost per port routers. Routers are desired because they enable administrative control over the campus network not enabled by low cost network switches. While there is a bit of debate on the number of these devices needed at Cornell, to date CIT has elected a conservative approach that maximizes the number of router ports required in order to provide high operational control over the larger network.

Wide Area Network

Like the Campus Area Network, Cornell's connection to the "Internet" is provided by a routed interface. However, the primary element that drives cost in the wide area is the expense of the data links that Cornell leases from Internet service providers. The cost of "internet bandwidth" is orders of magnitude more expensive than the aggregates cost of data services provided by the Campus Area Network hardware. For example a dedicated 100Mbs switched port in a LAN, if capitalized over 3 years would cost ~ $30 a year (Port fee does not reflect media and associated costs such as middle-ware, etc.). In turn, a 155 Mbs Internet link currently costs ~ $800,000 per year!

Base Costs Associated with All Network Services

There are two additional classes of costs associated with network use: media and general service costs. These are described in the two sections that follow.

Media (Copper/Fiber) and Costs:

Implied in the above outlines of CAN and LAN hardware costs is the additional cost associated with the media used to transport the signal to and from devices. On the Cornell campus LAN connections to clients are based primarily on TP copper, and CAN interconnection is done by fiber. Generally fiber is capable of carrying larger amounts of data longer distances than TP copper and therefore is ideal for interconnecting distant locations. In turn, copper is excellent for horizontal building distribution where distances tend to be less than 100 meters. The cost of provisioning and maintaining these media plants are part of the larger network service delivery costs. Fortunately the usable life span of fiber and copper is very long so the capital costs can be spread out over an extended period (in excess of 15 years). Unfortunately at Cornell the building wire installed is very old (more than 15 years) and limited in the total speeds at which it can carry data to ~ 10 mbs. In addition, despite the relatively long life and low cost per port when capitalized over 15 years, network services fees have not reflected any media expense to date. Given these facts, Cornell is faced with the quandary of how to best replace the existing older "cat 3" wire. If this were to be done en masse the total one time costs could be significant and would impact any final rate structure. For additional information about campus rewire directions please see http://www.cit.cornell.edu/oit/Cornell_Network_Futures.pdf.

Other Associated Costs

In addition to media expenses, it can be argued that there are general service costs directly associated with network service delivery. These often include:

"Middle Ware" such as:

• Traditional network Name Services (DNS).
• General directory access tools such as LDAP.
• Network authentication and encryption tools - such as Kerberos / PKI.
• Dynamic IP number allocations tools such as DHCP.

General support services such as:

• Network Operations Services (NOC).
• Security / incident response.
• Network Help Desk Support.
• Network Research and Development
• Maintenance

While some may debate what should or should not be included as a general network cost, once defined the final list is universal and must be reflected in any network costing model along with CAN/LAN/WAN hardware and media expenses. Clearly a failure to maintain currency in evolving middle ware applications or evolving network service demand will impact Cornell's ability to participate in larger "global" exchange of information with peer institutions. For example, unless Cornell participates in the development of 'standards" based inter-domain authentication schemas, Cornell will not be a "trusted" peer allowed to exchange "authenticated" information. The ongoing research and development of these tools and related expense is an assumed part of the larger middle-ware base universal expense of Internet service provision at Cornell.

Estimated network port provision element costs as a % of total expense is as follows (Graph represents % total cost associated with deployed hardware not % of "capacity" for WAN/LAN/CAN resources):

With the above outline of the nature of LAN/CAN/WAN deployments, it is self evident that the cost of a given network services at Cornell is a function of the path a given data communication takes, plus any other associated networking costs. If a patron of Cornell's network only communicates within a switched LAN, the cost of service can be VERY low, however, the same amount of data pushed across the CAN and WAN will be much higher.

Conceptually these cost zones can be modeled as follows:

Client Impact on Network Resources as a Function of Application Type

In addition to the varying costs based on network zones, not all clients with the same connection to the campus network impact the network equally. The point is not to belabor the obvious notion that a "video" application running on a computer connected to the network would use more than the same computer simply running a mail client, but to note the fact that a computer with a "100 Mbs" connection to the network viewing a video from a remote location, may often use fewer WAN resources than a "server" or multi tasking computer connected to the network with only a 10 mbs connection. TCP communications across high latency Internet links may actually penalize connections attempting to run a single large bandwidth tasks.

In short, due to the nature of TCP/IP, compounded by a given Internet latency and Non QoS enabled network hardware, when it comes to a client's ability to consume WAN resources, the number of unique network connections per client can often be more important in determining impact on WAN resources than the size of the LAN port connected to the client. A wonderful example of this phenomenon is the impact of "shared " 10 Mbs connected "Napster" servers in the residence halls and their consumption of outbound Cornell WAN resources.

Multiport Repeater Impact on CAN and WAN Resources

Adding complexity to modeling data transmission costs is the growing use of multi-port repeaters, or "hublets" by departments to increase the number of connections within an office to support additional devices. Hublet impact on a given network WAN/CAN resources parallels that of multitasking vs. single tasking clients. If hublet installation enables multiple communication connections across the campus CAN / WAN the impact on campus and Internet resources will be high. However, if the hublet is used to interconnect devices within an office, such as a computer and printer, then the traffic is local and there is no additional impact on the larger campus resources. However, it is arguable that every device connected to a hublet, regardless of local vs. remote data paths, still consumes "base line" service, i.e., DNS, Security, Helpdesk support, etc. And data network connections connected to hublets consume more resources on average than data network connections connected to single clients. But of course, like single clients, the nature of the use can dramatically change the quantity of CAN and WAN resources hublets consume.

Cost Modeling Alternatives

As is apparent from the above overview, determining the true cost of a given network connection at Cornell is not as simple as charging based on the quantity and size of a given LAN connection. Network consumption or cost per user is a function of network port speed, datapath, application type, the technical nature of the IP Wide Area Network resources used to transmit IP plus any universal base support costs.

Summary of Current Average Cost Model

To date, cost recovery at Cornell has been rooted on a simple average cost model that takes the total number of network users and divides that into the total cost of the network service to determine the network "bill" per connection. Clearly this does not reflect actual consumption - it does not attempt to do so, it only reflects the average cost of the larger service per activated jack. Some patrons are getting a great deal, others are paying far more than they are utilizing. OIT and CIT are exploring refinements to the current average cost model, but the fundamental premise remains the same.

Complications in the Average Cost Model

One of the first challenges of setting up an average cost recovery process is the attempt to find the best measure for total user count. This is not as straightforward as one might initially assume. To date CIT has elected to use active ports as the base number (divisor) to determine average cost. However, with the growing use of hublets in departments to connect additional resources to a single "billed" port, the average cost derived income is gradually becoming threatened. To aggravate matters, the use of average "port" charge has placed a false economic incentive for departments to lower their total network costs by accelerating the installation of hublet and wireless repeaters to lower total port counts they are billed for. Clearly the continuation of this practice will prove unsupportable by Cornell at large as it undermines the ability to deliver advanced network services such as e2eQoS and required funding streams to maintain base, wire, CAN, & WAN network infrastructure.

Other Average Cost Models

Given the failings of the use of ports to determine the divisor for average expense, thought has been given to alternative average cost schemas. These include:

• Total IP number count
• Total MAC address count
• Total head count

Each has weaknesses as outlined below:

Total IP Number Count: While using IP numbers may well be a better current measure than port counts it is easily subverted by the use of NAT enabled hublets. NAT or "Network Address Translator" enabled devices would present only one IP number to the larger campus resource while supporting multiple hidden IP numbers to the installing departments. Again, as the port count schema seems to have stimulated the installation of hublets, using IP numbers as the base count would likely stimulate the use of NAT devices. For an IP count process to be effective there would have to be a university policy that made it illegal to use NAT devices and a process to police network installations.

Total MAC address count: Each hardware device on an Ethernet network has a unique identity, known as a MAC address. It is possible to write applications that "sweep" or "police" the network and to get a total device count. However, you can only assure sweeps through routers that are controlled by CIT. In addition, MAC addresses could be hidden by NAT devices. Therefore to be effective CIT would have to mandate access and control of all network routers on campus and have the authority to sweep networks within departments to get accurate device counts.

Total Head Count: This process is based on the assumption that on average the cost per person is a good measure of average network consumption at large. The advantage is creating a process that does not encourage "cheating" or false economic incentives to hide total network utilization. The disadvantage is this billing process is often perceived as a central "tax" on departments for a service they may argue they do not use or that does not provide enough value to justify the tax. In addition, head count or tax systems often encourage abusive consumption, as there is no direct economic consequence for excessive utilization.

Though alternative divisor count strategies are still being actively considered, at this time it is not clear that changing to MAC or IP count strategies would offer enough value to justify the operational expense incurred.

In the near term CIT and OIT has elected to address the proliferation of hublet installation and the higher average cost per network port they create by advocating a higher fee for ports supporting hublets. This has been referred to as a "single circuit gateway" rate. This higher fee is based on the assumption that on average hublets will consume more network resources than a single client does. Unfortunately as with clients, there is tremendous variation in the actual consumption of resources by hublets and, in turn the proposed "ISP" fee for hublets and or wireless repeaters has been meet with considerable community angst. This angst is not, per se, unjustified as many higher performance servers supporting multiple WAN sessions connected to a single port can and will use the same amount of network resources as a hublet supporting multiple "average" clients. This fact is not lost on many departments.

But, somewhat antithetically it is also true that multiple departments have installed hublets precisely to connect "average" clients with the sole purpose of lowering local networking costs. This process violates the fundamental fairness of average rate based billing fees to the larger Cornell community.

The bottom line is that all average rate based cost models are fundamentally limited and in turn subject to legitimate criticism. However, it is also clear that departments have been installing hublets and wireless systems simply to avoid legitimate network fees. If any average cost system is to succeed, Cornell must face the realties and limitations of this billing strategy and in the spirit of a larger community "play fair" and pay their fair share, once a process is adopted.

Alternatives to Average Cost Models

Despite all hope and good will, history at Cornell has taught us that departments will go to extraordinary lengths to figure out how to pay the lowest network fees possible regardless of the impact on the larger community. This is not meant to be a derogatory comment about departments, it is natural for departments to attempt to lower costs, and average cost billing based on ports, IP numbers, and MAC addresses creates a clear incentive to do so. Given this, short of a mandated head count based average cost system, Cornell may have to enable a non-average rate based system or a system that penalizes or limits collective abusers of the inherent weakness of a given average rate fee.

Proactive Identification of High-Use Users and Departments

In order to provide fairness within an average cost based billing system, there is active consideration of policing connections and if abuse is found, restricting access to broader CAN and LAN resources. An alternative but similar theme would be to average the "income" associated with a given LAN and then allow access to CAN and WAN resources reflecting that % of the total income. For example RESNET patrons would be allotted CAN and WAN resources to reflect the % of the total CAN and WAN expense RESNET subscriptions cover. This same model could be applied to all Cornell departments. If departments persist in the deployment of hublets or use excessive amounts of CAN and WAN resources, the larger department connection to the campus could be limited to the share funded via official port subscription.

Of course this is a rather draconian approach to network cost recovery. It implies an active policing process and would punish all members of a given "department" regardless of individual use. Alternatively there is a growing belief that short of a head count based cost recovery system, only a true "consumption based" or "rate based" approach to network fees will be viable in the long run.

Rate-Based Billing Applications

Cornell has recently entered into an agreement with Apogee Networks, Inc. to acquire Apogee's rate based billing application. Once fully deployed this application will allow CIT a fine tuned understanding of network traffic flows on campus. In turn, this information could be the basis for generating usage based network "bills." Depending on the final configuration, usage charges can be a function of:

• Time of day
• Class of users
• Data Type
• Prorated utilization of the three network cost zones (outline earlier in this paper)

An advantage of rate based billing not enabled by average based billing schemas is the proactive feedback it provides to consumers. It is empirically clear to all who bother to dig beneath the network covers a bit at Cornell that a significant percentage of networking costs are related to activities that departments may not desire to fund. For example, with rate based billing, departments would be empowered to encourage users who stream audio content across the Internet to "buy a CD player" as the CD player cost far less than the network resources used to provide the same. A little proactive communication within a department might have a dramatic effect on a given departments network bill. In turn as departments rationalize their network use, CIT could rationalize its future network development and optimize its investments.

There is a recognized concern that a rate based billing model might exacerbate the notion that "rich departments" will be able to consume all resources needed while "poor" departments' access to Internet resources will be stifled. In addition, some members of the community have expressed concern that a rate based billing model might have a "chilling" impact on the scholarly use of network services at Cornell. This general debate over the network as a "common good" vs. a more capitalistic approach of allocation is sure to cause interesting dialog. Regardless, it is also clear that the existing models for network billing based on average rates have failed to assure long-term funding for this critical Cornell resource. Possibly, if the Cornell community is unwilling to move to a more inviolate "head count" billing process, rate-based billing, despite the impact it may have on given departments access to internet resources, may be the only viable alternative.

Closing Thoughts and Observations

There is a spectrum of approaches that could be applied to network cost recovery systems. These range from simple and effective fees derived from total headcount, to the implementation of a rate based billing application. It should now be clear that given the nature of IP networking and the great variation in network consumption by given applications, average port fee models may always be challenged by individual users.

Pragmatically, if Cornell elects to maintain an average rate billing system, only a total head count (or general "tax") model may be immune from the enterprising individual attaching additional ports or hiding IP numbers in an attempt to lower network costs. Alternatively Cornell could embrace an empirically defensible rate based billing system for network service delivery.

For any funding system selected, it is important to understand its inherent limitations and ramifications. Traditional port/IP/Mac count based systems encourage ad hoc deployment of network hardware by departments that in turn threaten required income streams and future advanced network services. In addition, loss of income diminishes Cornell's ability to underwrite the cost of middle-ware and general support requirements of all network users. Simply stated, attempts to avoid paying legitimate costs impact advanced network services development-services that will be commonly delivered at every other major research university.

To avoid a failure to maintain prudent investments in the network infrastructure at Cornell, OIT, CIT and departments must work together to forge and fund an advanced network that has the potential to deliver new services and has a fair and supported process for cost recovery. Once network costs, application use, and the limitations of alternative models are understood, the best cost recovery system may well be limited to either rate-based billing or a simple "head count" based fee.

 

 

Return to Papers Page