Note: The Network Operations Center was moved to the seventh floor of Rhodes Hall in December 1997.
By Nancy A. Flynn
On a typical Friday night, while many of us are busy making weekend plans, staff at CIT's Network Operations Center (NOC) are doing what they do seven days a week, 24 hours a day, every day of the year: keeping the Cornell campus network--the highway of wires, routers, voice switches, concentrators and servers that support central applications and link most of the campus buildings to the global Internet--up and running.
In a dark, climate-controlled room in the basement of the Computing and Communications Center on the Ag Quad, NOC staff work in the network's "nerve center," an area that looks like the stage set for a futuristic movie, with its U-shaped console of computer monitors stacked two high, row of keyboards and telephones with multiple lines.
Whether they work first shift (8 a.m. to 4:30 p.m.), second (3:30 p.m. to midnight) or third (midnight to 9 a.m.), the nine NOC staff members (and two students who work two days a week) are busy around the clock. They answer phones that sometimes ring off the hook with users who can't connect to the network and get their work done. Staff members log problems, check databases and monitor systems to determine the cause; when necessary, they dispatch on-call technical staff to the problem site.
"The NOC's goal is to keep network and central system downtime to a minimum," said Andrea Beesing, NOC manager.
Late afternoon and early evening are the network's busiest time--for everything from logging into Cornell Library's online catalog to checking e-mail from home. Juan Salomon, NOC coordinator for the second shift, explained, "There's never a dull moment. That's one reason why we have overlapping shifts--so we can discuss the previous shift's activities and review any outstanding problems."
Cornell's Network Operations Center has been in this fast-paced business for seven years. In the days before the explosion of the Internet, it was one of the few places that could monitor external connections to national and international sites. Its 24/7 operations schedule helped the NOC secure contracts from organizations such as Sprint to monitor their international connections to Stockholm's NORDUNET, Inria's ICM-Sophia (France) and, briefly, to Japan.
When a connection to any of those sites went down, NOC staff worked to determine the cause. If the equipment was located in the United States, understanding the root of the problem--and potential solutions--was fairly easy; troubleshooting the telephone line and router on the international end was more of a challenge. According to John Becker, third-shift network operator since 1991, "The whole concept of such interrelated service was fairly new at the time--we often spent too long on the phone while the phone people tried to locate our circuit numbers in their database or while waiting for colleagues in faraway time zones to return from lunch or locate office mates fluent in both network acronyms and English." As the Internet evolved in the early 1990s, groups such as Sprint began to monitor their own worldwide connections. By that time, Cornell had enough sophisticated networking equipment and sufficient campus connections to keep NOC staff busy taking care of business on campus.
At its inception, the NOC mainly managed and monitored the central data network and Cornell's external connections to the Internet. In recent years, responsibility has widened to include key components of the campus telephone system, such as Blue Light emergency phones; central servers supporting applications such as e-mail; local department network components; and the Cornell mainframe, CornellC. Because the campus depends on the network, any downtime can quickly assume the status of an emergency. NOC staff need to be responsive to all problems, and operators on every shift must be able to identify the nature of many technical problems, notify critical system users and administrators when an outage occurs, and interact with the appropriate technical staff in CIT and at NYNEX, Applied Theory (formerly NYSERNet) and IBM who may be needed to troubleshoot and resolve problems as quickly as possible.
As more services are added to the campus network (911, Project 2000 applications and video, for example) and service reliance increases, the pressures on NOC staff continue to mount. "A sense of humor is a highly desirable trait in someone working on our area," said Michelle Mogil, first shift operator. She and Jenny Signor, another member of the first-shift crew, get their daily laughs from Dilbert cartoons.
True teamwork is more than a buzzword for the Network Operations Center. According to Beesing, the ongoing work of Cornell's NOC could not be accomplished without the expert assistance of technical staff throughout CIT.
"The average campus customer may never meet any of these individuals," said Beesing, "but working behind the scenes as a team with NOC operators, they play a critical role in delivering these service-often at all hours of the day and night."