Company Accelerates Application Performance Troubleshooting Customer Challenge
Headquartered in Herndon, Virginia, Boxwood Technology provides job boards and other online career center services for associations. The company is endorsed by the American Society of Association Executives and is a charter member of the International Association of Employment Websites.
Boxwood employees in two locations depend on business applications housed in a collocation facility, including a ticketing system, customer databases, Microsoft® SharePoint, accounting software, and an IT logging tool. Previously, if users reported slow application performance, Boxwood’s systems administrators used the “show IP NBAR” command to investigate. “But the command returned limited information about only a few protocols, and interpreting the data was time-consuming,” says William Bordeau, systems administrator for Boxwood Technology. “To accelerate troubleshooting and provide a good user experience, we wanted deeper insight into end-to-end application performance, and an easy-to-use graphical user interface.”
Complicating the challenge, Boxwood was preparing to replace a traditional phone system with an all-IP voice system. “We knew we needed the ability to visualize voice traﬃc from end-to-end, because voice quality aﬀects customer satisfaction when they call for sales or support,” Bordeau says. The existing, third-party NetFlow traﬃc analyzer would not work for voice traﬃc monitoring, because it relied on traceroute, which only shows traﬃc between routers, not between switches. This meant that a portion of the traﬃc ﬂow for interoﬃce calls, which goes through the Multiprotocol Label Switching (MPLS) cloud, would be hidden. To see the end-to-end ﬂow, Boxwood needed a router that supported Performance Monitoring technology that discovers switches as well as routers. Cisco® Performance Monitoring technology follows ﬂows hop-by-hop, collecting statistics across the ﬂow path. It leverages another Performance Monitoring technology, which collects diagnostic data including packet loss, jitter, hop-by-hop latency, and response time.
Ease-of-use was critical. “Our staﬀ is busy and needs a simple tool that displays actionable information without requiring them to jump through hoops,” Bordeau says. “We wanted an easy-to-use visualization tool that would require less detective work.”
Boxwood found a comprehensive solution to visualizing application ﬂows by using services on the Cisco Integrated Services Router Generation 2 (ISR G2) in conjunction with LiveAction (now known as LiveNX*).
- Cisco Application Visibility and Control (AVC) is a suite of services in Cisco network devices that provides application-level classification, monitoring and traffic control. It uses deep-packet inspection to identify more than 1000 applications, collecting performance statistics such as bandwidth use, latency, and response time. On a recent day, Cisco AVC reported that the top 10 types of traffic by volume included HTTP, Common Internet File System (CIFS), Exchange, Active Directory, Cisco WebEx®, Simple Network Management Protocol (SNMP), and YouTube.
- Performance Monitoring technology supported in Cisco ISR G2 routers, monitors voice flows from point to point, across routers as well as switches. “With Performance Monitoring, nothing is blocked from view, including voice traffic traversing our MPLS cloud,” Bordeau says.
- LiveAction provides an intuitive GUI for filtering the information from Cisco AVC and Performance Monitoring and presenting it in an easy-to-understand, visual format. If an employee reports slow network performance, Bordeau can visualize current or historical network activity on the LiveAction interface, which color-codes traffic links by volume and protocol. This capability makes it easy to see the protocol responsible for the saturation, such as CIFS for file transfers, and the originating endpoint. “LiveAction is a single interface we can use with Cisco AVC and Performance Monitoring to see congested areas, visualize flows from end-to-end, and apply changes that LiveAction pushes to our routers,” Bordeau says. “The visual representation helps us pinpoint the location of a network incident that’s affecting the user experience in just minutes, compared to hours or even days before we had the tools.”
Successes to-date include quickly discovering the sources of network congestion, detecting and mitigating an attack against the company’s web server farm, and identifying a misconﬁgured ACL before it caused voice quality problems.
Accelerated Troubleshooting of Network Performance Issues
Boxwood uses Cisco AVC and LiveAction primarily for diagnostics, to identify the cause of slow performance for TCP-based applications. “We use LiveAction on-demand anytime we need to troubleshoot or diagnose network performance issues,” Bordeau says. “If a user reports slow application performance, I can apply a ﬁlter in LiveAction to look at metrics from that server. Being able to visualize network activity by application, not just the port, is very valuable for us.” If LiveAction shows zero retransmissions for a large ﬁle transfer, Bordeau can narrow down the problem to the server, not the network. The combination of LiveAction and Cisco AVC also makes it easy to see if slow application performance is a result of client network delay or server network delay.
“LiveAction is a single interface we can use with Cisco AVC and Performance Monitoring to detect congested areas, visualize flows from end-to-end, and apply changes that LiveAction pushes to our routers.”
– WILLIAM BORDEAU, SYSTEMS ADMINISTRATOR, BOXWOOD TECHNOLOGY
Boxwood experienced the value of Cisco AVC and LiveAction soon after implementation, when email traﬃc slowed abruptly. Cisco AVC reported congestion in an uplink to the MPLS cloud, clearly visible on the LiveAction interface. From within LiveAction, Bordeau built a ﬁlter to look for ﬁle transfers with a few clicks, quickly identifying the source. Further investigation revealed that an employee in the sales and marketing oﬃce was backing up large, oﬄine mailboxes.
Blocked Network Attack
LiveAction and Cisco AVC also helped Boxwood’s IT team detect a Structured Query Language (SQL) injection attack on company web servers. The ﬁrewall at the facility reports the source country of all connection attempts. When Bordeau used the LiveAction to view historical data, he noticed multiple connections in a country where the company does not do business, always from 2 a.m. to 3 a.m. With this information, he was able to construct a ﬁlter that blocked the traﬃc.
Averted Voice Quality Problem by Discovering Misconﬁgured ACL
When preparing for the migration to uniﬁed communications, Bordeau connected an IP phone to the network for testing purposes. Although the phone worked, LiveAction showed that the connection was going across a diﬀerent interface than expected. With further investigation, Bordeau discovered that an access control list (ACL) was helping to enable calls to travel over the public Internet instead of just the MPLS VPN. “Without Cisco Performance Monitoring and LiveAction, we might never have discovered the error, allowing some calls to travel on a path without QoS,” he says. “Discovering the problem helped us protect voice quality for our employees and customers.”
Today, Boxwood uses LiveAction and Cisco AVC primarily for diagnostics. Later, the company might use the alerting feature in LiveAction to gain early awareness of issues before they aﬀect the user experience.
Bordeau concludes, “We’re excited to have the tools to collect detailed information about all application ﬂows and view it from one easy-to-use interface. And visualizing the network is helping us provide a great quality of experience by identifying and resolving issues more quickly.”