AT&T Synaptic Compute as a Service Case Study
Detroit Water and Sewerage Keeps its Data Flowing by Migrating to a Cloud-based Solution (Cont'd)
We don’t have to worry about maintaining the hardware or the operating system – we rely on AT&T to continue to provide secure and reliable service.”
- Anil Gosine, Process Control System Administrator, Detroit Water and Sewerage Department
About Detroit Water and Sewerage
Detroit Water and Sewerage Facts
Maintain high-quality public drinking water and wastewater services with flat or declining revenues
Ethernet-based enterprise network, Network-Based Firewall, AT&T Synaptic Compute as a ServiceSM with VMWare vCloud® Datacenter Service
Ability to deliver cost-effective and exceptional service with fewer resources
Municipal water authority
The Detroit Water and Sewerage Department (DWSD), one of the largest municipal water authorities in the nation, is charged with providing water for the city of Detroit and 126 suburban communities representing 40 percent of the state’s population. In addition, DWSD handles wastewater services for the city and 76 neighboring communities. Its employees maintain a system comprised of remote pumping stations, water treatment plants, wastewater facilities, and some 7,000 linear miles of water and sewer lines.
As a not-for-profit entity, DWSD is only permitted to charge its customers what it costs to provide its services. The Department is faced with a shrinking population and revenue base, but its aging infrastructure requires continual maintenance and upgrading. DWSD operates sophisticated networks to support business, process control and security communications. DWSD is careful in the selection of hardware, system solutions and providers for these business critical services. The Department operates a 24x7 operation and extends the life of the equipment through hardware and software upgrades to gain maximum return from the capital investment. However, periodic technology refreshes are required to maintain the performance and service ability of the system and expand its capabilities.
Over the past few years, DSWD has migrated from an older point-to-point network to an AT&T Ethernet network service. This has dramatically improved network uptime and lowered administrative costs, and supports the process control systems, business systems and security systems to meet the operational needs of the Department. In addition to the scalable network infrastructure, AT&T has also provided a network-based firewall to enable added security. Most recently, AT&T helped DWSD transition to cloud computing in order to quickly turn up new applications without investing in or maintaining hardware. These solutions allow for enhanced service at a reduced cost.
Managing the Full Water Cycle, Coming and Going
At first glance, the task of delivering drinking water, then carrying it away may seem like a basic, low-tech operation that relies mainly on big pipes in the ground, storm drains, sewers, pumps and valves. The reality, however, is different. Managing this critical “endless loop” requires a meticulous and highly precise orchestration of processes that span hundreds of square miles. And it all must run flawlessly and reliably, 24 hours a day, 365 days a year.
Being able to deliver exceptional service with less manual intervention and less overhead is the key for organizations like ours.”
- Anil Gosine, Process Control System Administrator, Detroit Water and Sewerage Department
DWSD’s overall operation starts with drawing fresh water from both Port Huron and the Detroit River. This is then continually processed into potable drinking water at a number of water treatment plants at the rate of about 596 million gallons a day. From there, more than 200 sites transmit, distribute and meter the water to the millions of customers in the Department’s service area.
DWSD is also responsible for collecting wastewater and storm runoff for much of its service area. This is carefully treated through a complex system of retention and treatment facilities, in compliance with Federal and State regulations. The fully-treated water is ultimately returned to the Detroit River.
Managing this process involves a Distributed Control Systems and supervisory control and data acquisition (SCADA) solution that provide monitoring and controlling capabilities for several hundred individual locations, pumps, meters and control devices – all of which are automatically adjusting to customer demand throughout the day. From its System Control Center, DWSD relies on a combination of a radio system and the AT&T network to see into virtually every remote site, allowing staff to address issues that need attention and to fine-tune processes as required.
Staying Efficient and Highly Secure with Technology
Like many cities in former industrial areas, Detroit has experienced hard economic times. Its population has dropped, moving the city from the fourth largest in the U.S. to 17th. The demands on DWSD remain high even though revenues are flat. This challenges the Department to find ways to deliver must-have services as its plants and infrastructure ages.
More often, DWSD turns to smarter technology – such as advanced water meters, new monitoring devices in the field or a more modern phone system – to increase efficiency and reduce costs. When its old point-to-point network showed signs of age, DWSD migrated to a newer technology. The Ethernet-based service provides redundant connections to the majority of the Department’s remote pump stations, water treatment plants and control centers so they can be operated and controlled without onsite personnel 24x7.
In addition to substantially lowering maintenance costs, the network created a more flexible platform for the operation. Today, DWSD operates its entire process control systems over the network, as well as its business and billing operations, remote security systems and everyday voice communications.
For security purposes, DWSD configured the network so that its various operations – security, business and process control – are functionally separate and cannot communicate with each other. If one system is ever compromised or is affected by a virus or worm, it won’t compromise the other networks. DWSD will be further enhancing its operation by moving to a newer Switched Ethernet offering in the near future to provide class-of-service capability.
Realizing that AT&T could offer more robust protection than anything the Department could build internally, DWSD deployed AT&T Network-Based Firewall Service to help protect the network where it connects to the Internet.
“We are now less concerned about a cyber threat getting past us as it would need to get past the AT&T network first,” said Anil Gosine, Process Control System Administrator for the Department. “Just as important, I don’t have to devote any full-time staff time to managing and monitoring our own firewall. We manage it and AT&T maintains it.”
Streamlining Field Work via the Cloud
Most recently, DWSD had been dealing with two operating issues that were solved with the help of an AT&T cloud-based solution.
For one, a number of the utilities that DWSD supplies had been requesting real-time, process control data — such as flow rates — from the pumping stations serving their areas. The information would help them manage their own operations more effectively. Since that data lives in the DWSD process control systems, there was no easy way to deliver it, short of having an operator read it off a screen or allowing outside access to DWSD systems, which would present a security problem.
Secondly, whenever a DWSD field technician visited one of the hundreds of remote sites to perform preventive maintenance or calibrations, it could involve a 30-minute cell phone call with an operator at the process control center. This was to verify that what the technician was seeing on site matched what was appearing in the process control data. While necessary, the call was tedious and intrusive for the control center operator. On the other hand, allowing the field technicians to access the data remotely could pose a security risk.
When Gosine discussed these challenges with his AT&T team, they settled on an idea that solved both issues. The process control systems now transmit the remote data in near real time to virtualized servers in the AT&T cloud using the AT&T Synaptic Compute as a ServiceSM with VMWare vCloud® Datacenter Service.
Since it is strictly one-way communication, this solution allows easy access to the data, without compromising the security of the Department’s process controls. DWSD customers can now log into the ‘cloud’ location for easy access to flow data.
Likewise, field technicians can access the data via netbook computers over the AT&T wireless network, without calling into the center. By working from a duplicate feed, they can see any adjustments reflected in the process system almost immediately. Technicians in the process control center are no longer interrupted by lengthy phone calls.
Gosine estimates that using the AT&T cloud solution is about half the cost of deploying such an application in-house, considering the savings in hardware, maintenance and management time. The results were so favorable that DWSD is now experimenting with AT&T Platform as a Service to allow for easy development of applications in the cloud for fast deployment into its systems.
As Gosine sees it, technologies like cloud-based services will play a critical role in DWSD’s future. “We don’t have to worry about maintaining the hardware or the operating system – we rely on AT&T to continue to provide secure and reliable service. I only need to have one person work on our application on a part-time basis so it minimizes the staff commitment.”
As the DWSD data centers get closer to their end of life, Gosine plans to review long-term strategies to determine where it might be advantageous to move services to the cloud-based infrastructure. This would reduce the overall infrastructure and resource footprint as well as the investment in hardware.
Regardless of its size, the demands for the services that DWSD provides will not shrink. “Being able to deliver exceptional service with less manual intervention and less overhead is the key for organizations like ours,” said Gosine.