In the Trenches

A Modest Proposal for 2009

Mon, 05 Jan 2009 13:21:27 -0800

Happy New Year to all.

As we begin the new calendar year, our budgets are still a mess, all new initiatives are on hold, and we continue to find ourselves in "maintenance" mode. There are a number of technologies that would benefit our organization and the provision of service delivery to our community, however the capital needed to invest in these technologies does not exist. With some down time over the holidays I started to ponder the potential impact of the recession if it does last long into 2009 and into 2010, and what the implications are for local government IT.

Most of use have already explored having our applications hosted or the SaaS model, but what about taking the service model a step further and applying it in some fashion to data centers and networks for example. Generally speaking IT spending in local government is not on par with the private sector. In my case the City IT budget is approximately 2.5% of the total general fund budget. Tax dollars are used by the City, County, K-12, Community College, and surrounding towns to build, maintain, and support separate data centers, networks, IT departments, and licensing of applications. Wouldn't IT dollars be better spent on building one network infrastructure for all of these entities? Couldn't one ERP application be licensed and used for all of these agencies?

I think that you get where I am going with this. By pooling resources and eliminating redundancy of effort economies of scale can be realized and each governmental agency would be better positioned to use technology as a means of reducing costs and providing a higher level of service to internal and external customers. Yes there are tremendous political hurdles to clear, but given the fact that budget and revenue shortfalls will be the norm for the foreseeable future, the political will might be building for moving towards such a solution. In my simple mind, as an IT professional, tax payer, and civil servant this makes sense on so many levels.

I know that indeed there is some consolidation of this type happening around the country. I would be interested to hear from anyone with a success story or lessons learned from such an effort. I believe that this service model is the future of public sector IT and that the current economic conditions could be the catalyst to make this happen sooner.

Desperate times call for drastic measures - well maybe. I prefer to believe that every crisis provides an opportunity. In this case it is an opportunity to do the right thing.

A Fistful of Dollars

Tue, 02 Dec 2008 12:34:42 -0800

Our municipality, just like other cities, counties and states across our nation, is dealing with some serious budget shortfalls. I have been told that our situation isn't as bad as the estimated shortfall in other communities, but that isn't making it any easier to "find" the 50 million dollars that it is currently anticipated we will need to balance our budget at the end of our fiscal year.

We have been through the drill twice already. You know the one where every department must cut their budgets by some arbitrary percentage and freeze all hiring. All of our projects have come to a halt, our budget has been slashed to the point where all we can do is meet our basic contractual obligations, and we are expecting to hear of another round of cuts after the start of the calendar year.

While our current budget shortfall has left us with our heads barely above water, I really believe that our city is missing an opportunity (I hesitate to say yet that the opportunity has already passed us). I believe that during times such as these investment should be made in technology to increase and improve the services we provide and to do so for less than it currently costs. An initiative such as this will take analysis of current processes and improvement of or elimination of many of these processes. (Yes, I know what you are saying, we should have been evaluating and improving our processes all along, but truthfully how many of your organizations follow this business best practice? As long as times are good there is no motivation or the leadership to do so ).

How are you handling your budget crisis? How has it affected your operations, projects, and plans? What lessons learned can be shared to maybe ease some pain?

As in the spaghetti westerns - where is our man with no name who will come riding into our town to save us from our budget woes - for a fistful of dollars.

A Lesson Learned

Wed, 19 Nov 2008 08:03:55 -0800

One of the projects that we have been working on over the past 2 years is the migration of our myriad voice systems to a VOIP implementation. As I have mentioned in a past post we have a very extensive city owned fiber network that of course we are using as a transport for our voice communications. To date we have replaced over 70 of our 130 phone switches ( yes I know that we needed to go to a central system some time ago - just never the money or there were other more pressing priorities) and we have migrated some 3000 users to the VOIP system. I have been amazed at how smooth this has gone to date and I am pleased with the cooperation and enthusiasm shown by our users.

As with every project there have been some issues that have been raised that we did not plan for. We planned work arounds for the fact that some of our facilities do not have cat 5 cabling. We planned for some facilities and locations that would have to remain on leased circuits because we do not have the ability to make connections any other way. The thing that has really bitten us on this project are the upgrades that have been required to some of the communication rooms.

The project has required the upgrade of core layer and distribution layer network equipment. While this was planned for, it is also where our problems started. IT is not responsible for the power, cooling, ups or generators that feed our communication rooms. When we started to plan the deployment of equipment we were informed that a number of facilities would also require one or multiples of the following: More electrical power, more cooling capacity, larger ups's or larger generators. It was eye opening to me that we (as an organization) do not plan appropriatly for these rooms and that in some cases backup battery systems and generators were undersized to power the equipment that was already installed!

As a result we have learned a very valuable lesson and we are now working more closly with the people who are responsible for the care and feeding of rooms like this in all our facilities. I am determined to ensure that this is not an issue in future projects, and that we, at a minimum, review the capacity and loading requirements for these rooms and that we budget appropriatly (either in our O & M budget or in future project budgets) so that we do not neglect this part of our infrastructure again.

Technology and the Next President

Mon, 03 Nov 2008 09:13:31 -0800

With only a few hours until Election Day, I was pondering what the impact would be of a McCain presidency or of an Obama presidency as it relates to technology issues. Yes, both have stated their positions on Net Neutrality, but beyond that what policies would be instituted that might address other technology issues.

On other technology issues neither has really offered anything substantive as best I can discern. Both say that we need to have a National Broadband Policy in place that provides ubiquitous access, but while this certainly sounds good how will this look and how might it benefit each community? Certainly every urban and rural locality has different challenges. Some will require infrastructure development while some might require more competition for affordability etc. We are seeing government control of ISPs such as in Australia where the Australian Communications and Media Authority has been testing and will be implementing mandatory network filtering. Is this something that either candidate supports and at what level? Where does each stand on the 700MHz spectrum auction and the availability of sufficient public safety and first responder bandwidth for interoperable communications?

I am generally a very optimistic person, but I do not see with either candidate that there will be much change in these overarching policies. I hope that I am wrong and that significant strides can be made, but with all of the other issues that the next president will need to address I once again see technology issues getting some lip service, but not much more.

Photo by Chesi - Fotos CC. Creative Commons Attribution-Share Alike 2.0 Generic

ITIL v3

Thu, 16 Oct 2008 12:38:57 -0800

For the past 3 days I have been attending a ITIL v3 Foundations in Service Management class. What is ITIL v3? The ITIL website offers the following overview: "The IT Infrastructure Library® (ITIL) is the most widely accepted approach to IT service management in the world. ITIL is a cohesive best practice framework, drawn from the public and private sectors internationally. It describes the organization of IT resources to deliver business value, and documents processes, functions and roles in IT Service Management (ITSM)".

Our IT department is embracing the ITIL framework and best practices. We believe that ITIL processes and best practices will ensure that we are continuing to provide value to our customers, even as our budgets are cut and our open positions are frozen. I will periodically provide updates in this blog regarding our ITIL journey; the challenges that we encounter and the impact of changes to our proceses.

I am interested to know how many other agencies and jurisdictions have embraced ITIL. What obsticles; technical, cultural, etc. have been encountered?

For those wanting further information on ITIL the following site is a good starting point - www.ogc.gov.uk/itil - or post your question and I will be happy to find the answer.

Virtually Virtual...

Thu, 09 Oct 2008 10:58:04 -0800

For those of us who have been around long enough to remember the days when the mainframe - or "big iron" was the only real platform that serious business applications ran on, virtualization of system resources was the norm. Each application ran in its own "machine" and more system resources could be added as needed. This provided an efficient use of system resources when hardware costs were quite large as compared to today. Application software vendors supported their applications running in these virtual machine environments without question.

Fast forward to the present. Over the last few years the concept of virtualization has taken hold in PC based servers. Most organizations - such as mine - have implemented or have a strategy to implement this technology as a natural efficient use of the computing resources, as a way to better manage these systems, and to battle data center issues such as power consumption and data center cooling. Unfortunately our experience with the software vendors has not been one of support in terms of their applications running as a production system in these virtual environments. I am assured that this will be changing over time with the introduction of Microsoft's virtual solution and software vendor experience in this virtual world. In the mean time we hear statements like: "Well you can run your test and training in those environments, but we will not support your production in that environment", or " why would you want to incur the cost of training staff and capital costs of that hardware? Individual servers are inexpensive." When I have tried to place the requirement of supporting virtualization in as part of our RFP language I am told that such a requirement is too restrictive and that I am eliminating vendors who otherwise can provide adequate solutions.

I am curious if anyone has been successful in getting software vendors to fully support virtualization in a PC server environment. What specifications or requirements were listed as part of the purchasing or contract process? Am I the only one seeing this?

The Lay of the Land

Mon, 29 Sep 2008 10:22:27 -0800

Before we get too far into the trenches, I would like to put things a bit into perspective and lay out a little about what we have done at the City of Tucson. A lot of what will be written here is from the experiences that we have encountered implementing technological solutions. We have planned well and we have also been extremely fortunate to have city managers over the years who understood the value of technology.

For those not familiar the City of Tucson is the second largest City in Arizona. It spans an area of 250 square miles and has a population of approximately 547,000 making it the 30th largest City in the United States. The city employs approximately 6000 people not including the uniform police and fire personnel. Tucson has placed in the top ten of the Digital Cities survey each of the last 7 years.

The City embarked on its broadband strategy in 1999, and over the last 9 years has created a robust communication network. This network interconnects all major city facilities such as fire stations, police stations, libraries, community and recreation centers. The technologies used are a combination of wired and wireless. A fiber optic backbone encompassing approximately 500 miles of fiber is connected redundantly as OC-48 and OC-12 SONET rings, and gigabit Ethernet pipes. An OC-3 digital microwave network provides connections to sites that are not accessible via fiber. The city has a wireless mesh network (for municipal use only) that covers all 250 square miles of the city. This mesh network was built as part of a project call ER-Link that provides video from a paramedic unit (while in transit) to the local trauma center.

The city owns, manages, and maintains the network. This network is used by other government and quasi governmental agencies at a cost that is less than contracting these services from a provider. The city currently has service contracts with the local community college, the county, and a local school district to provide wide area networking services across this network

The city continues to take advantage of this infrastructure for the provision of technologies and applications that offer a greater level of service and reduce costs. Some examples include: VOIP, NOVA (our CRM implementation), streaming video, traffic signal management, and a number of e-services. In the final testing and implementation phase are projects that provide GIS information to utility workers and first responders in the field, wireless applications for permitting and fire inspections, and consolidation of data centers.

Some other high profile projects that are in progress are: use of open source software, outsourcing of the city payroll application, ITIL, a countywide public safety radio system, and the expansion of the city network to be a regional system.

This obviously is a very general overview of technology and projects in Tucson. These are not unique initiatives, they are common to most municipalities and counties. As we explore each of these over the next few months, I welcome your experiences, both good and bad as they relate to these technologies and initiatives.

Photo by Joe Brent. Creative Commons Attribution-Share Alike 2.0 Generic

The Data Center of the Future?

Mon, 15 Sep 2008 11:11:27 -0800

An interesting article in the Times Online about a solution proposed by Google for their new data centers.

According to the article, Google is considering deploying the supercomputers necessary to operate their Internet search engines on barges anchored up to seven miles (11km) offshore.

The floating data centers could use wave energy to power and cool their computers. And if they had an "offshore status," the company would no longer have to pay property taxes on its data center properties around the world, an additional saving.

I have heard about the generation of electrical power from the motion generated from ocean waves, but this certainly solves issues associated with current data centers: power costs and cooling.

While there are a whole hosts of issues associated with a scenario such as this that must be addressed, (and yes I know that Google is looking to address legal issues as well) but from a technical perspective I am interested to know what your thoughts are regarding the use of data barges as future data centers.

The Cost of Wireless

Tue, 09 Sep 2008 06:29:04 -0800

Municipalities and counties have come to realize the value of wireless networks for, among other benefits, communicating with field workers and providing a better level of constituent services. While the discussion of these networks has evolved away from "free" to the understanding that implementing these networks must be funded by the local jurisdiction, there are a number of operational costs and requirements that are initially overlooked as part of the need analysis and justifications. Some of these overlooked items are:

Maintenance: Most wireless networks pose a major problem for municipalities in that they require continuous maintenance. Access points are mounted on light posts or traffic signals which are susceptible to the elements, wildlife, and the occasional stray vehicle that veers off the road and takes out one of these mounting devices. In Tucson we have even found that our devices and aerial fiber cable have been used as targets for those who cannot make it to the local shooting ranges. While some will have the staff and equipment to replace and fix this equipment, others will have to outsource this function.

Staff Training: Most staff are not experienced in managing and supporting wireless networking equipment. Using Tucson as an example our network engineers are trained and experienced supporting switches and routers. In our case we not only would need to spend a considerable amount of money to train staff, but we need to hire 2 new staff members who (once trained) would be dedicated to supporting and managing the wireless network. Again the alternative to building this in house capability is to contract for these services.

Administration and Monitoring: Along with the addition of and training of staff, there are costs associated with the tools necessary to appropriately monitor and administer the network.

Technology refreshes and upgrades: As with all technology wireless components have a fairly short life cycle and their upgrade and/or replacement must be planned for and funded. You can bet that the wireless network will become as critical as your wired network and the demand from applications for bandwidth and functionality, not to mention the demands from the first responder needs, will not allow anyone to consider a life cycle of more that 3 or 4 years.

Addition of new applications: Adding new applications to the network or expanding its reach can result in significant costs if not planned for initially. Depending on the network design and concentration of access points new applications might require additional access points, this will require additional backhaul, and gateway locations. While a laptop might indicate that you can "see" the wireless network the wireless radios built into most laptops might not send a strong enough signal to be received by the access points without some sort of signal amplification. Networks that were built with one application in mind will invariably require re-architecting and more equipment as other applications are introduced.

As with all technology projects the ongoing operational and maintenance costs should be analyzed upfront and budgeted for, While wireless technology can be very effective in extending the reach of a network and access to applications and information where and when needed, without the proper care and feeding wireless networks can also quickly become a white elephant.

Welcome to "In the Trenches"

Mon, 25 Aug 2008 22:41:28 -0800

Thank you for joining me in the trenches - the place where the action is.

One of the constants of technology is that it is always changing. While new technologies have the potential to change the way communities are governed, they also stress government IT departments as they struggle to stay abreast of new technologies and build the necessary skills to effectively implement and support them . All of this is occurring while budgets continue to shrink, it is harder to recruit and retain staff with the necessary skill sets, and most government IT shops are not staffed to do anything other than support existing applications and systems.

This blog will discuss some of these technological trends and how they are advancing government at all levels. We will focus on the issues facing those that are doing the work and use some of the real world experiences and lessons learned that we have encountered at the City of Tucson as well as the experiences of those in other jurisdictions.

Some of the topics that I plan to cover in the coming months include: Municipal wireless systems, Open Source software, Data Center initiatives,(virtualization of servers and desktops, "green" issues etc.), Public Safety radio communications, VOIP, WEB 2.0 issues / non issues, telecommuting, and disaster recovery. This is just a start and I welcome any suggestions for other topics that should be part of the conversation.

Please join in on the conversation so we might all learn from each other Thank you for joining me "In the Trenches"... .