CityTime was launched in 2003 at a budget of $63 million, but costs swelled dramatically as the project stumbled along for nearly a decade.
From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.
In these types of projects it seems like no one wants to point the finger. Once costs have doubled, or tripled, everyone is afraid of taking the blame for wasting all that money. So they keep 'working with' the vendor.
Clearly fraud and kickbacks kept this stumbling along much longer than a normal failing IT project would. I'm not sure why or how an expense like that on an overage like that on a major payroll system would not be on the mayor's daily agenda. The right people must have been getting those kickbacks.
I've been on the wrong end of a couple of projects like this, and in my experience (not necessarily indicative of this case) the customer can be VERY at fault.
I've been dragged into 'feature acceptance meetings' that literally dragged out for weeks when testing of a new chat component devolved into a days-long discussion on whether the graphic in the header (provided by the customer) was appropriate. The unspoken issue proved to be that they didn't want us to use 'barren trees' in the picture of their building. We switched the photo used from one taken in winter to one taken in summer. This took something like 400 man-hours to resolve, after which we achieved signoff, but at the expense of having met budget.
Another instance on the same product was feature creep to the extreme. We had originally won a firm-fixed contract, meaning we would build products a, b and c for x dollars, and if it cost us more, it came out of our pocket. If it cost us less, we pocketed the profits. The customer decided, on day 1, that they actually wanted a lot more features than were described in the project scope defined, and that we won a legally binding contract to perform. Out of accomodation, we changed their product to a time-and-materials contract, meaning we had a fixed markup on labor and a minimal markup on hardware / tangibles. This then involved them dragging the project out for years longer than anticipated and signing invoices all the way, then griping at us on the budget near project completion and asking for a bunch of free labor (which is illegal under their new contract). The problem in that case was that the project management and senior execs changed repeatedly. We were at first working with their Office of Science and Technology, then with Information Services, then with Public Affairs. Each of these department changes involved swapping our equipment out so that we could have appropriately charged-back equipment to the appropriate department, which meant weeks of non-work. Each department had a different spin on what they wanted out of the project. We spent 8 months and thousands of man-hours implementing a very elegant services oriented architecture that could surface all of their legacy systems to a single web portal. Then we changed departments, and they literally told us to turn that feature off, and instead focus on a visual redesign. We implemented a single sign on service that we integrated with our product, but could be extended to work with literally every system in their enterprise. It was abandoned due to a department change.
During the first phase of the project, we'd worked with the customer to build out their data center. They wanted a new data center because the division that owned the project was going to be adding a bunch of other programs to integrate with what we were building. They spent over a year on construction, converting a service garage into a data center. They bought millions of dollars worth of EMC storage hardware and Cisco routers. Then they realized that they didn't have any budget planned for out-year maintenance, so they moved the venue. That was somewhere in the realm of $50 million dollars that was literally just thrown away, as the last I heard, they were re-converting the data center back into a garage.
In short, government contractors catch a lot of flack, because they control the visibility of these things. The contractors often can't speak to details of these projects due to security clearances or NDAs, and at the end of the day, the government is always happy to blame the vendor. It's well practiced, and designed with a variety of scapegoat clauses built in.
... it's just the article notes a subcontractor has already been convicted of fraud and kickbacks, so this might not have been a project quite like the projects you describe (though if this is a Terry Childs-type case, I'd love to hear about it).
Sorry, I meant to imply that that's just the way things are, however, after the realization that he'd been convicted, it's likely that whether he's done anything wrong on this particular project, he's drawn the scapegoat straw.
I meant to say those things, but got lost in typing so much other nonsense and trailed off. Thanks for bringing it up.
I wouldn't be so sure. One thing that working in government has taught me is that the bad contractors seem to be equal or greater proportion to the good.
They would bid on something if they weren't going to find a way to make more than what the original contract was for because in the bid wars they often lower the bids so much they have to recoup somehow. In this case they probably did not build the recoup method into the contract and did it illegally.
Everywhere I go I end up on SSO projects. I always swear I'll avoid them, but they seem to jump out at me when I least expect it.
Your assessment of the software and services seems fairly accurate to me (there are a couple of exceptions: CAS is simple enough to be easily understandable and works ok, and Atlassian Crowd looks ok too).
From personal experience I can confirm that this is exactly how it works when governments try to do things. There's such a compulsive need to negate risk that everything is all about the process instead of the results - and inevitably this leads to huge amounts of wasted time and money while people try to decide the right thing to do.
On a previous project for a very large automaker, it took me 5 full days plus a 1 week trip to 4 cities in China to change a single graphic. (The rational was that they wanted the client to talk directly with the engineers instead of having to go through all the corporate bureaucracy. So much for that. ;)
Replacing the logo itself took all of about 30 minutes (receive new picture, digitally retouch, overlay onto background image, resize, play around with transparency, save to png.)
Also, bear in mind, the 400 hours is 400 man-hours, which is a cumulative number based on the number of people involved. If you've never done business with government, let me assure you, it was a lot more people than necessary.
Absolutely. The man hours can easily pile up when you do a conf call with a half dozen folks where majority of them are on mute 90% of the time.
I recently did contract work for an agency and had instances where a simple spacing or image resizing took days of work because the client or the agency would understand and then forget that I must crop and/or resize images for them all to be of one size.
This is a common scenario. Government picks the lowest bidder, but the price can go up every time there is a change in specs. Then every government agency imaginable wants to decide what color to paint the shed and the whole thing bogs down, sometimes permanently. It's amazing that on this project they eventually managed to deliver a working system.
That's only about a 30% growth per year though, which may not seem so ridiculous on a year-over-year basis. I can see how such a mistake could be made, though it is pretty crazy.
Although New York has "received a working system that will advance our management ability ... because the project was apparently tainted by fraud and kickback schemes, the City must be made whole"
...
The city is also planning to launch a request for information (RFI) process in January in search of alternative timekeeping systems.
Wow. So they eventually got a working system. Now they're throwing it away and going to purchase a new one? Meanwhile turning 83 contractors into FTE's to maintain the existing system? This seems like complete craziness for something as commoditized as timekeeping software.
Timekeeping systems are actually extremely complicated. EXTREMELY sophisticated. The level of detail it takes to get one right is painstaking, and it is key to a city's survival. It's complex enough that you wouldn't understand it, nor would any auditor.
For that reason, I just started a new startup company: TimeKeep NYC. We specialize in timekeeping software for NYC area cities. Looking for employees now, especially ones who aren't snitches. Based on past products in this market, I think we can do quite well.
To be fair, timekeeping software really isn't as trivial as you might like, at least assuming that we want to move beyond the raw recording of who worked when out into actual management of such things. Requirements come from federal, state, and local laws, as well as negotiations with all relevant unions. So even before you've gotten to the usual BS requirements laid on by management, you've already got a large pile of requirements that A: are legally mandated and B: written without so much as five minutes consultation with a software engineer.
But, yeah, on the other hand that's just the usual business software story. 10x budget overruns are probably not called for.
But, really.... timekeeping and payroll were some of the FIRST business applications developed. It's a solved problem at this point, even with all the nuances. Developing this from scratch for NYC seems like the epitome of reinventing the wheel.
No enterprise software is a solved problem yet, there's a lot of bad software dominating at the moment (look no further than sage for accounting for example), any enterprise software you randomly pick will suffer from at least one of:
- Have bloody awful UIs
- Aren't extensible
- Need a massive amount of expensive consultant days to setup
- Are very brittle
- Have an overwhelming list of features which make them very hard to use
- Have bizarre restrictions on certain data (e.g. max 100 chars for a comment)
- They're not capable of talking with common system X
- Don't expose an API
- Have atrocious db structure making reporting extremely expensive in dev time
- Use a home rolled DB (oh. god. no.)
As for the problem of timekeeping being extremely complex, it's not at all. It's difficult, but solvable with one or two decent developers/project managers in a few months. I know as I've seen it done twice now.
It's only extremely complex if you try and include every weird quirk and feature that every client has ask for. Sometimes the solution isn't extending the software, it's changing the business process or simply leaving them on the old paper system.
If you take a 37signals-esque path you've got something that'll sell to 80% of businesses, let niche software cover the rest.
"It's difficult, but solvable with one or two decent developers/project managers in a few months. I know as I've seen it done twice now."
It was actually my first task that I ever did "for real", though we did only the raw recording; the processing was done by older code that had all the quirks built in. It was a success, too. The problem itself is easy, but it's definitely in the class of things that can be made arbitrarily hard by added requirements with no thought given to costs/benefits. Fortunately on my project, the devs and the management were on good terms, so much thought was given to cost/benefit tradeoffs, but I can easily see cascading failure in a different environment. Even the easiest software projects can and are sunk by that.
Also, if you ever get in the position of saying "X is extremely complicated and I can't explain it to you" it is because you stand to benefit by designing a system poorly, or you aren't bright enough or don't have the experience to realize that you can reframe the problem. Algebraic geometry is also complicated, but these days there are plenty of books that will explain it to you, and the tools (like sheaves) are well-established and standard.
Obfuscation is the MO of installed, configured enterprise software. If there were an incentive to create a simple, effective solution, then, lo and behold, one would be created. The incentive is almost never there.
Take Unix for example. It is simple and effective (and flimsy as all heck) but before it came along, obtuse operating systems were the norm. It is worth pointing out that A) Thompson and Ritchie are geniuses and B) there was no financial incentive.
They are more complex than this. Not just anyone can start a company to provide 99.9% reliable enterprise-level management of this problem. A recent study indicated that a major city would have to spend at least $100M to tackle this systemic issue. NYC's mistake was trying to cut corners with $60M.
Have you already looked into DCAA compliance? If not, your system can't be used for any federal systems, and likely aren't available to be used by most state and municipal government agencies.
Things like varying overtime rates depending on time of day, accumulated hours, whether it's a holiday, different rates for depending on specific work assignments, etc. It's not as simple as a flat rate x number of hours. But it's nothing that hasn't been done before, either.
Agreed. You also have to think about who can approve the hours booked. So you would need to have some sort of hook into the HR / Org database which determines each users entitlements. It gets very very complicated! Technically, the problem is nothing new. The difficulty lies in designing the system and getting it to work correctly with an organizations existing business processes and systems.
In this case, one of the issues was that worker's were incensed over biometric scanners proposed for the system intended to fight pervasive time reporting fraud.
The City of New York employs about a quarter of a million people, which is around the same size as General Electric or GM, worldwide, performing a wide range of job functions.
Also: the particular system under development included hand scanners, so you should include developing for that.
Those job titles are set by the government and specify what pay is going to be. My job title at my current company is Computer Analyst, yet I am a computer programmer. The gov't contract we have mandates that specific title.
Yeah, those look like outrageous salaries. But, to be fair, I doubt they are salaries. They're probably derived from hourly consultant rates of around $250 / hour.
It would seem that way, but this is municipal government, so there is likely local code (law) that imposes requirements that will boggle your mind and are likely unique to NYC. Take any trivial software task, target municipalities as your customer, and you'd better triple your budgets.
Municipal governments are a large portion of our business, so I'm intimately familiar with the challenges. What's unfortunate is that it's infrequently the fault of the individuals running the municipality. We meet a lot of really great people out in the field. It's just that they're doing their job with one hand tied behind their back. They have to operate within a framework that is (mis)guided by public interest.
This is fairly common; a decent, working solution already exists, but (the) government wants to do its own instead of adopting the existing system, and the bills begin piling up.
Do payroll slips issued by the system to part-time workers have the option to translate the statement of worker's rights mandated by NY city law on the back of the slip into Mandarin Chinese? No? "Did you even read the bid document? It was right there on page 487 where it said 'The system must be fully compliant with NYC labor law.'"
There are literally a hundred thousand requirements like that one.
Worked for subsidiary of SAIC contracting out to NYC government over a decade ago. Trust me when I say this is nothing out of the usual... project will go order of magnitude over budget constantly.
As for the requirements that drive these projects, typically they are written specifically to present the appearance of an open bidding process while allowing the departments to know ahead of time who will actually implement the contract because they are so specific in certain areas that only one contract can possibly fulfill them.
Those who say that the city's payroll needs are complex are entirely correct. A city government that size will inevitably have a mind-boggling number of variations in the business rules that the software must model.
Yet $600 million? I've built accounting software, and though it wasn't as complex as what NYC needs, I'm 90% sure $600 million is astronomically higher than what a reasonable price would be.
What about the original price of $63 million? That still seems awfully high to me, but I'm not sure. Does anyone have an opinion on that?
I've never built a time keeping system, but this is how I put that number in perspective. For $63 million I can hire everyone who's ever been a Django core developer (people I've worked with, and who are excellent) for 5 years, at 500k a year... and there'd be money left over.
Building a time-keeping system for large city like NYC is many,many times more work than building web-framework like Django. It's not a technically complicated problem, but it has a different kind of complexity. That complexity comes complex business requirements, which have to be analyzed and implemented piece-by-piece.
My university spent $10 million updating their student information system. It's not hard to hit eight figures when you're paying an army of extremely unproductive consultants $500-$1000/hour.
$600 million is extortionate, of course - hence the demand for reparation. That's why bloated consulting companies like this are able to demand such high prices. Their size provides a target for the inevitable lawsuit if things go south.
How complicated must that piece of software have been for it to cost that much? For that kind of money it better be at least as complex as any OS out there.
I would wager good money that you could supply thedailywtf.com with entries every day for a month from the code base of this project, easily. When you do the math it works out to something like $100 million a year, which implies a fairly large development team (hundreds of people, at a bare minimum).
It's really easy to waste money on big development projects of this sort. The bigger the development team the harder it is to manage, the less well it is managed the less productive it is in aggregate, and thus the less productive it is per salary-dollar. And these are not small effects. A small team of very talented devs can quite easily compete head to head with a giant team of hundreds of devs, precisely because of these scaling and management problems.
Michael Bloomberg might have a point, but he really isn't in a strong position to complain about shitty overpriced software. Glass houses and all that.
The role of Bloomberg the businessman is independent of Bloomberg the public servant. He can sell overpriced software as a businessman all he wants, but once he becomes mayor his responsibility is as a steward of the public's money and to combat ridiculous overpriced projects such as mentioned here.
I don't think his complaint is so much about shitty software as it is about criminal activities relating to its creation. Carl Bell (the executive in charge of the project) plead guilty to approving charges from a contractor for years when he wasn't even on the project. I think Bloomberg has a strong argument that there was fraud going on here.
Just so people can put this in perspective, a Bloomberg terminal license costs approx $1750 per month, with a two year contract. The software basically gives users a fairly raw interface to financial data, analytics and an email/IM system.
In a sales call, the Bloomberg sales reps will demonstrate endless functionality of the terminal : there's a huge amount there. The dirty secret is, though, that most people use only 4 different functions (different people, different functions, of course). The big switching cost is that the email/IM is entirely proprietary : Everyone has to have a BBg terminal, because everyone else has one.
The software is probably as well written as you would expect from a company in a monopoly position. One thing in their favour : If they fundamentally changed the way the 'MSG' button worked, 50% of their users would complain.
Also, it should be mentioned that it has over 30k functions. Some of these are so obscure, that if you happen to only use one of these functions, the high monthly cost could be justified.
You're essentially correct on all points but clients tend to use many more than 4 functions though significantly less than the total number.
And with regards to IM and email, it should be noted that any user of a Bloomberg terminal can IM any other user anywhere in the world (there's no 'friending' or anything). Many, many trades are conducted over IM or 'IB' as it's called.
The most accurate analog I can think of to the Bloombeg terminal is Emacs. It has everything and the kitchen sink (News, dining suggestions, email, IM, sports updates, etc.) and clients are extremely particular about functionality. I have a cousin that's been a subscriber since 1997.
And with regards to code quality, I'm not sure how much I'm permitted to say, but the first Bloomberg terminal was released in the 1980s (this was before SQL existed). Also they maintain the largest private network in the world (they run a line to every terminal)... so all the things that go with that.
> The most accurate analog I can think of to the Bloombeg terminal is Emacs. It has everything and the kitchen sink (News, dining suggestions, email, IM, sports updates, etc.) and clients are extremely particular about functionality.
Of course, Emacs is free. Do you think Bloomberg is susceptible to competition from a much lower-priced and much leaner startup?
The point of a bloomberg terminal is not the software, it is the data. The monthly fee is for a one-user license of the data that they provide, and it is quite vast.
From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.