When it comes to software, one of the wonderful things about it is, there is generally zero cost to add an additional user to an already built system.   The effort that went into the system can be magnified many times over in terms of the value it provides to the world. No wonder software developers are such a prized resource!

The write once sell many attribute of software is something I inherently realized, even from the early days of booting from a floppy disk.  With SaaS, and now cloud services out there, it is all the more relevant.  I recently read the book Free – The Future of a Radical Price by Chris Anderson, which covers this scaling idea from just about every angle: technology, business, economics, society, even history!

The book is an easy going read, enjoyable, insightful, though a bit repetitive at times. It brought clarity to me personally in terms of what is going on when a company ‘gives’ me something, when I buy something, and when I avoid buying something and look for a free alternative.

Free illustrates two economies colliding:

• The atoms economy, like oil, steel, lumber is inflationary.
  • As time passes, these goods generally become more expensive in nominal terms (due to the Fed’s policies).
  • The price of these goods can also increase in real terms.  Oil for example, is growing in demand due to population growth and global modernization.
• The bits economy, like storage, processing power, and bandwidth, is deflationary.
  • As time passes, these ‘goods’ become relatively less expensive.
  • Moore’s law drives this for processing power, and it also holds true in storage capacity.
  • The book argues that these goods don’t just become cheaper, they become so cheap they essentially become rounding errors – too small to even worry about measuring.

When someone reads this blog, the marginal cost to me is zero.  If a million people were to read the blog, yeah, a server upgrade would be required, but it would not be 1 million times the cost, it would probably be more like ten times the cost.    This is what is putting news papers out of business.  Companies like Amazon and Google saw it coming, and positioned themselves to capitalize on the trend by pricing ahead of the curve – good for them.

So, what about software developers, inflationary or deflationary?

I would say our field is inflationary.  Demand is wildly increasing, supply is increasing (outsourcing), but with current technology there is no way to get exponential productivity gains out of software developers. The best you can hope for is a team of 10x programmers, but there are no 100x programmers, or 100,000x programmers on the horizon.  Besides, when there are more than 3-4 programmers on a team, the communication burden starts to become a drag on productivity.

Computers like IBM’s Watson change the game. When computers start writing software on their own, we’ll have some interesting issues to deal with (skynet?)…

Free removes “purchase decision” stress:

• By making a service free, the number of potential users is maximized.
• When a product is being evaluated for purchase, a special psychological thread kicks in (the reptile brain). From a marketing standpoint, it is nice to avoid that whenever possible.
• The book talks about how huge the difference is.  In one study, Hershey Kisses were offered against a superior Swiss chocolate at $0.25 each, $0.01 each, and free.  Even though the monetary difference between $0.01 and free is pretty small, there was a huge difference in the consumption of the free chocolates vs the $0.01 chocolates.

What the book doesn’t talk about enough I think, is the downside of free:

• When a product is free, people just don’t value it. They still expect it to work, but in their minds they have no serious investment in it.
• When we buy things, they become part of who we are. As human beings we want to value ourselves through our choices, including the brands we pick. Free products don’t tap into that psychology, except for cheap skates – but they are not potential customers anyway.
• A premium can be had for exclusivity.  Free products are by nature not exclusive.  For a brand like MIT, which is now giving away free courses online via MITx (but not college credit).  They are playing a dangerous game with diluting their own mystique.

Here is my take away – Free is here to stay:

Just like having a URL, a twitter account, a Facebook page, and mobile app – introducing some kind of free offering that differentiates a company is a required element going forward in business.  Free is raising the table stakes.  Delivering ‘free’ is getting cheaper thanks to the bits economy. However, building a free offering is not cheap.  Thus, software developers are more in demand than ever before.  It is a good situation for us, pretty frustrating for everybody else.

Everything is getting capital ‘S’ Smart these days. Smart phones, Smart homes, Smart cars, in EPIC there are ‘Smart phrases’, and even a local cafe chain has something called Smart beans….

The Smart trend lumps together intelligent networks, big data, bio-metrics, domains that end in .io, and apparently organic farming practices.  Makes Web 2.0 and the ‘e’ and ‘i’ nomenclature seem so yesterday.

The Smart trend actually takes a lot of smarts to solve and will no doubt take many iterations to get right.  Sounds like a lot of fun stuff to work on!

What is driving this change:

The cost of smartness is dropping based on two principles:

• Sensing:
  Physical instruments that collect data are becoming more sophisticated, widely available, and easily networked. I personally love the idea of easily and cheaply connecting the world of software to the real world of atoms, photons, temperatures, pressures, ppms, and anything else that can be measured.
• Storing:
  The cost of storing most ‘facts’ is already a rounding error. So, why not just store everything? If you doubt this, see my review of the book “Free”.

The ‘Smart Milk Carton’ concept:

There is a romantic idea of the future where the milk carton tells the fridge it is running low or expired. The fridge then adds milk to the household shopping list. The grocery store sees this and gives you a coupon for milk as you pull up to the store. Or, the delivery service drops off more milk automatically.

That is a novel application of technology, but I’m not sure it solves a need that bothers people enough. Maybe for a restaurant or cafeteria this makes sense, but it would be a luxury for the average household.

Smart becomes practical when the information flow has economic significance:

As a thought experiment, apply the ‘smart milk carton’ technology to a hospital’s store of medications. The hospital’s inventory of drugs could become Smart. When a medication runs low or expires more is automatically ordered. Orders to suppliers happen automatically reducing human error and staff overhead. Analyzing the flow of medications can help reduce costs, ensure vital drugs are fully stocked, and forecast which drugs are needed when.

The information flow surrounding a $4 gallon of milk once or twice a month is not that interesting or economically viable. The information flow surrounding the millions of dollars worth of medications used everyday is just one example of how hospitals will get ‘Smart’.

Smart devices measure you:

You know how if you don’t pay for something (like Facebook or Google) you are the product?

Well our DNA, vital signs, and behaviors are about to be measured and commoditized in the coming decades like never before. Human biosensors are coming out that track every facet of our health (sleep patterns, nutrition, digestion, respiratory, cardio, etc).  Two popular products that already track health stats and sync to the cloud are fitbit and jawbone.

In practice biometric data analysis has the potential to substantially lower the cost of overall treatment (and perhaps save lives). Right now, measuring things like a person’s vital signs or EKG is still somewhat intrusive. Eventually it will be a matter of swallowing a capsule or having a subdermal implant that syncs to a smartphone.

Smart and the changes ahead:

There are social implications as the Smart revolution rolls out. The ‘haves’ and ‘have nots’ of this technology will be in two different and very unequal worlds. Those who have access to this technology will be able to remove variability from their business, maintain their health to a higher degree, and be in greater control of their environment.

A side effect is monopolies may stay in power longer because of the extra edge that comes from wielding the most data and having the best toys. This will create additional barriers to entry for market newcomers.  Maybe this has always been true though?

In 50 years today’s world will look like the dark ages in terms of all the advanced knowledge we will have about our health, our potential, and our interactions as a society. Can you imagine what it was like 50 years ago to go on vacation to another country with just a guide book and paper map? We’ll look back and say, gosh can you image what it was like not to know your blood chemistry stats in real time on your phone? Yes, cholesterol really does spike after eating a burger and fries!

For more reading:

The Human Face of Big Data by Rick Smolan and Jennifer Erwitt has dozens of examples of the application and promise of big data. The book lives up to its name. It is huge, twice the page size of a standard book!

Milk and Cat image by ‘the bridge‘ on Flickr
Sensor image by Huskeflux on Flickr

For anyone interested in code, networking, and finance, Flash Boys is a real page turner. For me personally, with interests in all three, it sent chills up my spine. I could not put it down!!!

Flash Boys is a fascinating, informative, and thoroughly done edge of your seat ride through the modern world of technology driven high frequency trading. I really enjoy Michael Lewis’ works and this is the best so far. This being his latest work, published March 2014.

The Back Story:

Since the mid 1980’s trading has been increasingly handled by computers instead of people. Starting around 2007 there was a huge disruption in the way stocks are traded on exchanges. It was precipitated by new SEC rules and advances in fiber optic networking. This lead to a surge in trading volume, all of it automated by computer. Clever ‘high frequency’ traders figured out how to exploit the slower players (everyone else) based on network latency and order manipulation. In a fine example of capitalism’s creative destruction, the high frequency trading firms were able to exploit a weakness in the market and shave billions in profit. It is unfortunate for investors that the new players in the market did not correct the inefficiency, but instead used it for exploitation and in some senses made the problem even worse.

I don’t want to give away any of the story because it was such an enjoyable read. You won’t be disappointed by the way tech talk is presented. Some of the heroes in this true story are developers and systems administrators. Go nerds!

Lessons for all Software Professionals:

One issue the book brings to light is the economic consequences of ignoring software quality and long term vision when it comes to system maintenance. Many of the trading platforms and exchanges out there were not written to cope with the complexity and speed of today’s world. This may not come as a surprise, but non-technical wall street managers are driven by short term personal gain in the form of fat bonuses. As such they end up with core systems that are done completely piecemeal, each feature bolted onto the next. Sound familiar?

The piecemeal approach to building software is commonly found in any non-technology company that uses technology. All companies in today’s world are forced to use technology to stay competitive, but few are good at managing that technology for the long term. On wall street it has become a systemic problem and is to blame for what are becoming consistent ‘system glitches’ that send markets spiraling for ‘inexplicable reasons’. The Knight Capital ‘glitch’ that lost $440 million is a great example. NASDAQ and other exchanges routinely have serious flaws that are now looked at as the cost of doing business. It is SCARY to think how much money flows through these systems each day. The planet’s economic security depends on these systems. Technology is easy to blame (especially for managers who don’t understand it). What is actually to blame is the way in which the technology is being managed. The book goes into this issue in detail from multiple view points and I was refreshed to see it brought up.

Hope you enjoy reading Flash Boys!

I have been told “there is no such thing as green field development”. While that statement is false for the hobbyist developer, in the business world it is nearly true. Those who code for a hobby or for pure enjoyment often start from scratch, as evidenced by the explosion of unmaintained projects on Github. See my article about software ghettos for more on that. When it comes to software used in the real world, open source or not, maintenance is an everyday task.

Consider what goes on between the 1.0 and 1.1 release. Was that 100% new work or did it include some maintenance to allow the 1.1 features to fit with the 1.0 architecture? Now fast forward to the 1.8 release, was the ratio of maintenance higher? Almost certainly.

An article by Robert Glass in IEEE Software May/June 2001 called Frequently Forgotten Fundamental Facts about Software Engineering states maintenance is 40-80% of software cost, and enhancements contribute to 60% of new maintenance costs!

Why care about quality?

Consider that businesses are not interested in (and probably can’t afford) a monument to computer science. What the average business demands is functional code. I have been involved with dozens of businesses, small, large, tech centric, and technophobic – none have asked for fancy or perfect code. Anything beyond functional is seen as a waste, and I agree. This is not a license to take shortcuts and hack things together. If shown the distinction a business doesn’t want a ghetto code base with anti-patterns everywhere that will soon become unmaintainable and cause developers to run and hide. In spite of this, it turns out a lot of systems are managed in a manner that contributes to major system outages, security holes, developer attrition, and occasionally huge monetary losses. Google ‘stock market glitch‘ for examples.

How can software maintenance work be done efficiently?

A great developer won’t make much of a dent if they are blocked from doing so. The product owner should have a long term plan for the system which includes keeping the system healthy and maintainable. That plan should favor fixing existing bugs (see #5 on Joel’s list) and allocate time for paying down technical debt in each release. Technologies such as source control, a suite of unit tests, code linting and build automation are extremely helpful. Policies on code style, documentation, learning, and knowledge sharing make a big difference too.

A team composed of a mix of veterans, mid level staff, and junior developers makes for a healthy balance. The developers should be allowed to think they own it (a variation on a famous quote from Bill Gates). A culture of knowledge sharing should be encouraged and rewarded. Assumption checking should be considered normal and non-threatening. Have you ever read a spec that was 100% free of half baked assumptions? Individual performance should take a back seat to team performance. Otherwise silos form, the incentives become twisted, and so does the code.

On the individual level a developer has three hills to climb to become maximally efficient:
1) The languages, libraries, and technologies used in the system.
2) The domain (the nature of the business).
3) The way the system was setup.

Languages and libraries should be a relatively low hurdle if the technologies used are ubiquitous and the right skills are hired for. Domain knowledge is harder to come by. In some areas such as insurance, finance, education, or ERP a person with the right experience is attainable. The third hurdle is by far the least visible to the business and the most challenging. It ultimately comes down to what is stuck inside the developer’s head that makes them efficient at maintaining the system. If the developer wrote the system from scratch, they get past that for free. That assumes they haven’t already moved on… perhaps washing their hands of a mess?

“Debugging is like farting – it’s not so bad when it’s your own code” – Unknown

The time it takes to attain mastery over a code base is proportional to its size and complexity. The best approach is to start with an easy task, then something slightly more complex in a different part of the system, then something in a third area, and finally circling back to the first area for a real challenge. This way confidence is built up steadily and the risk of breaking something critical is reduced.

The first few days to several months of working on an unknown system are the most stressful and error prone for a developer. Without knowing every aspect of the system it is easy to accidentally write new bugs. Without a senior developer or product manager to explain things it can be very confusing and frustrating to make headway. This is where developers with solid people skills and high self esteem will shine because they are not afraid to ask for help and are effective at getting good answers.

Development efficiency increases over time then plateaus:

The orientation phase and steepness of the growth phase increase relative to the size of the system. They can be shortened with documentation, clean code, but most importantly friendly and knowledgeable team members. Hiring for a person with knowledge of the languages, libraries, and domain also helps.

Let’s say things go well, and the developer climbs the efficiency curve after X days or weeks. Now they are really ‘making money’ for the business. This is the most efficient place for the developer to be business wise. The length of time a developer spends on top of the curve depends entirely on the company’s ability to retain that developer. The going advice is to pay at least a fair wage, be flexible, be organized, then stand back and let them go. Make sure to let them do interesting things from time to time. Offices with windows, sit stand desks, and flexible hours are nice perks that don’t cost much when averaged out. The alternative is to loose the developer and go back to square one in the orientation phase with someone new.