Rationalizing Software

In 1776, Adam Smith wrote Wealth of Nations. His treatise changed our understanding of the economic world just as Newton's Principia changed our understanding of the physical world and Darwin's Origin of Species revolutionized our thinking regarding all living organisms based on his theory of evolution.

Adam Smith's remarkable book was published at a time when governments granted monopolies and gave subsidies to protect their own merchants, farmers and manufacturers against 'unfair' competition. The guilds operated stern local cartels: artisans of one town were prevented from traveling to another to find work. Local and national laws forbade the use of new, labor-saving machinery. Not surprisingly, poverty was accepted as the common, natural, and inevitable lot of most people. Adam Smith railed against this restrictive, regulated, 'mercantilist' system, and showed convincingly how the principles of free trade, competition, and choice would spur economic development, reduce poverty, and precipitate the social and moral improvement of humankind.

According to Adam Smith, rationalizing production increases quality, lowers cost, and improves efficiency. The expansion of industrial capitalism, the rise of bureaucracy, and the emergence of scientific management have all accelerated the momentum for creating defined, quantifiable, repeatable production and organizational processes. Today, it's a generally accepted principle that open market competition has proven the power of rational systems.

The logic of rational organization and production exists in almost every facet of modern industry. Yet, the creation of software has somehow managed to elude an engineered approach to its production. Software developers cannot agree on the details of a defined, quantifiable, repeatable process. Instead, highly skilled professionals still craft most of the world's software. The notion of a software factory remains as nebulous and unattainable as ever.

The following edited excerpts were taken from Software Development: An Outsider's View, Kyle Eischen, p36--44, Computer 35(5), May 2002, IEEE:

Can software be rationalized?

Rationalization assumes a quantifiable process, maximized for efficiency by a distinct division of labor, with defined inputs and outputs, managed by an effective rule-bound bureaucratic structure -- all of which results in a process capable of being engineered.

Given the rationalization of multiple craft industries over the past two centuries -- all of which were once considered impossible to manufacture or mass-produce -- and the resulting benefits in efficiency and quality, we can reasonably expect that software can and should become engineered as well.

Rationalization leads to rule-bound, fixed parameters and hierarchies that place control, knowledge, and power in institutions and not individuals. Bureaucracy is essentially an institutionalized algorithm that takes general inputs and produces fixed and anticipated outcome.

Thus, rationalization and bureaucracy -- the underpinnings of modern manufacturing -- have consistently emphasized management objectives and goals as the drivers of industrial development. The more rationalized and explicit a process, the more it can be managed, moving control of the process from producers to managers.

Why hasn't software development been rationalized?

Why, even with a tremendous effort to engineer the process from within the profession and from the industry overall, does software development continue to confront the same issues it did 30 years ago? Why haven't basic industrial patterns -- software industrialization, software manufacturing, software engineering, and software assembly lines -- become dominant within the industry?

By the time the automotive industry had reached its 30th anniversary, around 1930, the fundamental issues of production, as exemplified by Ford's assembly line; product, as exemplified by standardized design; and industry, as exemplified by the few dominant national players like Ford and GM, had all been clearly established.

Patent or copyright?

Is software development an invention derived through a scientific method, or is it an act of speech and creativity?

Is software an act of engineering or communication? If software is a rational endeavor, improving quality involves better and more resources: better management, better tools, more disciplined production, and more programmers. If software is a craft, improving quality involves the exact opposite: focusing on less hierarchy, better knowledge, more-skilled programmers, and greater development flexibility.

As early as 1974, Fred Brooks (The Mythical Man-Month: Essays on Software Engineering) stated that merely adding people to a software project didn't increase its productivity -- a classic diseconomies-of-scale phenomenon. The essential limit of software development is communication.

Communication is inherently difficult, mediated by always-contextual codes, norms, culture, and perceptions. Translating knowledge from one context to another, like translating any language involves not just basic grammar and syntax rules, but also issues of meaning and intent that are contextual and subjective.

Software is the Execution of Knowledge

Much of knowledge is tacit, undefined, uncodified, and developed over time, often without being explicit even to the individuals participating in the process. More importantly, such knowledge and practices are dynamic, constantly evolving, and transforming. It should not surprise us that modeling these processes proves exceedingly difficult and that such efforts are often incomplete, impractical, or unsatisfactory.

The further away from broadly accepted and understood domain knowledge we move, the more difficult translating that knowledge becomes. The issues are inevitably about context and communication.

Context and communication are the most salient features provided by ITatlas, ITguide, and ITscout -- Flashmap Systems' products. I can't ever overemphasize the importance of getting an IT organization to improve its communication with its user constituency. Invariably, the key to effective communication depends on context management. The reason why this is so important is because you can never ignore the cultural reality that no one likes to admit what they don't understand. Context is what's missing from an information dissemination strategy based solely on search engines. Search is a terrific starting point, but once information is found what people additionally need is an understanding of its context.

For additional discussions about rationalizing software, see Software -- Engineering or Craft?.