Notes from ‘Where good ideas come from’



Image Courtesy: Amazon


Where Good Ideas Come From‘ is a book about the space of innovation. The argument of this book is that a series of shared properties and patterns recur again and again in unusually fertile environments. The more we embrace these patterns—in our private work habits and hobbies, in our office environments, in the design of new software tools—the better we will be at tapping our extraordinary capacity for innovative thinking.
When life gets creative, it has a tendency to gravitate toward certain recurring patterns, whether those patterns are emergent and self-organizing, or whether human agents deliberately craft them. These patterns turn out to have a long history, much older than most of the systems that we conventionally associate with innovation.
Both the city and the Web possess an undeniable track record of generating innovation. The metropolis shares one key characteristic with the Web: both environments are dense, liquid networks where information easily flows along multiple unpredictable paths. The history is particularly rich because it is not exclusively limited to human creations like the Internet or the metropolis.
How are ideas formed
A new idea is a new perspective on a problem or a recognition of a new opportunity that has gone unexplored to date. A good idea is a network.
Two things govern the formation of a new idea: Context and Analysis.
a. Context of Innovation
If we want to understand where good ideas come from, we have to put them in context. For instance, Darwin’s world-changing idea unfolded inside his brain, but think of all the environments and tools he needed to piece it together: a ship, an archipelago, a notebook, a library, a coral reef.
b. Analysis of Innovation
When we analyze innovation, it takes the shape of the hourglass. At the center of the glass, the perspective shifts from nature to culture, and the scales widen: from individual thoughts and private workspaces to the major cities and global information networks. That is, it goes from generic to specific and then to generic again.
When we look at the history of innovation from the vantage point of the long zoom, what we find is that unusually generative environments display similar patterns of creativity at multiple scales simultaneously.  Seeing the problem of innovation from the long-zoom perspective does not just give us new metaphors. It gives us new facts.
Analyzing innovation on the scale of individuals and organizations—as the standard textbooks do—distorts our view. It creates a picture of innovation that overstates the role of proprietary research and “survival of the fittest” competition. The long-zoom approach lets us see that openness and connectivity may, in the end, be more valuable to innovation than purely competitive mechanisms.
The following are some of the ways through which we form ideas:
1. Brainstorming – It opens up the flow of ideas and hunches in a more generative fashion than is customary in a regimented workplace meeting. However, one trouble with brainstorming is that it is finite in both time and space.
2. Serendipity – It is built out of happy accidents, to be sure, but what makes them happy is the fact that the discovery you have made is meaningful to you. It completes a hunch.
Serendipity needs unlikely collisions and discoveries, but it also needs something to anchor those discoveries. The challenge, of course, is how to create environments that foster these serendipitous connections, on all the appropriate scales: in the private space of your mind; within larger institutions; and across the information networks of society itself.
Therefore, instead of cloistering your hunches in brainstorm sessions or R&D labs, create an environment where brainstorming is something that is always running in the background, throughout the organization. Your discovery may well be interesting and informative, but it will not be truly serendipitous unless it helps you fill in a piece of a puzzle you have been poring over. By making the ideas public, and by ensuring that they remain stored in the database, these systems create an architecture for organizational serendipity. They give good ideas new ways to connect.
Serendipitous discoveries often involve exchanges across traditional disciplines. For instance, the Web is an unrivaled medium for serendipity if you are actively seeking it out. The serendipity engine of the Web suggests look everything up.
However, the irony of the serendipity debate is that the thing that is being mourned has gone from a fringe experience to the mainstream of the culture.
3. Errors
Truth is uniform and narrow; it always exists and does not seem to require so much an active energy, as a passive aptitude of the soul to encounter it. However, the error is endlessly diversified.
The error is needed to set off the truth, much as a dark background is required for exhibiting the brightness of a picture.
Fertility of imagination and abundance of guesses at truth are among the first requisites of discovery, but the erroneous guesses must be many times as numerous as those that prove well founded. The trouble with error is that we have a natural tendency to dismiss it.
The history of being spectacularly right has a shadow history lurking behind it: a much longer history of being spectacularly wrong, again and again. Innovative environments thrive on useful mistakes and suffer when the demands of quality control overwhelm them. Error often creates a path that leads you out of your comfortable assumptions. However, we must turn the errors to insights.
For instance, the error is what made humans possible in the first place. Sex helped harness the generative power of mistakes while mitigating the risks. Sex lets us learn from the mistakes of our genes. Another example is the Audion, which was not so much an invention as it was the steady, persistent accumulation of error.
4. Adjacent possible – The adjacent possible is as much about limits as it is about openings. The phrase captures both the limits and the creative potential of change and innovation. What the adjacent possible tells us is that at any moment the world is capable of extraordinary change, but only certain changes can happen.
Each new combination ushers new combinations into the adjacent possible. Indeed, one way to think about the path of evolution is a continual exploration of the adjacent possible.
You can see the fingerprints of the adjacent possible in one of the most remarkable patterns in all of intellectual history, what scholars now call “the multiple”: A brilliant idea occurs to a scientist or inventor somewhere in the world, and he goes public with his remarkable finding, only to discover that three other minds had independently come up with the same idea in the past year. The telephone, telegraph, steam engine, photograph vacuum tube, radio—just about every essential technological advance of modern life has a multiple lurking somewhere in its origin story.
Technological (and scientific) advances rarely break out of the adjacent possible; the history of cultural progress is, almost without exception, a story of one door leading to another door, exploring the palace one room at a time. However, of course, human minds are not bound by the finite laws of molecule formation, and so now and then an idea does occur to someone that teleports us forward a few rooms, skipping some exploratory steps in the adjacent possible. However, those ideas almost always end up being short-term failures, precisely because they have skipped ahead. We have a phrase for those ideas: we call them “ahead of their time”.
Therefore, the trick is to figure out ways to explore the edges of possibility that surround you. Challenging problems do not usually define their adjacent possible in such a clear, tangible way.
5. Hunches – Sustaining the slow hunch is less a matter of perspiration than of cultivation. You give the hunch enough nourishment to keep it growing and plant it in fertile soil, where its roots can make new connections. Then you give it time to bloom. The slow hunch is the rule, not the exception. This is how slow hunches often mature: by stealth, in small steps. They fade into view.
Most great ideas first take shape in a partial, incomplete form. They have the seeds of something profound, but they lack a key element that can turn the hunch into something truly compelling. More often than not, that missing element is somewhere else, living as another hunch in another person’s head.
For example, the Web arose as the answer to an open challenge, through the swirling together of influences, ideas, and realizations from many sides, until, by the wondrous offices of the human mind, a new concept jelled. It was a process of accretion, not the linear solving of one problem after another.
Keeping a slow hunch alive poses challenges on multiple scales. For starters, you have to preserve the hunch in your memory, in the dense network of your neurons. Most slow hunches never last long enough to turn into something useful, because they pass in and out of our memory too quickly, precisely because they possess a certain murkiness. So part of the secret of hunch cultivation is simple: write everything down.
Reading and writing were, therefore, inseparable activities. They belonged to a continuous effort to make sense of things, for the world was full of signs: you could read your way through it; and by keeping an account of your readings, you made a book of your own, one stamped with your personality.
6. Recycling – Good ideas are not conjured out of thin air; they are built out of a collection of existing parts, the composition of which expands (and, occasionally, contracts) over time. Nature’s innovations, too, rely on spare parts. Evolution advances by taking available resources and cobbling them together to create new uses. Old ideas can sometimes use new buildings. New ideas must use old buildings. Platforms recycle much more than just architecture.
Part of coming up with a good idea is discovering what those components are, and ensuring that you are not just recycling the same old ingredients. The trick to having good ideas is not to sit around in glorious isolation and try to think big thoughts. The trick is to get more parts on the table.
Factors promoting innovation
If life is good, keep doing what you are doing. Don’t mess with success. However, when the world gets more challenging—scarce resources, predators, parasites—you need to innovate. Moreover, the quickest path to innovation lies in making novel connections. When nature finds itself in need of new ideas, it strives to connect, not protect.
Here are some factors that foster innovation.
1. Big cities
Necessity is the mother of invention.
This quote cannot be more appropriate especially in the case of major cities where there is scarcity in space. Despite all the crowding and distraction, the average person residing in a metropolis is three times more creative than one living in a town of hundred thousand.
The main cities nurture subcultures much more efficiently than suburbs or small towns. Lifestyles or interests that deviate from the mainstream need critical mass to survive; they atrophy in smaller communities not because those communities are more repressive, but rather because the odds of finding like-minded people are much lower with a smaller pool of individuals. Wherever lively and attractive parts of cities are found, the small much outnumber the large.
For example, there is a thriving Indian music scene including artists such as Prasanna, Rini in NYC, Berklee Indian Music Ensemble in Boston, and Gautam Tejas Ganeshan in SFO.
According to the quarter-power law governing innovation, “a city that was ten times larger than its neighbor was not ten times more innovative; it was seventeen times more innovative. As cities get bigger, they generate ideas at a faster clip. This is what we call “superlinear scaling.”
Cities, then, are environments that are ripe for exaptation, because they cultivate specialized skills and interests, and they create a liquid network where information can leak out of those subcultures, and influence their neighbors in surprising ways. The cultural diversity those subcultures create is valuable not just because it makes urban life less boring. A world where a diverse mix of distinct professions and passions overlap is a world where exaptations thrive.
Cities and markets recruit more minds into the collective project of exploring the adjacent possible. The innovation power of the marketplace derives, in part, from this most elemental math: no matter how smart the “authorities” may be, if they are outnumbered a thousand to one by the marketplace, there will be more good ideas lurking in the market than in the feudal castle. Libraries make this possible by democratizing knowledge.
2. Work environment
A work environment that carved out a space for slow hunches cordoned off from all the immediate dictates of the day’s agenda is ideal for innovation to thrive. It is true that ideas happen inside minds, but those minds are invariably connected to external networks that shape the flow of information and inspiration out of which great ideas are fashioned.
3. High signal-to-noise ratio
We value good ideas because they tend to have a high signal-to-noise ratio.
For instance, without noise, evolution would stagnate, an endless series of perfect copies, incapable of change. One way or another, the noise had to be preserved, because, without it, evolution would grind to a halt.
4. No permission
When you do not have to ask for permission, innovation thrives. The natural state of ideas is flow and spill over and connection. It is the society that keeps them in chains.
For example, in modern tech-speak, markets allowed innovation to flourish at the edges of the network. Planned economies were more like the old mainframe computer systems that predated the Internet, where every participant had to get authorization from a central machine to do new work.
Another example is the issue of copyrights. There is nothing “natural” about the artificial scarcity of intellectual property law. Those laws are deliberate interventions crafted by human intelligence and are enforced almost entirely by non-market powers. If you want to get into a debate about which system is more “natural,” then the free flow of ideas is always going to trump the artificial scarcity of patents.
Ideas are intrinsically copyable in the way that food and fuel are not. You have to build dams to keep ideas from flowing. We are often better served by connecting ideas than we are by protecting them. Good ideas may not want to be free, but they do want to connect, fuse, recombine. They want to reinvent themselves by crossing conceptual borders. They want to complete each other as much as they want to compete.
5. Cross-pollinating ideas across disciplines
There is as much value to be found in seeking the common properties across all these varied forms of innovation and creativity as there is value to be found in documenting the differences between them. The poet and the engineer (and the coral reef) may seem a million miles apart in their particular forms of expertise, but when they bring good ideas into the world, similar patterns of development and collaboration shape that process.
The coffeehouse model of diversity, unlike the “melting pot” political kind, is about the diversity of professions and disciplines, not of race or sexual orientation. Many of history’s great innovators managed to build a cross-disciplinary coffeehouse environment within their individual work routines.
Legendary innovators like Franklin, Snow, and Darwin all possess some common intellectual qualities—a certain quickness of mind, boundless curiosity—but they also share one other defining attribute. They have many hobbies.
Chance favors the connected mind. For understandable reasons, we like to talk about artistic innovations regarding the way that they break the rules, open up new doors in the adjacent possible that lesser minds never even see. However, genius requires genres.
Genres supply a set of implicit rules that have enough coherence that traditionalists can safely play inside them, and more adventurous artists can confound our expectations by playing with them. Genres are the platforms and paradigms of the creative world. They are almost never willed into existence by a single pioneering work. Instead, they fade into view, through a complicated set of shared signals passed between artists, each contributing different elements to the mix.
For instance, Gutenberg’s printing press was a classic combinatorial innovation, more bricolage than a breakthrough. Each of the key elements that made it such a transformative machine—the movable type, the ink, the paper, and the press itself—had been developed separately well before Gutenberg printed his first Bible. Evolutionary Biologists have a word for this kind of borrowing, first proposed in an influential 1971 essay by Stephen Jay Gould and Elisabeth Vrba: exaptation. An organism develops a trait optimized for a particular use, but then the trait gets hijacked for a completely different function. If mutation and error and serendipity unlock new doors in the biosphere’s adjacent possible, exaptations help us explore the new possibilities that lurk behind those doors. The history of human creativity abounds with exaptations. The history of the World Wide Web is, in a sense, a story of continuous exaptation. The literary historian Franco Moretti has persuasively documented the role of exaptation in the evolution of the novel. New genres need old devices.
Modern indexing software like DEVONthink’s learns associations between individual words by tracking the frequency with which words appear near each other. This can create almost lyrical connections between ideas. There are few actions as commonly connected to the pursuit of creativity as free-associating.
“All decisive events in the history of scientific thought can be described regarding mental cross-fertilization between different disciplines.”
How ideas spread
1. The 10/10 rule –  A decade to build the new platform, and a decade for it to find a mass audience.
Examples: Cell phones, personal computers, GPS navigation devices
2. Dense networks – In a low-density, chaotic network, ideas come and go. In the dense networks of the first cities, good ideas have a natural propensity to get into circulation. They spill over, and through that spilling they are preserved for future generations. For reasons we will see, high-density liquid networks make it easier for innovation to happen, but they also serve the essential function of storing those innovations.
Before writing, before books, before Wikipedia, the liquid network of cities preserved the accumulated wisdom of human culture.
The network needs to be densely populated. The network has to be plastic, capable of adopting new configurations. A dense network incapable of forming new patterns is, by definition, incapable of change, incapable of probing at the edges of the adjacent possible.
Economists have a telling phrase for the kind of sharing that happens in these densely populated environments: “information spillover.” The idea, of course, is to strike the right balance between order and chaos.
To conclude,
Go for a walk; cultivate hunches; write everything down, but keep your folders messy; embrace serendipity; make generative mistakes; take on multiple hobbies; frequent coffeehouses and other liquid networks; follow the links; let others build on your ideas; borrow, recycle, reinvent. Build a tangled bank.