Random, Living, Error
Serendipity is the relationship between a surprise and a beneficial outcome. The source of surprise can be something random, leaving, or an error. It can also be all three at once, like when we spill coffee on a stranger, and this leads to something good.
This essay is part of the serendipity series. In a previous post, I described three patterns arising from the distinction between a serendipity event and a related serendipity episode. This one is about another set of patterns, but this time, it is related to the source of surprise.
Random
Mild success can be explainable by skills and labor. Wild success is attributable to variance.
Nassim Taleb
Our society loves order, control, and predictability, but its mere existence is a matter of randomness. Randomness plays an important evolutionary role from mutation of bodies (diversity and adaptation) through mutation inside bodies (B-cells to fight unknown pathogens) to bodies and minds coming up with smart ways to deal with complexity (e.g., Monte Carlo method, simulated annealing).
The recent worldwide recognition of randomness is associated with the work of Nassim Taleb. However, its importance for adaptability became clear to some seventy years ago. The first machine to show adaptive behavior was Ashby's homeostat, built in the 1950s and described in Design for A Brain. The homeostat was a system with two connected feedback loops. The first one was based on deviation-correction feedback, keeping essential variables within viability ranges. When going beyond these ranges, a step function in the second one tried random new configurations until the stability was restored.
Ashby's work was influenced by Shannon's information theory. However, while Shannon's was concerned with noise reduction, Ashby's step function demonstrated the value of something similar to noise: randomness. Despite this, in the decades since, noise has consistently been viewed negatively, with ongoing efforts to find new and improved ways to minimize it. Yet, noise can be functional and bring value in domains of optimization and computation1 and also, for the primary reason to fight against it, weak signal detection.2
Today's LLMs are good proof that noise should be explored and not eliminated. As Stephen Wolfram said,3 when Aristotle discovered logic, he "stopped too quickly." Our language has regularities beyond grammar and logic. Until we have the computation means to "hear" them, they are disregarded as noise.
Apart from science and technology, randomness has made a mark in art. Jackson Pollock’s use of random drips and splashes in his paintings led to a new art movement known as Abstract Expressionism. His painting "No. 5, 1948", sold for 140 million dollars in 2006, is among the most expensive artworks ever sold.
Randomness might also be the key to solving the current crisis of legitimacy and efficiency in Western democracies. Sortition, the selection by lottery practiced in early democracies, seems less prone to perversions than a system based on elections.4
But if, despite the above examples, noise and randomness are still generally underappreciated, that's not the case when it comes to serendipity. It’s well-known that randomness brings surprises, some of which can be serendipitous. We see that common understanding manifested in various designs and practices. It could be used to break creative blocks by encouraging lateral thinking, which is the case with oblique strategies. Or for group creativity techniques, where it appears as non-sensuring rules. It is there even in the most boring of all, the brainstorming. Another more specific example is Nesta's randomized trial coffees (RTC). Randomness is relied upon when designing a physical space to increase the chances for serendipitous encounters. Black Rock City, the temporary city for the annual Burning Man event in northwestern Nevada, is designed to promote chance interactions between people.
In good recommender systems, randomness helps break bubbles, balancing accuracy and exploration and ultimately leads to serendipitous discoveries. That is the case even for self-curated content such as a collection of annotations. For example, Readwise daily review mixes random highlights with those determined by a spaced repetition algorithm and user feedback. There is a growing appreciation of randomness in Personal Knowledge Management tools. The Smart Block extension for Roam Research has a set of serendipity commands incorporating random functions.5 Even simple tools like Lumen have a "Roll a dice" button next to search.
Random events are unavoidable and can cause episodes of serendipity. All we need to do is notice them and act. But in developing products and practices, to be a catalyst for serendipity, randomness shouldn't be used at random. It takes experimentation and tuning to make it a reliable source of serendipity.
Living
As a result of extensive work with this technique a kind of secondary memory will arise, an alter ego with who we can constantly communicate. It proves to be similar to our own memory in that it does not have a thoroughly constructed order of its entirety, not hierarchy, and most certainly no linear structure like a book. Just because of this, it gets its own life, independent of its author.
Niklas Luhmann
Living creatures are a constant source of surprise. Most serendipitous episodes in our lives are tied to others: those we see daily, an old friend we reconnect with, a new acquaintance we meet at a dinner party or conference, or even a complete stranger sharing an elevator with us by chance. The Path and Pair serendipity patterns discussed in a previous post of this series are triggered mostly by other people.
Not only real but also imaginary people can surprise us. Common for novelists. Once characters have enough characteristics and history, they take a life of their own. According to Stephen King, that’s not exceptional but a sign of a writer’s work well done:
And if you do your job, your characters will come to life and start doing stuff on their own. I know that sounds a little creepy if you haven’t actually experienced it, but it’s terrific fun when it happens.6
In a recent interview about her new book, Sally Rooney was asked if it was a particular challenge to have male characters central for the first time, and she answered:
It definitely was not a conscious project of trying to write about men or trying to write about masculinity. I just felt my way through the story that seemed to emerge when I encountered these characters and I kind of followed where they led me.7
Characters can do unexpected things and lead their authors to places they did not anticipate, but they can also change the story completely. That was what Sara from Colorless Tsukuru Tazaki and His Years of Pilgrimage did to Haruki Murakami:
In almost an instant, the words that Sara spoke totally changed the story’s direction, nature, scope, and structure. This was a complete surprise to me.8
But not only fictional characters take on a life of their own. As the quote starting this section says, it could be a big collection of interlinked notes that come to life.
Luhmann developed and collaborated with his Zettelkasten over a period of 45 years between 1952 and 1997. In the end, it contained over 90,000 interlinked cards.9 The smart indexing system allowed quick finding and growth from every place. Pulled by prompts, links and curiosity, one can make frequent serendipitous discoveries. What made that collection of index cards a serendipity generator was its network structure. Luhmann's Zettelkasten should be regarded as the first personal knowledge graph (PKG), as I claimed in the second chapter of the PKG book. But what is so special about PKGs?
In a library, the answer to a query is the end of the journey; in a graph, it's the beginning. An idea, question, or random browsing can lead users to a note, which may then link to adjacent or distant notes or to a note via a backlink, enabling free exploration of the knowledge network. In other knowledge graphs, this is called follow-your-nose browsing. However, there, it's indeed limited to browsing. In a personal knowledge graph, like Luhmann's Zettelkasten, users can not only gain new insights but also create new notes, connections, or classifications as they explore their graphs. Even what defines a node is up to the user. As Luhmann said, "we can connect [notes] anywhere — even to a particular word in the middle of a continuous text," making that word a node.
New associations and insights make each exploration different, even without changes in the meantime. As Cevolini explains, following "the same searching route" can lead to new meanings due to new links formed by fresh readings or speculations.10 In two sessions, a user might reach the same node through different paths, starting with the same or different queries. This feature of knowledge graphs distinguishes them from SQL and other noSQL models. Graph query languages go beyond standard relational operations, supporting path traversal and arbitrary-length queries.11
The phrase "laying down a path in walking," inspired by the poet Antonio Machado, reflects the idea that engaging with the world creates new possibilities for action and perception.12 There is no pre-given path, as "the path is our footsteps, laid down in walking".13
Similarly, a user walking through a personal knowledge graph is both following and creating a path. Current steps, past ones, and newly made connections will shape future walks.
The user and the personal knowledge graph co-adapt and evolve together, forming a structural coupling. Structural coupling, autopoiesis, and operational closure are foundational to both Luhmann's social systems theory and enactive cognitive science. Though they diverge, either can explain how a personal knowledge graph takes a life of its own.
Enactive cognitive science studies mainly biological individuals and the cognition arising when they engage with the world. However, there are other systems with a self-sustained organization that exhibit autonomy and agency. These can be transient systems like the one emerging when two people try to pass each other in a narrow corridor, or they can be long-lasting, like habits14 and emotions.15 Systems with a self-sustained organization can emerge in situations like interaction with some classes of software tools such as video games.16
What is common between all these systems is that they are operationally closed:
A system is operationally closed if, for any given process P that forms part of the system, (1) we can find among its enabling conditions other processes that make up the system, and (2) we can find other processes in the system that depend on P.17
That is what goes on in a Zettelkasten system, where a card “receives its quality only from the network of links and back-links within the system”,18 which explains why it behaves like a living thing.
That was about the second source of serendipity: everything that has a life of its own: people, fictional characters and non-human collaborators like personal knowledge graphs.
Error
To err is to wander, and wandering is the way we discover the world; and, lost in thought, it is also the way we discover ourselves.
Kathryn Schulz
We try to avoid errors. That's understandable since some are costly, and for the rest, just admitting we were wrong is difficult. But the fear of error can preclude learning opportunities and grow into a risk-averse attitude. We can miss cues, leading to surprising discoveries. Errors can be a source of serendipity. That can’t be such an obscure idea considering the popularity of similar ones like “When life gives you lemons, make lemonade.”
Often, errors are simply deviations from intent. And intent is the first thing to be suspicious of since it's based on assumptions and beliefs. One way to challenge assumptions is to embrace errors as learning opportunities. It’s not just about learning from the error itself — whether about our assumptions or how to avoid it in the future — but about letting the error push us into unfamiliar situations. A risk-averse attitude is symptomatic of overexploring and underexploiting. Worse, it leads to a reduced agency, tolerating a well-paid boring job instead of living a full and exciting life.
When it comes to what errors can bring, we can put them roughly (and often only retrospectively) into three categories. The first is costly errors that have to be avoided. Transport and healthcare errors are clearly in that category.19 The second category is for errors that are not so costly, and we can learn from them. Learning can be about ourselves (assumptions and beliefs), the world, or how to avoid errors in the future. There is a third category of errors that we find ourselves lucky to have made later on. That's the category of serendipity.
Jazz musicians, for example, effectively use that third category of errors. Errors come from risk-taking and active exploration, but improvisers go a step further. They see errors as inevitable when improvising, so they incorporate them into their performance. Herbie Hancock once played a wrong chord, which Miles Davis took as a prompt, played the wrong notes, "embellishing them, using them as a creative departure for a different melody".20 It is a whole different way of reacting to errors. During live performances, instead of ignoring them and moving on, good jazz musicians repeat the errors and further develop them, taking a path that can potentially lead to something better than what would have come out if it had been played as intended.
Serendipitous errors are well known in software development: it's not a bug, it's a feature. It is again about attitude and reaction to errors. Some bugs are not just turned into features but become part of the core functionality of what we associate some software with. The Incognito Mode of Chrome, Firefox's View Source Code, and GitHub's Forking are all bugs that have been turned into features.
Many discoveries are failed attempts to achieve something else. Post-it notes came out of a failure to create a strong glue; the first plastic, Bakelite, was a failed attempt to produce a substitute for shellac, and the first synthetic dye, mauveine, was a failed attempt to find a treatment for malaria.
An error looked upon differently may not lead to a grand discovery but can be a micro serendipity important at a personal level and potentially contributing to something bigger. Some weeks ago when a cupture workflow on my mobile failed, instead of moving on with frustration which is what I'd usually do, I made the capture using an alternative, supposedly inferior method. It turned out so much better than the usual one, than it is now my primary method for that workflow.
Awareness of these three sources of serendipity can influence internal serendipity factors such as mindset, habits, curiosity, and risk appetite. When it comes to practices and product design, there are additional considerations to take into account. Randomness will be of different value depending on where and when it is introduced. For example, when it comes to personal knowledge graphs, it takes some tinkering to find which subset to apply it to, when to bring it, and to which workflows. In the forthcoming essay on graph pruning, I’ll give some concrete examples. Then, for the second, the viability of your graph will depend on three essential balances: autonomy-cohesion, stability-diversity, and exploration-exploitation. And when it comes to errors, there are no recipes apart from one related to size: run small, safe-to-fail experiments, possibly in parallel.
Kahneman, D., Krakauer, D. C., Sibony, O., Sunstein, C., & Wolpert, D. (2022). An exchange of letters on the role of noise in collective intelligence. Collective Intelligence, 1(1), 26339137221078593. https://doi.org/10.1177/26339137221078593
Lex Fridman (Director). (2023, May 9). Stephen Wolfram: ChatGPT and the Nature of Truth, Reality & Computation | Lex Fridman Podcast #376 [Video recording].
Reybrouck, D. V. (2016). Against Elections. https://www.penguin.co.uk/books/434541/against-elections-by-david-van-reybrouck/9781847924223
All functions in programming languages that are called “random” are actually pseudo-random. They use pseudorandom number generators.
King, S. (2020). On Writing. https://www.simonandschuster.com/books/On-Writing/Stephen-King/9781982159375
Marchese, D. (2024, September 21). Sally Rooney Thinks Career Growth Is Overrated. The New York Times. https://www.nytimes.com/2024/09/21/magazine/sally-rooney-interview.html
Murakami, H. (2022, October 24). Where My Characters Come From. The Atlantic. https://www.theatlantic.com/magazine/archive/2022/12/haruki-murakami-book-novelist-as-a-vocation/671845/
Most of them are now digitalized and available online.
Cevolini, A. (2018). Where Does Niklas Luhmann’s Card Index Come From? Erudition and the Republic of Letters, 3(4), 390–420. https://doi.org/10.1163/24055069-00304002
Angles, R., Arenas, M., Barceló, P., Hogan, A., Reutter, J., & Vrgoč, D. (2018). Foundations of Modern Query Languages for Graph Databases. ACM Computing Surveys, 50(5), 1–40. https://doi.org/10.1145/3104031
Varela, F. (1987). Laying down a path in walking: A biologist’s look at a new biology. https://www.semanticscholar.org/paper/Laying-down-a-path-in-walking%3A-A-biologist%E2%80%99s-look-a-Varela/495fc912cb6ef16da41a75c7385f5a5a1fc5db3d
Thompson, E. (2007). Mind in life: Biology, phenomenology, and the sciences of mind. Belknap Press of Harvard University Press.
Egbert, M., & Cañamero, L. (2014, August 1). Habit-Based Regulation of Essential Variables. https://doi.org/10.7551/978-0-262-32621-6-ch029
Colombetti, G. (2017). The Feeling Body. The MIT Press. https://mitpress.mit.edu/9780262533768/the-feeling-body/
Vahlo, J. (2017). An Enactive Account of the Autonomy of Videogame Gameplay. Game Studies, 17(1). https://gamestudies.org/1701/articles/vahlo
Di Paolo, E. A., Rohde, M., & De Jaegher, H. (2010). Horizons for the Enactive Mind: Values, Social Interaction, and Play. In J. Stewart, O. Gapenne, & E. A. Di Paolo (Eds.), Enaction: Toward a New Paradigm for Cognitive Science (p. 0). The MIT Press. https://doi.org/10.7551/mitpress/9780262014601.003.0003
This is from Luhmann, N. (1981). Kommunikation mit Zettelkästen. In H. Baier, H. M. Kepplinger, & K. Reumann (Eds.), Öffentliche Meinung und sozialer Wandel / Public Opinion and Social Change (pp. 222–228). VS Verlag für Sozialwissenschaften. https://doi.org/10.1007/978-3-322-87749-9_19, translated by Manfred Kuehn.
From a certain perspective, an exception might be a medical error that costs a life but leads to a discovery that would save many more lives.
Barrett, F. J. (1998). Coda—Creativity and Improvisation in Jazz and Organizations: Implications for Organizational Learning. Organization Science, 9(5), 605–622. https://doi.org/10.1287/orsc.9.5.605