Fallacies of Distributed Systems — Applied to Teams
Introduction
I recently attended a presentation by Andrew Harvey, CTO in Residence at Microsoft of Startups. This presentation was titled Your Team as a Distributed System, and in it, Andrew talked a bit about the 8 Fallacies of Distributed Systems and how they apply to a team. In this series I have summarised his key points, added additional points from my knowledge and drawn my own conclusions about what this means for managers.
I hope you find the content useful and gain as much from it as I did from Andrew’s presentation.
Fallacy 1: The network is reliable
The first fallacy is easy to prove in a team environment. We just need to remember the last time we accidentally missed an email, forgot to respond to a Slack message, or forgot something that was said in a meeting. Once we accept that our network is unreliable (if it were a computer network, we’d fire the network admin because it’s so bad) we can start to recognise that we need to use different communication protocols for different types of information and for different recipients. If the wrong protocol is used, then the communication will likely be lost. Again, in networking terminology, we need to use TCP not UDP; if we use UDP we have no idea if our communication was received. With TCP we have some additional overhead but get confirmation that the message was received.
Fallacy 2: Latency is zero
If you need proof that latency isn’t zero, simply send an email to someone and don’t do anything else until you get a response, we can quickly discover there can be significant latency in our human network. The members of our team have to prioritise their tasks, the latency caused by a communication delay can trigger team members (or teams) to perform conflicting work, it can create deadlocks where people are unable to proceed due to unfulfilled dependencies and can become quite wasteful without appropriate management.
Fallacy 3: Bandwidth is infinite
Unfortunately, as humans, we do not have unlimited bandwidth. We have a limited ability to express ourselves, our ideas and our knowledge. Our communication mediums are also limited; when we talk face-to-face we have more bandwidth available as we can get a better idea of body language, in video calls we lose the body language but still have access to facial expressions, on a phone call we can still hear vocal inflections even though we can’t see the person’s face, and when we get down to text communications we often have to rely on emojis. As we receive information, we fill in the blanks to make up for lost bandwidth. This can lead to miscommunication and misunderstandings.
Fallacy 4: The network is secure
In most situations we can make an assumption that our human network is relatively secure and no one is actively sabotaging it, although sometimes people are looking out for their own self-interests which can be a form of unintentional sabotage.
What is more likely is that our network is prone to corruption. Every time information is communicated it changes to some degree, the more points of relay, the less likely that the information being conveyed will be accurate. It is important to ensure that we verify what we heard with the person telling us some information to reduce the chance of corruption.
Fallacy 5: Topology doesn’t change
The topology of our human network changes frequently. This could be driven by corporate restructures, it could be the scaling (either up or down) of a business or team, or it could be due to staff turnover. I’ve never seen a team remain stable, there will always be changes in the topology of our human network, we need to ensure this is accounted for in all actions of the team. This could mean documenting processes, removing single points of failure, and having easy onboarding processes to ensure a change in team topology has the minimum possible impact.
Fallacy 6: Only one administrator
Many businesses are structured to have a single administrator, but in practice there are always multiple people controlling the flow of information. The board or CEO may set a direction for the business, but at the message is communicated down through the layers of the hierarchy, each layer adds or removes information to better align to their own agenda. Even at the individual contributor level, each person will focus on what’s important to them. Because of our ability to individualise each requirement we need to ensure that consensus is reached and that all parties are working toward the same goal.
Fallacy 7: Transport is free
Many of us are acutely aware of the cost of communications, rom the time spent in a meeting, to the context switching and loss of productivity from interruptions. It is important to balance the need for communication and the required productivity. With too little communications everyone is working for a different goal, with too much communication no one can achieve anything.
Fallacy 8: Network is homogeneous
The final fallacy is that the network is homogeneous. Every team and every team member has a different driver or motivator, we need to learn what each of these is and to optimise the interactions as early as possible. The closer we can get to a homogeneous environment the easier the communications become. We don’t have to align the drivers of every team member but helping them to understand each other will help to make communications more effective.
Conclusions I’ve Drawn from the Fallacies
Recognising that our team or human network is a distributed system and is subject to the fallacies of a distributed system leads me to the belief that the primary role as leaders and managers is to coordinate the communications between the components of our system (the people). If our team is not delivering in a coordinated fashion at the expected rate of delivery, we are failing as managers to optimise our network.
Our primary role as a leader or manager is to manage and facilitate communication. To achieve this we need to ensure that messages are delivered clearly and via appropriate means, that delays are kept to a minimum, that messaging is understood and acknowledged, that we can cope with change, that we don’t under or over communicate, and that we align as much as possible.
If we treat our team as a distributed system, account for the deficiencies of these systems in all that we do, then we can be successful as a leader and our team can be successful. If we fail to allow for the deficiencies, or if we are unable to negate the negative impacts of these deficiencies, then we have failed as a leader and manager.
This article is one in a set of 3. The others can be found at:
This article originally appeared on my personal blog, AlphaGeek.
