This piece was originally written for the Global Institute for Women's Leadership's new Essays on Equality publication, which features contributions on a range of different issues by leading researchers and figures working on gender equality.

Read the full collection of essays

Leadership, the process that enables individuals to work together in the pursuit of a common goal, has been a critical resource throughout the evolution of humankind. Every significant accomplishment in human history – the use of fire, the invention of writing, the mapping of the human genome, and so on – sprang from collective action that could not have occurred without leadership.

A century of science has provided an enormous amount of evidence about what good and bad leadership looks like, but this does not erase the archetypes of leadership in our minds. Our gut feeling for what constitutes good leadership is shaped much more by our ancestral and evolutionary roots than by the latest research on leadership, as our brains have been shaped by millions of years of evolution. Even when old models of leadership are no longer effective, they still match our imaginary leadership archetypes fuelled by our instincts, and it is not easy to unlearn them.

In my view, this is the main reason for the uneven sex ratio in leadership and management across the board, throughout countries worldwide. And part of this is our inability to discern between confidence and competence – that is, because we commonly misinterpret displays of confidence as a sign of competence. This is consistent with the finding that leaderless groups have a natural tendency to elect self-centred, overconfident and narcissistic individuals as leaders, and that these personality characteristics are not equally common in men and women. So, when it comes to leadership, the only advantage that men have over women is the fact that manifestations of hubris – often masked as charisma or charm – are commonly mistaken for leadership potential. And again, these occur much more frequently in men than in women.

So long as we continue to associate leadership with “masculine” features, we can expect female leaders to be evaluated more negatively, even when their performance is higher than that of their male counterparts, and even when those who evaluate them are women. For instance, a recent study on social sensing, in which male and female leaders were tagged with sociometric badges that monitored everything they did and said for weeks, showed that despite a lack of behavioural or performance differences between men and women, men were promoted to leadership roles much more frequently than women were.

Yet this tendency to uphold typically male leadership styles as more desirable is costing us, not just in terms of gender equality, but also in terms of lost revenue. There is now compelling scientific evidence for the notion that women are more likely to adopt more effective leadership strategies than men. Most notably, in a comprehensive review of studies, Alice Eagly and colleagues showed that female managers are more likely to elicit respect and pride from their followers, communicate their vision effectively, empower and mentor subordinates, and approach problem-solving in a more flexible and creative way, as well as fairly reward direct reports. In contrast, male managers are statistically less likely to bond or connect with their subordinates and are relatively more inept at rewarding them for their actual performance.

Indeed, whether in sports, politics or business, the best leaders are usually humble – and whether through nature or nurture, humility is a much more common feature in women than men. For example, women outperform men on emotional intelligence, which is a strong driver of modest behaviours. Furthermore, a quantitative review of gender differences in personality involving more than 23,000 participants in 26 cultures indicated that women are more sensitive, considerate, and humble than men, which is arguably one of the least counterintuitive findings in the social sciences.

An even clearer picture emerges when one examines the dark side of personality: for instance, our normative data, which includes thousands of managers from across all industry sectors and 40 countries, shows that men are consistently more arrogant, manipulative and risk-prone than women. We also need to acknowledge the importance of emotional intelligence – or EQ – which should be a core competency in any data-driven model of leadership potential. Paying more attention to EQ would augment both the quality of leaders and the number of female leaders, increasing the overall levels of personal effectiveness, self-awareness and transformational leadership in organisations.

So how do we fix this to not just increase the representation of women in leadership but also improve the quality of our leaders overall? As I argue in my latest book, Why do so many incompetent men become leaders (and how to fix it)?, organisations can take concrete steps. For example, they can stop interpreting displays of overconfidence, narcissism, psychopathy, and charisma as a sign of leadership potential. In an ideal world, leaders would follow science-based practices and prioritise engaging with and inspiring their employees, and providing them with a sense of meaning and purpose.

So we need to properly understand leadership talent and learn how to measure it. These solutions are easier said than done. Too many decision-makers overrate their intuition, and political agendas interfere with the selection of talented leaders, especially when the decision-makers are more interested in their own agendas than their organisation’s wellbeing. Knowing how to detect true leadership potential is clearly not enough. We also have to introduce measures that place better leaders in key roles and promote a culture that helps them succeed.

While it is certainly a sign of progress that a growing number of organisations are putting in place deliberate interventions to increase the proportion of women in leadership, a more reasonable goal would be to focus instead on selecting better leaders, which would also take care of the gender imbalance. There is no conflict between boosting gender equality and boosting leadership quality. On the contrary, it is harder to improve the standard of leadership without improving the number of female leaders.

Adapted by permission of Harvard Business Review Press. Excerpted from Why Do So Many Incompetent Men Become Leaders? (and How to Fix It) by Tomas Chamorro-Premuzic. Copyright 2019 Tomas Chamorro-Premuzic. All rights reserved.

Tomas Chamorro-Premuzic is a Professor of Business Psychology at University College London.