1)❓ What is The Grid?

The Grid is the longest functioning machine built by mankind. It is also the most complex, and the most expensive (at ~$5 Trillion replacement value in the US alone it dwarves the International Space Station and, frankly, anything else). The Grid is managed by the Grid Operator (often referred to as TSO), which essentially has two jobs.

Their first job is to balance Supply and Demand. Everyday, the market operator in Spain organizes an auction, asks consumers and utilities to forecast demand, fixes energy prices, and defines a production “program” to meet the anticipated demand. This is called P48: 48 half-hourly periods that divide a full 24-hour day for energy scheduling and balancing purposes. If everyone followed their plan, the system would be balanced (and the TSO would be very happy and have less work). But this, never happens, and the real demand and generation almost always differs from the programmed plan. So, the Grid Operator needs to correct these balances by dialing up or down capacity or playing around with demand.

Their second job is to regulate Grid Frequency. Electricity isn’t just about voltage (how strong the push is) but also how steadily and in in what rhythm that push happens. In Europe, systems are designed to keep frequency close to 50 Hz constantly. Keep this number in mind, it will comeback soon. Also, energy sources that rely on controllable inputs (gas, coal, hydro) are much better at regulating frequency than non-controllable ones (solar, wind).

To make things more complicated, the Energy Grid isn’t a single, unified system. It’s made up of many different nodes, each covering a region with its own electricity supply and demand. These nodes must stay in sync, which makes the Grid harder to manage. On top of that, some grids are interconnected across countries to share electricity. For example, Spain’s grid is interconnected to those of France, Portugal, Morocco and Andorra. This is good because it helps with stability but brings in new challenges (when things change in neighboring grids).

‍

2)💥 *** happened?

Under normal circumstances, the Grid Operator has multiple tools at their disposal to balance the network: it does so by adding dispatchable capacity (e.g.: turning on a gas power plants, commonly referred to as “peaker” plants), shutting of large industrial consumers, and performing frequency regulation (e.g.: turbines in large gas, nuclear power plants, large scale generators).

Unbeknownst to most, the Grid Operator does this multiple times a day. All to ensure the smooth operation of the Grid. However, yesterday’s misbalance, was more than the Grid Operator could handle.

At 12:33 pm, the frequency, dropped from the target 50 Hz to 49.85 Hz. This drop looks really small, but in the context of grid operation it is catastrophic. The Grid is designed to keep 50 Hz with an extremely tight margin of ±0.1 Hz. Anything outside this margin triggers an emergency shutdown.

This oscillation led to Spain’s grid disconnecting from the European system. Power Plants and relays at the different nodes started disconnecting to protect themselves, and this cascaded into a widespread collapse of the system. In under 5 seconds, Spain lost 15GW of capacity (equivalent to 60% of the overall demand). Portugal and Spain’s mainland went dark.  All the 7 nuclear power plants shut down. Some analysts even claim that if the drop had been closer to 0.2 Hz we would be talking about a continent-scale blackout.

‍

3)⌛Why it took so long?

The challenge with Spain is that it’s a Grid with extremely high solar and wind penetration. In this specific case, this is bad because most renewable capacity, (unlike, for instance, gas) cannot be dispatched on-demand. There is, literally, no precedent in history of such a large Grid, with such penetration of renewables, being reenergized. We will soon become a case study. The response and root causes will surely be studied by TSOs around the world.

It took 17 hours for the system to restore 90%+ of the Grid (at 5 am today), way more than the planned 6-8 hours. The reason for such a slow restart is that the Grid Operator needs to do this node by node, doing so little by little to avoid the very same frequency imbalances that would bring us back to square one. Also, as crazy as it sounds… many power plants (here and elsewhere) require external electricity to restart from this state (called a black start), they cannot restart on their own.

To make things worse, the longer it takes to restart everything, the worse. It is a compounding problem. Substations at the different nodes are equipped with batteries that allow for remote restart of switches. After 10+ hours, there is risk of depleting this batteries, and reenergizing nodes could have required some manual switching (watch this, the safety and PPE involved, it will probably blow your mind).

We also were lucky we had help from our neighbors with energy is flowing from France and Morocco to help wake up the Grid. Also, water reservoirs were at record levels, and Hydro which is fast, great at frequency regulation and voltage stabilization, has been fundamental to restore the grid overnight. Gas Combined Cycles did most of the heavy lifting too, and saved the Grid - despite coming out of a wild winder that pushed them through extremely demanding operating regimes.
‍

‍

4) 🔍 What is the root cause?

Let’s preface this by saying, "Nobody Knows Anything" (yet). But multiple hypotheses have floated as plausible explanations for this incident. Let’s review them, in no particular order:

1. Cyberattack: Cyber warfare against energy assets and the Grid has happened before. Israel and the US allegedly used a computer worm to delay Iran’s nuclear program by months/years. Spain’s Grid suffers ~1.000 cyberattacks/day, so a highly successful one is not entirely unreasonable. However, the European Comission, Portugal and Spain’s Intelligence agency have downplayed this so far.‍
2. Disturbance of the Interconnections: In the early hours of the blackout, some media outlets pointed at a potential wildfire in the southwest of France that damaged a high power line between Narbonne and Perpignan and disrupted the French Grid and spread to Spain through the interconnection. Also this was identified as the first root cause by the responsible from the TSO. However, France’s TSO (RTE) scrambled to disputed this. While the French interconnection did play some role in yesterday’s episode, our Portuguese neighbors are in the clear: the issue certainly originated outside of Portugal.‍
3. ‍Technical fault: A fault or fire in one of the substations in the nodes has also been considered as a possible explanation. This however seems unlikely, as Spain’s TSO is no stranger to this events, and this can be easily solved by isolating the affected node.
4. “Atmospheric Event” affecting Power Lines: This hypothesis was first raised by Portugal, who blamed it on an “atmospheric event” affecting Spain, specifically on a phenomenon called “Induced Atmospheric Vibration”, which, apparently, is a thing in physics. In English, this means, that under certain conditions (humidity, temperature), very high-voltage lines experience oscillations and changes in conductivity, which can disturb the Grid. This phenomenon, while well documented, rarely manifest at such a scale.
5. Renewable-induced: Due to extreme variations in the availability of renewable resources a sudden and major shift in wind or solar generation, can cause a frequency problem. Because this energy sources cannot be regulated, this can cause a blackout (they almost did in Germany in 2017). This is the hypothesis that carries the most weight among experts.

It is still early to say, but if we had to bet our money we are inclined to think this was not a product of a cyberattack, and is more likely to be technically related, likely any of the options 2 to 5.

5) ⏭️ What’s next? What to expect?

‍
We are certain this will send shockwaves through both Spain and Europe. As we said before, this will likely be a textbook example on multiple topics like Grid Design, “Black Start” (re-powering a Grid after shutdown) and Energy Policy. In summary we believe we will this episode will ignite all of the following:

• Renewable Energy Debate: This blackout has happened shortly after Spain hit its first weekday of 100% renewable energy powered grid. Some voices in the sector have grown wary of the challenges of having a Grid with such reliance on solar and wind, and on the coincidence that episodes like this or that of 6 days ago (when Repsol invoked a force majeure clause in one of their refineries due to an electrical failure) happened in days with extraordinarily high solar generation. Regulators should, and will, become more open to discuss the major challenges of Grids so aggressively skewed towards solar and wind, and add further counterarguments to the question of shutting down nuclear power plants.‍
• Grid redesign: The German grid operator issued a statement claiming such a situation could never happen in Germany. Whether true or not, their claim was that the German Grids are better equipped to deal with black restarts, through equipped batteries. ‍
• More talks on interconnections: Spain is an electricity island. It is unclear the role that interconnections played in the blackout, but in any case they seem to be helping with the restart. Spain interconnection capacity is slim, sitting at 2%, well below the 10% EU target. Expect talks on this, although these cross-country infrastructure projects are extremely lengthy and complex. ‍
• Solar and BESS as tools for energy independence: We learned firsthand, from many of our customers, that even those companies with rooftop solar did not escape from the blackout. We also learned from another Barcelona-based startup that many companies with Energy Storage Solutions did not escape either. How come? Most of these systems rely on inverters to produce electricity, and most of these inverters are “on-grid”, requiring a steady grid supply to generate electricity. The reason for this, is that these systems were not designed as with energy independence in mind, but with energy savings in mind. We expect this to change and demand for “off-grid” inverters to pick up.‍
• Energy Agenda: In the short hours after this happened we have had multiple companies reach out to us inquiring about procurement, energy security, and much more. Similar to what happened in 2022, we expect Energy to be bumped to the top of the agenda. Expect CEOs, CFOs, Energy Managers and Procurement Managers to talk about it. We also expect growing interest in systems like UPS (Uninterrupted Power Supplies), gas and diesel-powered generators, voltage regulation devices, and interest in EMS like the one we have built at Evoltio.

Altogether, we believe this is a great opportunity for startups like ours (evoltio.es) and for Founders building in the sector. Energy is back at the top of the agenda.

Let’s build 💪!

‍

‍