Portland VPP Supporting The Grid During Heatwave

Portland is having *another* heatwave. And our local utility, Portland General, is dispatching our Virtual Power Plant (VPP) to help alleviate the grid strain that the additional air conditioner usage will cause. 

As I pointed out during the last time the VPP was dispatched, in our situation, this dispatch operation can actually increase our grid load. That's because we are load-shifting and reducing our electricity bill. 

The VPP control software will improve. Next summer, they'll likely be able to account for our use case. In the meantime, our minor increase in load will be more than made up for by others in the VPP. 

Alternatively, I might disable the VPP temporarily to stay "islanded" so that we don't increase our grid load. After all, I volunteered our batteries to help the grid, not increase the burden on it.

It Takes Talent - How Tesla is different #95

Most companies advertise their products. Outside of a college recruiting event, most companies don't advertise much if at all to attract talent.

The small amount of advertising that Tesla has done, has primarily been to recruit talent, rather than to sell their products. This is true at Tesla's "Day" events (Battery Day, AI Day...) as well as in their social media. This philosophy extends into other Musk-run companies too. The Neurolink Launch Event did demo the product but they spent an equal amount of time talking about the roles in the company and how they need to hire animal caretakers, programmers, chip designers, signal integrity engineers...

One recent interesting example of Tesla "advertising for talent" was a video that starred Tesla's chief vehicle designer, Franz Von Hausen. The video looks more like a cyberpunk video game than real life. It starts with Franz spraypainting Chinese characters on a wall. Someone hands him a tablet, as the glow from the tablet illuminates his face, you can see that he is both impressed and intrigued by the design. He asks, "Who did this?" as the caption asks, "What will you design?" 

Certainly an enticing thought. If you are a designer, you'd want to work at a company that's trying new things, this might catch your attention. The Cybertruck is featured prominently in the ad as an example of something that likely would have never been attempted at another company. 

Tesla plans to design an affordable car for the worldwide market in the Chinese design center that they are beginning to staff with this ad. 

The specifics of this effort are part of the bigger picture. Tesla knows that to achieve things that have never been done before, you have to hire talented designers and engineers. You have to hire people that are going to push things too far (in a controlled environment), see where they break, learn from that, and use that knowledge to make something that not just a derivative product, but a disruption.

100,000 kWh!

We installed our first solar photovoltaic (PV) system in November of 2007. We used a local PV installer company called Mr. Sun Solar. Other than a few down days when an inverter needed to be replaced, the system has been in operation for 4975 days. In 2015, we added a second PV system to our home. This time we used SolarCity (now Tesla). 

In the 8 years from 2007 to 2015, solar costs had dropped significantly and PV efficiency had improved. Meaning that our 2015 system was twice the size at half the cost of our older system.

Today, these two systems have each generated about 50 MWh; collectively generating just over 100,000 kWh (100 MWh) from the sunlight hitting our roof.

Solar energy production on our roof

Looking at the above chart, you can see that, despite the older system having an eight-year headstart, each system has generated ~50 MWh. The smaller system took 13 and a half years to reach this milestone. Whereas the larger PV system arrived in just 5 years, 10 months. I been watching for this crossover point, where the larger system would eclipse the smaller one, but I didn't know it would happen so close to this major milestone of production.

How Much Is 100MWh? 

Okay, we've made 100MWh, but how much energy is that? Let's look at it a few different ways.

The average US home uses 10,399 kWh annually. This means that our 100 MWh could power the average home for 9.6 years. 

The 2020 Tesla Model 3 SR+ is a highly efficient EV with a 239 Wh/mile consumption rate. At that rate, 100 MWh could propel this vehicle for more than 400,000 miles; enough to drive around the planet 16 times.

Looking at it one more way, the EPA says that a gallon of gasoline has 33.7 kWh of energy. This means that our 100 MWh is equivalent to nearly 3000 gallons of gas (however, with zero emissions from our solar).

From 2015 to today, solar's price has continued to drop. If you want solar on your home, you can use our referral code.

The Reason The Boring Co Will Win (That No One Understands)

The Boring Company recently unveiled the Las Vegas Loop. Riders will use an app and select their destination. The app will direct them to a stall number and car; riders pile in and they are whisked off to their destination.

Currently, the cars are human-piloted and only driving at a max of 35 MPH. This, of course, was trumpeted by the Musk detractors. The criticism goes something like this: A car can only hold a few people; a train would hold many more. So, they conclude, install the track and make it a tried and true subway like so many other cities have. This would work much better.

I find this trains-thinking to be stuck in the 1900s. You might even say, they have "tunnel-vision." I'll explain.

Centralized vs Decentralized

Long ago, my day job was as a network engineer. I worked at a company developing network infrastructure products. We analyzed traffic flows (network packets) and methods to reduce latency and optimize throughput. We had contracts with NASA, The Olympics, most of the hyperscaler datacenters, and most of the server OEMs. The network traffic analysis I did there was not the same as automobile traffic, but there were some important lessons. When I started Token Ring was the cash cow, but things changed quickly.

Token Ring

This type of network is not all that different from a train route. Without going into the technology, nodes on this type of network are logically organized into rings. A circulating token controls access. This is not all that different from a train going around a loop, you can only get on the network/tracks when the train/token arrives. The nodes in this type of network are even called stations. This type of network eventually failed because it was not scaleable. As more nodes were added to the network, the effective throughput of any given station slowed. 

Ethernet (Half Duplex)

The technology that succeeded Token Ring was half-duplex Ethernet. This type of network allowed many more stations to be added to the network without choking throughput (with caveats). During those half-duplex days, networks had limited uses. On half-duplex networks, all of the nodes share a common communications media called the bus. Half-duplex networks function well when there are just a few, short-lived, traffic flows; which is why they worked well back then when most network traffic was periodic client-server activity (like fetching email or printing a document). However, as network communication became more essential with the rise of the internet and streaming, this type of common bus network collapsed under the pressure. This is like a walkie-talkie network with everyone on the same channel. You can have as many people as you want on that channel as long as most people are just listening most of the time; however, as soon as everyone wants to start talking frequently, you just have a jammed-up unusable bus. 

Switched Ethernet

We've looked at two network types (Token Ring and half-duplex Ethernet) that didn't scale (albeit for different reasons). Both of these technologies had a centralized control (the ring or the bus). Half-duplex Ethernet was quickly replaced by Full-duplex or Switched Ethernet. With switched Ethernet, there was no bus, no shared walkie-talkie channel, every node has its own dedicated channel. When node A is talking to node B the traffic flows from A to B. If C and D are on the same network, the conversation between A and B generally does not interfere with the traffic between C and D. This is the technology that is used today in everything from your home network to massive datacenters around the world. It is far more scalable and has much less congestion.  

So why didn't we just start here? The idea of switched networks has been around since the 1960s. The problem was the technology. At the heart of a packet-switched network is a switch that must look at every packet that comes in, determine its destination, and then send it out of the switch via the exact right port for that packet's destination. Multiply this by every port on the switch that is both and sending and receiving.  Then grow the network by having switches attached to other switches, path discovery, forwarding tables... the switches that allow this type of network to be possible have to be very advanced.

During the time that Token Ring and half-duplex, these full-duplex switches would have been very expensive (if even technically possible).

Comparing Network Topologies to People Mover Topologies

So let's tie this back to the topic at hand, The Boring Company Loop system. 

Token Ring is like the train or subway. It has a fixed route and the more stops you add, the more people can access it, but the slower the overall speed. More throughput means more latency (i.e., longer travel time).  

Half-duplex is like a bus. If you charter a bus and your whole party is going to the same place, it works great. But if you have a bunch of people getting in all with different agendas, things fall apart quickly.  

The Boring Company's Loop design is a switched network. Riders select their destination and are assigned a dedicated car. That car goes to their selected destination directly. They don't stop at all the points in between to allow people in and out of the car. The Loop system computes the most effective route for your car to your destination. There's no stopping along the way.

The Boring Company's solution allows new routes and stations to be added to the network without adding interim stops at which all passengers must stop even when this is not their destination. 

This is a scalable transportation network. And it will get faster. 

Switched Ethernet started out at a speed of 10 megabits per second (Mbps). This grew to 100 Mbps, then 1000 Mbps or 1 gigabits per second (1 Gbps), then 10 Gbps, 100 Gbps, and now 800 Gbps is under development. I'm not saying that the vehicles in these tunnels will be 80 thousand times faster than their current 35 MPH speed, but they can get 3 or 4 times faster in long straightaways. To be fair, even at 35 MPH, this is far faster than city street traffic. With the stop and go of traffic lights and congestion, city traffic averages about 14 MPH door to door. So even the initial Loop speed is more than twice the speed a taxi / ride-share could offer (although with more limited destination options). 

Today, the cars in the narrow Loop tunnels are piloted by humans. This will change soon. Solving self-driving in this controlled environment will be far easier than solving it in city street driving. As regular readers know, our prediction is that Level 5 driving will not be solved until 2027. These tunnels, on the other hand, are the best case for self-driving. They don't have to deal with rain, snow, sun directly in the lens, cross-traffic... This Level 4 solution could launch as soon as 2022. 

Scale-out vs Scale-up - How TBC Wins

Trains scale by going faster, adding more train cars, and/or more stations/stops. This is scaling up. There are limits to increasing the speed and limits to adding more cars. As the number of stations increases, it increases capacity and access at the expense of increasing the average travel time for everyone using the platform. Scaling up trains quickly hits real physical limits.

The Boring Company scales by adding more destinations, more tunnels, and more cars. This is scaling out and it allows for more parallel operation. It means that stations, routes, and cars can be added to increase capacity without impacting the throughput or latency of the existing routes. It also means that popular stops can increase capacity by adding more ingress/egress tunnels to/from that station, thereby forming superstations. Alternatively, popular locations could have multiple standard-sized stations (e.g., Convention Center North Station and Convention Center South Station). This is easy to do with the Loop system since additional stations don't burden the system.

Scalability Adds Flexibility

If you add a stop on a train route that has low utilization, then you've slowed down everyone for the benefit of a few, if any, passengers. Similarly, if you add a bus stop in an out-of-the-way place, you add cost for the bus to periodically drive past this location, even if no one is getting on or off the bus. 

This is different with Loop. All rides are point to point. If a hotel or casino that has low traffic pays for a station, then the network has grown and no one has been slowed down and there is no on-going fuel cost to drive to this out-of-the-way location unless it is actually needed. 

I've avoided diving into the network analysis (graph theory) math for this article; instead, trying to articulate the common sense case. If you view each station as a vertex and each tunnel as an edge, there's a vast amount of analysis that can be done to understand the traffic flow within the system. The Boring Company will know every ride that occurs within the system. They'll be able to use graph theory and congestion information to determine the best paths for vehicles to take. 

If you have n number of stations, the possible number of station-to-station connections is n * (n-1). They'll be able to use historical information to determine the best path for each new tunnel that they add to the network. According to the Vegas Unzipped image above, there are 17 stations currently planned for the Vegas Loop. That's a possible 272 tunnels that could be dug for full-mesh connectivity. This would, of course, be overkill at the start of the system, but it demonstrates their ability to scale capacity as needed.

The Time Is Right

If this is clearly the best method for public transportation, why hasn't it always worked this way? The answer is the same as it was for Switched Ethernet: technology. Imagine if everyone that got on a train told the engineer where they wanted to go and then the route was computed and the tracks were switched in real-time. That wouldn't have been possible with last century technology. Today, however, route planning is trivial. 

Next on our 'the time is right' list is EVs. When a tunnel is designed exclusively for EVs, it doesn't need elaborate ventilation systems. Tesla's EVs will have the range to have a full day's service. They have the performance to whisk you at speed quickly to your destination. 

Our third item on this list is Apps. Today, nearly everyone has a smartphone, with the LV Loop app, you'll be able to schedule your ride and pay. The route will be calculated before you even click your seatbelt. There's no need to queue up to buy tickets... this is just another way that Loop systems will allow parallel operation.

Conclusion

Loop is in its beta phase, it will improve greatly over the next few years. As it matures, it will become the most efficient public transportation system we've ever built. This is not simply because it uses electric vehicles, it's because they have a significant technological advantage over the legacy competing technologies.

This is the same formula that other Musk Co. endeavors use. SpaceX's reusable rockets have a massive cost advantage over "disposable" rocket companies. Tesla's electric cars have a massive efficiency and performance advantage over their fossil-burning competition. 

The Boring Co. has "packetized" transportation and made a scalable "switched" network. This gives them a technological advantage over their last-century-based train competition. 

It's time to change from tunnel-vision to visions of tunnels.

Heat Impact on Solar Production

In my last post, I quipped that our solar energy production dipped because of the recent high temperatures. Thinking more about this, I decided that I had only looked at a couple of days. This is one of the most common human fallacies. I had an idea, I looked for confirming data, found some, and assumed that means I was correct.

You overcome this, not by trying to prove yourself right, but by looking for credible data to prove youself wrong. Well, it didn't take long. 

Day	Fri	Sat	Sun	Mon	Tues
Temp (F)	93°	108°	111°	115°	91°
Solar (kWh)	75.1	50.5	73.5	49.4	75.2

Looking at the table, if you compare Friday and Saturday, it sure seems like the increased temp resulted in lower solar production. Saturday was 15°F hotter and it had ~25kWh lower output. 

Comparing Monday and Tuesday is a similar story. Monday was 24°F hotter and had about 25kWh lower output. 

Problem solved right? Nope. Sunday shatters this correlation. Sunday was hotter than Saturday but had solar production closer to the cooler Friday and Tuesday. Perhaps I was too quick to blame the production fluctuation on heat. 

It is well known that heat can impact solar efficiency, but the impact is not as significant as I thought. Each brand/model of solar panels is a little different, but they all publish the heat impact by listing the efficiency impact for each degree Celsius above 25°C. One reasonable example is negative 0.258% per degree C. So going from 93F to 108F is about a 7°C change. This would be about a 2% efficiency change. This alone does not account for the ~30% decrease from Friday to Saturday. Clearly, something else is happening too. 

Looking at weatherunderground and other sites, these days went from "passing clouds", to "clear", to "sunny." Meaning that, despite all of them being hot, hot days, they had varying levels of cloud coverage. Sunday had the least cloud coverage and solar production remained high later into the evening (8PM) than other days.

So there you have it, the variance can primarily be blamed on clouds, not the heat.

120 Hours of AC - Excessive Energy Usage

The heatwave hitting the Pacific North West seems to have finally broken. This event set new record highs for many locations in the region. British Columbia set a new record for the entire country of Canada.

Some have called this event a "once-in-a-millennium" occurrence. This would likely true if we had a stable environment, but we've added a lot of energy into the climate system and, in a perturbed system, you see many unexpected results. We've entered into an era of "global weirding." 

Roads Buckling in Heatwave - via @wsdot_north

Several roads in the region buckled and cracked due to thermal expansion. The trolley cars of Portland had to shut down due to damage to the high power cables. Similarly, the light rail passenger train, MAX, shut down due to cable expansion and sagging. Much of Oregon has had drought conditions for the last three years. Then, compounding the problem, it was hit with temperatures higher than any that have ever been recorded in the area.  

For our little home, we had a few little challenges, but overall we were rather lucky. Some of our neighbors lost power for a few hours. This is not surprising given the load that all the additional air conditioner use caused. However, these outages were in small areas and short-lived. The Western Interconnect grid held up to this stress test far better than the Texas grid has held up to their recent hot weather. Kudos to everyone in the region that helps keep the lights on (and the AC running when we needed it most). 

Speaking of AC, our AC unit is sized for a typical Oregon summer. It is, however,  significantly undersized for extreme heat like this. Even with our AC running all-out, the temperature in our home slowly continued to increase throughout the day. As I write this the AC has been running for 120 hours, non-stop. And it looks like it will be running for at least another 10 hours. The higher highs this heatwave brought were bad, but the higher overnight lows were worse; they meant no relief overnight either. On a typical hot summer day, the AC will run about 16 hours, not for days straight.

Portland weather graph via timeanddate.com

We (and much of the region) are going to have a big power bill this month. Compounding the energy problem is that despite all the sunshine, our solar energy production was reduced. Solar panels have a preferred operating temperature; when they get too hot, their efficiency is reduced. For example on June 28th (one of the hotter days), we generated 49.4kWh. On a more typical summer day, we typically generate ~80kWh. This is a notable reduction in output at a time when we most needed it. Just another reason that you should oversize your solar PV system whenever you can (even if it results in some solar clipping on good days).

Have a great summer and stay cool!

Virtual Power Plant Performs Suboptimally During A Heatwave

UPDATE (6/27/2021): Title updated for accuracy. Details at the end.*

We had our first real virtual power plant (VPP) event and it didn't go as intended. 

The point of a pilot project is to "learn by doing" on a small scale. Lessons learned on a small scale can prevent problems in a bigger program later; so, from that perspective, this was a victory. 

Before getting too much further into this, I should explain what happened. 

On June 21st, one of the longest days of the year, we were having a heatwave here in the Northwest. In response, Portland General Electric decided to put its new VPP into action. Our batteries would be discharged to help offset the expected increase in air conditioner use. Perfect, this is why we signed up. If this helps the utility avoid using diesel generators and peaker plants, that's great.**

On the surface, this seems like a great plan. At 5PM PGE is going to take over the battery's operation. BUT two hours before, at 3PM, peak time starts. Our battery is configured to discharge during peak hours and remove our home's load from the grid. So at 3PM, the battery responds as expected and our home is off-grid (sometimes referred to as islanding). Actually, our home is better than off-grid. The battery is running our house and the solar panels are feeding the grid. 

Then at 5PM, PGE takes over operation of the battery. Up until this point, the battery had been discharging at a rate of about 7 to 10kW (adjusting up and down with our home's needs). When PGE took over, they had the battery discharging at a steady 2kW (see graph below).

Home Energy Flow: (grey is the grid, green is Powerwall, yellow is solar)

This was 5 to 8kW lower than it had been discharging. This increased the grid load, exactly the opposite of the intention of the program.

Looking at the graph, you can see that after ~9AM, when the battery was full, our home became a negative grid load. Our solar panels generated enough to run our home, air conditioning and all. In the times the AC cycled off, we were feeding the grid. Then starting at 3PM our battery took over and we continued to be a negative load. It was not until PGE took over at 5PM that we started to use energy from the grid and add to the demand. 

If our battery had been in standby/backup mode, just sitting at 100% charged up, waiting for an outage, then this VPP plan would have worked fine. However, that was not the case. 

The SmartBattery program needs to add another level of "smarts". For example, setting up the battery to discharge at least 2kW, that would have worked better. Alternatively, they could have requested that the battery discharge 2kW more than the home required, thereby guaranteeing some level of feed-in. Perhaps the simplest option would have been to have the battery discharge at a higher level, e.g., 8kW. This is well within the 15kW that our system can sustainably supply. 

It may be that such modes are not possible with the APIs available to VPP operators. Requesting 8kW works fine for our system, but if an owner only has a single Powerwall, 8kW is not an option. The VPP does not currently customize the request for each home. Tesla Powerwalls are not the only home battery system in the mix, so they may need to adhere to a lowest common denominator mode... 

It could be, that the net result was still positive, just sub-optimal. For example, say there are 500 homes in this pilot. 50 are in a state similar to mine. Each added an average of 4kW of load to the grid for a total of 200kW more load. The other 450 homes, however, added 900kW of relief to the grid. This means that the VPP added 700kW of net relief to the grid. Still a net gain, but not as good as adding 900kW or more grid relief.

It looks like it is going to be a hot one this summer, so you can expect that this will not be the last VPP call-to-arms. Perhaps they will make some improvements before the next event.

Ω

* UPDATE1: The initial title was "VPP Fails During A Heatwave". As I explained in the article, the VPP didn't perform as intended at my house, but that does not mean that the VPP as a whole failed. Some readers, rightfully so, called me out on this clickbait characterization and I've updated the title to be more accurate and less clickbaity. One other minor update: the original article referred to June 21st as solstice. June 21st is often the solstice, but this year, in N. America, the solstice occurred on June 20th.

** Sidebar1: Global warming is causing hotter summers, which increases energy demands, which (when energy is sourced from fossil fuels) increases emissions, which increases global warming... This feedback cycle can/must be broken. Summertime is when the sun shines and solar energy production scales well with AC usage. Combine this with just a few hours of energy storage and you can time-shift loads as needed to stabilize grid demand.