use-cases.org – Matthew Tippett's Blog

Disclaimer: This post was generated via ChatGPT (GPT-4) based on the following prompt:

I’d like you to write a blog post in the style of Matthew Tippett from use-cases.org. I’d like the article to be long form, and to be centered around using the terms headwinds and tailwinds in the context of software engineering. The general concept is that things that slow engineers down are headwinds, and things that speed engineers up are tailwinds. Lack of documentation is a headwind that increases confusion. Just enough documentation is a tail wind that finds the balance between bootstrapping understanding for an engineer, and wasting too much time writing documentation. I’d like to also bring in the sailing concept of beating or tacking where a headwind doesn’t need to be fully removed if the engineering team can adapt and tack to still obtain some forward momentum.

When it comes to software engineering, you often hear terms like “agile” or “scrum” that emphasize the importance of flexibility and adaptability. But have you ever considered the concept of headwinds and tailwinds in this context? In this blog post, we’ll borrow these terms from the world of sailing to better understand the challenges and opportunities that software engineering teams face daily. Just like sailors, engineers need to navigate headwinds and tailwinds effectively and utilize tacking techniques to make progress, even in the face of obstacles.

Headwinds: The Challenges Slowing Software Engineers Down

In software engineering, headwinds are the factors that hinder productivity and slow engineers down. They can take many forms, including:

1. Lack of documentation: This headwind increases confusion and makes it difficult for engineers to grasp the ins and outs of the project. Insufficient documentation can lead to delays, miscommunication, and rework.
2. Technical debt: Accumulating technical debt due to rushed development, outdated technologies, or suboptimal coding practices can slow progress and make it harder to maintain and improve the software.
3. Poor communication: Ineffective communication among team members or with stakeholders can result in misunderstandings, missed deadlines, and duplicated efforts.
4. Inadequate resources: Insufficient hardware, software, or staffing resources can hinder the team’s ability to develop, test, and deliver quality software on time.

Tailwinds: The Factors Accelerating Software Engineering Progress

On the other hand, tailwinds are the elements that propel software engineers forward, boosting productivity and enhancing project outcomes. Some common tailwinds include:

1. Just enough documentation: Striking the right balance between too little and too much documentation can provide engineers with the necessary information to understand the project without wasting time on excessive details.
2. Continuous integration and delivery (CI/CD): Implementing CI/CD pipelines enables faster development, testing, and deployment of code changes, reducing time-to-market and improving software quality.
3. Effective communication: Clear, concise, and timely communication among team members and stakeholders fosters collaboration, streamlines decision-making, and ensures everyone is on the same page.
4. Adequate resources: Providing the team with the necessary hardware, software, and personnel resources can facilitate smoother development processes and faster project completion.

Tacking: Adapting to Headwinds for Forward Momentum

In sailing, tacking is a technique used to sail against the wind by moving in a zigzag pattern. By applying the concept of tacking to software engineering, teams can adapt to headwinds and still make progress, even if they can’t eliminate them completely.

1. Prioritization: Focus on the most critical tasks and features first, allowing the team to make the most significant impact on the project, even in the face of headwinds.
2. Incremental improvements: Instead of trying to tackle large, complex problems all at once, break them down into smaller, manageable tasks. This allows the team to make continuous progress and reduces the impact of headwinds.
3. Cross-functional collaboration: Encourage team members to share their expertise and knowledge to address challenges more effectively, boosting the team’s ability to tackle headwinds together.
4. Regular retrospectives: Hold regular retrospectives to evaluate the team’s performance, identify headwinds, and implement adjustments to improve productivity and overcome obstacles.

Conclusion:

Navigating headwinds and tailwinds in software engineering requires a keen understanding of the factors that hinder and accelerate progress. By acknowledging these forces and implementing tacking strategies, software engineering teams can adapt to challenges and continue making forward progress.

This is a short youser style guide on diffs. Some background thoughts are in an older blog post about Code and the Written Word.

As an engineer, your legacy within Meta is primarily the code that you create. The oldest thing that will come back to haunt you years in the future is your code. What do I mean about legacy?

• We regularly come across code that was written years ago.
• We do not regularly come across docs written years ago.

So, through that lens we should consider about how that legacy is created. If we work back from the code, we see diffs, near-code documentation, and design notes. I’d argue that diffs and near-code documentation are close peers in what we reference, but for the purposes of this note I’m focusing on diffs.

Hands up who has ever looked at code and thought “What were they thinking when they wrote this?”. The context for the change is probably the singular best answer to that question.

Diffs are Context for A Code Change

Diffs and the associated meta data (tasks, tests, summary, etc) all provide a context for the change.

• Title provides the one-liner view of what’s happening – it’s almost the topic sentence in a paragraph. See the blog post from above
• Summary provides the details on what the diff does that won’t be obvious from the diff itself. See below for a comment of design discussions in this section.
• Testing is actually important in two ways. It allows other engineers to walk through your final testing before you checkin. It also provides a reference about what to test if you need to modify the code in the future. Remember that the familiarity you have with the code and the way to test it is not shared by others and won’t be shared by you in the future.

All these sections provide context on the change. It’s the perfect place to reflect on the maturity of the environment, critical factors affecting how the code is created.

Diffs are a Watercoolor for the Code

Diffs are where we get to know each other and what’s important and what is not important for individuals and the team. We get to know each other’s engineering style, our triggers, our level of pendantism. What motivates us when we look at code.

For example, my style of code review is very sensitive to the “future self“. These are more or less the same things that any other future engineer that intersects with your legacy is going to ask as well.

• What’s likely to happen to the code?
• What’s going to be confusing?
• What’s going to fail?
• What’s going to age poorly?
• When I’ve forgotten I’ve written the code, what’s information will I need to quickly come up to speed.

As we converse and discuss the code, we begin to develop a Theory of Mind for the engineers that we will review the code with. This is important when preparing a diff.

When receiving feedback on the diff, accept graciously any feedback as well intentioned. However you should feel comfortable accepting the feedback and incorporating it, considering it for the future, or respectfully rejecting it. Likewise the reviewer should acknowledge that the 3 paths for the feedback and accept the best intentions of the Author.

Rarely should there be disagreements in the diffs. If there are unresolvable differences it’s time to step out and write something down and agree on those principles.

Diffs should be a Reflection of the Community

It’s never your code. If you are listening to the Watercooler discussions on the diffs, and begin to determine who is going to respond to your diff in different ways, you begin to incorporate your teammates considerations into the way you present your code. When I’m preparing a diff, I’m either conciously or subconciously thinking about what’s important for my team mates and how do I intercept with their concerns.

To be clear, I’m not saying to pander to everyone’s concerns within a diff. I’m saying consider what’s important and when you know there is something relevant that another engineer would care about and be deliberate about if you respond to that or not. If Sarah cares about architectural clarity, and you for good reason are violating that clarity speak to it directly in the diff, or followup outside of the diff. If Paul is concerned about shotgun surgery needed for small changes think about what needs to change for likely small changes.

Having these other voices in your head makes you a better engineer and allows you to live vicariously through others and get their experiences. I find this one of the strongest accelerants for engineer growth.

What doesn’t belong in diffs?

I’ll get a bit controversial here. This is my starting point, but the community around the codebase will have their own norms.

Design Review in Code Review The later changes occur in development (requirements, design, code, deployment, production) the higher cost to make the change. Sunk cost is a real killer. when designs are considered late in the cycle, code may land to avoid rework. Fortunately, there is an easy solution here is similar approach to diffs but for design – small targeted design notes that address specific concerns. Either as part of a bigger design doc, or individual docs.

We’ve all seen it. The engineer who almost feels how a system works and can within seconds can identify what’s going on. Ask them how they do it, and most of them will say “I just know”. Not helpful at all.

Then there are the mere mortals who struggle to get some things debugged and wander in circles trying lots of different things and most of the time stumbling on the problem.

I spent a while observing the intuitive engineers, observing myself, and observing others, and here is how I do systematic debugging. It’s very close to my understanding of the technique of differential diagnosis. Apparently I’m not alone (link, link).

Worked example: let’s consider a web server that is not returning data.

Step 0: Start with a System View

Ironically, this is one of the biggest gaps I see.

What’s the first thing the SWAT team pulls out when they are trying to work out what’s happening in a building? It’s the floor plan. Why? So that they can conceptualize where people would be, where the weak and strong spots are. Without a floorplan they are going in blind.

A lot of engineers are either unwilling or unable to communicate how they think a system works. This is the most critical part since the theory of operation that is in your head will guide the following steps. It ultimately doesn’t matter if the system view is right or wrong – it will evolve as more information is found.

A System View doesn’t have to be detailed, it can be superficial but you build up as you go down. It can be a set of interconnected black boxes, that can be simply accepted as working consistently and as intended, unless data indicates otherwise.

Step 1: Gather Information (Gather)

Gather whatever information is available. Logs, Failure Modes, Reproduction Scenarios, whatever is available. It doesn’t really matter how much you have, and sometimes too much makes it harder to diagnose. This starts you going.

Worked Example:, this could be error codes present in the response, amount of time that it takes to fail, do other requests work or not. In this case, we don’t have any return code, it just times out.

Step 2: Create Multiple Hypothesis (Guess)

So based on the available information, you can consider about what may be the cause. As you consider each of the potential causes, don’t worry too much about pruning too quickly. By nature you will automatically prune out some scenarios based on the available information. (See Other Thoughts).

Worked Example: There could be a server failure, server overload, bug in the page rendering, network issue, etc.

Step 3: Work out how to Prove or Refute each Hypothesis (Plan)

For each hypothesis, consider what you can do to either prove or disprove that hypothesis. As you consider what can prove or refute, there will be items that are either easy to test, or split the hypothesis. Keep in mind on these. Don’t worry too much if there is information that missing, or you can’t get completely prove or disprove it.

Worked Example: What error codes are returned (decides between server failure/overload), are other pages working (may split between server side code and/or static code), server load values.

Step 4: Conduct your Experiments

Now that you’ve got your experiments, you are ultimately either removing a potential hypothesis, or your have proven that it’s at least true. If you have refuted a hypothesis, then you can remove it from your list of things to debug. You’ve proven it right, you now have more data that helps provide a better picture. Just like 20 questions, you need to choose what experiments to run that confirm/refute as many of your hypothesis as possible. This mixes in a healthy dose of effort, information gained, number of hypothesis.

Step 5: Repeat Step 2-4

As you work through the debugging, you will continue to get more data. as you get more data, you will possibly come up with other hypothesis. This is more complex than a simple decision tree, since it’s dynamic and not designed. By cycling through multiple times you will be getting more data, and isolating where the problem is.

Note that if it was easy, it would be linear, directed and you would be done. Life isn’t that easy, so there will be mistakes, incorrect assumptions. Always keep a beginner’s mind where you observe, re-challenge and ensure that you don’t take hard positions on things that you may have excluded, new information may make a big difference.

Other Thoughts

There are a number of other things that are happening in the background here. I’ll explore some things here.

Intuitive Engineers: We’ve all seen them, the engineers who sit down with a problem. Tap a few keys, look at a couple of logs, and magically zero in on the root cause. In my experience, you can usually ask questions like “What do you know it is not?“, or “How did you determine that it wasn’t or was that?“. Most engineers will have immediate answers to those questions. If you ask them directly about how they get to the right answer, most won’t have a strong view on how they got there.

Systems Thinking: A strong part of this is process is to construct a theory of operation about how the system works. A lot of the software industry just plain suck at maintaining good architectural documentation. Independent of that, constructing a System Model of how the system works is a critical step. Like the Beginner’s Mind – allow your view to be fluid and learn and respond to new information.

Remember 20 Questions: As a kid, I’d play 20 questions with the family. The critical part to quick isolation is to segment the solution space as quickly as possible. Through the power of bisection, 20 questions allow you to segment into 1,048,576 different sub-domains. Well constructed hypothesis and experimentation will apply the halving in an ideal world. Real world is messy, but the rule still applies.

Depth vs Breadth First: There is no prescription on driving down a line of experiments – or knock off a different set of experiments first. The best choice comes with practice and improvements in intuition and judgement. Generally, if you keep getting hypothesis confirmed, continuing depth first can help isolate. When there is confusion, breadth helps first.

Beginner’s Mind: Shoshin is a Japanese term for beginners mind, similar in concept to growth mindset. However within that concept you drop preconception, biases and be in the moment with the data that you have.

Allow Yourself to Be Wrong: This is critical, if you don’t take risks, it will be hard to be learn. Learning is best when you make small mistakes that help shape your best path. You learn best when you are wrong.

Practice, Practice, Practice: I personally feel that intuition comes from improved judgement and experiencing patterns. You can only get better if you practice.

Electric Vehicles represent a challenge for the future. This blog post imagines an EV future and what it may mean. I’m an outsider to the EV space, having had a used 2017 Nissan Leaf for just over 4 months.

I’m also going to look beyond the irrational and skewed EV world that we live in currently, with free Supercharging and heavily workplace subsidized EV charging. In this future, people won’t be sitting in their cars for 30 minutes on a Friday evening to fill their car for free. Neither will we be getting juiced at work for free, having never charged your car at home or at a EV charging station. You have to pay for electricity in this world, one way or another, grounding this post in a mostly rational economic world.

Charging Options

In this post I will explore the two fundamentally two types of EV charging available today. Although there are others in the future, and have been available in the past, the two most common forms available now in the US are 6.6kW Level 2 (L2), or DC Fast Charging (DCFC).

Charging at home – L1 or L2

In this first scenario, which is mostly a real world scenario now, homes are equipped with a L2 chargers. You install the number of chargers that you need for your car. L2 chargers provide around 6.6kW, or about 25 miles per hour.

In a simplistic scenario, you plug in when you get home, and unplug when you leave for work. This would give around 12 hours of charging per day, or about 300 miles of charging overnight PG&E, my local power supplier, has an EV rate which provides a heavily reduced rate from 11 PM to 7 AM, leaving about 8 hours or 200 miles of charge each night. That’s sufficient for a very large commute or a reasonably sized car.

For small commutes, L1 also works, but at 3.3kW, you will only be able to recharge less than 100 miles overnight which places you at range anxiety if you can’t charge above your previous day’s charge. Electricity rates that make charging very cheap between 11 pm and 7 am, put the consumer in a difficult “do I pay more” situation. Don’t underestimate the psychological barriers to paying double the off-peak rate, even if it is much cheaper than the retail rates at public stations.

Charging on the Road – DC Fast Charging

The across the country road trip in a Tesla is a clear possibility with the Supercharger stations. These DCFC chargers range from 20 kW all the way up to 350kW. The common rules of thumb range from 1 hr/100 miles, up to 1 hr/300 miles for 150 kW and 20 minutes at 350 kW.

Note that most of these DCFC systems will charge quickly up to 80% and then move to trickle charging for the last 20% (see https://use-cases.org/2019/05/05/real-ev-charging-rates-from-the-bay-area/). So the math becomes complicated unless you discount that last 80%.

Most of the DCFC in the bay area charge you per minute, rather than per kWh, so there is a strong disadvantage to go beyond the 80%, in which case you are starting to pay for parking rather than charging. DCFC that charge per kWh usually charge a premium, likely balancing the time of use with the charge they deliver.

Either way, the DCFC is the closest that we will get to the gas station experience unless we go down the battery charging or start pushing up to high hundreds of kW charging. Which when you think about how electricity behaves at high power levels, the safety interlocks, cooling and so on will make high kW charging a challenge to install, maintain and keep safe.

Charging Scenarios

At this stage, I can’t see too many other fundamental options, cheap and slow, or fast and expensive. Time may prove me wrong, but unlikely in the next 5 or so years.

Home Charging

Home charging with L2 charging is likely to be the most popular and predictable charging experience. Your car is charged when you get up in the morning, you do your driving during the day, come home and plug your car back in and it’s ready fully charged by the next day. If your daily commute and errands sit below the night time charge rate of around 100 miles, you should be fine to continue with this model for as long as you want.

Road Trip Charging

The Tesla Supercharger or the Electrify America network are good example of road trip charging networks. The DCFC chargers on these networks vary between 150 kW to 350 kW. Most of the reports from people using these networks indicate that in most cases the trip stops (typically after 4-5 hours of driving) are completely reasonable to charge while having a short break. Most online articles (mainly Tesla) indicate that the the car is typically charged before the driver has finished their break.

Destination Charging

Destination charging covers the scenarios where you go somewhere and you plug the car in at the destination. These destinations are typically work, business or other activities. Most of these are L2 chargers for a typical commute of 10-20 miles, a car is recharged from the trip in 1-2 hours.

Work Charging

Work charging is an interesting challenge, the car is going to be at the office for about 8 or 9 hours. If the car is charged at night is above 80% when destination charging starts, it may take 2-3 hours to complete the charge. At many workplaces there are 2 or 3 shifts of people shuffling cars and coordinating with each other through corporate chat systems. I usually go for a car spot that doesn’t have enough available charger, relying on colleagues to move the charger over when they finish the first shift, leaving me the option of moving the car early afternoon or if there are spots open – not at all.

Shopping or Leisure Charging

Destination charging at a shopping center or leisure location is a slightly different scenario. Although most destination chargers are L2, the amount of time spent at the location may be measured in a few hours. For the purposes of this post, I’ll assume that the average distance to shopping or leisure will be around the same 20-40 miles from home.

In most cases the amount of time that a car is parked and charging will be slightly or considerably less than the amount of time needed to get the car fully charged.

Future Charging Directions

Charging at scale will present some critical challenges for electrical infrastructure. A large number of DCFC charging status can easily overwhelm the available power within a particular area. Commercial installations for power charging will invest to support what is needed. Destination charging may struggle with the electrical installation requirements for charging at scale.

Battery Smoothing

For home L2 charging, solar and house batteries like the Tesla Powerwall will be extremely useful for ensuring spare charging capacity is available. Although the Powerwall 2 is only around 13.5 kW, much less than the capacity of the EV battery, it does help reduce the dependence on the network for charging.

Another benefit of energy storage is that it allows short term peak consumption of energy above what would be available from the utility.

Adaptive Charging

As described above, most of the destination charging has a difference between the amount of time that a car will be charging and the amount of charge that will be required.

Allowing cars to have a target time and adjusting the charge rate to ensure that the car is fully charged at the time that the driver is ready to depart. If a car needs 10 kWh of energy, it can charge at a very low rate for 8 hours. Being fully charged by the end of the day. Likewise at a shopping center, declaring a departure time allows slower but consistent charging to get a car fully charged by the time that a customer is ready to leave.

This has two benefits:

• Allowing more charge points to be installed
• Reducing the electrical infrastructure required to support the peak load from the number of charge points installed.

Powerflex offers such a system, having installed some of these systems in the Los Altos/Mountain View school districts. The powerflex system is captured well with the graphs below.

Obviously, adaptive charging will not be suitable for road-trip style charging where the starting.

The Future – Energy Storage and Adaptive Charging

Ultimately, the charging future I see is a blend of energy story and adaptive charging. As the EV revolution continues, with places in Europe expecting to be weaned off internal combustion engines by around 2025. I look forward to driving into a car park and be presented 50 to 100 L2 charging stations and then selecting when I think I’ll be leaving the car park, knowing full well that the car will be mostly charged by the time I get back.

What price should we expect to pay for this sort of charging? It will likely be per kWh plus a small space usage rate. I’d be happy paying a 50% to 100% premium on the best charge rate that I’d be able to get at home.

For just over a year now, I have been involved with Plato as a mentor.  If you don’t know, Plato is a subscription platform that links engineering managers to mentors.  As a Mentee, you sign up for unlimited mentoring sessions and access to a Slack channel.   As a Mentor, you make yourself available for one or two 30 minutes slots per week.  Plato’s system allows Mentees to connect with Mentors who are likely able to provide insight or guidance on their particular leadership problems.

I’ve been asked by a few people regarding the lack of symmetry between the Mentee, Mentor, and Plato.  I’ve thought about this quite a lot to ensure I have a mental model for the collective benefit 3 parties get.   Clearly, for the Mentee and Plato relationship is clear.  Plato provides access to Mentors, and the Mentee (or Mentee’s company) pays for the privilege, fairly simple charge for facilitation.

What about the Mentor?  On the surface, the Mentor receives no direct monetary reward, they have the opportunity to expand their network through Plato’s networking events.  Plato obviously receives access to the top tier of engineering leadership all quite happy to provide time to the paid Mentees.  On the surface, the Mentor receives very little as part of this relationship.

Why have a Mentor?

Personally, I moved through most of my career without what I would call a strong mentor or coach.  My lessons were learned the hard way, and I apologize in retrospect to those employees who helped me shape my management views and understanding of how business works.  As I’ve progressed in my career, I’ve become a more experienced manager, I’ve come to strongly support coaching and mentoring.  It’s a way to help avoid some of the leadership potholes that as humans, we are prone to stumble into again and again.  Guidance or advice at just the right the time can help immensely.

I strongly recommend that people find Mentors to help navigate their career.   Even if you are a naturally introverted person, you should still look to find a Mentor, it’s hard, it’s awkward, but it is still valuable.

One comment I’ve heard from a couple of Mentee is that they are ultimately concerned that they will be imposing on the time or advise of the Mentor.  Generally Mentors see Mentoring as a professional gift to the industry.  Assuming a formal or semi-formal Mentoring relationship, unless you have an agreed cadence of communication, contacting them once or twice a month is likely what is expected by being a Mentor.  Most Mentors are senior professionals, they will know how to ask you to back off.

Why Mentor?

As a Mentor, I have the opportunity to pass my experiences forward.  I can see new engineering managers making the same mistakes that I made, bringing forward the same assumptions.  Finding small ways that I can help the engineering management community not re-learn the lessons the hard way.

Why do I Mentor through Plato?  

Plato provides a very interesting value proposition for a Mentor.  I personally have 1 mentoring session each week, with a monthly panel mentoring session.  So I’m working with about 7 Mentees each month.  Each of these Mentees is at the point where they are having a particular problem that they want or need to solve, and so they are real problems needing real advice and guidance.

This speed-dating approach to mentoring actually has some interesting benefits for the Mentor.  The biggest benefit that I get is the need to think through rapid-fire management scenarios during the 30 minutes, typically 2-3 topics are discussed in each session.  This type of mental engagement and scenario practice would typically take months of real-world management to experience.

Muscle Memory and Tuning Judgement

Like any skill, practice makes perfect.  When a martial artist practices moves and scenarios over and over, they are training their subconscious to recognize the scenario and automatically respond in a practiced way.  Similar near automatic or intuitive judgment comes to others like chess masters who can see a layout of chess pieces on a board and have a good idea about the state of the board and likely good moves that can follow.  Grandmasters can usually detect how close to an end-game the board is.

What appears as expert intuition, can usually be attributed to gut feel, but rather practiced judgment.  This practiced judgment comes from automatically recognizing a scenario and automatically knowing sensible next steps. The more experience you get, the better your recognition of a scenario, and the better you are able to respond.

Managers go through a similar process.  A manager either needs to have a large amount of experience, or have a way of mentally exercising these scenarios again and again.  In the real world, a manager deals with their staff with issues coming up a few times a week and needing months to resolve.

When mentoring, a Mentor is able to quickly go through the larger number of these scenarios at a rate faster than they would experience in their normal professional life.  This helps train the practiced judgement, making the manager more effective and faster on their feet.

But be Aware of Bias

A highly self-confident Mentor, can train poor judgment in the same way.  If a Mentor isn’t careful, their responses to the Mentee’s problems can drift from what works well for the Mentee to what is simply trained into the Mentor’s mental model.  This is where both the Mentee and Mentor have some responsibility.

The Mentee’s first responsibility is to take the advice, as a well-intentioned response to the scenario presented.  The Mentee must integrate the advice into their real world scenario and take actions as appropriate.  Each Mentor’s solution may be only a partial solution to the Mentor’s problem.

The Mentor’s responsibility is to both appraise the scenario and determine what they might do in that situation.  The appriasal stage is the most critical.  A Mentor that doesn’t adapt to the Mentee’s actual sitation may end up pushing the Mentee down the knife edge that represents the Mentor’s history.   Hoping that you can take the same life steps of your favorite artist and repeat their results, is folly.  The linear path that their life has taken also down a knife edge.  Any variance, be it a chance meeting or a different decision and the path is different.