jamesmunns

Doing embedded stuff

Be nice, work hard. Build tools, tell stories. Start five year fights. Kein Ort für Nazis.


A list of all the other services I'm on around the internet


Posts with code highlighted with codehost by @wavebeem.


All posts licensed under CC-BY-SA 4.0, unless otherwise stated.


Writing intros are hard, so lets cut to the executive summary.

I work as a consultant. I usually work on projects that involve hardware (electronics and mechanical parts), but I usually help on the software side, from microcontroller code, up to desktop stuff, to server stuff sometimes.

Sometimes I get brought in to implement specific things, like drivers or proof of concepts. Sometimes I get brought in when people are looking to move to Rust. Sometimes I get brought in when people are looking to turn a "prototype" into a "product".

Usually, what I end up spending the most time helping people with are the fundamentals of systems engineering.

Maybe this is a case of "when the tool you have is a hammer, all problems look like nails". Maybe I've overdrank the Systems Engineering kool-aid.

That being said, it keeps working and being helpful, so I'm going to keep swinging that hammer.

I might want to write a book on this some day, but here's the start of some unsorted thoughts in the mean time.

I have nothing to sell you (yet), and I promise there are no magic bullets in these writings. Just some things to think about that might help you or your projects.

I am also not a process purist, or religiously tied to the concept of systems engineering. Sometimes doing a little of this work goes a long way. Sometimes (especially if you are doing something safety critical) you need to do everything to the full extent. Do what feels right. If it stops helping, find something else to do. Ideally, all of the concepts I'll describe should start paying dividends almost immediately.

What is Systems Engineering?

I dunno, NASA or someone probably has a better definition, but to me, it covers the domains of:

  1. Be explicit about what you are building
  2. Be explicit about how you are building it

That seems simple, right? Well, sometimes it is.


Where do things fall down?

Once your project has multiple people working on it, sometimes with widely different experiences (e.g. the 'software folks' and the 'hardware folks' and the 'manager folks' and the 'product folks' and so on), the concepts of "what are you building" and "how are you building it" seem to get away from teams and organizations.

Or, your project gets a little bigger than the last one, or you need to tackle some new domain you aren't super familiar with. Things that felt simple start getting complicated very quickly, and despite working very hard, the finish line doesn't get any closer.

Like many other "best practices" - you can usually power through on smaller projects or with smaller teams without running into any problems.

But the problems in this domain are like a sneaky O(n^2) term. It'll be fine at first, until one day, things get exponential, and your schedule slips a month, or three, or twelve.

Why aren't more people doing "systems engineering"?

Your mileage may vary, but I've found that software folks have very little formal training in tackling these topics. They might think about breaking a problem down in to parts as an individual engineer solving problems, but don't have a lot of tools for breaking down "team sized problems".

"Classical engineering" folks in electrical and mechanical domains tend to have seen some of the processes and formality around systems engineering before, but haven't necessarily internalized why certain approaches exist.

If your organization doesn't already have a "culture" of systems engineering, there's usually no one to introduce it! This is usually where I get brought in

That being said: Things don't have to get to an emergency state to benefit from a little systems engineering. Sometimes it's nice to just have a more predictable development process, with fewer surprises.

James, your definition of system engineering is still super vague

Yeah. I was trying to be succinct. Let's expand a little bit.

Be explicit about what you are building

Formally, this typically covers the topics of requirements, specifications, architecture, and/or acceptance criteria. There is a lot more to explain about what I mean on those topics, but that's for a later day.

Informally, this means "everyone who is working on your project should unambiguously know what it is supposed to do".

NOTE: I'll use "specification" and "requirements" somewhat interchangeably here. In this section, I mean it to generally describe "what your system is supposed to do".

I'm going to say this a lot, but if information isn't written down, it doesn't exist.

  • You never know if two people have the same mental model. This gets even worse with more than two people.
  • You never know if YOUR mental model is consistent. Sometimes things make sense in your head, but when you write it down, it's self-contradicting.
  • You can't guarantee your understand doesn't change over time, or you forget parts of the mental model.

The solution to this is that you must write down what your project (code, hardware, whatever) is supposed to do. It doesn't have to be super formal or anything.

Use a markdown document. Use a word document. Use an excel spreadsheet. Use whatever, but write it down. It's always easier to move from one text medium to another, than it is to capture all the information in the first place.

But whatever you use, this brings me to the next point: your whole team must know where the source of truth is, how to view it, and have access to it at all times.

If you keep the definition in a git repo, and the project managers don't have access to it (or don't know how to view it), then your specification is useless to them. If your company uses a fancy requirements tool, but doesn't have licenses for everyone to view it, then your specification is useless to everyone.

This specification is only useful when someone can say, "wait, how does this work?", and look up the information in less time than it takes to guess wrong.

This specification is only useful when two people disagree on how things are supposed to work, and they can look it up, instead of having an argument, or needing to schedule a meeting "to get on the same page".

For this bet to pay off, we hit the next rule: your specification must be accurate.

This gets a little weird in the time between "the requirements say what the system will do", and "the requirements say what the system does do". That's a longer topic for another day.

But, at all times, your requirements/specification must be accurate.

If the requirements say "the system does X", the electrical engineers must be free to use this assumption, without talking to the software engineers. The sales people must be able to put this in the marketing docs, without having to guess.

If two (or more) people ever disagree on what the system should do, the requirements should be the tie breaker.

I don't have a good segue for this, but your specification should be easy to read, search, and get information from.

Some folks track "what their system does" in things like JIRA or GitHub tickets. But often these are written over a long period of time. How do you know which ones are relevant? How do you know which ones changed later? If you have a question like "how many whatevers can my system handle at one time", how will you find an answer to this?

For this reason, I strongly recommend keeping a single document up to date.

If the requirements are unclear, or just plain wrong, everyone should know how to update the requirements.

Requirements should be a living document. Even if you subscribe to the "waterfall process" sort of thing, your understanding of the system will change over time. People might require new features. You might realize certain performance goals aren't possible or necessary.

We'll get into how you should change requirements later, but whatever you choose, everyone should know how to "fix" or "improve" the requirements when necessary.

Finally, you should be able to prove your system matches its requirement/specification.

This might mean testing, this might mean review, but basically: you should be able to "gut check" every part of your requirements/specification, so you know if it is accurate or not. More on that later.

TL;DR: To be explicit about what you are building:

  1. You should write down what your system should do
  2. It should be in a single, living document, that is kept updated
    • It should be as complete as possible (you can do it incrementally!), BUT
    • It should ALWAYS be accurate
  3. Everyone should know where to find it, how to read it, and it should be easy to find answers from
  4. Everyone should know how to change it, if/when things are incomplete or inaccurate
  5. You should be able to prove your system does what you say it should

Be explicit about how you are building it

Formally, this covers topics like System/Software Development Plans, System/Software Verification Plans, Design/Coding Standards, Review Standards, and Document Control.

Informally, this means "everyone who is working on your project should unambiguously know how they are expected to work".

There is... a lot going on here with respect to process. How you design or verify a system, how you capture requirements, down to little details like how code is formatted. This is the sum total of things that someone like a new hire will need to learn to be productive as a member of your team.

I'm going to repeat this, if information isn't written down, it doesn't exist.

In the ideal world, you should be able to:

  • Have someone come in for their first day
  • Know what they need access to, before they start
  • Point them at one document (which might link to other ones)
  • They should be able to read this document
  • They should know generally how to start working with your team

Again, you can capture this information however you want. Markdown documents work great. Internal wiki pages are just fine. These documents end up acting a bit like "process requirements":

  • They should be a living document over time
  • Everyone should know where to find them
  • If there's a disagreement, the "process requirements" should have the answer, or if it doesn't, you should come to a decision and write it down
  • Incomplete is okay (but improve it as you go!), BUT
  • Inaccurate is never okay

You don't need to legislate your process to death. Most teams don't need safety-critical-formal process definitions. But if you do something every day (like make PRs, or review code, or design circuit boards), maybe capture some of that "process how-to", so it's easier to onboard the next person.

If you have two people disagree on "how to do X", sit down as a team, make a decision, and write it down. Then don't argue about it, unless there's a good reason to change it again in the future.

At the end of the day, having a written process helps you be consistent. Consistency is great, because it means people spend less time being confused how to do something, and spend more time solving the actual problem.

TL;DR: Write down how you do stuff.

At some point in the future, I'll write down certain ways of doing things that you might want to use as a "reasonable default", but honestly, do whatever works for your team. And write it down, so everyone is on the same page.

Summary

Systems Engineering helps you with the "meta" of solving problems. If you have a big team, or a big problem, it's worth sitting down and "solving the problem of how to define and solve a problem".

If you can, spend some time BEFORE you start working, to make sure everyone is ready to solve the same project, and everyone knows what they are supposed to do. If you've already started, it's not too late, you still might catch problems earlier than you would have otherwise.

You don't need formal training (but if you want help with this, send me an email!). Just start writing things down, and make sure everyone is on the same page.

The Only Difference Between Screwing Around and Science Is Writing It Down

If you have questions, feel free to leave a comment here, or shoot me an email. Hopefully I'll get motivation to write more soon.


This post is made available under the CC-BY-SA 4.0 license.


You must log in to comment.

in reply to @jamesmunns's post:

This is super interesting! At the start, I felt like this was super intuitive and that of course, this makes sense, this is how good teamwork works. I read more, and reread, and realized that while I intuitively agree with the general principles, like everyone agreeing what we're building, you've written a ton of specifics that are entirely new to me.

For example "if [it] isn't written down, it doesn't exist" feels invaluable - like a crystallization of a general idea I like, but much more based in experience, way simpler than a big abstract push, and actually actionable.

I think I'll have to revisit this a few more times and see if I can internalize this more. I'm an individual contributor in my team, so I would not be instituting these as policies, but I hope I can bring suggestions that improve cohesion in how we operate as a small team.

my 2 cents:

  • dogfooding documentation is very beneficial for software and documentation, and onboarding is the best chance you get to find any holes. If the new person can't figure something out, the knowledge gap MUST NOT be filled verbally, instead either write documentation as needed and point them at it, or if applicable, let them write it themselves!
  • change propagation seems unsolved even by big corps, I routinely see things like MS end user help pages 404ing, Apple docs falling apart, etc. In my mind a thorough documentation tool would include some kind of dependency graph that automatically invalidates anything that's stale (and rejects in CI?), but modeling this properly is an enormous task and probably not solvable in the abstract - maybe nix(os) is moving us in the right direction here?

In my mind a thorough documentation tool would include some kind of dependency graph that automatically invalidates anything that's stale (and rejects in CI?), but modeling this properly is an enormous task and probably not solvable in the abstract

These are called "traceability tools", they exist, but in my opinion:

  • They are targeted at safety critical customers with a very specific (regulation) use case
  • The okay ones are very expensive. The sucky ones are also very expensive
  • The open source ones are okay, but don't have a good UX, so are unlikely to be used more widely

It is very feasible to detect when docs, code, and tests change. The "state of the art" is typically to say that a change at one level requires you to re-check that the relevant items above/below are still valid, or whether they require changes (and then whether those changes propagate to more surface area).

The way I’ve always explained systems engineering is that it is Applied Engineering. You should have background in one of the typical engineering disciplines (mechanical, software, hardware, etc.) so you know how to design products. Then we can start learning about requirements decomposition, flow down, verification, validation, etc.

Great description!