How to structure machine learning work effectively

Let’s flesh this out with an example. Say we're building the detector service that recognizes potholes on pictures of the road. Now let's write an example user story for the machine learning team according to the "classic" template:

"As an HD Map user, I want to have the 'deep pothole' detection model optimized so that a pothole is located on the map more accurately."

This sounds a bit outlandish, doesn’t it? A map user would never ask for a machine learning model to be optimized because the user has no idea of its existence in the first place.

Given this, let's just call them stories to reduce the ambiguity.

The stories in this setup are usually quite technical. Still, it’s important to describe them in a neat, concise and easily understandable way so that they can be taken up by any developer on the team. They must also be understandable by the product owner, new developers or anyone else with limited knowledge of the particular context or technical details.

One of the possible alternatives to the common "as a user" pattern discussed above, is the so-called WWW template. The same example would sound like:

Apart from that "what, why, who" formula, a good story should have detailed acceptance criteria so that it is clear what exactly should be the result of the work, and how to assess that result. This is especially important for more research-oriented work because sometimes the research tends to be never-ending.

Acceptance criteria of the story should draw the boundaries of the research to get a definite result from the work and help to eventually evolve from a research idea to a product feature.

In the "pothole" example above, the acceptance criteria would describe what the expected impact of the model optimization is, how to measure that impact and possibly what the necessary experiments that should be done are.

Of course, due to the nature of research-oriented work, it is often not possible to define all the acceptance criteria up-front, so making room for exploration and experimentation is a good idea. However, that doesn't mean that a machine learning team can drop any attempts at defining what the scope and expected result of the work should be. Whatever is certain during planning about the expected result of work should be reflected in the acceptance criteria.

There are also other measures for effective research and development stories. While this will be the focus of a future article, here are two to already consider:

Stories can be further classified into "new knowledge" and "new feature" stories.

Let me explain.

The high uncertainty of whether a machine learning idea will work often makes it risky to combine the feasibility investigation (and other kinds of research activities) with the implementation of that idea within the same iteration of the plan. Thus, the epics contain multiple backlog items of a different kind.

If the team has an idea on how to improve its machine learning product often, it makes sense to make a "new knowledge" story, which would have knowledge as a result. One important benefit of this approach is that many new ideas tend to be ineffective, so it is good to carry out some investigation and experimentation, then report on what has been discovered, present the results to the team and then iterate further.

As I mentioned before, the epic could end up with a negative result (i.e. the idea didn't work out). By dividing the "new knowledge" and "new feature" stories, we make sure this fact is discovered as early as possible. In case the idea is actually effective and feasible for the product, this distinction would still help to achieve the desired result sooner.

The "new knowledge" idea is somewhat contradictory to the notion of a user story, as it implies that every iteration of work should have a direct impact on the customer. However, I believe this is the "lesser evil" given the benefits that approach gives.

In the case of "knowledge" stories, it is especially important for the team to properly manage that knowledge in a clear, consistent and persistent way. In the next article, I'll reveal how we manage knowledge in TomTom Autonomous Driving machine learning teams.

An example of a "new knowledge" story would be:

Acceptance criteria:

Yes, it's the developers who benefit most of all from that work.

Then the "new feature" story would look like:

So far, I haven’t mentioned the definition of done (DoD). Unlike the acceptance criteria, the DoD is not specific to one story, but rather a generic checklist of all items to be done before a story of a certain kind can be considered complete. That checklist is created and maintained by the team and is usually specific for different projects. For "knowledge" and "feature" stories, the DoDs are also supposed to differ for obvious reasons.

In the first article of the series, I explained the "why", the conceptual part of the "Agile machine learning" mindset. In this article, we focused on the "static" side of the story, namely, how to structure, divide and formulate the work for machine learning research and development teams.

Next time, I will unveil some "dynamic" aspects. I will tell you how to apply specific Agile methodologies like Scrum and XP in the machine learning setup. I will also share some home-grown best practices to gather, grow and manage the knowledge in order to apply the research ideas in real software projects.

Stay tuned!