Innovation accounting and portfolio management

The other week at the Lean Startup Summit I introduced our ROI calculator in a couple of innovation accounting roundtable sessions. The sessions were overbooked in general since innovation accounting and the connection to portfolio management was a topic many were interested in. Most of the people joining the tables were looking for ways to deal with getting allocated budgets and manage top level expectations. This seemd to be a recurring problem.

For this reason I started of by introducing the ROI calculator right away. Starting of the story by having them play with the calculator and having them fill in the actual investments in the amount of innovation initiatives they were doing right now. And the effect of using the same budget on more innovation initiatives. Seeing the numbers actually will help you realize that what others have been saying is actually true.

The Calculator

Let me start by explaining the Calculator. As an investor you know that investing in startups is like betting. There is no formula for succes for any startups and even the most promising teams can turn out not to work in the end. VC’s have known that for quite some time. They have learned the hard way, through investing and losing money while only sometimes getting returns. We have taken these numers of VC investments and their returns over the time of some years and looked at the way those returns are distributed over how high and how often they occured.

The calculator itself is using these actual numbers to run scenarios calculating chances by using the Monte Carlo model. So what are the chances of return if I invest this amount of money based on this real data.

If you have a budget of 5 million Euros that you are dividing to give to the 5 most promising internal startups, this will give you a almost 60% chance of losing almost all of that money. Now this is only one scenario but running it again will give you the same sort of outcome almost all of the time.

Changing the amount of startups to say 100 and leave the budget the same, will give you a very different outcome.

Investing in more startups will give you a better chance at return than in only a few. Even with the same budget, investing less in each startup. This is good to know, for a healthy innovation portfolio, you need a larger number of startups to begin with. There is nothing you can do to make the portfolio more succesfull. There are however ways to reduce the risk of loss in that portfolio.

Stage Gates

Now that it is clear that you need more balls in the machine, lets look the seperate balls whilst they are going through the process of searching for a business model that works. Plinkromatic is a useful tool to illustrate this.

Each startup is represented a little ball dropping down this board of pins. We have already seen that the chances of returning zero are the biggest so therefor there are more of those on the board. Dropping down a ball will sometimes give you return sometimes not. But if we have a way of stopping the balls that seems to drop towards the zero return, we have a way of spending less money on those and spending more money on the ones that looks like they are going to fall towards return. Stage gating your investments may help you reducing the risk of spending to much on something that probably will not return anything. Probably because it is all still chance.

Now how can we insert those stage gates and control the health of your overall innvation portolio. This is where innovation accounting comes into play.

That path of the startup is its lifecycle and only a few will make it till the end. It makes sense to look at the stages in this lifecycle and determine based on what is known at that point in time wheather or not it makes sense to keep exploring on that path.

Asking the right questions

The questions you ask at each stage to determine if a startup can continue, are different for each type of startup as well as different for the stage that they are at. The main goal of these questions (or KPI’s) is to learn and make decisions. At the first stage you want to learn if an idea has the first signs of the promise of something to help people with in a potential market. In additionalignment with the vision and strategy of the company at a global level are important. These innovation KPI’s are never as fixed as they are with financial accounting and project management. They could differ for different projects in different market contexts. It is important to keep in mind why you want to know certain things and what answers will give you a more informed way to make decisions for the continuation of a startup.

Levels of accounting

In terms of innovation accounting we recognize three levels of accounting within a large company.

Lets start with the team level. For individual teams and the coaches assigned to these teams it is important to keep track of the learnings. It is how the team can make decisions to continue something or maybe pivot into a different direction. Coaches can help them and need the input to help them steer and make these decisions.

Since we have established that we need multiple teams, and a way to kill startups that managed to validate that there is nothing to solve, it is important to have reporting at management level. Guarding the stage gates and determining if there is enough data to either go ahead, stop or try again. This is innovation accounting at Management level.

Also important is a clear overview of the overall company portfolio and the part of innovation in that. How is that portfolio performing and how many startups are in what stage. With innovation accounting at strategy level will help make decisions towards a healthy and future facing portfolio. With a business strategy that is connected to the innovation strategy.

The result of our summer break is online!

The summer is getting near the end, most of you have been on holidays or wel deserved breaks. We have been on a different kind of break. Whilst working with our NEXT Canvas with the startups we help, we were seeing a way to help both startups and companies to actively keep track of what is happening within a startup or innovation team. Right now we use a process of using the canvas to ‘see’ actual progress and interaction with, as well as keeping track of, experiments and validation. It is on the wall with the paper version of both the card and the canvas. While we store experiments and progress over time in a Trello board. We figured it could be made so much clearer if we could get the mental model out of our heads and into a digital tool to guide you. We decided to act upon that and dedicated two weeks in August not going to Bali or Ibiza. Working solely on this online tool during our ‘internal summer hackathon’ ( with just the two of us 😉 ).

As a result a first minimal version is online, based on what we have learned from the paper version of our canvas and experiment board so far. The tool needs to be tested by a few startups right now, so we can learn as much as we can from the start. Things like; How do we convey our mental model to the users and how can you benefit from this. Right now it has a few basic functionalities. You can add a project that contains one or more canvasses. Each segment of the canvas can hold assumptions. To each ‘sticky’ with an assumption you can add one or more experiments. There is an experiment board for an overview of running experiments as well as an experiments inbox.

You are welcome to give it a try, provided you ask us when something is unclear and tell us when something is wrong. We want to learn, learn, learn, so we can continue to integrate and optimize. To be able to build something that will help you build a viable business model.

Progress board: tracking learnings over time

This is Part 2 of our Innovation Accounting series.

For the sake of learning and progress it is important to be able to keep track of what happens in terms of the changes and adaptions to your Business Model in every stage of your startup. Every update due to an experiment, the things you have validated or dismissed over time. The closer you get to product market fit, the more likely it is that you can show metrics like traction or retention, but are those the metrics you can account your decisions on, put milestones on or determine your roadmap on? From the very first start you want to be able to account for the decisions made towards building a viable business model, you want to see the progress you are making towards a this. Most likely (in the very first fase of your startup) you will start off with multiple possible options and models. The goal is, over time, to validate and de-risk as much as possible, ending up with one that actually works. Going through the phases, experiment by experiment, means you are making decisions and setting milestones along the way. To account for those, it makes sense to document the learnings in such a way that it gives insight in this learning progress.

At NEXT we are using a Trello board for now. We have changed our approach over time and probably will optimize it when we have learned more. Playing with the structure is easy with a flexible tool like Trello and we don’t have to build anything yet ourselves. Although it misses certain other things, this still gives us a flexible approach towards something workable.

Setting up the Trello board

Create lists that correspond with experiment loops. These are preferably two-week time slots, but sometimes something happens and the timeslot changes to a longer period, so it makes sense to number the lists. Something like ‘Experiment loop #’, but weeks or dates work as well, whatever gives you insight.

Every Trello card within a list corresponds with a Business model canvas that is still in option. Experiment loops at the very beginning of the board can have multiple canvas cards, since there are often a lot of options in play. Some are discarded on gut feeling or research, some on experiments. We use a picture of the business model canvas (lean canvas or our own canvas) as a cover for the card, adding color to the canvas segment that is specifically being tested. This makes the overview more clear. There can be more than one experiment in play on a specific card. Note also that in a two-sided businessmodel, you will need a canvas for each side, each is put on its own card.

Canvas cards

Whenever an experiment is run, we add the experiment to the corresponding canvas card. In a comment we upload a picture of our experiment card, with its unique number. (Make sure that the canvas remains set as cover on the card.) The due date of the experiment is set in the calendar. Once an experiment is finished, the end result (validated or invalidated ) as well as the metrics or documents that proof this, get uploaded or linked to this card as well. The updated Business model (Canvas) is than made into a new card in a new experiment loop.

Stickers and labels

Every card that has an experiment attached to it gets a triangle/exclamation mark sticker. If an experiment fails it gets a thumbs down sticker, if it validates a thumbs up or checkbox sticker (whichever you prefer). Running experiments get a rocket sticker and business models that are still in play but without experiments currently being done get a clock sticker. For experiments that have proven inconclusive or those that need follow up experiments, either more experiments are added to the card, or the card is copied to a new loop and experiments are added there. Sometimes we assign a confused smiley sticker to show this.

Labels are very useful especially for two-sided marketplaces or more complex models that need several canvasses for one option. Colored labels can be assigned to cards indicating the two sides of the market place, or the slices that belong to different users within a certain segment when you are dealing with complex markets/customers that need separation into user, influencer etc.

Progress reports

If you keep updating your board over time this will create a very visual overview of your learnings. Building up from one side, that has many options and assumptions, working towards something that has fewer assumptions and is less risky. It will give you some insight into your learnings and a way to proof your progress. Whenever there are periods of time when you have not been able to take steps in proofing your businessmodel, this will also be visible on the overview, just as pivots or new options.

You can work from left to right or the other way around, the result is the same. A progress over time, validating towards a working businessmodel. With everything that is related to this, the exeriments and their corresponding results attached to it. Thus accounting for decisions and steps made along the way.

We’d love to hear how you keep track of your learnings and if and how you have made adaptions to make this work for your startup? Let us know in the comments!

Learn more about NEXT Amsterdam →

Introducing our innovation accounting framework

This is Part 1 of our Innovation Accounting series.

We have talked about the importance of Continuous Experimenting and working on a sustainable business model, experiment by experiment. Even more important is a way to keep track of these experiments and the progress made. Not only to see what you have learned over time, but also to account for the decisions that are made in the process and why certain things need priority over others. In his book Eric Ries calls this innovation accounting:

To improve entrepreneurial outcomes, and to hold entrepreneurs accountable, we need to focus on the boring stuff: how to measure progress, how to setup milestones, how to prioritize work. This requires a new kind of accounting, specific to startups.

Keeping track of your learnings

Usually, as a startup, you can start tracking as soon as there is a version of your product out there and some metrics are in place. This is why it is important to get an MVP out there. But your much craved hockeystick curve is usually mainly flat in the very beginning. Not because founders are not doing anything, but because looking for a working business model takes time and is hard. You usually get the most valuable insights from the most failed experiments, but this doesn’t show up anywhere in our metrics. How do you keep track of the experiments you have done, and the learnings from these experiments even before having a MVP out there?

How we do innovation accounting at NEXT

Identifying priorities and risks in your businessmodel is hard, setting up and formulating an experiment is even harder. We have found out working with a lot of startups that once you have a framework in place, it is more manageable.

For every experiment sprint or loop, we look at our canvas and decide on the things that are still risky and decide on experiments based on that. At NEXT we like to work very visually so that every team member knows what is going on and why. The starting point is a canvas. We use the Lean Canvas instead of the Business Model Canvas in early stage startups because it helps to focus on the problem and solution rather than strategy and product. We also have developed our own NEXT Canvas, which we will share with you soon.

A physical version of the canvas is always on the wall and gets updated with every single insight from experiments. The canvas helps identify risks and priorities and decide on the next experiment (or set of experiments) to validate assumptions. Once we have decided on an experiment and what it is that needs validating, we formulate an hypotheses or learning goal.

To help us formulate an experiment and to define its succes criteria upfront, we have designed this experiment card:

The card helps us to be scientific about every single experiment. It helps formulate an hypothesis and learning goal, forcing us to think about what it is that we are testing. It also helps being consistent across experiments.

How to use our experiment card

The upper row:

Used to fill in the date of designing the experiment.
A number for this specific experiment. This way you can attach this experiment to a canvas or other events in metrics that this experiment relates to. See it as a unique ID for this experiment
The actual cost of the experiment, this could be time, development time or actual hard cash.

Hypotheses: A falsifiable statement. (What is it that you want tested)

Context: With which segment on the canvas does this experiment correspond. What risky assumption are you trying to validate and why? We put dots in the areas of the canvas to visually mark the segments the experiment is related to.

Learning goal; what is it that you want to learn from this experiment. (If the hypotheses is not really clear, for instance in an exploitative customer interview, this one should be even clearer.

Experiment: It is important to time-box every experiment. Set both a start and end date.

Describe the experiment; Is it an interview? A fake button experiment? Describe how you are testing the hypothesis. In relation to this, which metrics are being monitored to determine succes?

Result: When is the experiment a succes? Write this down before running it. When the experiment is done make sure that the actual numbers are updated.

Next step: Was the hypotheses validated? Did you learn anything? And what does this mean in terms of next steps. Pivot, persevere, or do another experiment because this one wasn’t really testing what you wanted to test?

To do, done

All running experiments are put up on the wall next to the canvas. Once an experiment is finished it is taken from the wall and archived. But not before we have decided on a next step, and updated the canvas or the product from the results of the experiment. The experiment cards are then archived together with the corresponding canvas ( the one just before we updated it with the results of the experiment). In this way we can see which experiments accounted for which decisions and if there are any metrics of importance that relates to them.

Avoid going around in circles; track your progress.

In the next blogpost we will dive into how we use Trello as a tool to keep track of current and past experiments and the changes of the canvases in a ‘timeline’ of sorts.

Learn more about NEXT Amsterdam →