ROI Dashboard

I attended five social media, intranet or other online media conferences this year. I met some hundred experienced media managers, online directors, software providers or consultants.
And I asked every single one of them (if I talked to them more than three or five sentences), how they measure the ROI of their 2.0 intiatives and what business cases they used to get their funding.

Not one did express it in cash.

Some mentioned ROC – Return of Change, some took the low costs as an argument for not caring about revenues, others took detours with more or less complex thought experiments, others use a lot of opportunity costs in their reasoning (what would you loose if you are not among the first movers).

That’s not really surprising. But actually – it is surprising, Businesses do run on money, they need to make more than they spend, so why could they be so relaxed in measuring success? Im am afraid, that they are not. Maybe some initiatives have been started without business case or roi calculation, but they will probably be the first to be stopped as soon as there are any restrictions or cost cuttings coming up.

Easy starting, low entry costs, lightweight scalability are good attributes of 2.0 media. But every minute that our employees spend in the office counts.
The ROI for entire portal projects may be based on usability improvements that save 2 minutes per employee per day (=app. 40 minutes per month, =400.000 minutes per month for 10.000 employees, =6.666 hours per month =533.333 Euros per month). And on the other hand we should invite people to play?
If each employee spends only three hours on a tool we have to throw away after three months, that are 30.000 hours per tool (or per three months), 120.000 hours per year which are 9.600.000 Euros per year (more than 9 millions). So even if only 5 percent of employees use it – that is a lot of money (480.000 Euros).

Is that a cannibalization of business cases? Are we saving and wasting the time of our employees in the same project, at the same time?
This is an issue of responsibility in planning, of scalability and sustainability in our tools. Lightweight trial and error processes are great in small environments; they don’t work in enterprise environments:

• It is a not so lightweight process to find testers
• enterprise usecases need big documents, big processes and big communities for testing – or they will just not be realistic
• introducing a tool (even as friendly user only, limited support not warranty testrun) can be a big pain; removing a tool can be pain as well (users will keep asking for the old one instead of using the new one)
• How do you want to decide on if it is a success or not, how can you analyze usage data – once your audience has been more than 20 people? Sounds like another lot of work.

In the enterprise, you may feel more comfortable with scalable integrated solutions that can be extended using plugins, adding templates or investing some development work – but then can be extended somehow. You may want to keep your data in the same database with the same database model, you may want to continue using the same editor instead of doing new training for your employees, introducing new support structures etc.

That sounds like an old integration imperative taking over on 2.0
Actually I look at it as a way to ensure the success of 2.0 against or in the enterprise… You have to talk the language of the enterprise if you want to be successful there. ROI or Satisfaction Dashboards would be cool features of 2.0 software. Stay tuned to learn more.

Part 1: ROI Dashboard – Introducing the User Experience Indicator

Part 2: ROI Dashboard – User Experience Indicator 2 - A Sample Project

Part 3: ROI Dashboard – User Experience Indicator 3 – Bring in the money

Part 4: ROI Dashboard – User Experence Indicator 4 – Aggregate and simplify

We need to measure what we do – that’s what we agreed on in part 1. What do we need to measure? Let’s assume we’re introducing a small state of the art 2.zeroey employee-portal in a not not small company.

The company does already run an intranet, it’s almost ten years old. People are used to it, they always complain but there’s nothing special about it.

What can be improved?

• With the new solution, it’s easier to create content, so editors can work faster.
• They have a better flow in their work, so they make less mistakes.
• The new portal offers a better navigation, so people should find content easier.
• Tags are introduced, so that should also increase findability.
• Search is improved.
• Some basic content features have been introduced, so now it is possible to create slideshows, embed videos, audios and other media – that improves the user experience and saves the editors worktime.
• Basic statistics are part of the tool (tracking the backend and the frontend).
• Wiki functionalities allow fast editing for a bigger and not so skilled audience – that saves training, editor time and user time and reduces errors.
• Blog functionalities introduce new possibilities and enhance communication.
• RSS is used for feeds – in the portal, in blogs, wikis, the other way round; they optimize the use and reuse of content.
• Comments are introduced and are a simple feedback tool for users.
• Tags, clouds, categories in the fronted are just some next generation tools in the frontend, they enhance and train the employee’s media literacy.

These are just words… They have to be transformed into measurable numbers, the numbers have to be interpretable so that they relate to values, and finally the complete package has to be transformed somewhere into money.

The desired improvements have to be transformed into trackable metrics: What are the indicators, can they be found in the cms/portal?

The metrics need to be clustered into rememorable topics, and they need a visualisation: create charts, get sample data, build words and their stories.

Finally, an obvious connection between the indicators, their behaviour and the financial development of the project has to be made visible.

Visualization: Key Values based on easily trackable indicators.

But step by step.

• In the current project, four main values could be identified: Effiency, Satisfaction, Quality, Impact. Each of these values is based on several metrics, these metrics can be combined to several graphs that indicate trends and development.
• This leads to a balanced-scorecard-like environment, where small changes on a basic level are aggregated to effects on a visible level, on a level that can be communicated on a senior management level: You don’t have to say “We are having less usercomments than last month”, but you can say “Our portal is loosing on impact”; you don’t have to talk about painful cms editor-tools, but you can talk about efficience in the work of editors or about increasing or decreasing cost per content or cost per user.
• Changes on the value level finally have to be translated into financial dynamics: where do changes in the techy metrics in the underground make you loose money, where do they make you win some?

Relations and data are quite complex by now, but it is really important to keep the dynamic parts really simple and related to as little data sources as possible – preferrably only one: Everything that comes out of the portal can be measured in the portal; everything else should have to be defined only once.

This is what we will cover in the next part.

Part 1: ROI Dashboard – Introducing the User Experience Indicator

Part 2: ROI Dashboard – User Experience Indicator 2 - A Sample Project

Part 3: ROI Dashboard – User Experience Indicator 3 – Bring in the money

Part 4: ROI Dashboard – User Experence Indicator 4 – Aggregate and simplify

 

The big advantage of well designed KPIs is their immediate connection to simple and basic values. But what we actually want to talk about is ROI, and ROI is always about money. Don’t say ROI unless you really mean it.

We have defined values, we found KPIs that could be assigned to values, and we have taken care that KPIs were really easy to measure. You will need them frequently, so it should not be a big effort to keep them up to date.

What we need now is money. The values and KPIs we have defined must be transformed into easy understandable terms that can be visualized and that have a clear financial connotation.

So there are three things to do:

• Find your financial values
• Create charts
• Fill in the data and talk about money

Financial values are relations, ratios and terms that are related to time, work or directly money. Some examples: The value is efficiency, the KPI is the time editors spend to create content, so the financial value is the amount of money you can save if editors spend less time for editing then before. Another KPI for the value efficiency is if the search works well, if the users know for which terms to search for and which contents can be easier and faster accessed using the menu. If users spend time using the search instead of just getting to the content, we can assume that they loose two minutes figuring out a search term, reading the search results and deciding where to click on. So the cost of bad search terms is the number of bad search term occurrences times two minutes times the average salary for two minutes work.

On the other hand we can assume that good searches save two minutes of time: Users don’t have to browse around, don’t have to test using trial and error – so it takes them faster to their results. The financial gain of good search is thus the number of searches (with clicked search results) times the average salary for three minutes of work.

Care about the details and really question your data. But stick to things you can measure, don’t try to add metaphysical value – that just leads to trouble and discussions. Another example: Is it efficient to use Wikis, what is the use of having nonprofessional authors publish content? To answer this question, reduce it to it’s really simple financial dimensions. How many pages does an editor publish during his work time? We assume it’s approximately 30 pages per week or 120 pages per month. If we relate this to 100% of an editor’s salary, each of the 120 pages costs 0,83 % of a salary.

Compare this to Wikis. We assume, that Wiki authors contribute around five pages per week and spend and our per page. So five pages take 2,5 % of an author’s work time, that makes 0,5 % of a salary for one Wiki page. Thus we can indicate that wiki pages are cheaper than CMS pages – but we also need to relate that to user figures to deliver indications for whether Wikis are just cheaper or really more efficient. This is why charts and visualizations are really important to support the financial indicators.

Some values can not be measured without the help of additional criteria. User comments do show participation, rss subscriptions show that users really appreciate your content – but how does that relate to money? In our sample project, we introduced a participation value of 12 $. Check out advertising fees your company is actually spending, when you have to define your participation value: How much money does your company spend to reach users with advertising, how much does it spend to activate users eg to fill out a form or to submit data? Get these values, compare them to your usecases, and define your own participation values. This not only gives you figures to operate with, it also brings you in a position to defend your choices: You did not invent these values, you are not responsible for the amounts – they just reflect a real amount of money your company is actually spending in order to achieve efficient communication.

We used participation values to rate the participation of users through comments, rss subscription or also consumption of non-mandatory contents like blogs, wikis or also movies.

Creating charts is the easy part once all criteria have been defined. Nevertheless, it is extremely important to have good visualizations – not only to support the formulas and foster understanding, but also visualize ratios and illustrate the true dimensions of the defined values. Charts need to show development over time, but they also need to illustrate dimensions and ratios.

And they are a good means to simplify things: it’s impossible to get rid of all uncertainties and stakes are high that you will have to discuss the same things over and over again -charts are very good in getting faster to the end. If they can be explained in one line, they take you directly where you want to be. (If they can no be explained in one line, they are bad charts anyway.)

Now as we have money in the game and we know that financial dimension of everything – do we know if we are doing good or bad? We can fill in some values in our formulas and we will get results; sometimes it’s money, sometimes it’s just some ratios or some indexes.

There are values that we want to be as high as possible, but there will also be values that should be as close to zero as they can get; some values even have to be as close to a certain amount as it’s possible.

So it’s not won yet. The values we have calculated are monetized, but we need to relate them to our general targets. The indication if this value should be higher or lower is one of the few things that has to be added from an external point of view; it can not be derived from the system we have defined.

Setting the targets should be easy. Most of them can be derived from common understanding, and they should fit to your general business strategy. This is important: Targets have to be compatible with your business strategy. That’s not only self explanatory, it is extremely important that you can show and explain this relation if you want to sell your dashboard to the upper management. This supports our business – that must be the main message of everything you say.

Actually – you should not say anything more at all, unless someone asks.

That means, that you don’t only have create the target driven relation, but you also have to go into one more round and aggregate a first-glance-overview of all the values you have defined and calculated so far. Something anybody can look at and decide immediately, if this company is doing good or bad.

This will be the main topic of the fourth and last part of this post – watch out for it.

Together with the last part, I will also publish all the background material that in detail illustrates the processes described so far.

Part 1: ROI Dashboard – Introducing the User Experience Indicator

Part 2: ROI Dashboard – User Experience Indicator 2 - A Sample Project

Part 3: ROI Dashboard – User Experience Indicator 3 – Bring in the money

Part 4: ROI Dashboard – User Experence Indicator 4 – Aggregate and simplify

Up to now, we have found values for our project, we have defined KPIs that support these values and we have created financial relations to emphasize the business value of our core values.

That are way too many indicators; we have to look at so many things that we might loose the big picture easily. This is not a good dashboard; dashboards stand for simplicity and for information delivered at one glance.

On the other hand, information is only good, if it is based on many and widespread indicators; you can’t control your business with a narrow minded view on what’s going on.

Another missing part of information in our sample project is the expense-side: What is your invest, what is it you want to get a return on? This is something we have to clarify before we can proceed to aggregate our financial values.

A first issue that will occur especially in media projects is dealing with the relations of onetime costs vs. running costs – what do you want to related and how do things belong together?

If we go back to our example project of running and relaunching an intranet, the expenses we need to consider are:

• implementation of the relaunch project
  • planning, designing
  • developing
  • training, roll out
  • contentmigration
  • evtl. software licenses, evtl. new hardware
• operational costs
  • ongoing software and hardware maintenance, support
  • hosting expenses
  • application maintenance (SLAs, software upgrades)
  • security and backup costs
• editorial costs (time the editors spend on creating and managing contents, evtl. costs for buying/producing specific content (eg. movies, audios etc.)
• usage costs (time users spend on the intranet)
• eventually promotional expenses (internal marketing)
• additional expenses (travel related to the project, education for project- or editorial staff etc.)

Most of these are ongoing costs that can easily be monitored on a monthly basis. The project itself is a onetime investment with a lifetime of (hopefully) several years. How can we integrate this in our picture?

All financial values we have defined so far are short term and frequently changing factors; they probably look very small if compared to major onetime invests.

There are several ways to deal with this:

We can split the invest into monthly amounts for it’s assumed lifetime

We can calculate the total costs (onetime + ongoing costs) for an assumed lifetime and split the total amount into monthly amounts

We can add the project price as an entrance fee and add a timeline similar to a break-even-calculation: This will clarify, that the ROI-figures will look pretty bad for a certain time, but that we can expect changes, and it will tell us how we are doing on our way towards these changes.

In our intranet project, we decide for the third option. This may need some additional explanation for user who see the ROI dashboard for the first time in an early stage (why is it so negative?), but it simplifies the handling of data over time: In fact, we now have to deal with monthly costs only; and the first month is a very expensive one.

That said, how can we proceed to get a better overview of things? A first step is to cluster expenses – there again, we have several criteria. An option may be to group them by cost centers, so that the responsible managers feel addressed. The disadvantage here is that the organizational division ist not always the most logical one, it is not always the best to capture the real processes. Another option is to sort costs by how they are caused:are they caused by the daily use of the system, are they caused by more users or more traffic, are they caused because of regulatory issues or just because of weaknesses in the organisation etc. The advantage here is that we have a chance to make obvious were the costs come from (they are not always due to additional wishes of editors or developers…), but the problem is, that you may have endless discussions on the origin of costs – that quickly turns into a philosophical discussion.

My favorite setting is to just focus on technical costs and usage costs. This covers the most important stakeholders in the project, and it somehow mirrors the idea of onetime and ongoing costs in the project.

Technical costs – in our project – would be

• project implementation
• operational costs

Usage costs would be

• editorial costs
• usage costs (in the narrower sense as defined before)
• promotional expenses
• additional expenses

Technical costs can be assumed with 1,5 million euro for the initial project and 20 k euro monthly operating fees

Monthly usage costs sum up from 12 k euro for editorial costs (three editors who spend 50 % of their time creating and maintaining content) 3 million euro for usage costs (10.000 users using the intranet for 20 minutes a day on 20 days per month), 2,5 k euro for promotional expenses (creating training materials etc.) and 8 k euro for additional expenses.

 So if we look at the true and actual costs, there are huge amounts of money. The biggest amount – user costs – are actually savings, because we assume that using the intranet (especially a good intranet) is cheaper then always asking colleagues for help, browsing in printed archives (or badly designed intranets) or informing employees on employee events. But this is what we have to prove.

 The clustered revenues look similar:

That now allows us to compare costs and revenue

This tells us several things:

we are not quite there yet (as long as the ROI is smaller then 1)

but we know our handles and we can watch how they affect our ROI

and we get some insights on the actual financial dimensions of media projects

An easy overview of costs and ROI-development can look like this.

The data obtained that way show that online business affects the whole company, that the financial value of media is not always related to advertising and that there is a way to determine values – and that seemingly small changes can have really big effects.

The ROI Dashboard is a product, it is not just a concept or an idea, it is a product that can be customized for several projects and that can grow over time. I think of it as a relative of the balanced scorecard: It sums up a lot of different aspects, it adds insights from different views and it includes more than just the obvious financial dimensions. – But we should not forget, that after all, everything should lead us to values we can express in hard currency. Even if we want to add more, we can only estimate the benefits of our project, if we can finally express things in cash.

I started a few discussions on this and will get back with some updates.

The data sheet with all calculations can be found here, an overview of the values, their charts, their monetization formulas and the overall outcome can be downloaded from slideshare.

Part 1: ROI Dashboard – Introducing the User Experience Indicator

Part 2: ROI Dashboard – User Experience Indicator 2 - A Sample Project

Part 3: ROI Dashboard – User Experience Indicator 3 – Bring in the money

Part 4: ROI Dashboard – User Experence Indicator 4 – Aggregate and simplify