How Can You Measure a Wiki's Worth? 78
moldar writes "I have been involved in a major project to migrate documentation from multiple sources to a wiki (Media Wiki, if you must know). Now, the PHBs are all asking questions about how organized the data is. I've already googled for various wiki metrics ideas, but mostly they focus on page counts, average page sizes, rates of edits, etc. Can anybody suggest better ways of measuring the quality of a wiki? Things like uncategorized pages, articles that are too small, etc? Any help or fresh ideas would be appreciated."
Turn it off. (Score:5, Insightful)
Re: (Score:2)
See how many edits there are on the "Who shot first, Han or Greedo?" entry.
Re: (Score:2)
He should do some research [uncyclopedia.org]
Close! His problem is backwards. (Score:1, Insightful)
The problem is that he's trying to measure the wrong thing. He's trying to get metrics from the wiki itself. He should be getting them from the users.
So the simplest solution is to do a user survey about what's good and what needs improvement.
Here's some advice (Score:5, Funny)
Re:Here's some advice (Score:5, Funny)
I suggest a new strategy, let the Wiki win!
Re: (Score:3, Funny)
I thought it was Happy Fun Ball(TM) that you were not supposed to taunt?
Re: (Score:2)
Sell it? (Score:2)
It would be worth whatever you can get someone to buy it for.
Nice editing on the part of whomever took the firehose article. Changing the title changed how anybody is going to read the article.
Getting a single metric for the quality of a wiki is going to be really hard, since the number would just about be made up.
If you want to say what percentage of pages are uncategorized, below a certain size, or orphaned, then do that.
Worth vs price; Value != value (Score:2)
People are very resistant to pay for something that they feel should be free. The same people that would pay a hundred dollars or more for MS Office would feel screwed if they paid $10 for OpenOffice.
Value to the economy is very different to the value marketeers attach to something. Wikipedia, Linux and many other free products are very valuab
In soviet russia... (Score:1, Offtopic)
your wiki measures you!
Metrics (Score:5, Insightful)
PHBs will always insist on metrics for things that can't properly be quantified.
What I would do is suggest that he task a team from another department to run a survey on satisfaction amongst wiki users. Has the wiki helped them? How often do they use it? What would they like to see changed.
This should distract the PHB for a while, while diverting your responsibility to come up with a metric.
Re: (Score:2)
Re:Metrics (Score:4, Insightful)
Re: (Score:2)
Of course, in the design of the survey, you should make sure that the result "people are not using it but they would if they knew about it" and "people are not using it because they tried and it sucked" aren't compounded into a single result. They are very different, but given the wrong set of questions, will be very hard to differ from each other!
Re:Metrics (Score:4, Interesting)
That's actually a pretty useful suggestion. Think about it, the real question the PHB is asking is, why should we continue to fund this thing? Are we getting our money's worth? The best way to answer that is to see how much, if at all, people are using it, and what they think of it. Your survey may just be the metric the bosses are looking for.
Re: (Score:2)
It is likely the closest "metric" that will answer the question.
Usage pattern statistics (Score:5, Insightful)
It depends a lot on what your wiki is for, but usage pattern statistics might be a good metric. For example, repeat visitors are people who are getting some value out of the wiki. Trampoline pages (pages people hit and then bounce elsewhere from a link in the page) might be another good thing to track. As well as abort-and-retry pages, where instead of following a link the user goes to the search box and tries something else.
You get the idea.
If the wiki is valuable to the people who are using it, you should be able to tell this from the logs--even if no one ever mentioned wikipedia in public, they could tell how much we care by noting that we keep coming back.
--MarkusQ
Don't underestimate "easy" (Score:2)
That's a valid part of the metric. Easy is good. When you're figuring benefits / costs, "easy" reduces the denominator and makes the total score much higher.
When you just compare the accuracy of weather forecasting methods, "looking out the window" doesn't rate very high. But it's generally so easy that in practice it's the technique of first recourse and you'd be a fool not to use it. The great thing about "easy"
Re: (Score:2)
It depends a lot on what your wiki is for, but usage pattern statistics might be a good metric. For example, repeat visitors are people who are getting some value out of the wiki. Trampoline pages (pages people hit and then bounce elsewhere from a link in the page) might be another good thing to track. As well as abort-and-retry pages, where instead of following a link the user goes to the search box and tries something else.
One can transform these types stats into quantified estimates of savings vs non-wik
Impossible! (Score:3, Interesting)
Re: (Score:1)
Re:Impossible! (Score:4, Funny)
I don't know what the other questions are, but I guarantee the last one is "Where's the quickest place to get laid on a Friday night?"
Re: (Score:1)
Re: (Score:2)
What is one plus one, for sufficiently large values of 1?
Re: (Score:1)
Unless you work for yourself, you have to *learn* to put up with PHB bullshit. That's just life, dude.
The PHB's at our pad wanted to give a group of people web cams so they could watch their faces in conference calls. The users for the most part don't really want them and don't want to mess with them. But if the big-wigs want
Re: (Score:1)
The PHB's at our pad wanted to give a group of people web cams so they could watch their faces in conference calls. The users for the most part don't really want them and don't want to mess with them.
I can see why. That sounds kinda perverted. I'm guessing that it's not two way...
If it's not voyueristic, it's at least a bit extreme. Are they trying to fix a problem, or create one?
it must be a strange situation...
Organized? (Score:2, Interesting)
The whole point of a wiki is that it doesn't have to be organized. It doesn't have a table of contents, it has a page list -- which are not organized in any sort of temporal or incremental way, but rather alphabetically.
You don't read a wiki start to finish like most PHB's think documentation should be read. Instead, try to make him understand that relevant links (internal link count metric) and search indexing (search count metrics) are what make a wiki organized.
Which is the whole point. You go from an or
Re: (Score:2)
The whole point of a wiki is that it doesn't have to be organized
Very good post!
At my company (one of the major international technology SI's, population >40k peeps) we've just moved our entire culture across to a global Wiki (Confluence) with a lot of MediaWiki imports from various wikis (our company's wikis, that is) around the world.
The existing knowledge base portal just didn't quite catch on, was too bureaucratically structured and just never achieved critical mass -- behind the scenes, a lot of individual Wikis had spread up to address the content shortfall
obvious (Score:3, Funny)
Re: (Score:1)
by using metrics of course
Or you can use imperial units if you want to torture yourself.
Re: (Score:2)
by using metrics of course
But imperialistics have been working so well.
"`What's the Wiki worth?' It exists and it's useful. That's enough! Off with his head!"
I've done this one a hundred times (Score:5, Insightful)
The truth of the matter is: You're asking the wrong people. You should be asking the suits what they would imagine in a report and what kinds of numbers they're looking for. If they're general and obtuse, tell them straight up they'll get a general report, or just make up numbers to keep them happy. Create a widget they can access from their desk that shows them numbers generated from the database that hardly matter. Does it matter what you deliver if they don't know what they want?
They're trying to imagine some great power-point presentation you'll be showing with pie charts and red/green/yellow/blue graphs popping out and wowing them? Make it so. They're trying to imagine a spreadsheet? Easier and more concise. They just want a status report at the end of the day on what percentage of the documentation has been migrated? You can probably get away with a few page count written on a napkin.
Remember the Alice's conversation with the Cheshire cat:
"Would you tell me, please, which way I ought to go from here?"
"That depends a good deal on where you want to get to," said the Cat.
"I don't much care where -" said Alice.
"Then it doesn't matter which way you go," said the Cat.
Re: (Score:2)
Create a widget they can access from their desk that shows them numbers generated from the database that hardly matter.
I have an RRD graph titled "Processed Internet Packets" that just graphs /dev/random every 5 minutes. The Manager I built it for contentedly uses the graph for all sorts of things, and I can get back to my work.
Re: (Score:2)
Re: (Score:2)
/dev/random will block.
Only if you use a Linux system. FreeBSD /dev/random doesn't block.
Simple (Score:3, Funny)
eBay
ask the users to survey it ... (Score:4, Interesting)
Depending how much of a BOFH you want to be you could ask users to rate the wiki, either a rating for any page they use (hard to enforce): MS have something like this as too do IBM, IIRC, "rate this article on ...".
Alternatively you could do an exit survey, when a user leaves teh site you pop-up a survey to ask about their experience.
Perhaps you could simplify adding templates and get users to add class markings (I think that's what they call them) for things like "this is a lame article", "this article needs editing for brevity", etc. and markings of "this is a B grade article", etc., then run a test of how many of what grades are given and how many templates are used.
Note that templates describing failings will be more prevalent as the wiki gets more use (per user and overall user count) a PHB might not twig this and will think quality is going down.
You could also measure uptake versus the previous sources.
Re: (Score:1)
Make it up (Score:2)
1. Pretend to invent your own metric by lack of pre-existing satisfactory metric
2. Describe it using esoteric terms you know your audience won't grasp/buzzwords
3. Make up phony results
4. ???
5. Ask for a raise, errr, I mean, profit!
Couple ideas (Score:1)
I was on the same boat not that long ago; here are a few ideas:
1. Wiki documentation to non-wiki documentation ratio. Compare how much documentation there is between the wiki itself vs. other project documentation like your word documents and pdfs (assuming they are not just copy/paste stuff from the wiki). A simple word count as measure will probably please your PHBs.
2. Assuming that this is a programming project that you are in, you can also compare the % growth of the wiki vs. the % growth of the codebas
Similar Situation (Score:5, Informative)
I'm actually in a situation which is very similar to the OP's. The company I work for has a number of training/procedural documents which currently exist in .mht format (of all things), and we're in the process of migrating those documents into the wiki which comes built into MOSS (the latest version of SharePoint). I won't go into detail about how horribly crippled the MOSS wiki is, except to say it is. Anyway, I was asked to write a "Wiki Best Practices" document and I tried to include a few ideas on what makes a successful wiki because I knew the business users were pretty clueless about them (not that I claim to be an expert here). These were some of the points I tried to address:
First and foremost are the users able to find what they are looking for? If they can't find it in the wiki they'll go somewhere else for it, and the wiki will fall on its face.
Is the information accurate? If you're not meeting these two criteria your wiki is probably worthless, and your users will almost certainly resort to another source of information.
Does your wiki exhibit signs of growth? By this I mean new pages/sections/categories being added when new information becomes available, and edits being performed to improve the quality of existing information. I think one of the biggest hurdles companies will have to overcome for this criteria is the fact that there tends to be a very top-down approach to the distribution of new information (This is definitely the case in my company). In order to allow the wiki to grow there must be enough users with create/edit privileges to keep it up to date. The more users there are who are actively involved in the wiki, the more likely it is to be successful. Of course, if management doesn't want to allow the user populace to be able to edit the wiki the task will probably fall to a select few, and they will inevitably become involved in their other tasks at which point the wiki will fall by the wayside and die a quick death. This is the battle I'm currently fighting, and I'm not entirely sure I'll win.
Ask your users what they think. This one is pretty obvious, but easy to overlook if you're doing things like checking page hit counts and sizes. Put together a survey and ask them if they find the wiki to be easier/more useful than the previous method, if it saves them time (and if so how much), and what they think should be done to improve it. Ask them if the information they find is accurate or not. Finally, ask them what they did/didn't like about the previous method, and whether or not that issue has been addressed by the wiki.
Re: (Score:1)
Does your wiki exhibit signs of growth? In order to allow the wiki to grow there must be enough users with create/edit privileges to keep it up to date. The more users there are who are actively involved in the wiki, the more likely it is to be successful.
Does it not defy the purpose of a Wiki in the first place if not everybody has edit privileges?
After all the ability to see all edits in public and to see the edit history is deterrent enough in a corporate organization where everybody (theoretically) knows everybody.
If management is concerned about editing important documents, then they should encourage authors/owners of documents to keep tabs on edits and try to "investigate why the change has been made and if it might be actually a welcome change." (if
Re: (Score:3, Interesting)
I completely agree with you. This was one my biggest concerns when the idea of using a wiki was brought up at my company. We know that the real power of a wiki comes from the users, but management likely won't see it that way. You didn't quote the rest of that point where I indicate that I'm pushing for the user populace to have edit privileges ;-)
Re: (Score:2)
Consider a hypothetical situation in which procedures may be changing semi-frequently in a customer-facing department, such as a call center. You want all procedures to be up-to-date, and it shouldn't take a lot effort to publish and re-publish the content, but you also don't want Joe McNewbie making changes when he may not completely understand what he is talking about. From the time he makes the (poor informed and/or possibly incorrect) change to the time at which someone spots the mistake and corrects it any number of other reps may have read the same material and used it during their calls. So now you have reps who are providing incorrect information, and possibly forcing the company to spend time/money/resources correcting those mistakes.
I'd thinking out loud here, but do any wikis support a policy where anyone can edit, but a one set of users have approval rights on any edits, and a different set can only see approved pages? Or even anyone can approve, but you can't approve your own edits. Wikimedia already supports part of this, where users can watch pages for changes.
Re: (Score:2)
Ah, good idea! Actually the MOSS wiki has this functionality; the "Workflow" feature allows you to do exactly that. I said the MOSS wiki is crippled (and I still stand by that), but that is one of the nice things about it. That would definitely open up the possibility of allowing all users to edit with an approval process in place.
Re: (Score:2)
Ah, good idea! Actually the MOSS wiki has this functionality; the "Workflow" feature allows you to do exactly that. I said the MOSS wiki is crippled (and I still stand by that), but that is one of the nice things about it. That would definitely open up the possibility of allowing all users to edit with an approval process in place.
It took me a minute to find (but I learned a lot about Moss, the plant, on various wikis), but I finally identified it as Sharepoint [microsoft.com]. Yeah, this is probably the way to go if (a) you're a business, (b) you've bought into the whole MS ecosystem, and (c) you want an otherwise crippled wiki that you can't improve.
I dug into Wikimedia's database schemas and it shouldn't be too hard to add an approval mechanism. I'd add just one table with pointers to a particular edit, the approving user, and the date approved
Re: (Score:2)
Connectedness (Score:2, Interesting)
Wrong target (Score:2)
You can use page size, amounts of edits, amount of pages, etc, but that will not imply that things are right or wrong. Sometimes a single word in the right context could say all you need. I remember that a page on wikipedia had
Web stats probably best (Score:2)
I'd say the best measure of a wiki's effectiveness is probably a thorough analysis of the access logs for the web server where the wiki lives. Run them through something like awStats [sourceforge.net] (personal favorite) and see what you get. Are people in your organization actually visiting the wiki? Which pages are most frequently accessed, and by whom? How many users are hitting the Edit or History portions of a page, as opposed to simply viewing static content? These are solid, quantitative numbers, and presented in a "d
Why isn't this answer included in every Wiki? (Score:1)
I can understand why you'd be confused - trying to measure the quality and worth of information is a big problem (just look at Google and AI), let alone how it is organized.
There are a lot of problems with Wikis - even the benefits, that you can grow your base of information, result in problems: who is going to reorganize it so it is usable again? What are good lists and bullet points to divide gobs of text into into usable chunks of information?
Someone has already mentioned accurateness, but what that rea
Re: (Score:2)
I agree with most of your post, but you are basing this statement off of the assumption that the content of a wiki is open to in
Re: (Score:1)
Well, this is where we start crossing the fold.
Opinion and interpretation are factors in every case of communication. That's why EULAs and company policies have official rewrites every year, quarter, or incident. That's also why there is always an HR department whose significant responsibility (50% of time?) is to respond to questions from employee, customers, etc.
Granted some of that time is because people don't bother reading documentation or the search doesn't work, but one thing many, many documents d
There are two possibilities (Score:5, Insightful)
Look, the pointy haired boss doesn't really want or expect a numerical metric of organization. Immagine asking him if he has a function in Word that will tell him how organized an outline or document is. He doesn't, and if he never needed it for Word, he doesn't need it for a webpage.
Possibility 1: he wants to kill the project, and came up with this as a way to find an excuse. The best response to this is to get your resume out there on monster right now, and walk into his office tomorrow morning and say "do you want to kill the wiki, and is this metric a way to get an excuse ?" . Let that conversation go where it will.
Possibility 2: he read in USA Today that anyone can add stuff to wiki's and they are chaotic, so he's worried. He's never actually looked at any wiki including this one in his life. This is the most likely possibility. If you still have a job after asking the question in Possibility 2, go for this, or maybe start with this.
Here, you want to talk to him for about 5 minutes using the following words as much as possible, without actually lying: "review", "control", "process" and "catagorize". If you want to can reveiw everything he has ever said and come up with your own set of buzzwords, based on the words he himself likes to use, but these will do. Then you say things like "We have a PROCESS to CONTROL the QUALITY of employee submissions. I occasionally REVIEW the most visited pages in the wiki, and make sure they are ACCURATE and properly CATAGORIZED. I also REVIEW the least visited pages, and if they are not visited because they are improperly CATAGORIZED and linked, I fix that." Actually talk like that, and kind of shout the buzzwords at him. He's either stupid enough that he needs it, or smart enough to figure out what's going on and move on to something important.
All the geeks in this discussion who started actually talking about measuring connectedness of graphs and crap are totally off in the weeds.
Re:There are two possibilities (Score:5, Interesting)
A good PHB is mostly concerned about 3 things:
1. A decent ROI. ( From the business as a whole down to the paper-clip supplier and the birthday party commitee.)
2. Making the people above him/her look good.
3. Control. (So #1 and #2 above can be maintained over time and change. You have to stay organized when you delegate or it's impossible to manage anything)
If you concern yourself with the exact same things in the exact same order then it should be easy for you to figure out what to do and what buzzwords to say.
The doing might be hard, but the "what to do?" should become easily apparent.
In other words answer these questions as asked by your boss:
What's the ROI on your wiki project?
What's the weakest link to me (PHB) not worrying about this..meaning what person or machine or database do we need to protect and make redundant?
Who can I trust (to make me look good) when they say it's a success. I will keep asking dumb questions until I find a leader I can trust.
Who should I scold if the ROI on this project starts tanking?
In 2 or 3 short sentences, what does it do so I don't look stupid when people ask me. And when I ask "what does it do?" I do NOT mean how does it work. I mean what Return does it provide on what Investment?
Careful though, thinking like this will get you promoted FAST.
What category do you put CATAGORIZED in .. (Score:2)
English Language
|_ American Spellings
|_ Misspellings
|_ C
|_ CATAGORIZED, replace CATEGORIZED, British CATEGORISED, not shouting "categorised"
If I spelt anything wrong I'm sure you guys have my back ...
Same way with any website... (Score:2)
Install some analytics tags and look at the data. How many people are using it? How many pages are accessed every day/week/month? etc.
If it's public-facing, put ads on it and see how much money you make. :) Can't beat that for a "how much is this worth?" question.
So, in other words... (Score:1)
... your bosses are demanding citations, and you're pissed about their lack of NPOV.
Slug it out in a good ol' fashioned edit war.
My experience... (Score:1)
Usege graphs (Score:2)
I manage a large intranet wiki at a university, based on TWiki. We also do some research on its use. The main metrics I would suggest :
* number of pages (cumulative graph over time, and per month/day)
* number of edits, idem
* number of views , idem
* Number/size of attachments, idem
* a histogram of edits/views per person
* a histogram of people who only read and never contribute.
you can obviously anonymize the graphs if
Irritating pop-up survey window. (Score:2)
When a user leaves your site, just pop up one of those irritating survey windows with a 'how useful was this wiki to you?' (You might have to say 'web site' if 'wiki' means nothing to them)..
Log some answers, or just plain old make them up, and put them in an Excel spreadsheet which you then print out and give to your PHB.
Don't use a wiki... (Score:2)
Use mind mapping software... they make professional mind mapping software for businesses.
http://www.thebrain.com/ [thebrain.com]
And it organizes information much better.
Support staffers (Score:1)
A few essential metrics (Score:1)
I'd look at a few measures:
Also, institute some meetings where people go through the Wiki and try to eliminate "old news" that has only historic value. This keeps the
~ot (Score:1)
See point #3 on the 10 Tips for a Successful Wiki [flickr.com] poster from gdc;
Invest in your wiki
Nothing is perfect off the shelf and your wiki will be no different. Most wiki packages are designed to be customized and extended, so take advantage of that by investing in a few days of developer time to modify it to suit your needs. Re-evaluate wiki usage at regular intervals to gather feedback on the most common gripes and dedicate resources to adjusting the wiki accordingly. The few hours or days spent tweaking it will come back tenfold in time saved by users.