Ideation does not occur in a vacuum and is impacted by a large range of factors. Some of the primary factors include organizational culture, challenge framing, community size, and community engagement. Factors can vary over time (for example, workload) and be influenced by events outside of the sponsor’s control (such as world events). For these reasons, it is not possible to establish hard practices that will guarantee a fixed level of idea quantity and quality for every challenge and community.
This document discusses the factors that should be considered when defining your idea graduation criteria and will give you some baselines to start with for your campaigns.
In December of 2021, the Automated Graduation feature was relocated to Administration > Setup Settings. The aim was to provide more control of the challenge event lifecycle for innovation teams. There are often scenarios within the process where a "pause" is needed while ideas are evaluated. This update focused on improving the user experience in that scenario.
Previously, this setting was only available to Planview IdeaPlace Administrators.
The setting is enabled by default, which means that the ideas automatically graduate according to the configuration that is set in the Graduation Thresholds section.
If this setting is disabled, idea graduation will cease to be automated. All settings configured in the Graduation Thresholds page will no longer result in ideas being graduated. When turned back on (from the disabled state), idea graduation will resume.
How it Works
Factors to Consider
Not all people invited to participate in a community respond to the invitation. Of those that respond, some only lurk (look around but don’t contribute). Others contribute infrequently and some will contribute regularly.
Generally speaking, the larger the community, the larger the activity level. However, the activity does not scale linearly with community size. Small communities tend to have a high participation rate, as the sense of community is high. Small communities are often hand-picked or share a common goal or culture. Therefore member participation rate in the community is high. Very large communities tend to have a lot of inactive participants: people who either don’t participate at all or only participate infrequently.
Below you see the results of an analysis of small to mid-sized IdeaPlace communities. The sampling is low enough that there is a lot of noise in the data, but the trend is clear: small communities have a relatively active community base while this tapers off with an increase in size.
When considering the impact of community size on member activity level, use the following table as a guideline:
For internal communities, organizational culture can stimulate or dampen community activity. If your organization uses open management techniques, empowers members, relies on the crowd, and recognizes contributions, your activity level will be above average. If your culture is more restrained and controlled, your community response will be lower than average.
The challenge at hand is painfully obvious to the sponsor looking for a solution. However, when asking a crowd for solutions, the crowd will inevitably know less about the problem or opportunity than the sponsor. Spend time framing the challenge so that the community members understand it, care about the solution and know how to participate in being part of the solution.
In the worst case, poorly framed challenges can kill a campaign. If you find your participation level unusually low for a campaign, revisit the challenge framing.
How sticky is your community site? How visually appealing is it? How effective was your marketing leading up to the campaign? How much do you stimulate the community during the life of the campaign? What is your plan to draw them back in day after day: to make your community one of their regular destinations? All of these can factor into community participation.
For internal campaigns, clear executive support for your campaign significantly increases participation. To maximize participation, executive support should be evident for all business units that are participating in the campaign. Having the executives pop in on a campaign and ask open-ended questions on a sampling of ideas is a recommended best practice to boost enthusiasm.
Having visible leadership support will always increase participation.
Middle Management Support
For internal private organization campaigns, don’t forget to make sure middle management has buy-in as well. Middle management can undermine campaign participation either intentionally (for political reasons or because they don’t view it as valuable) or unintentionally (for example, not allocating breathing room for staff to participate).
The timing of your campaign can impact the participation level. Try to avoid particularly busy times of the year like the holidays, year-end/tax season for finance employees, or periods where your business cycle causes particularly high workloads.
In addition to the time you run the campaign, the duration of the campaign can impact participation levels. Shorter campaigns drive a sense of urgency and priority. Members are more likely to participate in a campaign when you know a) they won’t have long to contribute their two cents and b) they only have to focus on this for a short period of time and can get back to “that big project” soon.
The best motivation is intrinsic but some extrinsic incentives can increase your participation as well. To maximize intrinsic motivation, identify people that care about your challenge (have skin in the game) and invite them to participate. Let them be a part of the solution. That will drive the highest participation levels. However, back it up with formal recognition programs for not just the winning ideas, but for high levels of participation as well. Reward results, but also recognize effort.
Graduation Criteria Options
Age (in days) is how long the idea has been published/posted.
Qualified Page Views
A user view, by session, for a particular idea. When a user views an idea for the first time, a qualified page view is counted. For the same user to log another qualified page view on the same idea, the user's session would need to be ended and restarted (like signing out and back in), otherwise any subsequent views in the same session for the same idea will not be counted as a qualified page views.
Votes is a straight measure of how many people have voted on an idea. The up/down/star rating does not come into play in these criteria. This setting ensures that the most basic level of input is being contributed to the idea. You are asking the crowd “how does this idea look?” Note that in contrast to Qualified Page Views, this ensures UNIQUE idea voters since each can only vote once.
Star Rating uses averages without taking reputation into account.
The pairwise ranking criteria is based on the Pairwise Leaderboard metrics. Set the criteria to the ranking desired and all ideas equal to or greater than the ranking will graduate.
The predictions ranking criteria is based on the Predictions' overall leaderboard metrics. Set the criteria to the ranking desired and all ideas equal to or greater than the ranking will graduate.
Number of Posts
In this context “Posts” refers to the number of comments made on an idea. The setting encourages qualitative input into an idea; for example, an explanation about a flaw in the idea or a proposal for an improvement. Note that this criterion is limited in that there is no systematic way to measure the value of a comment on an idea. For example, is the comment something like, “I like this idea,” or is it a very meaningful suggestion?
A conversation is a communication between two people. The conversation level on an idea is a measurement of the number of interactions that have occurred around an idea. Two people could generate a conversation level of 0 (no interaction - just independent comments on the idea) or 1 (a comment and a response to that comment). Three people can generate a conversation level from 0-3. Four: 0-6. Etc.
The Review Count is a simple measurement of the number of reviews that an idea has. The review results do not come into play with this criterion. This setting ensures that an idea has more detailed quantitative input, since a review is multidimensional and encourages the reviewer to consider the merit of the idea from different angles. You can allow reviews from the general crowd or restrict it to designated subject matter experts. IdeaPlace supports multiple review forms and allows you to have separate crowd and expert review criteria.
The Review Rating is a reputation-weighted review average. This criteria ensures that the idea passes merit with detailed reviews. Note that the Review Rating changes with each new review and may change AFTER an idea has met this criterion. See the Side Note below.
The Team Size criterion is a measure of the number of formal team members an idea has. Teams can improve the quality of an idea by including multiple perspectives in its development. This is a more formal means to encourage collaboration around an idea. Team members are motivated to improve an idea through their association with the idea.
The Role Approval criteria is a simple way to ensure someone from a select list has approved an idea to move to the next stage. In the simplest implementation, the approver ensures the idea is appropriate for the business goal and appears to have some validity. However, offline processes could require the approver to evaluate the idea using more formal means before approving it to move forward.
Frequently Asked Questions
Q. How long does it take for a change in graduation criteria to take effect?
A. When graduation criteria are changed during an active challenge, it may take up to 24 hours for the ideas to graduate based on the new criteria.
Q. Should I set a low bar for the first stage?
A. Having ideas graduate quickly encourages participation. People start to get excited about the possibilities that their ideas hold. They see them becoming a reality. One goal underlying every crowd based-innovation campaign is to encourage participation in this and future campaigns. Easy Stage One graduation is a free way to support this goal by generating excitement.
Q. I need one of my Experts to review the ideas. When should I bring in manual work?
A. The early stages of a campaign have higher idea volumes. Let the crowd do the work for you and your team. Apply automated, crowd-sourced criteria to the first stage or two to narrow the ideas down. Believe and trust in the wisdom of the crowd. You can always go back later and manually graduate ideas that you think the crowd missed. Alternately, if ideas that you don’t think are appropriate have made it through to later stages, you can stop them there. They don’t need to be implemented.
Leave criteria like expert review and manual approvals for stages where fewer ideas will require your effort.
Q. Is it better to communicate the graduation criteria?
A. It is best not to make the graduation criteria a mystery. However, you don’t want to be overly specific, either. In your challenge framing, explain in general terms what types of evaluation methods will be used when considering ideas, but stay away from specific details. Not including any information on idea evaluation criteria may turn off some participants that feel the system is rigged, but being too specific doesn’t allow for adjustments later on and can encourage some community members to try to game the system.
For example, explain that stage one uses crowd voting to evaluate ideas but don’t say an idea must have 10 votes to graduate.
Q. What if I have too many ideas getting through?
A. Raise the thresholds to slow down the idea flow. You can either do it in an early stage to slow down new idea graduation, or you can do it in later stages to stanch the flood before it gets to the manual work.
Q. What if there are too few ideas getting through?
A. Ease the criteria.
Q. I’m thinking of running an Always Open Suggestion Box. Is there anything I should consider differently?
A. Ensure there is a sense of urgency around idea promotion by setting idea expiration periods that are similar to the timing you use for your challenges. Then remind idea owners that they have a limited time to promote their ideas and see them graduate through the system. Without the sense of urgency, ideas will languish and not get the attention they need to develop.
Generally speaking, the community as a whole will not have the same activity level since they won’t feel the sense of urgency that a timed challenge brings on. On the other hand, they won’t have the same demands on their attention since ideas will flow into the system sporadically (and with less volume than a timed challenge). The owner will have to encourage the crowd to ensure proper attention.
Monitoring and adjusting idea stage levels within an always open system will be different than timed challenges. Instead of using absolute terms as described for challenges, use flow rate goals: ideas per stage per month.
Start with the graduation criteria used for timed challenges but lower the bar a little, perhaps 20%. Then open the suggestion box and check the flow rates over time and make adjustments. Remember that since ideas will be expiring out of the queue, you will want to monitor idea stage population throughout the month to ensure you account for ideas before they expire.
Q. How/where does one accomplish this, please: "You can always go back later and manually graduate ideas that you think the crowd missed. Alternatively, if ideas that you don’t think are appropriate have made it through to later stages, you can stop them there."
A. As an Administrator on the ViewIdea page, check the Control Panel. There you can select Move to Stage and select the stage you would like the idea to graduate to. The idea owner will receive an automated email that their idea has graduated.
If Pairwise criteria are not specified, ideas will still graduate if other criteria specified are met.