Let’s say you publish a new update that bundles together several improvements. You hit the deploy button and monitor zendesk for feedback. It slowly starts to trickle in… but its not exactly what you were hoping for.
“I DON’T LIKE THE NEW UPDATE!” someone says. “How do I revert to the previous version” someone else writes. Hmm. Could this be a vocal minority? It’s possible that the loudest customers are also the most critical of your product. But more negative feedback accumulates. It starts to manifest in the App Store. A few users even take to twitter to announce their dislike. Company executives see this and send you terse emails late at night asking you what is happening.
These situations happen more often than most product manager’s expect.
So what do you do? Do you consider it a mistake and roll it back? Do you iterate on the feature to address user feedback? Or do you stay true to your vision despite what customers are saying?
Having gone through this myself a couple times, i’d like to share a few solutions that i’ve found helpful, which i’ve packaged into an easy to remember acronym … DISWACC.
Channel the Power of DISWACC
Define the problem. The first step is to succinctly define the problem. What exactly about this update did users dislike. Did you take away a favorite feature in the spirit of simplification? Is it a confusing interface? Was there an incorrect assumption in your value hypothesis? You can gather this information by collating from all sources (eg. social, zendesk, App Store reviews) and looking for intent patterns. Another technique is to incorporate in your product a surveying tool, such as an app prompt. App prompts ask users to rate their experience on a scale of 1-5. Ratings of 3 or below ask a user to fill out a feedback form for richer explanations about his or her experience.
Next, identify who is affected. Use customer data to drill down into the profiles of those who are complaining. Again, look for patterns. Are these complaints coming from a small number of new accounts? Then investigate if you attracted a cohort of users from a new marketing initiative. This could have been unrelated to your product launch. Are these complaints originating from a new accounts with no other reviews attached? It could be a smear campaign initiated by a competitor. This is rare and pretty unethical, but it’s been known to happen in competitive industries. Unpacking these details should reveal whether issues are troll or legitimate complaint. I’ve conducted similar analysis in the past which revealed that many complainants did not align with our core mission. This prompted discussion which led to the decision that it was ok to to lose certain types of customers. We didn’t want to spread our focus thin building a product for everyone, which would have ended up mediocre. We preferred to build something for a smaller group of people that was amazing.
Understand the severity of the issue. If a nontrivial number of customers have been affected, then what is the severity of the issue? Identify if it caused a drop in metrics such as retention or engagement with a core feature. It’s important to learn if the quantitative story supports the qualitative story. A cohort analysis tends to be more beneficial in this step. Worst case: metrics decrease in line with the qualitative feedback. This may prompt a reconsideration on the benefits of this release. More likely case: metrics improve despite qualitative feedback. It may just be a matter of allowing more time pass so users can acclimate to changes.
With the analysis done, you’ll soon need to make a decision. In this step it is important to widen your decision set. Avoid defaulting to extremes, such rolling back the version or blindly sticking to your conviction. It helps to have a wide lens as you brainstorm alternative solutions. Is it possible to come up with a solution that gives you the best of both worlds? In the past, I’ve launched a recommendation feature that appeared to have a flat metrics impact. But further analysis revealed more nuanced findings. There was a pronounced lift in one group, and a sharp decline in another group. This caused the topline impact to look flat because it was averaged across groups. We figured out a way to personalize the experience. One segment was shown the old recommendation, while the other segment was shown the new recommendation. This led to both improved user experience as well as metrics impact.
Attain distance. If personalization is not an option then get some distance. Think about what would be the best decision 10 months or even 10 years from now. Is there an option that would be best for the user over the long term?
Compare alternatives. A best practice within gaming product management is to run expected outcome models for all new features. If you have an existing growth model for your product, plug these tradeoffs into your model as inputs. Run scenarios for what would happen, so you can predict if each option will harm the business. Temporary pain can be forgivable. But permanent pain, especially if you’re a startup, may not be a viable option.
Core priorities. Last but not least, clarify core priorities. Every company seeks to fulfill some higher mission. If none of the above works, then you should consider the decision that best services this mission. It’s a good idea to run this rationale by other executives to make sure you’re actually channeling the mission. You’ll get confirmation from people who have more experience than you, which should help build confidence in the decision. They can also provide support when further bad press or negative reviews occur.
One Last Thing
Sitting in the hot seat is tough, but all product managers eventually face this if they’re shipping new features at high velocity. One final bit of advice is to always look out for your team. Harsh words in a public forum sap morale. As you rally the troops to work on new features, it’s important to frame this experience as a learning milestone. One that is being internalized for future roadmap planning.