Design Discussion - Quality Improvement System

This is a different type of blog post for us than what we’ve been doing lately. Almost all of us like to design and/or use useful solutions. That’s why we use SugarCRM, after all. In this post we thought we would share some of the design decisions we are facing as it may be interesting and you may have some ideas of your own to share. If this sort of post is something that you like let us know in the comments below and we may continue to do this for other areas we are working on.

Our goal is to always offer the highest of quality add-ons for SugarCRM. We opened in September with what we felt was a strong starting architecture that could immediately help reach that goal. However, maintaining a certain level of quality is a very tricky thing to do as versions change along with user expectations. It requires an ongoing effort to ensure that every add-on continues to meet and exceed the expectations of users. This is where we want to target next.

Currently, we offer a pretty standard review system that helps users determine how well polished an add-on is or isn’t. Along with that, we offer a verification system where users can verify for specific versions of SugarCRM whether the add-on does or doesn’t work. With that info we calculate a “verification score” that lets users quickly see if an add-on is a dud or, excuse me, a stud.

This is nice and has some immediate benefit for users, but it is of very little help to the developers, which in the long term hurts users. There needs to be a better way to close the communication loop for both methods of providing feedback (reviews and verifications). Here is where we are heading along with a short blurb on how we feel it will help:

Reviews

• Allow users and/or developers to reply to a review, much like how comments currently work.
  • The review may really be a misunderstanding or a call for help.
• Upon a reply, notify the original reviewer along with the developer.
  • This one is pretty obvious. Closing the communication loop.
• When the reviewer is notified an option will be given to change or remove the rating or remove the review altogether.
  • This will allow self calibration of ratings to provide more accurate ratings.
• Provide a clear description of what each star value represents. This will help better clarify the goal of ratings as a way to represent whether the add-on does what it says it does.
  • 0 Stars - Does not work at all. For example, could not install.
  • 1 Star - It sort of does what it says it does or is just packed full of critical bugs that make it unusable.
  • 2 Stars - It basically does as it says but is functionally unusable (many bugs)
  • 3 Stars - It functionally works as stated and has minimal bugs.
  • 4 Stars - Works very well. Solid implementation. There may be some non-critical bugs.
  • 5 Stars - Meets and exceeds expectations
• Allow a developer to dispute a review in addition to being able to reply. A dispute would be marked to all users as so and would not be used in the calculated overall rating until settled.
  • To be fair, developers need a way to combat either complete misunderstandings or outright incorrect rating/reviews.
• The review will remain in dispute unless the original reviewer confirms or changes the rating/review.
  • This will help prevent hit-and-run reviewers who have no interest in actually providing useful information, but instead post slanderous and untrue reviews.
• If the developer still feels a rating/review has wrongfully been posted after the original reviewer confirms it can be escalated to SugarOutfitters for review.
  • Unfortunately, there are people that just enjoy being a pain.
• Create support case option.
  • Developers should be able to easily create a support case based off of any review. This should notify the reviewer and it should be indicated on the review along with a link to the created case for all to see. This shows an active commitment to the product by prospective users.

Verifications

• If “Doesn’t Work” is choosen, require a description of what doesn’t work
  • Who hasn’t been told it “doesn’t work” without any helpful information?
• Based on the description given a developer can: 1) create a case 2) reply 3) dispute
  • Basically, the same reasons from the Review counterparts apply here.
• Much like a rating, a disputed verification will not count against the “verification score” until resolved.
• A user can change or remove a verification for any version, just as they currently can.

With something like this in place it will hopefully help to foster more communication around addressing the issues with any given add-on. In addition, it’ll help to weed out non-constructive and unhelpful criticism. We’re asking more of everyone involved, but if it means better products we feel it’s worth it.