I got Beru ready for the Ubuntu App Showdown and submitted it. I didn’t win, but I did…. Well, I don’t really know how I did. There were no scores published or reported privately, no rankings, and not even a list of all the contestants. All I got, a month later, were brief comments from three of the five judges. While I appreciate this, I had been expecting to get scores on the five criteria being judged. Disappointed, I wrote an email to Jono Bacon:
I just got back from a trip, got the feedback on my app submission, and was disappointed to learn that I didn’t get the scores on the various criteria. I’ve seen your explanation on Google+, and quite frankly I find it lacking. Please allow me to explain my disappointment.
On Google+, Jono Bacon wrote: > It would be unprofessional of us to share people’s scores with other app authors
I can only speak for myself, obviously, but when I submitted an open source app to an open contest for an open OS, I fully expected it to be scored in the open. Sharing my scores with others doesn’t strike me as unprofessional; I was assuming it would happen. By the Copernican principle, I can only assume that this attitude is typical. :)
Scores only make sense when related to others scores. As an example, if your app got 20/100, that may seem disappointing, but if the top app got 22/100, it puts it in a new perspective.
Even if you don’t want to publish all the scores, you could still publish the mean and standard deviation. This would give me most of the information I’d want from the full list. You don’t even need to give us the raw scores; just saying that I got +0.5 sigma in foo and -0.2 sigma in bar would be enough.
everyone definition of a score differs, and even though we were very clear with the judges of how to rate apps (to get consistent scoring), every app developer’s perception of that number will differ, thus causing some potential concerns.
Well, then tell us what rubric the judges were using. That’d resolve those potential concerns.
This would also be mostly addressed by my suggestion above to publish the mean and standard deviation.
but the value in the feedback is not the number…it is the written feedback
Honestly, the written feedback I got wasn’t so great. Of five judges, only three wrote notes, and one of those was “nice treament of downloads and progress”. (Good to know, but not chock full of actionable items.) The other two had comments that were so vague in places that I’m not entirely sure what the problem was; I’ve followed up with David to try to get more details.
[Nothing forthcoming yet. —Ed.]
Moreover, this written feedback is more prone to bias than the scores that you’re afraid to share. Some judges may only comment on what they liked; some on what they didn’t. Numeric scores would at least give an overall idea of the judges’ opinions on the various criteria.
And even if the written feedback were more valuable than the numeric scores, getting both together can’t be less valuable than just one of them.
Additionally, not seeing all the scores prevents us from figuring out which apps are good models for us to copy. If I knew I got a low score in foo, I could go look at the apps that scored highest in foo and see what they do differently. You wouldn’t have to publish the full scores to allow this, you could just list the names of the top five (say) apps for each criteria. Surely no one will object to being held up as a paragon, but this would be few enough that not being on the list wouldn’t be a mark of shame.
While I have you, there are two other semi-related issues that bother me:
1) Why isn’t there a list of all the apps submitted to the showdown. Or if there is, why can’t I find it? It’d be fun and maybe educational to look through and check out apps that I hadn’t seen in development. Moreover, I was hoping for some advertising for my app as a result of participating, and I’m sure other authors feel the same way.
2) I was hoping to use the feedback from the judges to improve my app and release a 1.0 before 13.10. But I got the feedback on 10/16 and 13.10 landed on 10/17, so this was obviously not possible. I realize that y’all had a lot to do in the past month, but surely a simple shell script could have emailed out all the feedback as soon as it was collected.
I hope I don’t come off as too negative. Overall, the Showdown was a positive experience, and I’m glad I participated. But I’m not entirely sure that I’d do it again.
No answer from Jono (or anyone I CCed) yet. If I get a reply, I’ll update this post to say so and include the reply if permitted.
My suspicious side can’t help but wonder if the reason that no scores have been announced is that there are no scores. Maybe the judges got together for a discussion and picked the three winners. That would be disappointing, but it wouldn’t be wrong. We weren’t promised any particular scoring system. But if this is the case, why give us this phony story about why they can’t release the scores?