Measuring the responsiveness of open source ERPs
A couple of years ago, it occurred to me it might be possible to compare open source ERPs, ranking them not by total number of downloads (a number which is easily skewed and/or misinterpreted) but instead by measuring how responsive we are to our respective communities. To get a reliable unit for measuring responsiveness, I decided to focus on bug reports. I liked bug reports because a) they are easily quantifiable and b) the information about them is readily available on open source ERP community websites.
There were three basic data points I was looking at:
- Total bugs reported (all time)
- Total bugs resolved/closed (all time)
- Total open bugs (current)
Once I'd collected this information, I did some basic division (total resolved / total reported) to arrive at what I started calling the responsiveness ratio. To my mind, this ratio is like a batting average in baseball. The better you're responding to incoming bug reports, the higher your average will be.
Here at xTuple, we've been "batting" for almost 12 years now. During that time, responding to our community has been a priority for us. I've always sensed intuitively that we are doing well at that. But until I started measuring our responsiveness, I didn't realize how well we were actually performing.
Like the other open source ERP I looked at for this study, the xTuple bug list is publicly available. That's part of the beauty of open source: its transparency. None of us is hiding this information. Anyone is free to inspect it, analyze it, and reach their own conclusions.
My initial findings in 2011 both surprised and impressed me. xTuple's responsiveness ratio turned out to be .957. To put that in some perspective, that meant that since the beginning of time (2002), xTuple developers had resolved and/or closed almost 96% of the bug reports they received. That .957 batting average was enough to make xTuple the clear leader in the league of open source ERPs.
Fast forward to October 2013. My statistics were two years old, and I decided to dust them off and see where things stand now. Was xTuple continuing to perform at the previously high level? How did we compare to other projects? The following table shows the current results for xTuple:
|As of Oct. 18, 2013|
Looking at the stats, it was clear xTuple was continuing its tradition of responsiveness. The company's ratio was down only .014 points over the two-year period—not bad, considering xTuple has been expanding its product line, including the addition of an entirely new Mobile Web client.
I turned next to OpenERP. Two years ago I looked for but had trouble locating information related to OpenERP bug reports. But this year I found the information easily on their site:
|As of Oct. 23, 2013|
By this measure, xTuple is almost 20% more responsive to its community than OpenERP. While it's true OpenERP's bug volume is nearly double that of xTuple's, the number of bugs isn't the key data point. The goal is to measure how responsive we are to the bugs being reported to us—and there's clearly a gap between the projects here.
Next I took a look at Openbravo. I found their responsiveness to be better than OpenERP's—but still trailing xTuple.
|As of Oct. 18, 2013|
Openbravo's responsiveness ratio two years ago was .878, so that's a meaningful improvement. Good for them!
The last project I looked at was ADempiere, the only of these four without a primary company coordinating the project. Their figures do not seem very robust, but I think that can be explained by their recent migration to a new bug tracking system. Legacy bug reports appear not to have survived the move. Two years ago I pegged ADempiere's responsiveness ratio at .856. As of now, they are significantly behind that previous high mark.
|As of Oct. 18, 2013|
Clearly, responsiveness to incoming bug reports is not the only measure by which open source ERPs should be judged—but I do believe the responsiveness ratio gives us one very valid metric to consider. I'm proud of the job we're doing here at xTuple, as we strive to be responsive to our community. Along those lines, if you have thoughts or comments on this methodology, please feel free to share in the comments below. Thanks!
Fri, 11/01/2013 - 11:59#1
Very cool! I have sent a link
Very cool! I have sent a link to this to one of my prospective customers who is considering xTuple.
It makes me wonder what the responsiveness would look like if proprietary software vendors were to publish this kind of information.
Fri, 11/01/2013 - 12:02#2
A couple of suggested considerations
Pierce, I would also consider initial response time and resolution time to get a clear picture of actual responsiveness.
Example 1: Two companies have closed 95% of reported bugs, but Company A's average initial response time is 1-3 days and average resolution time is 2 weeks, and Company B's are 7-10 days and 6 weeks. Company A is a heck of a lot more responsive than Company B.
Example 2: Company A has closed 75% of reported bugs with an initial response of 1-3 days and resolution of 2 weeks, and Company B has closed 95% of reported bugs with times of 7-10 days and 6 weeks. I would give Company A higher marks for responsiveness because while not closing as high a percentage of bugs, they are responding to and resolving bugs faster. Company B is doing a good job of closing bugs but a not-so-great job of being responsive.
I think this is an area where xTuple really shines, because I have reported bugs and had resolutions delivered in less than one day! This is not normal, but for it to happen even a few times is exemplary. When I report a bug, it's important for me to know that someone at the company has seen it and triaged it, and that it gets fixed promptly. Short response and resolution times are a sign of great service and a proof of the effectiveness of the bug reporting process. If I were to report bugs and not get reasonable response or resolution times, I'd probably stop reporting everything but major issues, since it would seem a waste of time to report the smaller ones. If I get good response and resolution times, I'm going to report everything because I'll know the process is working and that fixes will come.