I absolutely love comparison tables and I wish there were more of them out there. I definitely like deep tables and I do not mind at all the potential danger of bias. In this very long post (apoligies) I will give my arguments on why I like comparison tables so much!
But if this table has only width but no depth, it doesn’t really tell you any solutions. It only tells you about potential software that may potentially have a solution for you. You have to find out yourself whether any of the competing software are actually useful for you by checking out every single software.
Sure. But what’s the problem? Why is this bad? You still get plenty of extremely important information. You know for sure what the software that made this table is very good at, better than the comparison. If the thing you need this software is in this table, then bingo, you are done.
I find it very frustrating if a new software comes along, that implements things already existing in multiple other software, and it doesn’t do such a comparison table. Like, what’s the point? Why go into a field with competing alternatives if you are not doing it better or if you are not offering something that wasn’t offered before? The table is your chance to show this. Without such comparison statements (they don’t have to be tables necessarily), you are making the life of the users more difficult. Discovering software is already difficult, but now the user has the burden to also decide which one to use. Such tables ease this burden.
The majority of people that would actually use such tables do not care about “which software is the best”. They care about “which software is the best for them”. In this case, having a biased table that highlights all the good stuff of a software isn’t a problem at all for the typical user. If all this good stuff is what the user needs, that’s it. If not, well, the table already showed the user several alternatives to consider! So I really find it hard to complain. I think the people that complain about these tables the most are competing developers, see problem 3 below.
That being said, I definitely acknowledge problems with these tables.
Problem 1: they are difficult to make. I am not particularly convinced by this:
I am wondering if you have any particular example for this, or if this an opinion you hold universally for any table. Or in general if there is any logical argument why this should be the case? Your phrasing makes it sound like it is normal to assume that the creators of such table are sinister. (I am not targetting you, the poster, directly, to be clear, but generally voincing my wonder for similar comments I’ve heard throughout the years about how supposedly subjective and unfair these tables are)
I have made two such tables (see them posted in Comparison tables for various Julia packages ), and both of them were very hard work. We worked very hard to make sure that the statements we make are genuinely objective. We contacted the compared software and told them of this table and welcomed corrections (fun fact: practically no corrections were necessary, meaning that this hard work paid off). In all cases we wrote the code that compared the computational performance. So we even worked for the other packages as well. In fact, we still do! In our Agents.jl comparison with other software, we are the only people that are contributing, even though we have actively invited every other developer to contribute and enhance their implementations as they see fit!!! But they don’t contribute.
So, personally, I would be rather unhappy if someone told me that these tables are “artificially biased” to make our software look better, without clear evidence, after all this hard work that has gone into the tables.
(Just to be clear: I definitely acknowledge that making such a table has the potential to lead to biased results. After all the creators of the table know their software the best! So they may simply not know of a feature that exists in another software the their software does not have! I just do not see it fair to just assume that a table will be biased de-facto. If you are careful, and ask for feedback, you can be fairly confident that you are unbiased)
Problem 2: maintenaince. These tables naturally have information that is not forever true. Overtime, the compared software may improve. If one cares about having an objective table, then one has to update the table all the time. But this takes a lot of effort! So a very old table may not be useful to share with the users anymore.
Problem 3: inciting dislike or even hate. While people very often try to be in the mood “all software is good, everyone is great”, the reality is that some new software will be better in most aspects of a comparison than it’s not better. In fact, it should be otherwise why was the new software made in the first place? (if it is just a programming exercise for the creator then no need to attempt to attract users and be part of this comparison - we ignore this scenario). The competing software being compared with (or more accurately, their develoepers) may not like hearing about this truth. They may actively hate you and attempt to hurt you, inducing great mental stress and potential damage on your career track, if you show such a comparison table. I don’t know what is a solution that can alleviate this pressure.
Problem 4: some things are important but are difficult to make objective. A problem we faced in our Agents.jl comparison was highlighting the simplicity of Agents.jl. We believed (and certainly still believe) that it is the simplest software for the task. But simple is not so easy to make clearly objective (such as “has feature” or “doesn’t”, or “runs faster by 123.45x times”). (Also if someone already knows another language, isn’t a new language more complex by the get-go…?)
So these tables may have some rows that are more belief based rather than number based. Now if such a row is really important for the end user, I would prefer to have it in the table and be up for debate or up to interpretation than not have it at all. But fore sure this is a definite downside. In a table cell you cannot really provide in-depth explanation for a particular entry…