I just read the book The Leprechauns of Software Engineering from Laurent Bossavit. It talks about a a few common “facts” of software development that are not facts at all. They are just merely anecdotal. Unfortunately they are heavily used to support other claims.
These facts originate in some older papers that are referenced from other papers, which are referenced from other papers, which are…. I guess you get the idea.
By all this referencing the original information changed and also gained some kind of authority that is not justified.
Laurent tracked down the primary sources (listing the references) and their hypotheses. Showing that our “facts” are based on very limited data that magically gets generalized and that some papers used obscure metrics which compare apples and oranges.
The “facts” he dissects in details are the Cone of Uncertainty, 10x variation in software developers & cost of defects.
While I never thought of these “facts” and their pictures as “real facts” to the numbers (but more or less as some kind of trend) it is still surprising that their supporting data is so weak. We do not really know if the trends they describe are real or not. There is too less data to “verify” their claim even if they feel right.
Definitely an interesting read!