Organizing quality software competitions
In this talk we describe an approach for organizing programming competitions that aim to reward software qualities such as readability, maintainability, and so on.
At my university we are considering the idea of organizing a software competition with the objective of promoting, beside programming ingenuity, the production of software of good quality, that is, software that it is easy to read, maintain, modify, debug, ..., Such qualities are, by their own nature, quite "fuzzy" and ill-defined, so measuring them in an objective and quantitative way is not easy. In order to solve this problem we figured out an approach based on a competition between "spoilers" that introduce bugs in the software and "counter-spoilers" that try to debug them. Briefly, each spoiler is given a set of programs and s/he must modify them by introducing bugs, as subtle and difficult to find as s/he can. Each counter-spoiler receives a set of programs and s/he must find the bug(s) in them as fast as s/he can. Finally, every program is given to many different spoilers to get many different spoiled versions.
The competition turns out to be, actually, three competitions in one since we reward
<ul> <li>The best spoiler, that is, the one that introduces the most difficult to find bugs.</li> <li>The best counter-spoiler, that is, the one that is fastest in finding the bugs</li> <li>The best program, that is, the one that is most "resilient" to bug insertion.</li> </ul>
The "untangling" of the times required for debugging into a measure of spoiler, counter-spoiler and program quality is done by using a statistical approach (that we plan to describe in detail in the talk).
The aim of the talk is to describe in detail the technique, stimulate feedback from the community and, also, search for volunteers spoilers and counter-spoilers.
Speakers: Riccardo Bernardini