The program MathMoon (version 0.3.1) was tested at the Nybo school (Tibro, Sweden) in May 2005. The pupils were given this questionnaire (in Swedish), which contains questions regarding the program as a whole, whether it was fun playing, the difficulty and the graphics. The results are presented below.
The survey was conducted among 69 pupils in grades 7 to 9 (approximately 13 to 16 years old). Pupils from the sixth grade were also supposed to participate, but they have not tested the program yet. 33 boys and 36 girls answered, most of them in the seventh grade.
| Boys | Girls | Total | |
|---|---|---|---|
| Grade 7 | 20 | 20 | 40 |
| Grade 8 | 3 | 5 | 8 |
| Grade 9 | 10 | 11 | 21 |
All in all the pupils thought that it was a good program. The pupils in seventh grade liked the program significantly better than those in the ninth, which (considering that the program is intended for the fifth and sixth grades) seems quite natural. Every fifth pupil in the seventh grade even thinks that the program is "very good".
| The program was: | Very good | Pretty good | Pretty bad | Very bad |
|---|---|---|---|---|
| 7th grade | 8 | 26 | 5 | 1 |
| 8th grade | 0 | 7 | 1 | 0 |
| 9th grade | 2 | 10 | 4 | 5 |
The girls liked the program somewhat better than the boys, and hardly any girls think the program was "very bad". One reason why the program was "very bad" was, as some of the boys put it: "no shoot-'em-up game". Boys in the ninth grade must be a difficult group to program for...
| The program was: | Very good | Pretty good | Pretty bad | Very bad |
|---|---|---|---|---|
| Boys | 5 | 19 | 4 | 5 |
| Girls | 5 | 24 | 6 | 1 |
| Total | 10 | 43 | 10 | 6 |
Making a mathematical program boring is easy. Multiplication, subtraction, addition and division really invites the programmer to create lots of boring tasks, comprised by nothing but numbers. More difficult is it if you, without skipping these, want to make a mathematical program fun. My strategy has been to alternate between "number knowledge" and different "graphical" questions which also test mathematical abilities. The room where one is to guess the weight of different animals is probably a good example, where it would be much more boring if I had just jumped straight to unit conversion. Instead, the result is that seven out of ten think the program is fun, and quite many think it is "really fun". As one would expect, the program is more popular amongst the younger pupils.
| Really fun | Pretty fun | Not fun | Not at all fun | |
|---|---|---|---|---|
| 7th grade | 7 | 25 | 7 | 1 |
| 8th grade | 0 | 6 | 1 | 1 |
| 9th grade | 2 | 10 | 6 | 3 |
The biggest difference between boys and girls is a greater spread among the boys. Both sexes have about the same amount of "fun", but the boys are more inclined to choose the extremes: "really fun" or "not at all fun". All in all I'm happy with the result.
| Really fun | Pretty fun | Not fun | Not at all fun | |
|---|---|---|---|---|
| Boys | 5 | 18 | 6 | 4 |
| Girls | 4 | 23 | 8 | 1 |
| Totalt | 9 | 41 | 14 | 5 |
The program should ideally be neither too hard nor too easy for the pupils it aims at, in the fifth and sixth grades. In this group it would be desirable if those who think it was "too hard" are approximately as many as those who think it was "too easy", and the majority should be somewhere in between. This survey is done on older pupils, and therefore we don't want anyone to think the program was "too hard".
We can see that the program was unequivocally too easy for pupils in the ninth grade: more than 50% think so, and the rest think it was "pretty easy". Somewhat unexpected, however, was the reaction in the eighth grade; they seem to think it was harder than the pupils in the seventh grade. I seem to have received a weak group of eights. Trying to guess how the program is perceived in grades five and six, I think the program suits its audience pretty well.
| Too hard | Pretty hard | Pretty easy | Too easy | |
|---|---|---|---|---|
| 7th grade | 0 | 11 | 19 | 10 |
| 8th grade | 0 | 3 | 3 | 2 |
| 9th grade | 0 | 0 | 10 | 11 |
I am personally content with the graphics. I have found some nice pictures that works well in the program. In the seventh grade, a vast majority agrees: the graphics are "good" or "very good". The graphics were, on the other hand, less popular in the ninth grade; especially the boys (who I assume are used to three-dimensional shoot-'em-up games) were sceptical. But that doesn't really matter, closer to the intended audience everyone was happy.
| Very good graphics | Pretty good graphics | Pretty bad graphics | Very bad graphics | |
|---|---|---|---|---|
| 7th grade | 15 | 22 | 1 | 2 |
| 8th grade | 4 | 1 | 2 | 1 |
| 9th grade | 2 | 11 | 4 | 4 |
| Very good graphics | Pretty good graphics | Pretty bad graphics | Very bad graphics | |
|---|---|---|---|---|
| Boys | 8 | 13 | 6 | 6 |
| Girls | 13 | 21 | 1 | 1 |
| Total | 21 | 34 | 7 | 7 |
The pupils were encouraged to comment different aspects of the program. There were some questions to help them commenting, concerning usability and plot, for example. The fact that not a single pupil commented the usability (e.g. the program's reactions on key presses and mouse clicks) could be a sign that the program is well programmed from a user's point of view.
16 pupils spontaneously mention that they think the program should be longer. Since this is only an early version of the program, it is highly likely that it will be much longer when version 1.0 is released.
Other comments were "more music", "no exciting shoot-'em-up game", "better images", "bad resolution" (but I don't think that has to do with the program, rather with the screen and graphics card), "the tasks were more fun than the other games", "just like every other mathematical game", "I don't like mathematics", "could be better", "looks very good", "It was fun, but not having to go back to the previous room" (if you fail certain tasks, you have to redo some of the previous), "too little time", "one learns new things while playing", "a bit too much time", "the instruction screens were boring", "could be more colourful" (it can be hard finding colourful images of the moon, but some images of the moon must reasonably be included), "if you fall down one room, there should not be the same tasks again", "I don't like that the amount of time is limited", "guessing animal weights was difficult" (but that was given as one reason why the program was pretty fun). One group frequently complained that the minus sign was too short, so that it could be mistaken for a multiplication sign. That is currently controlled by the operating system, but perhaps the program should specify the fonts to be used.