School of Science and Technology 科技學院
Computing Programmes 電腦學系
| Programme | Bachelor of Science with Honours in Computing |
| Supervisor | Dr. Oliver Au |
| Areas | Software Engineering and Cloud Computing |
| Year of Completion | 2011 |
The aim of this final year project is to investigate how FIT table enforce readability of requirement specifications and to investigate how programmer choose a specification framework concerning maintainability. This project includes two experiments in which readability experiment compares FIT tables and word document. Secondly, the maintainability experiment compares requirement specification in FitNesse and Concordion.
This project is initiated due to an observation that many software projects fail to meet the customer requirements. Pure word document and oval meeting may not share a common domain language between customers and programmers. Therefore, one aspect of this project is to investigate whether FIT tables, which are in tabular format, can express the business logic in a more readable way by it executable specification.
The objectives of this project have been defined as follows:
FitNesse is a wiki-based test execution engine that allows users to write, edit, and run FIT tests. It enables developers and customers to express acceptance tests in a tabular format and execute them from a web interface. FitNesse consist of three major parts in its framework. They are System Under Test (SUT), Fixtures and FIT tables. The following shows the relationships between FIT table, fixtures, test runner and system under test:
As stated in Lisa Crispin 2010, FitNesse is unique in using a Wiki for creating and maintaining test cases. The Wiki can be used as a knowledgebase and repository for story and theme requirements. The ease of installation and relative smooth learning curve are the reasons that I choose FitNesse. The original FIT uses html as the test template. The wiki format in FitNesse shows results in tabular format. Its core concept is similar to the following diagram:
The goals of our experiment were to determine whether FIT tables can substitute the role of plain text document and to evaluate which frameworks are more suitable for implementing test cases and writing test cases. A project was simulated to develop an Automatic Transaction Machine (ATM) system. This system allows users to login and perform transaction process.
We adapt ATM system originated from Joshua Chuang. The major reason is that he provide detailed word document which can serve as traditional software document. We consider this is valid for us because we can not write a representative document within a short period of time. We modify ATM system and make this as reference for comparison of frameworks and document formats.
The major tools used in this project include Eclipse plugin com.bandxi.fitnesse_v1.0.2, FitLibrary and Fixture, MYSQL database, the concordion-1.4.1 Junit and the Focuses.
FIT tables are designed for specifying all necessary requirements mentioned in the word document. Different representation formats would be considered in the FIT tables. After consulting with supervisor, the requirement specification which modified to the most readable case would be used to contrast with the original document in the experiments.
ATM system functionality would be preserved to a great extent. Some features such as timeout session would be eliminated for easiness of experiment. New functionalities such as new reward schemes are written for evaluating maintainability of FIT tables in case of requirement change.
Whether test cases are readable and maintainable should be judged by people. Both maintainability and readability would be examined in the experiments. Ideally, subjects of readability experiment would be year two computing students and subjects of maintainability would be year three computing students. Year two students only have to verify which documents is more readable while year three students have to perform coding for new requirement. All subjects in experiments should be freshman in FIT tables formats. The following describes the design of the two experiments:
| Content | Readability experiment | Maintainability experiment |
|---|---|---|
| Comparison | Document | Framework nature |
| Material | FIT tables word documents | FIT tables test cases in Concordion |
| Subjects | Year 2 computing students | Year 3 computing students |
| Sample size | 69 | 10 |
| Duration | 1 hour each subject | 2-4 hours each subject |
| Focus | FIT tables can/ can not substitute document role | Which framework is better for 1)design test cases 2) implement test cases as code |
| Reasons | Reasons | |
| Expected Result | Value/limitation of FIT tables | Value/limitation of frameworks Improvement in future |
The following shows the differences between the designed frameworks of FitNesse and Concordion:
| Content | FitNesse | Concordion |
|---|---|---|
| Mapping document contents to Java code | Heading rows in tables from implicit mappings | Explicit but hidden instrumentation in the document perform the mappings. |
| Storage of specification | Wiki | Java source folder |
| Integration | Separate runner | Run using Junit |
| Platforms | Java, .net, Python, Ruby, perl, Smalltalk,C++ | Java, .net, Python, Ruby |
| Rule | Test cases and code naming and format should obey different parsing criteria | should use Junit (assertTrue, assertFalse or assertEqual) to test |
The following describes the details of the implementation efforts in the experiements:
To make a trivial system for both frameworks to be functional
This task started from October to November.
To design test cases for readability experiment and maintainability experiment
I need to write same documents in two frameworks for comparison. While we use wiki to write test cases in FitNesse, we use HTML to write test cases in Concordion.
To write fixture codes to bridge specification to system under test
One hundred and twenty two forms were received back from subjects. Out of 122 questionnaires, 69 questionnaires were selected to be valid. To ensure responses from subjects are valid, we filter out questionnaires that subjects did not take this experiment seriously.
All the 69 survey forms were de-identified and individually numbered prior to data entry for result analysis. A 10% random sampling check for accuracy against the data in the original survey forms was conducted after the whole data entry process had been completed. No apparent error was detected by the sampling check on forms numbered 2, 19, 25, 38, 46, 59, 64.
Our readability experiment has six questions. The first two questions are concerned with the document clarity. They are listed as follow:
Question 1: The description of FIT table was clear. (1-5)
Question 2: The description of original document was clear. (1-5)
A 5-point Likert scale was used for ranking the clarity of two documents.
1 ->Strongly disagree, 2 ->Disagree, 3 ->Not Certain, 4 ->Agree and 5 ->Strongly agree
We summarized these results for question 1 and question 2 as follow:
We observe that 39 subjects agree or strongly agree that FIT tables are clear. 18 subjects agree or strong agree that word document is clear. Most subjects concern that FIT tables are clear while they do not sure the word document is clear enough.
For question 3, we ask them to determine which document give them better understanding on system functionalities. From this question, we can see more subjects choosing FIT tables. The result is shown below:
Concerning system functionally, the information in FIT tables and word document version is the same. We are interested in how programmers make their decision on previous question. Therefore, we have follow up question to ask the reasons behind this decision. Since the questions are open ended. We summarized them into following categories:
a. show result /problem clearly with distinguish color
b. can more understandably present test cases with many data values
c. FIT tables are easier to read and clear
d. FIT table provide cases from user perspective/ more focusing
e. FIT is more interactive/repeat many time
f. personal preference to web page format
Subjects respond that FIT tables are more users oriented and understandable. It reflects the
potential usage of FIT tables to simulate functional requirements in system. The result for question 4 is shown below:
Apart from knowing why people choose FIT tables, we also concern why people choose word
document. Their response can reflect the limitation of FIT tables, which help us to make
suggestion. In question 5, there are 21 people that choose word document as better document. Their reasons are summarized as follow:
a: More familiar with reading document than form
b. The document format is clearer with diagram
c. There are troublesome when using FIT system
d. The word document is more detailed/ accurate
e. The word document has different method to present data and classes
f. FIT table require more programming knowledge to understand
g. Not enough time to explore FIT tables benefit
h. Easy to find information in word document
Most subjects think word document include more detailed information. They make this conclusion because they concern charts in word document are valuable. They think test cases in FIT tables are not enough for them to understand system completely. They want to know non functional requirement such as performance/scalability which has been included in word document. The result for question 5 is shown below:
Finally, for question 6, we want to know whether FIT table can substitute the role of traditional word document in requirement definition. This is also the core question we want to answer in this experiment. The result is nearly half and half which is shown below:
From readability experiment, most people think FIT tables are clearer than word documents. Subjects also respond positively that FIT tables give them better understanding on system functionalities. However, there is no strong tendency that subjects believe FIT tables can replace the role of traditional word document. It is interesting that even subjects think FIT tables give them better understanding on system functionalities, they don’t believe FIT tables can replace the role of traditional document completely.
Subjects think FIT tables can present functional requirements well by showing the desired output and actual output. Problem and result can be easily detected with the help of FIT tables. They also recommend that FIT tables work well in programmer to programmer communication.
Another task aims at simulating programmers create test cases in two different frameworks. The questionnaires have stated the purpose as follow:
Case 1: To calculate the score based on amount calculation in different factors
Case 2: To simulate the a group of people combination in room. Certain people can not live with other in the same room.
To allow easier analysis, we focus on four areas including display format of result, easiness of writing specification, easiness of organizing test case and source code readability. We calculate the average among all subjects’ responses. We can see FitNesse only score higher than Concordion in easiness of organizing test cases. Concordion scores higher than FItNesse in other three categories. The result is as follows:
Finally, we want subjects give a satisfaction mark to both frameworks from two perspectives. Again, Concordion scores higher than FitNesse. The result is shown below:
We had two experiments in this project. The survey is to find out information to answer some questions that we are interested in. We found table format in FIT tables may not be natural to all subjects. The conventional reading habit and additional programming knowledge are dominant reasons that subjects think FIT tables cannot substitute the role of plain text document. It seems harder to overcome these barriers considering FIT tables must involve certain programming knowledge to handle them.
We can conclude that FIT tables can complement the role of plain text document. Subjects respond positively to this idea. From their responses, FIT tables can present functional requirement and use cases which provide solid examples to word document description. The idea of checking the system functionaries in early stage is also attractive to them.
Related to framework comparisons, we cannot say which framework can substitute other one. They have different user targets. From our experiment, we can see programmers are much familiar with Concordion than FitNesse. They have this preference because test cases are written in HTML format in Concordion. We do not sure if this situation revises if FitNesse improve its format control. We can conclude that programmers should choose Concordion at that moment.
Our experiment design is qualitative. We consider this design is good for exploratory purpose to find limitation of FIT tables and frameworks. In future, we hope there would be quantitative experiment to gain statistics information. We suggest future work can be done on how difference between frameworks affects programmer productivity. From our experiments, we know the differences and subject respond. Our results can be served as foundation of future researches.
Copyright Fung King Ho and Oliver Au 2011
