After the 150 sentence are collected, we will manually read and tag the entire sentence with 6 categories in feature and 2 categories in opinion. To increase reliability, we will evaluate the agreement of annotations using the Cohen's Kappa coefficient. It means that the sentences will be tagged by two people instead of one. The Cohen's Kappa coefficient, the proportion of agreement corrected for chance between two judges assigning cases to a set of k categories, offers as a measure of reliability. It intends to give the reader a quantitative measure of the magnitude of agreement between observers. The calculation is based on the difference between how much agreement is actually present compared to how much agreement would be expected to be present by chance alone. The following two tables illustrate the results for the inter-annotator agreement for the feature classifier and opinion classifier respectively:

A/B	General	D&C	Camera	Sound	Network	Battery	Null
General	22	6	0	0	0	0	4
D&C	8	37	0	0	0	0	2
Camera	1	0	15	1	1	0	1
Sound	1	0	0	16	1	0	5
Network	0	1	0	0	9	0	0
Battery	1	0	0	0	0	8	0
Null	3	6	0	1	2	0	10

A/B	Pos	Neg
Pos	69	8
Neg	9	23

The Cohen's Kappa coefficient for the feature classifier is 0.6587 and for opinion classifier is 0.6205. Both of them are above 0.6. This means that they are substantial agreement and can use it for evaluation. The following table gives the matching result as well as precision and recall for the feature classifier:

Test/standard	General	D&C	Camera	Sound	Network	Battery	Null	Precision
General	10	4	0	0	1	0	2	0.588
D&C	0	17	0	0	0	0	1	0.944
Camera	0	0	13	0	0	0	1	0.928
Sound	0	0	1	18	0	0	0	0.947
Network	0	1	0	0	5	0	0	0.833
Battery	0	0	0	0	0	5	1	0.833
Null	16	11	1	4	1	0	13
Recall	0.385	0.515	0.867	0.818	0.714	1

The average precision is 0.845 and the recall is 0.716. Both of them are acceptable. However, we observed that some specific category, such as category 1 (general) have a relative low recall. The reason of the low recall may be caused by the implicit feature. Implicit feature is some feature may not appear in sentences and it is more difficult to identify than the explicit feature. Implicit feature usually uses the adjective as feature indicator. For example, “The phone is small”. The word “small” indicates the size of the phone. For the category 1, the features are mostly the attribute of phone. It has a higher chance to encounter the implicit feature and make the low recall. The following table gives the matching result as well as the precision and recall result for the opinion classifier:

Test/Standard	Pos	Neg
Pos	64	14	0.821
Neg	9	10	0.526
	0.877	0.417

The average precision is 0.6735 and the average recall is 0.647. Although the result is not excellent, the result is still valid for our application. However, our application hasn't any algorithm to deal with the neutral opinion and objective fact. Thus, if our application classifies the opinion orientation into 3 categories, the result will drop significantly. In summary, the classifiers we built are valid and acceptable for our application. Let us use LG optimus 2x as the example for the qualitative evaluation. We use the LG optimus 2X because it is the first phone which embeds with dual core processor. There are three type of summary and they are radar chart, bar chart sentence summary respectively. These three types of summary is two side of the same coin including radar chart, bar chart, and sentence summary. The following show some of the examples for each one of those: