The research paper published by IJSER journal is about Content Analysis of Rubric-based Online Discussion Forums for MIDT Students in Open University Malaysia 1
ISSN 2229-5518
Content Analysis of Rubric-based Online Discussion Forums for MIDT Students in Open University Malaysia
Dr.Purushothaman Ravichandran, Prof.Dr.Abtar Kaur
Abstract—In order to establish effective online communication in a learning management system (LMS), students need to be motivated to participate in discussion forums. Such motivation may be stimulated by the use of a Learning Management System (LMS), which i mprove access to learning materials, the provision of more timely feedback to students through on -line assessment (Breen, Cohen, and Chang,
2003), and improved communication among students and between students and faculty through the availability of bulletin boards , discussion forums and email facilities (Beard and Harper, 2002, Kang, 2001). However, it has always been a challenging task for a teacher, as to how to scale the students’ interactions within the discussion forums. In order to track the effectiveness of online co mmunication, content analysis servers as a technique, that enable researcher to study human behaviour in an indirect way through an analysis of their communication. (Fraenkel and Wallen, 2006). As such, data from 14 students from 11 different countries, who enrolled for post graduate course (MIDT) at Open University Malaysia (OUM), were used in this study, to find the significance of their online discussion forum (ODF) contributions against their final scores. In order to predict if students’ final scores did confirm linearly or not to forum scores, the predicted final score and the error associated with regression model were calculated. The normality test results (Shapiro-W ilk) indicated that forum score is normally distributed (significance value >0.05) and the final score showed some deviation from normality (significan ce value
<0.05). Also, the F-test results from this study indicated, that the regression model is significant in explaining the variance in the final scor e using the forum score as the predictor variables, F (1,12) = 9.469, p<0.01. The regression coefficient indicated that the final score increased by 1.09% units with each 1% unit increase in the forum score.
Index Terms— LMS, MIDT, OUM, Online assessment, Bulletin board, ODF, Content analysis.
—————————— ——————————
1 INTRODUCTION
IIn Open University Malaysia, OUM LMS serves as a com- munication tool for delivering the online course and to organ-
ize fully asynchronous discussion forums for the Master of Instructional Design and Technology (MIDT) programme. Since the discussion forum tool has the capacity to enable the facilitator to initiate high level asynchronous discussions, it is important to identify such interactions and assess the signific- ance for meaningful learning. In order to establish effective online communication, students need to be motivated to par- ticipate in discussion forums. Such motivation may be stimu- lated by the use of Learning Management System (LMS) as a result of improved access to learning materials, the provision of more timely feedback to students through on-line assess- ment (Breen, Cohen, and Chang, 2003), and improved com- munication among students and between students and faculty through the availability of bulletin boards, discussion forums and email facilities (Beard and Harper, 2002, Kang, 2001). Fur- ther, as mentioned by Fraenkel and Walen (2006), content analysis is a technique that enables a researcher to study hu- man behavior in an indirect way through an analysis of their communication. Thus content analysis becomes imperative in online discussion forums to gauge a student’s understanding of subject matter and critical thinking.
2. LITERATURE REVIEW
The potential of online discussion forums has always been a much debated area. For example, researchers have reported that messages were often left unanswered by fellow students (Vonderwell, 2002; Eliis 2001). At the same time, Vonderwel
(2002) narrates that, it promotes a collaborative learning envi- ronment where learners interact by negotiating, debating, re- viewing and reflecting upon existing knowledge, and are able to build a deeper understanding of the course content. Thus we can see two different statements from the same research- ers’ perspective about the impact of discussion forum in an collaborative learning environment. However, with the in- crease in the use of a Learning Management System (LMS), online discussion forums seem to have gained popularity in capturing the communication process between teacher- student, student-student, and student-teacher. As such, as mentioned by Schrire (2006), understanding the discourse that occurs in online discussion forums in the teaching-learning context requires some methodological approach to measure and analyze the data and information. However an earlier body of research is more positive in its description of online forums on meaningful learning. Specifically, these studies in- dicated that participation in online forums could lead to broader and deeper participation in group activities (Kiesler, Siefel, & McGuire,1984; Pullinger, 1986; Spitzer, 1989, as cited in Mazur, 2004). Therefore, it seems important to examine the impact of discussion forms with respect to students final score scored by them and look into the various parameters that promote meaningful learning.
3. METHOD
In this study, the discussion forums were conducted in a LMS- like social tool called Ning . This web portal was designed for the subject “Current Trends and Issues in Instructional Tech- nology”, which had 14 students enrolled for the full online
IJSER © 2012
http://www.ijser.org
The research paper published by IJSER journal is about Content Analysis of Rubric-based Online Discussion Forums for MIDT Students in Open University Malaysia 2
ISSN 2229-5518
Master of Instructional Design and Technology (MIDT) course, offered by Open University Malaysia. Out of the 14 students, there were 5 women and 9 men from different coun- tries. Discussion board participation was worth 20% of the final grade and consisted of discussions on the assigned read- ings, instructional design topics, and peer-to-peer reviews. The rubrics that were used to assess the online discussion fo- rums postings were based on the assessment criteria framed by Palloff and K Pratt, Jossey-Bass, John Wiley & Sons (2003) and that of Palmer, Holt, and Bray (2008). This is because, the online assessment for this study was completely based on asynchronous mode of communication, which used rubrics to capture the quality of postings, contributed in the discussion forum. Thus the rubrics for this study were based on the fol- lowing criteria.
4. SOURCE OF DATA
Since the main objective of this study was to investigate, if there exits any correlation between the students’ scores allot- ted for their contribution in the discussion forum against their final scores for the said course, the main source of data for this study was from the discussion form created using Ning portal (http://hmid6303.ning.com). The discussions contributed by the students were based on the book "The World is Open" by Professor Curtis Bonk. The book consisted of 12 chapters, which was referred by the author as “openers”. Omitting the introduction chapter, the entire book had 11 openers and the period of assessing the student’s contribution in the discussion forum pertaining to each opener began from 11th Jan 2010 until 30th April 2010.
5. FINDINGS
1. The normality test results (Shapiro-Wilk) indicate that fo- rum score is normally distributed (significance value >0.05) and the final score shows some deviation from normality (sig- nificance value <0.05). However, given the small sample size we can assume that the final score is also close to normal dis- tribution.
Further, the F-test (statistical test used here to measure the overall significance of the regression model) results indicate that the regression model is significant in explaining the va- riance in the final score using the forum score as the predictor variables, F (1, 12) = 9.469, p<0.01. In addition, the model pa- rameters suggest that the model (using forum score as a pre- dictor) is able to explain 44.1% variance in the final score. The regression coefficient indicates that the final score increased by
1.09% units with each 1% unit increase in the forum score.
2. The scatter plot diagram shows that the variables are linear- ly related.
If we have reason to believe that two variables are related, we can plot data values for these variables in a graph, called a scatter diagram. A scatter diagram is a graph, which show the relationship between two quantitative variables measured on the same individual. Each individual in the data set is represented by a point in the scatter diagram.
The predictor variable is plotted on the horizontal axis and the response variable is plotted on the vertical axis. If the scatter diagram roughly resembles a line (the points of the scatter diagram cluster roughly in a straight line), than there could be a linear relationship between the two variables.
Fig. 1Scatter plot showing variation of final score with the fo- rum score
3. The condition of homoscadasticity is also satisfied (ie. resi- dual variance is equally distributed or normally distributed).
Further, model parameters give the properties of the regres- sion model. In this study, the regression model is FINAL SCORE = 1.09*(FORUM SCORE) + 24.81.
4. Also, in order to find if the students whose final scores do not confirm (linearly) to forum scores, the predicted final score and the error associated with the regression model were calcu- lated. THE RESIDUAL = (PREDICTED VALUES - THE ORIG- INAL VALUE). The predicted final score and the error asso- ciated with the regression model are shown in Table 1.
Table 1: The predicted final score and the error associated with the regression model
IJSER © 2012
http://www.ijser.org
The research paper published by IJSER journal is about Content Analysis of Rubric-based Online Discussion Forums for MIDT Students in Open University Malaysia 3
ISSN 2229-5518
*The results indicate that the error associated with 50% of the respondents is greater than 20%.
6. SUMMARY
The regression model shows FINAL SCORE = 1.09*(FORUM SCORE) + 24.81, can also be used as a key indicator for online facilitators to identify the relation between the online discus- sion forum scores and the final scores. That is, the regression coefficient indicates that the final score increases by 1.09% units with each 1% unit increase in the forum score. Also the predicted final score and the error associated with the regres- sion model shows significant deviation, that is, the results in- dicates that the error associated with 50% of the respondents is greater than 20%.
7. CONCLUSION
The success of any online course relies on how efficiently the facilitators anchor the discussion forum to bring in meaningful and quality discussions. In order to assess the online discus- sions, facilitators use many methods, either qualitative or quantitative assessment methods. While quantitative assess- ment on discussion forums are widely used, it becomes im- perative to use qualitative assessment, as in-depth content analysis techniques in the form of rubrics can be used to cap- ture the actual quality of postings. This study showed a sig- nificant correlation between the discussion forum postings against the final score of the students. That is, FINAL SCORE
= 1.09*(FORUM SCORE) + 24.81. Further, the regression mod-
el is significant in explaining the variance in the final score
using the forum score as the predictor variables. Also the re-
gression model shows significant deviation, that is, the result indicates that the error associated with 50% of the respondents is greater than 20%. Although this study captures findings on rubric-based discussion forum assessment, the sequence of content analysis techniques that were used seems to make this study very unique. Further, capturing these discussion forum postings based on the rubrics is a challenging and time con- suming process. However, this study seems to serve as an eye opener for all those who wish to assess rubric-based online discussion forum postings. Further work is needed, as listed below.
(a) Find if the same correlation exits within a larger sample size.
(b) Comparing this study result with different rubrics.
(c) Find if the error associated with 50% of the respon-
dents is greater than 20% or not for a larger sample size.
(d) Find if their exits any correlation between the critical
thinking ratio of the respondents against their final scores.
REFERENCE
Beard, L. A., & Harper, C. (2002). Student Perceptions of Online
Versus On Campus Instruction. Education, 122, 4,
658-663.
Breen, L., Cohen, L., & Chang, P. (2003). Teaching and learning
online for the first time: Student and coordinator perspec-
tives. Paper presented at the Partners in Learning: 12th
Annual Teaching Learning Forum, Edith Cowan Uni-
versity, Perth.
Ellis, A. (2001). Student-centered collaborative learning via face-to-
face and asynchronous online communication: What’s the
difference? Proceedings 18th ASCILITE Conference
Melbourne, 9-12 December.
http://www.ascilite.org.au/conferences/melbourne01/
pdf/papers/ellisa.pdf
Fraenkel, J.R., & Wallen, N.E. (2006). How to design and evaluate
research in education. New York: McGraw-Hill.
Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical in-
quiry in a text-based environment: computer confe-
rencing in higher education. The Internet and Higher
Education, 2(2–3), 87–105.
Kang, S. (2001). Toward a Collaborative Model for the Design
of Web-Based Courses. Educational Technology, 41
(2), 22-30.
Kiesler, S., Siefel, J. and McGuire, T. (1984). Social
psychological aspects of computer- mediated
communication. American Psychologist, 39, 1123–
1134.
Mazur, J. (2004). Conversation analysis for educational tech-
nologists: theoretical and methodological issues for
researching the structures, processes and meaning of
on-line talk. In D. H. Jonassen (Ed.), Handbook for Re-
search in Educational Communications and Technology,
2nd Edition. Mahwah, NJ: Lawrence Erlbaum Associates.
Palloff, R., & Pratt, K. (1999). Building learning communities in
cyberspace: Effective Strategies for the Online Classroom.
San Francisco: Jossey-Bass.
Palmer, S., Holt, D., & Bray, S. (2008). Does the discussion
help? The impact of a formally assessed online dis-
cussion on final student results. British Journal of Edu-
cational Technology, 39(5), 847-858.
Pratt, K. (2003). The Virtual Student : A Profile and Guide to Work-
ing with Online Learners. San Francisco: Jossey-Bass.
Schellens, T., & Valcke, M. (2003). Collaborative learning in
asynchronous discussion groups: the impact on cog-
nitive processing. Paper presented at the Annual Meet-
ing of the American Education Research Association, Chi-
cago, IL.
Schrire S. (2006). Knowledge building in asynchronous
discussion groups: Going beyond quantitative
analysis. Computers and Education 46, 49-70.
Spitzer, M. (1989). Computer conferencing: An emerging tech-
nology. In G. Hawisher & S. Selfe (Eds.), Critical pers-
pectives on computers and composition instruction. (pp.
187–199). New York: Teachers College Press.
Vonderwell, S. (2002). An examination of asynchronous com-
munication experiences and perspectives of students
in an online course: A case study. The Internet and
IJSER © 2012
http://www.ijser.org
International Journal of Scientific & Engineering Research Volume 3, Issue 8, August-2012 4
ISSN 2229-5518
Higher Education, 6, 77-90.
IJSER 2012
http'lfwww .ijser.org