2016
Exploring Quality Teaching
in the Online Environment Using an Evidence-Based Approach
Elena Prieto-Rodriguez
The University of Newcastle, Australia,
elena.prieto@newcastle.edu.au
Jennifer Gore
The University of Newcastle, Australia,
jenny.gore@newcastle.edu.au
Kathryn Holmes
The University of Western Sydney, Australia,
k.holmes@westernsydney.edu.au
Recommended Citation
Prieto-Rodriguez, E., Gore, J., & Holmes, K. (2016). Exploring
Quality Teaching in the Online Environment Using an Evidence-Based
Approach. Australian Journal of
Teacher Education, 41(8).
Retrieved from http://ro.ecu.edu.au/ajte/vol41/iss8/2
This Journal Article is posted at Research Online.
http://ro.ecu.edu.au/ajte/vol41/iss8/2
Exploring Quality Teaching in the
Online Environment Using an Evidence-
Based Approach
Elena
Prieto-Rodriguez
Jennifer
Gore
The University of Newcastle
Kathryn
Holmes
University of Western Sydney
Abstract: Online learning is
increasingly ubiquitous in higher education. However, research regarding online
teaching often focuses on the affordances of the online environment rather than
on the quality of pedagogy. In this paper we consider how online learning could
be enhanced using rich pedagogical models that are consistent with a wealth of
existing knowledge on pedagogy for face-to-face settings. To do so, we apply an
established framework, the Quality Teaching model, to explore pedagogy in the
online environment and illustrate its potential benefits using a case study of
60 students in a tertiary mathematics teacher education program. We conclude
that the use of an evidence-based pedagogical model can help guide online
instructors in the development of high quality online courses.
Introduction
There is ample evidence that teaching quality is a
key determinant of student learning outcomes during schooling (Darling-Hammond,
1999; Fullan, 2007; Kyriakides, Christoforou & Charalambous, 2013).
However, in the higher education setting, particularly in the context of online
learning, the evidence supporting a similar link is less robust. Moreover,
given that technology acts as a mediator of the teaching and learning
experience within online learning scenarios, methods commonly employed to
determine the quality of teaching in face-to-face settings, particularly as it
impacts on the learning experience of students, are often seen as unsuitable in
this environment (Ginns & Ellis, 2007).
Defining and evaluating the quality of teaching in
online learning environments, which many have characterised as substantially
different from traditional classrooms, is a central focus of recent educational
research in online teaching (Garrison, 2011). Several examples of instructional
principles for courses were developed early for the online medium with clear
guidelines for staff-student interactions, encouraging cooperation and active
learning, giving prompt feedback, and setting clear deadlines (Graham,
Cagiltay, Lim, Craner, & Duffy, 2001). Whilst many of these earlier
guidelines acknowledge pedagogy as important, they have tended to focus on the
affordances of the specific online environment such as accessibility,
communication, reliability of the interface, and bandwidth demand (Herrington,
Herrington, Oliver, Stoney & Willis, 2001). More recent work centres on
general pedagogical principles that would be applicable across any online
delivery system (Kidney, Cummings, & Boehm, 2014; Margaryan, Bianco &
Littlejohn, 2015). Some authors argue for more work in the online context in
order to “inform learner outcomes, learner characteristics, course environment,
and institutional factors related to delivery system variables in order to test
learning theories and teaching models inherent in course design”
|
Vol 41,
8, August 2016
|
22
|
(Tallent-Runnels et al., 2006, p.93). Indeed, some
argue specifically that the development of such models could benefit from
drawing on the existing wealth of well-established research on classroom-based
pedagogies (Haythornthwaite & Andrews, 2011).
A major issue identified in early research on online learning when
utilising comparisons with face-to-face teaching is that in non-classroom based
environments the notion of a ‘lesson’ is substantially different (De Wever,
Schellens, Valcke, & Van Keer, 2006). Lessons in online and blended
environments can be understood as ‘units of work’ delivered via a range of
media including discussion forums, blogs and individual email communication
with the teaching academic. Hence, it is primarily the interactions occurring
through these media that can be studied and analysed. Typically, interactions
are student– student, student–teacher, or student–content, and it has been
found that there is an association between the frequency of interactions and
increased student achievement (Bernard et al., 2009; Tamin et al., 2011). Many
different instruments and measures are available to analyse the content of
online interactions, although concerns about the validity and reliability of
some of them have been raised (De Wever et al., 2006).
The purpose of the study reported in this paper is to explore the
applicability of an evidence-based approach to evaluating pedagogy in
classrooms for the review and refinement of teaching in online and blended
environments for pre-service teachers. In so doing, we explore the potential of
pedagogical frameworks for informing the improvement of teaching in the online
environment.
We undertake this exploration using the Quality
Teaching (QT) model (NSW Department of Education and Training, 2003b), a
conceptually and empirically robust model guiding developed to guide the
quality of teaching in primary and secondary schools. Very minor adjustments to
the wording of the coding instrument (NSW Department of Education and Training,
2003a) were made in order to apply the model to the specific features of
‘lessons’
and ‘interactions’ in the online environment. We use a case study of two online
courses to illustrate how a pedagogical model, in our case the QT model, can be
used to analyse teaching in the online environment. The elements that
constitute the model, described in the following section, provide a strong and
accessible conceptual basis and set of principles for guiding course
development and interactions online. These principles have been shown in
face-to-face environments to be linked with improved teaching, improved
outcomes for students and narrowing of equity gaps for students from
traditionally under-represented groups and equity target groups (Amosa, Ladwig,
Griffiths & Gore, 2007).
The Quality Teaching Model
Quality teaching and how we define and recognise it
has been the object of research for many decades. In the most populated state
in Australia, New South Wales (NSW), the Department of Education and Training
commissioned the development of a model for teaching quality, comprehensive in
scope, and applicable across all subject areas and grade levels, as a framework
for teachers’ professional self-reflection and for school improvement
practices. This Quality Teaching model (NSW Department of Education and
Training, 2003b) is now well established in NSW and Australian Capital
Territory public schools and there is growing evidence of its efficacy for
improving teaching and student learning outcomes (Gore, 2007; Gore, 2014; Gore
& Bowe, 2015; Ladwig, Smith, Gore, Amosa, & Griffiths, 2007).
The QT model is a refinement of
the Productive Pedagogies model (Hayes, Lingard, & Mills, 2000) which in
turn was an extension of Authentic Pedagogy (Newmann, 1996). It features
teaching practices that have been linked to improved student outcomes and can
be
Vol 41,
8, August 2016 23
characterised as representing three dimensions of pedagogy: pedagogy
that promotes high levels of intellectual quality, pedagogy that promotes a
quality learning environment, and pedagogy that develops and makes explicit to
students the significance of their work (NSW Department of Education and Training,
2003b). Each of these three dimensions is elaborated through six elements as
detailed in Figure 1. For brief explanations of each element please see
Appendix 1.
Figure 1.
Elements and dimensions of the Quality Teaching model (NSW Department of
Education and Training, 2003a, p.10)
Studies using the QT model are often designed around the observation of
teachers and their interaction with students in the classroom (NSW Department
of Education and Training, 2003a; Gore et al., 2015). However, as we argued in
the introduction, the ‘observation’ of virtual classroom practices requires a
different approach, involving the observation of student-student,
student-teacher, and student-content interactions through the systematic
analysis of discussion forums and other forms of online communication. In this
paper we use the QT model for analysing interactions in a virtual environment
in higher education. This investigation extends previous research which found
the model to be an effective tool with which to analyse the quality of
assessment practice in the social sciences in the tertiary setting (Gore,
Ladwig, Elsworth, Ellis, Parkes & Griffiths, 2009).
Case Study: Mathematics Online
Teaching mathematics online is a relatively new practice in which
educators need to be aware not only of the affordances of the online medium,
but also of issues inherent to mathematical concepts such as notation or the
highly structured way in which concepts need to be scaffolded. These issues all
play a major role in how courses are designed and delivered (Threlfall, Pool,
& Homer, 2013). In pedagogical terms, Engelbrecht and Harding (2005) point
out that while “little has been done in developing a pedagogy for online
mathematics courses, there are some clear guidelines. Care should be taken to
have a sound balance between teacher- and learner-centred activities and that
interaction should be carefully planned; interaction between learner and
content, between learner and instructor and between learner and learner” (p.
254). This is not specific to mathematics; in fact identical considerations are
used for all learning areas with the proviso that evidence-based approaches
Vol 41,
8, August 2016 24
for improving online learning are used (Abrami, Bernard Bures,
Borokhovski, & Tamim, 2011).
Our case study focused on the teaching of
mathematics in the online environment to explore how a pedagogical model, the
QT model, can be used for interpreting and evaluating teaching practices. Case
studies are generally undertaken to ‘describe, explain or evaluate particular
social phenomena’ (Gall, Gall & Borg, 2005, p.306) and in doing so, they
aid in understanding complex social situations. In this study we use the lens
of the QT model to systematically analyse the pedagogical features evident in
two online courses. The two courses form part of a Master’s level program,
intended for practising teachers and other educators who wish to gain
postgraduate teaching qualifications in mathematics education. The student
cohort we focused on for this study was comprised of 60 practising teachers who
were re-training to be high school mathematics teachers. There were 34 female
participants in the study (56.7%). The backgrounds of the students who enter
this program are diverse, but most are secondary school teachers who have
previously specialised in areas other than mathematics and believe that
re-training in mathematics will provide better career prospects at a time when
there is a shortage of mathematics teachers.
We selected two concurrent semester-long courses in the Master’s program
for the case study that had recently been updated with new technologies.
Previously, these courses had been taught online asynchronously, whereby
students were sent textbooks and other text-based course materials and asked to
submit two written assignments and sit a final exam. In this earlier version,
students were able to communicate via email with teaching academics to seek
help with mathematical concepts or request feedback. They also participated in
discussion board tasks in response to instructor prompts. In the revised
offering of the courses we aimed to provide a wider range of online learning
experiences for students. To do so, we utilised a range of digital resources
available for online teaching and assessment to enable interactions of students
to occur with each other and with the instructor in synchronous and/or
asynchronous fashion (Holmes, 2005).
The two courses were focused on mathematical
concepts. The first (Course 1) focused on Calculus concepts including limits
and continuity, derivatives and basic integration. The second (Course 2),
contained elements of Number Theory, Combinatorics and a thorough introduction
to Complex Numbers including their geometrical applications. Course 1 was
undertaken by 41 students (46% female) and Course 2 by 50 students (62%
female). There were 31 students who studied both courses (51% female).
In general, the teaching of mathematics in online
environments centres heavily on the mathematical concepts to be delivered.
Using the constructs of the QT model (see Appendix 1), the emphasis is
customarily on the Intellectual Quality dimension of the teaching. In
particular, the elements of Deep Knowledge, Deep Understanding, and Higher
Order Thinking are often favoured. When we set out to improve the online
delivery of the two courses in our case study, guided by the pedagogical
principles underpinning the QT model, our primary focus was to also achieve a
Quality Learning Environment, where Explicit
Quality Criteria, Engagement, High Expectations, Social Support,
Students’ Self-regulation and Student Direction would be more deliberate in our
teaching. We also aimed to increase the Significance of the concepts we taught
by including Narrative and Cultural Knowledge in our course design.
To progress towards an improved Quality Learning
Environment and increased Significance, we produced two types of resources
during the intervention. On the one hand, and to facilitate Students’
Self-regulation, Engagement and Social Support, a series of resources were
either specifically created for the course or externally sourced from the
Internet. Externally sourced materials comprised two open-source text-based
mathematics
Vol 41,
8, August 2016 25
books and many short videos and interactive demonstrations covering most
of the topics in the course. Our internally produced materials included:
A course blog, which integrated many of the externally sourced short
videos and interactive demonstrations. Our pedagogical aim with the course blog
was to promote Engagement using a Narrative created by the lecturer and to
include elements of the world history of mathematics thus promoting Cultural
Knowledge.
Pencasts, i.e. interactive documents containing both written text and
voice, were provided to students on request, thus promoting Student Direction,
and frequently involved further explanations of mathematical problems raised in
units of work, thus promoting Deep Understanding.
Discussion forums, aimed at promoting Social Support, were designed to
enhance Intellectual Quality through discussion of the concepts in each of the
units of work. The pencasts referred to above were included in these forums.
In
addition, to ensure Explicit Quality Criteria and High Expectations, we created
a
series of
assessment tasks to be submitted fortnightly by made available to students from
the beginning of the course. In previous iterations of these courses we had allowed
students to submit handwritten mathematical work covering all topics at the end
of the semester. This time we opted for two different forms of assessment to
make better use of the online environment: a timed multiple-choice test to
check for basic understanding of the topics presented in the preceding two
weeks, and a more challenging long-response question to ensure Higher Order
Thinking and Deep Understanding of the topics. The second task differed from
previous years as it was to be typed, and plotted if necessary, with
mathematical software provided to all students prior to the commencement of the
course. This type of formative assessment was designed to help these teachers
develop skills needed in modern-day technology-rich mathematics classrooms. The
software used was suggested by practicing mathematics teachers who expressed
how beneficial it would have been to learn to use it during their pre-service
years.
Methodology
All data collection occurred in the first semester of 2013. Our case
study comprises two separate courses, with different instructors, each
involving six ‘units of work’. Each ‘unit of work’ was delivered fortnightly to
students. Our university learning management system allows the running of
analytics concerning use of the different resources included in the course, and
we used these as the starting point for our analysis. In a previous paper
(Prieto & Holmes, 2014) we presented a comprehensive analysis of the
student activity and how it correlated with student achievement in the course,
finding a positive relationship between the time spent within the learning
management system and achievement in the course. In this previous paper, we
used Engelbrecht and Harding’s (2005) framework to classify the interactions
between students and academics in online mathematics courses. Their framework
is comprised of seven factors ranging from instructor facilitation to internet
resources, focusing mainly on the affordances of the online environment rather
than on the pedagogical approach to teaching online.
In this paper we analyse only the content of the
forums where students discussed the units of work. By content we mean all
written interactions occurring within the learning management system in the
fortnightly period when that unit was delivered. As discussed in the
introduction, content analysis of online interactions is often carried out by
creating instruments specific to the online environment. However, in their
review of content analysis schemes for discussion groups in online teaching, De
Wever et al. (2006) expressed concern
Vol 41,
8, August 2016 26
about the
instruments used. In particular they referred to the empirical validity of
instruments and the reliability of coding. Our methodology addresses those
concerns. By utilising the QT model, which has been systematically tested in
classrooms, we use a coding instrument tested and validated in pedagogical
research (Gore, 2007). The coding scales, used in relation with the focus
question for each element as well as descriptions of each element/construct,
are detailed in Appendix 2.
To address issues of reliability in our study, the
coding of content in the 273 posts contained in the discussion forums was
undertaken by an experienced coder using the QT model. A random sample of 44
posts were independently doubled coded by another experienced coder, achieving
an initial agreement of 85%. Subsequently, the two coders discussed the
disparities and came to an agreed ‘best’ code for all posts.
The forums were downloaded from our University’s
learning management system by using its “Collect” functionality and then
imported as text files into QSR NVivo 10 for coding and analysing purposes
(NVivo qualitative data analysis Software; QSR International Pty Ltd. Version
10, 2012). For each unit of work the coding was conducted by highlighting
portions of text in the discussion board corresponding to that unit, and coding
it in relation to relevant elements of the QT model. The codes were
pre-determined by creating “’Nodes’ within the NVivo environment. Double (or in
some cases triple) coding was allowed since several elements could be present
in the same portion of text. Examples of these extracts are given in the
Results section. This approach enabled us to produce an accurate analysis not
only of the degree to which the different elements in the QT model were evident
in students’ interactions, but also the amount of text devoted to each of the
elements as a percentage of the total amount of text students wrote in the
forums. NVivo’s analytical tools enabled us to determine the amount of text
that each of the different QT elements represented by using the Node summaries.
It also enabled us to see the proportion of text coded to each element relative
to the whole text for each forum examined.
We only report here the interactions occurring in
the forums that were not part of formal assessment for the courses. In other
words, the focus of our study was on those online interactions that mimic the
informal, but nevertheless crucial, connections between students and teachers
which occur as part of typical face-to-face instruction. The discussion forums
were organised with three different foci: assessment questions, mathematical
questions, and miscellanea. The first forum was designed for students to
communicate with their lecturer about all matters relating to assessment of the
course, including mathematical questions which were part of their assignments.
The second two forums were also monitored by the lecturer, but were used mainly
by students to communicate amongst themselves, sharing resources and ideas or
extending their learning beyond the course syllabus. All three forums have been
analysed for this study. All individuals were de-identified for the purpose of this
research. Human Research Ethics Committee approval at our institution was
obtained (Approval No. H- 2013-0023), with active consent from students for us
to anonymously report on their answers.
Results
In this section we provide results organised according
to three different perspectives. First, we focus on the overall coding of each
element in the QT model for each of the six units in both courses to get an
overall picture of the pedagogical quality of the courses. Next, we analyse the
proportion of text in the forums coded for each of the QT elements, and argue
the limitations of this type text analysis for online forums. Lastly, we focus
on in-depth content analysis of the text in the forums for each element of the
QT model.
Vol 41,
8, August 2016 27
Overall QT Coding
The agreed QT coding of the content in discussion forums for the six
units in each of the two courses is presented in Figure 2. It is apparent that the scores given for each QT element were consistent
across units demonstrating that certain features of the Quality Teaching model
are more evident in these online courses than others. In particular, the
elements of Deep Knowledge, Substantive Communication, Explicit Quality
Criteria, Student Support, Student Self-Regulation and Inclusivity, scored
highly in all units in both courses. In contrast, there was very little
evidence found of Higher-order Thinking, Metalanguage, Cultural Knowledge and
Narrative. In terms of the QT dimensions, Quality Learning Environment and
Intellectual Quality consistently scored higher than Significance across all
units of work.
Figure 2. Coding of all units of
work in Course 1 and Course 2
Vol 41,
8, August 2016 28
The mean scores for each element in each module of
each course are displayed in Figure 3. It is clear that although the courses were conducted independently by
two different instructors, the overall pattern of scores is highly consistent
across all 18 elements (r=0.96, n=18, p=.000).
Proportion of Text
Another measure of the prevalence of the QT
elements across units is the percentage of text explicitly devoted to each
element in the discussion boards. This measure could be considered equivalent
to measuring the proportion of class time ‘devoted’ to each of the QT elements.
The analysis was undertaken using text analysis
software by highlighting portions of text and coding them against one or more
QT elements as explained in the Methodology section. The software takes into
consideration the total amount of text in each unit of work and consequently
allocates a percentage to the text highlighted. Figure 2 presents the average
coding over the 6 units of work for each of the elements in the course and the
percentage of text devoted to each of the elements.
Figure 3. Mean scores and proportion of text for
each element
The elements of Deep Knowledge, Substantive Communication, Explicit
Quality Criteria, and Social Support are prevalent in both courses when
analysed using the proportion
Vol 41,
8, August 2016 29
of text as a measure of their presence in the forums. This aligns with
the overall coding presented in the previous section. Also similar to the
overall coding, Deep Understanding, High Expectations and Knowledge Integration
show medium levels of presence in the courses, and Metalanguage, Student
Direction, Cultural Knowledge and Narrative are virtually non-existent in the
discussion forums.
One interesting point arising when comparing the
two approaches to analysis of the courses, by overall coding and by proportion
of text, is that some elements (i.e., Engagement, Student Self-regulation and
Inclusivity), are coded higher in Table 1 than the percentage of text devoted
to it in the coded discourse would seem to indicate at a first glance (see
Figure 3). For these elements, we considered that unsolicited student
participation in the discussion board acted as a proxy measure of their
presence. As an example of this, the element Engagement does not appear
explicitly in any percentage of text in either of the forums, but we have coded
it as high as 4 depending on the level and quality of unprompted engagement
that students showed with the mathematical concepts taught that fortnight. If
we were to continue the analogy with face-to-face settings, ‘proportion of
text’ would be equivalent to ‘time spent in class’ in a setting where students
are highly engaged. Essentially, participation in the discussion forums implies
engagement with the course, and so, all students who post comments are engaged
to some degree. Therefore even though there is nothing in the text of student
posts that indicates engagement, the existence of the text at all indicates the
presence of this element. Taking this into consideration, we interpreted the
presence of the comments as a proxy to engagement, and coded accordingly for
the previous section.
In the case of Student Self-Regulation, the fact that there is no text
in the forums devoted to it is equivalent to having no time in a classroom when
the teacher has to discipline students or redirect their attention to the task
at hand. In this sense the absence of such text denotes high levels of
self-regulation by students. Similarly, Inclusivity was not mentioned by the
students but was considered evident in posts from diverse students including
males and females.
Analysis of Content
Analysis of the content of interactions in relation
to each of the QT elements will now be presented in turn, using examples where
available to illustrate in detail how the characteristics of Quality Teaching
can be observed in the online environment.
Dimension: Intellectual Quality
Deep
Knowledge
In both courses, across all units, the element of
Deep Knowledge was coded highly indicating that within each unit the
discussions focussed on a small number of key concepts and the relationships
between those concepts. In many cases the discussion began with a student
question:
looking at the nature of points
of inflection. a horizontal point of inflection occurs when both first and
second derivitives [sic]= 0 and there is a change in concavity. An oblique
point of inflection occurs when the first derivitive does not = 0 but there is
a change in concavity and second derivative =0. Is this correct?
Student post, Course 1, Unit 2
Vol 41,
8, August 2016 30
The
question was typically followed by an instructor response:
Points of inflection must all
have their second derivative =0 (this is a necessary condition). However, the
first derivative may or may not be 0. If the first derivative is NOT zero, they
are sometimes referred to as "oblique". The function y=x^3+x is an
example of this. If the first derivative IS zero, then they are sometime
referred to as "horizontal". The function y=x^3 is an example of
this. Please note that there must always be a change in concavity for it to be
an inflection point. So answering your question: yes, you are correct :)
Instructor post, Course 1, Unit 2
In some cases, other students also offered responses to student queries,
enlarging the discussion and thus demonstrating substantive communication as
defined in the QT model. The units within each course were often accompanied by
an instructor-written blog post (not included in the analysis conducted for
this paper), introducing the concepts for the unit and thereby providing the
necessary focus for the discussion that followed. These examples point to the
necessity of planning cohesive, carefully focussed units within courses in
order to promote discussions that go beyond superficial treatment of key
concepts.
Deep
Understanding
In comparison with the element of Deep Knowledge,
the coding for Deep Understanding across both courses indicates that most
students were demonstrating only superficial knowledge of the key concepts
under discussion. This is to be expected because the forums were designed as
spaces for the students to seek help. Instructors would be more likely to see
this element demonstrated through the formal assessment tasks associated with
each course, rather than in student posts. At times, however, the students did
demonstrate Deep Understanding, particularly when helping each other:
I think I can help with your question one query.
If you included 1 as a value (by
shading the circle) the equation would be undefined. This is because when you
substitute x=1 back into the equation you would get 1-1=0 for the denominator.
The denominator cannot be zero because anything divided by zero is undefined.
I hope this makes sense. :)
Student post, Course 1, Unit 6
Problematic
Knowledge
Mathematics as a school subject is rarely presented
as being socially constructed and/or open to question. It is not surprising
that the transcripts analysed here show little evidence of the Problematic
Knowledge element. There are several examples, however, where the instructors
encourage students to embrace the fact that there are many different ways to
complete most mathematics problems:
Sometimes the graphs are long and
skinny, you just have to tell me where the important features are. Sometimes
you label them a,b,c etc or you could just state it under the graph. Sometimes
we need a 1-1scale, we don't want graphs to be stretched (we don't want a
circle looking like an elipse). The question will be looking at your setting
out, how do you communicate all the data to me in a neat and easy to read
fashion. There may be different ways to do this, it is up to you how you set
everything out and format your answer.
Instructor post, Course 1, Unit 3
Vol 41,
8, August 2016 31
Higher-order
Thinking
There was little evidence of the Higher-order Thinking element in either
course, however, this could be absent for similar reasons outlined above for
Deep Understanding. The nature of online discussion boards to answer
mathematical questions in the course encouraged brief, rather than extended
interactions between participants, usually focussing on specific difficulties.
Therefore, responses indicative of higher-order thinking about the course
content, involving analysis, synthesis and evaluation, was not expected to be
prevalent in these forums but expected in responses to formal assessment tasks
which were not included in this analysis.
Metalanguage
While there was ample evidence of discussion using mathematical language, there was
little evidence of discussion about
mathematical language. One of the few examples of the Metalanguage element
comes from an instructor post:
Now into turning points: They’re
known as “turning points” as this is where the curve ‘turns’ from ‘going up’ to
‘going down’ (or vice versa). Mathematically we can see this happening because
the gradient changes from being positive to negative (or vice versa).
Instructor post, Course 1, Unit 3
Substantive
Communication
The Substantive Communication element is one of the
most prevalent across all units in both courses. This is not surprising as any
communication via an online discussion board, by necessity, has to be
elaborated sufficiently for other participants to make sense of the post.
Dimension: Quality Learning
Environment
Explicit
Quality Criteria
The Explicit Quality Criteria element refers to the
extent to which the characteristics of high quality student work are made clear
and are reinforced throughout the unit, ensuring that students are able to
assess their own progress against these criteria. We found a consistently high
level of evidence for this element across all units in both courses, typically
in the form of students asking for detailed clarification of assessment task
requirements or seeking feedback on their progress towards a high quality
product. Also, students posted work-in-progress for feedback which was given by
the instructor and in some cases by other students.
Engagement
The level of engagement of students in each course
varied significantly across units, with some units having most students
actively engaged in discussion and others with only a few engaged on a regular
basis. In the online environment, engagement is on the one hand very easy to
identify, as any posting indicates a conscious choice by the student to engage
in conversation. However, student engagement could be occurring ‘behind the
scenes’ with students accessing and reading posts but choosing not to engage by
posting themselves. It is
Vol 41,
8, August 2016 32
difficult
to determine the factors related to these different levels of engagement; this
may be a fruitful area for future research.
High
Expectations
This element refers to the degree to which students
are engaged in challenging work and are encouraged to take risks. In general we
found that only some students explicitly showed that they were engaging in
challenging work, and indeed, it could be said that these students were taking
a risk by posting their work publicly online as demonstrated by the following
post:
There is no solution for this
question so I don't know if I got it correct. If anyone else has completed this
question can you compare your answer to mine below, please. If you get
something different can you put it up. If you get the same can you let me know
too please.
(u^2-3v)^4 = u^8-12u^6v+54u^4v^2-108u^2v^3+81v^4
Thanks
(Student post, Course 2, Unit 5)
Social
Support
The Social Support element is related to the degree to which the
(online) learning environment is free from negative put downs and is an
environment where contributions are valued and encouraged. Across all courses,
no negativity was found; however, we generally observed a relatively neutral
environment, with some positive and encouraging feedback such as the following
comment from a student to the instructor:
Thank you for providing this
information and yes, I agree that a whiteboard would have been very handy. I
have made notes from the video and I understand it better now. I really appreciate
the time you take to answer our questions.
Student post, Course 1, Unit 6
And this
from one student to another:
Thank you for your brilliant
explanations. I get it !! Yeah!! Thank you again and congratulations for
working out 10d. I will need to read over it a couple more times before I get
it but thanks for sharing.
Student post, Course 2, Unit 5
Students’
Self-Regulation
The element of Students’ Self-Regulation focusses
on the extent to which students act autonomously when interacting in the online
environment rather than only participating when prompted by others. Given the
nature of the online learning environment, the element is coded highly across
all units in both courses. Inherently, the online learning environment,
particularly for adult learners, requires students to be self-regulating as
they juggle their daily responsibilities with their learning trajectory, even
when tight timelines are set for online activity.
Vol 41,
8, August 2016 33
Student
Direction
In the two courses under analysis there was very little opportunity for
Student Direction, with students demonstrating only minimal amounts of control
over the direction of their learning. These choices were mostly the pace at which
they went over mathematical concepts and the assessment tasks, or alternative
sources (videos, websites, etc.) they used for their learning. This has
prompted the academic staff teaching the courses to re-design the structure of
the discussion boards so that more input from students could be present. In
particular the “Miscellanea” forum in both courses has been altered to include
more Student Direction. We include an outline of those changes in the
“Implications for course design” section below.
Dimension: Significance
Background
Knowledge
The presence of the element of Background Knowledge
is measured through observation of references to students’ prior knowledge
obtained within or outside formal educational experiences. Across all units in
both courses there is little evidence of this element; however, occasionally
students do share pertinent personal details:
Thank you
for sharing this article, I have an interest in this area. I am Food Tech
trained and have always incorporated a lot of maths into my lessons. I plan to
do the reverse as a maths teacher.
Student post, Course 1, Unit 4
Cultural
Knowledge
The element of Cultural Knowledge is one of the
lowest scoring elements across all units of work. In general, there was no
acknowledgement made that alternative cultural approaches to mathematics are
possible. This finding possibly reflects on the nature of mathematics as a
discipline rather than on the online learning environment under analysis here.
Knowledge
Integration
Knowledge Integration was found to be one of the
most variable elements in the Significance dimension across all units. In one
unit, several meaningful connections were evident between topic areas; however,
in most of the units of work only trivial connections were made. For example,
the following instructor post is illustrative of the Knowledge Integration
element:
Last week we dealt with some very
important concepts when graphing functions: tangents and normals. This week we
will build on those concepts, mash them up with what we know about continuity,
intercepts and voila, we will be ready to graph any polynomial function that is
thrown in front of us. Isn’t it great?
Instructor post, Course 1, Unit 3
Vol 41,
8, August 2016 34
Inclusivity
The element of Inclusivity is evident in the online environment through
observation of the participation levels across units of work. In our analysis,
most students contributed to the online discussion, although there was some
level of variation in both courses. We did not find that the variation was
gender-related, and other social differences were difficult to appreciate in an
online environment.
Connectedness
Connectedness refers to the degree to which
learning is related to ‘real world’ settings and/or students are given the
opportunity to engage with an audience outside of the confines of the learning
environment. In the case of online learning, the latter could possibly be
achieved through engagement with the internet beyond the boundaries imposed by
the learning management system, however, we found no evidence of this in either
course.
Narrative
Across
all units we found no evidence of the Narrative element.
Implications for Course Design
Implications for course design were drawn from
applying the QT model to our case study of mathematics courses. First, the
analysis of the 12 units of work from two online courses, with different
instructors, produced remarkably consistent coding across the 18 elements of
the QT model which also aligns with what has been found previously in the
face-to-face classroom-based environment (Gore, 2014).
In terms of the Intellectual Quality dimension, we
found that the elements of Deep Knowledge and Substantive Communication were
most apparent in the online interactions across all units of work. While the
other elements in this dimension were detectable, they were not as prevalent,
possibly due to the concise nature of online discussions or alternatively due
to the nature of mathematics as a subject. However, each of the lower coded
elements, Deep Understanding, Problematic Knowledge, Higher-order Thinking and
Metalanguage, could be enhanced in the forums through careful planning. Using
the forums to pose higher-order tasks undertaken, as opposed to simply using
these forums as a springboard for asking lecturers direct mathematical
questions, could increase the quality of the courses.
Considering the Quality Learning Environment
dimension, the most highly coded element was Students’ Self-Regulation, however,
in the online learning environment this element must be assumed to be present
as adult students in higher education are, by definition, self-regulating.
Secondly, the element of Explicit Quality Criteria was coded reasonably highly
across all units, indicating students’ focus on explicit assessment
requirements as a key component of learning. Three elements, Engagement, High
Expectations and Social Support are all present in both courses, but do show
some variation in coding across the units of work. The element of Student
Direction is the lowest coded element in this dimension indicating a general
lack of planned opportunities for students to influence the direction of their
learning, with the exception of pacing. Opportunities to incorporate Student
Direction needed to be provided in the courses, and subsequently new learning
activities to achieve this goal have been incorporated in the Masters program.
An
|
Vol 41,
8, August 2016
|
35
|
example of these activities has been adding a ‘question time’ to the
weekly course blog and prompting students to ask about issues they would like
their lecturer to expand on in the following blog post. This has effectively
replaced the existing ‘Miscellanea” forum and added an extra component for
Student Direction to it. Students who have taken the course since this change
occurred have positively embraced this new feature and chosen topics relating
not only to mathematical concepts but also current issues in mathematics education.
Lastly we considered the dimension of Significance,
which was consistently coded at a lower level than the other two dimensions.
The highest coded element was Inclusivity which is observable in levels of
participation among students from different social backgrounds. It should be
noted that in our study, gender was the only obvious marker of social
difference and we found no pattern in participation by gender. We found varying
levels of the elements Student Background, Knowledge Integration and Connectedness,
indicating that it is possible to observe these elements in the online learning
environment. In contrast there was little or no evidence of the elements of
Cultural Knowledge and Narrative in any of the units of work.
The lack of certain Significance elements in the
discussion forums may have had an impact on the students’ annual evaluation of
courses undertaken by our institution. Using this official avenue, students
provided feedback at the end of the course suggesting that they would have preferred
the discussion forums and course blog to be blended into one interface. We
interpret this feedback as an indication that Significance and the other two
dimensions (Intellectual Quality and Quality Learning Environment) should have
been integrated to provide a more cohesive learning experience for students,
instead of being delivered using different media: the forums and the blog. It
is important to remember here that both courses included a weekly blog
delivered to students independently of the student forums. The weekly course
blogs were purposely created to incorporate Narrative and Cultural
Knowledge into the courses but since they didn’t include student
interactions, they are not part of the analysis reported in this paper.
Our exploration of the courses’ pedagogy using the QT model, through a
coding process that gives specificity and structure to key points of
consideration, reveals some consistent areas of strength in the methods
employed in the online teaching of mathematics, specifically a clear focus on
Deep Knowledge and Explicit Quality Criteria. Also, the nature of the virtual
environment, where students are only observable based on their online postings
and interactions, ensures that Substantive Communication, Students’
Self-regulation, Inclusivity and Social Support, are consistently displayed. In
contrast, evidence to support the presence of the remaining elements was
variable across units of work and clearly exposed areas ripe for pedagogical
improvement. Interestingly, some of these areas of improvement were
independently confirmed by student feedback upon completion of the courses. The
main such area identified was a need for a thorough integration of the course
blog and the student forums.
Conclusions
With this paper we demonstrate the applicability of
evidence-based methods for appraising quality teaching in face-to-face
classrooms to quality teaching in online environments. In particular, we
illustrate how the Quality Teaching model can be used to analyse, review and
improve teaching in the online environment using a case study of a series of
mathematics courses for practising teachers who wish to gain postgraduate
qualifications. While we acknowledge that our case study is small in scope and
results are not generalizable,
Vol 41,
8, August 2016 36
we contend that the Quality Teaching model can be a useful analytical
tool for improving online learning environments.
We utilised a range of digital resources available
for online teaching focusing on creating a Quality Learning Environment, where
the work of students is Significant and of high Intellectual Quality. This type
of environment is explained in detail in the Quality Teaching model (NSW
Department of Education and Training, 2003a). Our research indicates that an
evidence-based pedagogical model is a feasible model for analysing,
understanding and improving online teaching. The model clearly identified some
strengths within our current practices but also revealed some important areas for
improvement, in particular in the dimension of Significance.
Online instruction is often guided by the affordances of technological
tools; however it is clear that such a limited focus can omit vital components
of quality teaching. Our study supports the view expressed by Margaryan, Bianco
and Littlejohn (2015), that general pedagogical principles should be explored
and applied to any online learning environment regardless of size and scope. We
contend that a classroom-based pedagogical framework, the Quality Teaching
model, can be a comprehensive tool for directing pedagogical improvement in
online learning. This kind of analysis can assist online learning instructors
to supplement their instructional strategies with consideration of key
characteristics of quality face-to-face teaching which can be overlooked in the
virtual environment.
References
Abrami, P. C., Bernard, R. M., Bures, E. M.,
Borokhovski, E., & Tamim, R. M. (2011). Interaction in distance education
and online learning: Using evidence and theory to improve practice. Journal of Computing in Higher Education, 23(2-3),
82-103. http://dx.doi.org/10.1007/s12528-011-9043-x
Amosa, W., Ladwig, J., Griffiths, T., & Gore,
J. (2007). Equity effects of quality
teaching: Closing the gap. Paper
presented at the Australian Association for Research in Education Conference, Fremantle, Australia. Retrieved from
http://www.aare.edu.au/data/publications/2007/amo07284.pdf on December 15,
2015.
Angus, S. D., & Watson, J. (2009). Does regular
online testing enhance student learning in the numerical sciences? Robust
evidence from a large data set. British
Journal of Educational Technology, 40(2),
255-272. http://dx.doi.org/10.1111/j.1467-8535.2008.00916.x
Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R.
M., Surkes, M. A., & Bethel, E. C. (2009). A Meta-Analysis of Three Types
of Interaction Treatments in Distance Education. Review of educational research, 79(3), 1243-1289. http://dx.doi.org/10.3102/0034654309333844
Blitzer, & Skaperdas, D. (1968). PLATO
IV- An Economically Viable Large Scale Computer Based Education System. Paper presented at the National Electronics
Conference. Retrieved from
http://bitsavers.trailing-edge.com/pdf/univOfIllinoisUrbana/plato/PLATO_IV_Dec68.pdf
on December 15, 2015.
Bonham, S. W., Deardorff, D. L., & Beichner, R. J. (2003).
Comparison of student performance using web and paper-based homework in
college-level physics. Journal of Research in Science Teaching, 40(10),
1050-1071. http://dx.doi.org/10.1002/tea.10120
Vol 41,
8, August 2016 37
Darling-Hammond, L. (1999). Teacher quality and student achievement: A review of state policy evidence. Retrieved from
University of Washington, Center for the Study of Teaching and Policy website:
http://depts.washington.edu/ctpmail/PDFs/LDH_1999.pdf
De Wever, B., Schellens, T., Valcke, M., & Van
Keer, H. (2006). Content analysis schemes to analyze transcripts of online
asynchronous discussion groups: A review. Computers
& Education, 46(1), 6-28. http://dx.doi.org/10.1016/j.compedu.2005.04.005
Engelbrecht, J., & Harding,
A. (2005). Teaching undergraduate mathematics on the Internet.
Fullan, M. (2007). The new meaning
of educational change. New York, NY: Routledge. Gall, J.P., Gall, M.D.,
& Borg, W.R. (2005). Applying
Educational Research: A practical
guide, 5th edition. New York, NY: Pearson Education.
Garrison, D. R. (2011). E-learning in the 21st century: A framework for research and practice. New York, NY: Taylor &
Francis.
Ginns, P., & Ellis, R. (2007). Quality in
blended learning: Exploring the relationships between on-line and face-to-face
teaching and learning. The Internet and
Higher Education, 10(1), 53-64. http://dx.doi.org/10.1016/j.compedu.2005.04.005
Gore, J. (2007). Improving pedagogy. In J. Butcher
& L. McDonald (Eds.), Making a difference: Challenges for teachers,
teaching, and teacher education (pp. 15-32). Rotterdam: Sense Publishers.
Gore, J. (2014). Towards quality
and equity: The case for Quality Teaching Rounds.
Proceedings of the Australian
Council for Educational Research (ACER) Research Conference (pp. 86–91). Retrieved from
http://www.acer.edu.au/files/RC-Proceedings-2014-ONLINE.pdf on December 15,
2015.
Gore, J. M., & Bowe, J. M. (2015). Interrupting
attrition? Re-shaping the transition from preservice to inservice teaching
through Quality Teaching Rounds.
International Journal of Educational
Research, 73,77-88. http://dx.doi.org/10.1016/j.ijer.2015.05.006
Gore, J., Ladwig, J., Elsworth, W., Ellis, H.,
Parkes, R. & Griffiths, T (2009). Quality
assessment: linking assessment tasks and
teaching outcomes in the social sciences: Final Report. Melbourne,
Australia: Australian Learning and Teaching Council.
Gore J, Smith M, Bowe J, Ellis H, Lloyd A, Lubans D. (2015). Quality
Teaching Rounds as a professional development intervention for enhancing the
quality of teaching: Rationale and study protocol for a cluster randomised
controlled trial. International Journal of Educational Research, 74,
82-95. http://dx.doi.org/10.1016/j.ijer.2015.08.002
Graham, C., Cagiltay, K., Lim,
B., Craner, J., & Duffy, T. M. (2001). Seven principles of effective
teaching: A practical lens for evaluating online courses. The Technology Source, 30(5),
50.
Hayes, D., Lingard, R., & Mills, M. (2000). Productive pedagogies. Education Links, 60, 10-13.
Haythornthwaite, C., & Andrews, R. (2011). E-learning theory and practice. Sage
Publications. http://dx.doi.org/10.4135/9781446288566
Herrington, A., Herrington, J., Oliver, R., Stoney, S., & Willis, J.
(2001). Quality guidelines for online courses: The development of an instrument
to audit online units. In M. K. G. Kennedy, C. McNaught & T. Petrovic
(Ed.). Proceedings of 18th Conference of the Australasian Society for Computers
in Learning in Tertiary Education. (pp.
263-270). Melbourne, VIC.
Vol 41,
8, August 2016 38
Holmes, K. (2005). Analysis of
asynchronous online discussion using the SOLO taxonomy.
Australian
Journal of Educational & Developmental Psychology, 5, 117-127. Kidney, G.,
Cummings, L., & Boehm, A. (2014). Toward a quality assurance approach to e-
learning
courses. International Journal on
E-Learning, 6(1), 17-30.
Kyriakides, L., Christoforou, C., & Charalambous, C.Y. (2013). What
matters for student learning outcomes: A meta-analysis of studies exploring
factors of effective teaching.
Ladwig, J., Smith, M., Gore, J., Amosa, W., & Griffiths, T. (2007). Quality of pedagogy and student achievement: Multi-level replication
of authentic pedagogy. Paper presented
at the Australian Association for Research in Education Annual Conference,
Fremantle, Australia. Retrieved from
http://www.aare.edu.au/data/publications/2007/lad07283.pdf on December 15,
2015.
Margaryan, A., Bianco, M., & Littlejohn, A.
(2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80(0), 77-83.
http://dx.doi.org/10.1016/j.compedu.2014.08.005
Martindale, T., Pearson, C., Curda, L. K., &
Pilcher, J. (2005). Effect of an online onstructional application on reading and
mathematics standardized test scores.
Journal of Research on Technology
in Education, 37(4), 349-360. http://dx.doi.org/10.1080/15391523.2005.10782442
Newmann, F. M (1996). Authentic achievement: Restructuring schools for intellectual quality. San Francisco, CA: Jossey-Bass.
NSW Department of Education and Training. (2003a). Quality teaching in NSW public schools: A classroom practice guide.
Sydney, NSW: Department of Education and Training,
Professional Support and Curriculum Directorate.
NSW Department of Education and Training. (2003b). Quality teaching in NSW public schools: An annotated bibliography.
Sydney, NSW: Department of Education and Training,
Professional Support and Curriculum Directorate.
Prieto, E., & Holmes, K. (2014). Online students’ perceptions of interactive
tools to support postgraduate
learning of mathematics. Paper presented at the 37th Annual Conference of the Mathematics Education Research
Group of Australasia (MERGA), Sydney.
Richardson, J. T. E. (2003). Approaches to studying and perceptions of
academic quality in a short web-based course. British Journal of Educational Technology, 34(4), 433-442. http://dx.doi.org/10.1111/1467-8535.00340
Tallent-Runnels, M. K., Thomas, J. A., Lan, W. Y., Cooper, S., Ahern, T.
C., Shaw, S. M., & Liu, X. (2006). Teaching courses online: A review of the
research. Review of Educational Research, 76(1), 93-135. http://dx.doi.org/10.3102/00346543076001093
Tamim, R. M., Bernard, R. M., Borokhovski, E.,
Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says
about the impact of technology on learning a second-order meta-analysis and
validation study. Review of educational
research, 81(1), 4-28. http://dx.doi.org/10.3102/0034654310393361
Threlfall, J., Pool, P., &
Homer, M. (2013). Implicit aspects of paper and pencil mathematics assessment
that come to light through the use of the computer. Educational Studies in Mathematics,
66(3), 335-348.
Vol 41,
8, August 2016 39
Tidak ada komentar:
Posting Komentar