Increasing the Generalizability of Qualitative Research, J W Schofield, 1989, Vol 1 Reader, p91

What is, may be and could be - solving the riddle of generalisability.

At last a paper that says what it says it will say, says it succinctly and imparts a genuine new and useful understanding of educational qualitative research. The summary says it all really: the paper itself is full of reasoned argument and examples drawn from the author's own experience.

Janet Ward Schofield argues that there are two questions one should ask if one wants ones research to be transferable to others:

1. What do we want to generalise to?
2. How do we design to maximise generalisability?

If Schofield is right, generalisability "is best thought of as as a matter of the 'fit' between the situations studied and others ..." To be able to judge how fit a study is, one needs information about the studies' concepts and conclusion - a thick description.

There are 3 useful targets for generalisation:

1. what is (e.g. accepted best practice)
2. what may be (e.g. unusual, but possibly good practice)
3. what could be (e.g. practice that could take off, but might not)

I would re-write these as more the more mnemonic present, possible and potential practices.

An enjoyable paper with sage advice.

NB p107 has an error, penultimate paragraph where "may be" should be "could be".

Action Research, 1988, S Kemmis, Vol 1 Reader, p177

Participative Practitioner Action Research is good. Okay?

K Lewin used the term Action Research (AR) in a paper in 1946, illustrating it by describing the process of reconnaissance necessary for a WWII bombing run. His emphasis was that knowledge comes from the action taken by participants in research (fact finding) and influences their next action. As will be significant, the methodology is critical, not any technique employed to carry it out e.g. fact finding is a methodology; how facts are found is the technique.

Kemmis dislike the simplicity of AR defined by Lewin. Kemmis and W Carr developed 5 rules to determine legitimate from illegitimate contemporary action research. It reads like commandments, and starts with

“1. It must reject positivistic notions of rationality, objectivity, and truth;" (sic).

They also insist that the motivation must be emancipatory (rule 4). Any failure renders the research inadequate or incoherent they argue. Their paper was 5 years earlier, in 1983, “Becoming Critical, Knowing through Action Research”. Acceptance is presumed, so I must write at this time that the “requirements” in isolation come across as dogmatic laws without justification. However, Rules 2, 3 and 5 are just re-amplifications of Lewin but applied to education.

Action Research peaked in the 1950’s but had a resurgence last century when Kemmis wrote.

The objects of Kemmis’s AR (KAR) are restricted to praxis, an unconvincing term of specious etymology that seems to be a re-minted coin, with the word practice on its face and with a requirement to be "informed by theory" as the legend on its edge. Actually, praxis is an Aristotelian term. Theory is subordinate to praxis because reason can't always prevent contradiction, which is always overcome by praxis (history, in fact). Is this pertinent? Kemmis seems to misapply it and misunderstand the role of theory, which he makes dominant and a pre-requisite to praxis. This undermines the rest of his paper. He actually writes quoting “theorist of practical action” and seems quite unconscious of his superfluous and pedantic language in the eyes of a non-convert. The difference between this and reconnaissance is mystifying. Praxis is vital to Kemmis. And that is the flaw in his paper – he just does not prove it, but writes as if it is dogma. There is no practical difference between practice and praxis, and Kemis seems again to re-define a word idiosyncratically.

He spends time explaining what AR is distinguished by methodology, not techniques. This is self-evident if his laws are used to prescribe KAR. It is also inevitably so because AR, even as defined by Lewin, is a spiral (or cycle) of methodology – techniques are irrelevant. The methodology is practitioners interdependently apply iterative research and reconnaissance to hopefully attain enlightening conclusions.

He makes clear the AR is associated with Policy Research and Control of Education.

The paper is mostly unconvincing and verbose. I wish I could have read more of Lewin’s paper.

Lawrence Stenhouse, The Teacher As Researcher, Offprints Reader, P31 (1975)

DIY - The Teacher As Researcher into the curriculum

32 years ago Stenhouse wrote his book, and this paper was an extract. I added the italics above because he limits research by teachers to the curriculum. He wrote:

"... all well-founded curriculum research and development, whether the work of an individual teacher, if a school, of a framework of a national project, is based on the study of classrooms. It thus rests on the work of teachers."

Stenhouse has no issue with objectivity for the teacher engaged in research in his or her classroom because "the teacher's subjective perception ... is crucial for practice since he is in a position to control the classroom."

He saw the main barriers to teacher research, a.k.a. action research, to be psychological and social. Notwithstanding the tensions created trying to be both a teacher and a researcher, he sees this research as accessible to teachers and feeding teaching. Indeed the self-motivating teacher-researcher is essential "if education is to be significantly improved"

The observation of one teacher by another seems to be a groundbreaking idea; so much so, that Stenhouse cites a study that provided "the careful and sympathetic help when teacher-researchers are exposed to feedback on their own lessons for the first time".

It is common place today for lecturers in colleges to be observed by colleagues on an informal basis. I wonder if Stenhouse had something to do with that.

W Carr, 1987, What is Educational Practice?, Vol 1 Reader, p160

Education: Theory considers and practice delivers.

What is educational practice? It is the delivery of education. What is educational theory? It is any theory associated with educational practice by practitioners. I assert these statements here for the sake of clarity; and there is not much of that in Carr's article. He does not want to look at how educational practice is presently understood. That is the failure of his paper.

Theory considers and practice delivers education. They are related to each other: theory is the academic or philosophical tool of practice, usually to seek to improve it. Carr's worry seems to be that practice is becoming the tool of educational theorists and, in my words, the tail is wagging the dog. The engineering model of education research was in vogue at the time, and that may explain this odd paper.

Why does Carr have a problem with simple but effective answers? It may be that he is worried by educational research being ruled by practitioners (those practising it) and non-practitioners (theorists) being rendered subordinate. If so, he should have realised that things have their day. A balance returns in the long run, as it has. He references the Aristotelian praxis and tries to wrench it into a different meaning to practice, saying that it implies underpinning theory. Actually, theory and praxis were quite different to Aristotle.

Was Educational Practice being misinterpreted and applied beyond its traditional commitments and roles 20 years ago? Did it need to be rescue? Unfortunately, the best Carr does is to "spell out core concepts".

The elephant-in-the-room in this paper is theory of education: Carr finds it threatened and its role and the relationship between it and education practice ambiguous.

Carr is sure that practice "can not be reduced to a form of theorising" and that "it is always the achievement of tradition". He looks into the historical roots of the word, hoping it will help him understand the answer to his eponymous question. Carr wants to rescue "practice" from misuse by "those cultural tendencies which now undermine it and degrade it." (These seem to be its practitioners ...)

He sets 3 tasks, but whether he achieved them or not becomes irrelevant because he fails to create a rationale whereby educational practice is an inherently ambiguous term or widely misunderstood. The mind and the body are one in the healthy human. The body is subordinate to the mind at most times but sometimes the body rules, as in sleep when the autonomous systems keep one alive. Carr is contriving a rational that educational practice has been ambiguously defined to put theory into a subordinate role. He does not come out and say so. He does not mention the engineering model. He does not even show that the practice of education is more esteemed than theorising about it. His arguments are invalid because he does not prove there is any real misunderstanding, or that theory and practice are out of harmony in the first place.

Subjectivity and Objectivity: an Objective Inquiry, D C Philips, 1989, reader Vol 1, p57

The Best We Can Do - By Consensus

Denis C Philips objected to Elliot W Eisner’s rejection of objectivity; that all forms of representation are partial. Indeed the title is suggestive that an inquiry can be objective. Philips sees attacks on objectivity in educational research methods as undermining the value of qualitative research. He repudiates doubts about qualitative research in principle unable to be objective research as unreasonable.

Philips runs an ideographic trace on the origins of the intellectual attack on objectivity, dealing directly with the general acceptance that there is no certain foundation for all knowledge (foundationalism) – all knowledge is tentative – by reminding us that does not mean there is no such thing as objective truth. In my words, just because one does not know how one got to a place, does not mean that one is not in that place.

Philips sees objective inquiry as a label suggesting procedural propriety but not a guarantee that results have particular certainty. The truth is “out there” and neither a subjective nor an objective methodological research procedure is guaranteed to capture it. But it might. Thus qualitative research is a valid basis for pursing the truth by objective inquiry or subjective inquiry.

Philips admits that data is measurement-dependent; one measures what one can, not necessarily what is most significant, and one may only measure what one hopes will support a theory – a choice has to be made what to measure, and that choice may be subjective even to the extent that it is determined by a single researchers desires, fears or competence.

Philips alludes to the notion of others that objectivity on an inquiry may be conveyed by a community of inquirers but consensus is not required (it would be unlikely) but rather critical scrutiny. Even competing arguments could be accepted as objective because “objective does not mean true”.

Philips argues that even judgement within a framework or a paradigm may be construed as objective if it is reviewed by peers. This leads on to a surprising idea that novels (fiction) may be judged objective, and this has been supported by some institutions.

Nonfoundationalist epistemology requires judgement of objectivity to be conveyed only when the procedure is such that it offers peers opportunity to repudiate it or refute it. A continuum exists from subjectivity on the left through to objectivity with practical objective inquiry as close to the right as possible. Opposing arguments may be equally judged objective and neither may necessarily be valid.

Philips is rejecting extreme relativism where anything goes, as he calls that subjective. Eisner judges it worthless too.

Eisner objects to naïve realism, and Philips is not signing up to it here. Both Eisner and Philips are pragmatically recognising the limits of objective inquiry. Where they differ seems to be significant, but about how valid justification of qualitative research really is. In my words, Eisner claims it is polluted by interference and Philips says (my words), “So what if it is, it is the best we can aspire to; so long as we are careful and cautious, there will be no harm done.” Both philosophers agree that caveats have to be applied to so-called objective inquiry. Eisner says objectivity is impossible; by redefining what is meant by the label objective inquiry, Philips says it is possible but results are not guaranteed.

My Reaction To This Paper

Eisner’s and Philips’ arguments in their context are both valid. The difference between quantitative research and qualitative research in terms of objective inquiry has shifted through necessity over the decades. Eisner is right that people are too complex and bring baggage to research so that it is not ontologically objective and they change what they are researching so that research is not repeatable in the strictest sense. Philips argues that Eisner is undervaluing peer review and procedural rigour having worthwhile results that can be tested for repudiation or rejection. Objective does not mean true and is not dependent on an individual researcher. Consensus of peers can make a position objectively defensible. That too is valid.

Is life objective or subjective? You are born, right? You die, right? In between you do the best you can. That is the meaning of life.

If you agree, it is an objective argument; if you don’t agree, it is subjective. One still will not know whether it is true.

Objectivity in Educational Research, E. Eisner, 1992, reader Vol 1, p49

Step into the river, and it is a different river to the one you stepped into

15 years ago Eliot W Eisner objected to the notion that educational research can be done objectively in the fullest sense because procedural objectivity is not untarnished by the research. His argument is based on the fact that people are researched and people have their own frameworks. He considers ontological objectivity is impossible and impractical because the research process would have to be fully certain of its constituent frameworks and be amenable to testing to be verifiable as valid; to know its frameworks and complexities of person interactions in the first place renders the research superfluous. In my words, procedural objectivity is a useful attempt at control, but it is naïve to consider its validity unquestionable.

Eisner’s simple, but bold, argument is well supported with evidence and his argument seems valid. However, the truth is not fully encapsulated because Eisner is not offering a positive construction but, with some usefulness, deconstructing what he sees as invalid i.e. education research should not base itself on procedural objectivity in naïve realism.

My strap line is suggesting that when crossing a shallow river on foot one may entertain a naive realism that one is crossing the river, but because the river by definition is flowing, it is not the same river during the journey, and one steps out from different waters than one stepped into. Moreover, one changes the river itself, especially if unwashed or dressed in polluted clothing. The objection to the river-paradox is that a river is what we want to call it – a nomothetic rather than an ideographic approach. Could naïve realism be more pragmatic than trivialising well meaning procedural objectivity? And, for a bit of fun, place strategic stones and one crosses the river without changing it, although it has been changed by placing the crossing stones in it… Three years earlier D C Phillips suggested that procedural objectivity was okay as the best one could do in the circumstances and repudiated the idea that since one could not know everything one should not test anything or care what one tested – something he called extreme relativism.

Both arguments are valid.

We build bridges across rivers to avoid falling into them and getting wet, but sometimes rivers flood over bridges or bridges collapse. How we build the bridge matters (Philips). We do not trust the bridge absolutely, that would be naïve (Eisner).

Gender and the Sciences: Pupils’ Gender-Based Conceptions of School Subjects, Lynda Measor, 1983, Offprint Reader, p16

Sex Matters - more for Girls than Boys ...

This was a qualitative study 24 years ago and drawn from interviews and observations with pupils in a large comprehensive school about 27 years ago. The researcher believes that children build up gender identities at school and see certain subjects supporting a gender choice and others undermining peer acceptance, which are resisted passively or actively. The focus is on physical science subjects - the boys and girls traditionally seeing it as a masculine area, and domestic science, seen as a feminine area.

Interesting stories are recounted sometimes with pertinent quotations from pupils to support the original supposition. They are convincing. Some fascinating details are provided to show insight into why the pupils believed and acted as they did.

Measor speaks of negotiation of gender identities of adolescent pupils (11-13 years). She suggests:

“the girls actively use d aspects of the school to construct their identity, in this case their feminine identity… Science lessons provided an arena for acting out of feminine susceptibilities in a public setting.”

The girls used passive resistance – being late, packing up early, talking etc. She says the boys’ reaction to domestic science, which was more active and noisy resistance, was similarly constructive to a masculine gender identity. She writes:

“Pupils could use their hostility to one of the science subjects - “natural” or “domestic” – to signal an interest in things masculine or feminine. At the same time pupils used strategies that they felt to be appropriate to their sex.”

What the teachers thought also mattered. Measor thinks that community and teacher- expectations and prejudices make “a very solid wall of resistance to any change.”

Conclusions drawn included

• Original rationale that gender identities are constructed at school
• That pupils are not gender-neutral as learning theory had suggested
• Curriculum change, including
• Girls would benefit from single sexed schooling for science
• Science could be feminised for girls

This study was interesting for its approach, its readability, and its unambiguous discrimination in favour of curriculum change for girls, even to the point of single-sex lessons for natural science and teaching a feminised curriculum.

My reaction to this paper

It demonstrates the importance of not just the question asked, but rational behind the research in the first place, and selecting data to write the story you want to tell. It all makes sense within the report, in a similar way to a novel does. Whether it is truth is arguable. That it is valuable is undeniable. After all, another researcher could emulate the research but find boys need curriculum change to have a fairer chance of constructing their gender identities in a mixed school setting, especially today when the media and governments admit that curricula seem to favour girls over boys, al least in so far as examination results show.

The quantitative versus qualitative debate continues. What I see at the moment is that qualitative studies are probably easier to do, more interesting to write up and to read when published and more likely to be understood by interested parties because the question why is the focus, rather than the questions what and how much or how often. In education people and their hopeful transformation matter. Measor used names a lot, and that made the report about people.

In a dental analogy, the toothbrush may be electric with x thousand revs per minute, but the manual inter-dental brush can reach where it can’t, preventing gum disease that the more powerful tool alone could not. Both are necessary, but one is more frequently needed than the other. The toothbrush is qualitative reseach and the small brush for focus and discrete action is quatitative research.