Category Archives: Scholarship

Epistemology as ethics: Decisions and judgments not methods and solutions for evidence-based practice

Share
Common (rapper) Common Sense (rapper) Tufts Un...
Image via Wikipedia

Show me the money! Or so the saying goes. Implying that talk is cheap and facts are the only thing that matters. But there is another thing we are being asked to do with money and that is put it where our mouth is. So evidence is not quite enough. We have to also be willing to act on it and demonstrate a personal commitment to our facts.

As so often, folk wisdom has outlined two contradictory dictums that get applied based on the parameters of a given situation. On one account, evidence is all that is needed. Wheras an alternative interpretation claims that evidence is only sufficient when backed up by personal commitment.

I’ve recently come across two approaches to the same issue that resonate with my own, namely claiming that evidence cannot be divorced from the personal commitment of its wielder. As such it needs to be constantly interrogated from all perspectives, including the ones that come from “scientific method” but not relying solely on them.

This is crucial for anybody claiming to adhere to evidence-based practice. Because the evidence is not just out there. It is always in the hands of someone caught (as we all are) within a complex web of ethical commitments. Sure, we can point to a number of contexts where lack of evidence was detrimental (vaccinations). But we can equally easily find examples of evidence being the wrong thing to have (UK school league tables, eugenics). History of science demonstrates both incorrect conlusions being drawn from correct facts (ether), and correct conclusions being drawn form incorrect facts (including aircraft construction as Ira Flatow showed). I am not aware of any research quantifying the proportion of these respective positions. Intuitively, one would feel inclined to conclude that mostly the correct facts have led to the correct conclusions and incorrect ones to the wrong ones but we have been burned before.

That’s why I particularly liked what Mark Kleiman had to say at the end of his lecture on Evidence-based practice in policing:

“The notion that we can substitute method for common sense, a very wide-spread notion, is wrong. We’re eventually going to have to make judgments. Evidence-based practice is a good slogan, but it’s not a method in the Cartesian sense. It does not guarantee that what we’re going to do is right.”

Of course, neither is the opposite true. I’m always a bit worried about appeals to “common sense”.  I used to tell participants on cross-cultural training courses: there is nothing common about common sense. It always belongs to someone. And it does not come naturally to us. It is only common because someone told someone else about it. How often, have we heard a myth debunked only to conclude that the debunking had really been common sense. That makes no sense. It was the myth that was common sense. The debunking will only become common sense after we tell everyone about it. How many people would figure out even the simplest things such as looking right and then left before crossing the road without someone telling them? We spend a lot of time (as a society and individuals) maintaining the commonality of our senses. Clifford Geertz showed in “Common sense as a cultural system” the complex processes involved in common sense and I would not want to entrust anything of importance to common sense. But I agree that judgments are essential. We always make judgements whether we realise that or not.

This resonates wonderfully with what I consider to be the central thesis of the “Science’s First Mistake” a wonderful challenge to the assumptions of the universality of science by Ian Angell and Dionysios Demetis.

“…different ages have different perceptions of uncertainty; and so there are different approaches to theory construction and application, delivering different risk assessments and prompting different decisions. Note this book stresses decisions not solutions, because from its position there are no solutions, only contingent decisions. And each decision is itself a start of a new journey, not the end of an old one.

Indeed, there is no grander delusion than the production of a solution, with its linear insistence on cause and effect.”

All our solutions are really decisions. Decisions contingent on a myriad of factors both related to the data and our personal situation. We couch these decisions in the guise of solutions to avoid personal responsibility. Not perhaps often in a conscious way but in a way that abstracts away from the situation. We do not always have to ask qui bono when presented with a solution but we should always ask where from. What is the situation in which a solution was born? Not because we have a suspicious mind but because we have an inquisitive one.

But decisons are just what they are. They are on their own neither good or bad. They are just inevitable. And here I’d like to present the conclusions to a paper on the epistemological basis of case study in educational policy making I co-wrote a few years ago with John Elliott. It’s longer (by about a mile) than the two extracts above. But I highlighted in bold the bits that are relevant. The rest is there just to provide sufficient context to understand the key theses:

from the Summary and Conclusions of “Epistemology as ethics in research and policy: Under what terms might case studies yield useful knowledge to policy makers. by John Elliott and Dominik Lukeš. In: Evidence-based Education Policy: What Evidence? What Basis? Whose Policy?, 2008.”

Overall, we have informed our inquiry with three perspectives on case study. One rooted in ethnography and built around the metaphor of the understanding of cultures one is not familiar with (these are presented by the work of scholars like Becker, Willis, Lacey, Wallace, Ball, and many others). Another strand revolves around the tradition of responsive and democratic evaluation and portrayal represented by the work of scholars such as MacDonald, Simons, Stake, Parlett, and others. This tradition (itself not presenting a unified front) aims to break down the barriers between the researcher, the researched and the audience. It recognizes the situated nature of all actors in the process (cf. Kemmis 1980) and is particularly relevant to the concerns of the policy maker. Finally, we need to add Stenhouse’s approach of contemporary history into the mix. Stenhouse provides a perspective that allows the researcher to marry the responsibility to data with the responsibility to his or her environment in a way that escapes the stereotypes associated with the ‘pure forms’ of both quantitative and qualitative research. A comprehensive historian-like approach to data collection, retention and dissemination that allows multiple interpretations by multiple actors accounts both for the complexity of the data and the situation in the context of which it is collected and interpreted.

However, we can ask ourselves whether a policy-maker faced with examples of one of these traditions could tell these perspectives apart? Should it not perhaps be the lessons from the investigation of case study that matter rather than an ability to straightforwardly classify it as an example of one or the other? In many ways, in the policy context, it is the act of choice, such as choosing on which case study a policy should be based, that is of real importance.

In that case, instead of transcendental arbitration we can provide an alternative test: Does the case study change the prejudices of the reader? Does it provide a challenge? Perhaps our notion of comprehensiveness can include the question: Is the case study opening the mind of the reader to factors that they would have otherwise ignored? This reminds us of Gadamer’s “fusion of horizons” (“Understanding […] is always the fusion of […] horizons [present and past] which we imagine to exist by themselves.” Gadamer 1975, p. 273).

However, this could be seen as suggesting that case study automatically leads to a state of illumination. In fact, the interpretation of case study in this way requires a purposeful and active approach on the part of the reader. What, then, is the role of the philosopher? Should each researcher have a philosopher by his or her side? Or is it necessary, as John Elliott has long argued, to locate the philosopher in the practitioner? Should we expect teachers and/or policy-makers to go to the philosopher for advice or would a better solution be to strive for philosopher teachers and philosopher practitioners? Being a philosopher is of course determined not by speaking of Plato and Rousseau but by the constant challenge to personal and collective prejudices.

In that case, we can conclude that case study would have a great appeal to the politician and policy maker as a practical philosopher but that it would be a mistake to elevate it above other ways of doing practical philosophy. In this, following Gadamer, we advocate an antimethodological approach. The idea of the policy maker as philosopher and policy maker as researcher (i.e. underscoring the individual ethical agency of the policy maker) should be the proper focus of discussion of reliability and generalization. Since the policy maker is the one making the judgement, the type of research and study is then not as important as a primary focus.

And, in a way, the truthfulness of the case arises out of the practitioners’ use of the study. The judgment of warrant as well as the universalizing and revelatory nature of a particular study should become apparent to anybody familiar with the complexities of the environment. An abstract standard of quality reminiscent of statistical methods (number of interviewees, questions asked, sampling) is ultimately not a workable basis for decision making and action although that does not exclude the process of seeking a shared metaphorical perspective both on the process of data gathering and interpretation (cf. Kennedy 1979, Fox 1982). Gadamer’s words seem particularly relevant in this context: “[t]he understanding and interpretation of texts is not merely a concern of science, but is obviously part of the total human experience of the world.” (1975, p. xi)

We cannot discount the situation of the researcher no more than we can discount the situation of the researched. One constitutive element of the situation is the academic tribe: “[w]e are pursuing a ‘scholarly’ identity through our case studies rather than an intrinsic fascination with the phenomena under investigation” warns Fox (1982). To avoid any such accusations of impropriety social science cultivates a ‘prejudice against prejudice’, a distancing from experience and valuing in order to achieve objectivity whereas the condition of our understanding is that we have prejudices and any inquiry undertaken by ‘us’ needs to be approached in the spirit of a conversation with others; the conversation alerts participants to their prejudices. In a sense, the point of conversation is to reconstruct prejudices, which is an alternative view of understanding itself.

It should be stressed however, that this conversation does not automatically lead to a “neater picture” of the situation nor does it necessarily produce a “social good”. There is the danger of viewing ‘disciplined conversation’ as an elevated version of the folk theory on ideal policy: ‘if only everyone talked to one another, the world would be a nicer place’. Academic conversation (just like any democratic dialectic process) is often contentious and not quite the genteel affair it tries to present itself as. Equally, any given method of inquiry, including and perhaps headed by case study, can be both constitutive and disruptive of our prejudices.

Currently the culture of politeness aimed at avoiding others’ and one’s own discomfort at any cost contributes to the problem. Can one structure research that enables people to reflect about prejudices that they inevitably bring to the situation and reconstruct their biases to open up the possibility of action and not cause discomfort to themselves or others? We could say that concern with generalization and method is a consequence of academic discourse and culture and one of the ways in which questions of personal responsibility are argued away. Abstraction, the business of academia, is seen as antithetical to the process of particularization, the business of policy implementation. But given some of the questions raised in this paper, we should perhaps be asking whether abstraction and particularization are parallel processes to which we ascribe polar directionality only ex post facto. In this sense we can further Nussbaum’s distinction between generalization and universalization by rephrasing the dichotomy in the following terms: generalization is assumed to be internal to the data whereas universalization is a situated human cognitive and affective act.

A universalizable case study is of such quality that the philosopher policy maker can discern its relevance for the process of policy making (similarly to Stake’s naturalistic generalization). This is a different way of saying the same thing as Kemmis (1980, p. 136): “Case study cannot claim authority, it must demonstrate it.” The power of case study in this context can be illustrated by anecdotes from the field where practitioners had been convinced that a particular case study describes their situation and berating colleagues or staff for revealing intimate details of their situation, whereas the case study had been based on research of an entirely unrelated entity. In cases like these, the universal nature of the case study is revealed to the practitioner. Its public aspects often engender action where idle rumination and discontent would be otherwise prevalent. Even when this kind of research tells people what they already “know”, it can inject accountability by rendering heretofore private knowledge public. The notion of case study as method with transcendental epistemology therefore cannot be rescued even by attempts like Bassey’s (1999) to offer ‘fuzzy generalizations’ since while cognition and categorization is fuzzy, action involves a commitment to boundaries. This focus on situated judgement over transcendental rationality in no way denies the need for rigour and instrumentalism. We agree with Stenhouse that there needs to be a space in which the quality of a particular case study can be assessed. But such judgments will be different when made by case study practitioners and when made by policy-makers or teachers. The epistemological philosopher will apply yet another set of criteria. All these agents would do well to familiarize themselves with the criteria applied by the others but they would be unwise to assume that they can ever fully transcend the situated parameters of their community of practice and make all boundaries (such as those described by Kushner 1993) disappear.

Philosophy often likes to position itself in the role of an independent arbiter but it must not forget that it too is an embedded practice with its own community rules. That does not mean there’s no space for it in this debate. If nothing else, it can provide a space (not unlike the liminal space of Turner’s rituals) in which the normal assumptions are suspended and transformation of the prejudice can occur. In this context, we should perhaps investigate the notion of therapeutic reading of philosophy put forth by the New Wittgensteinians (see Crary and Read 2000).

This makes the questions of ethics alluded to earlier even more prominent. We propose that given their situated nature, questions of generalization are questions of ethics rather than inquiries into some disembodied transcendent rationality. Participants in the complex interaction between practitioners of education, educational researchers and educational policy makers are constantly faced with ethical decisions asking themselves: how do I act in ways that are consonant with my values and goals? Questions of warrant are internal to them and their situation rather than being easily resolvable by external expert arbitration. This does not exclude instrumental expertise in research design and evaluation of results but the role of such expertise is limited to the particular. MacDonald and Walker (1977) point out the importance of apprenticeship in the training of case study practitioners and we should bear in mind that this experience cannot be distilled into general rules for research training as we find them laid out in a statistics textbook.

Herein can lie the contribution of philosophy: An inquiry into the warrant and generalization of case study should be an inquiry into the ethics surrounding the creation and use of research, not an attempt to provide an epistemologically transcendent account of the representativeness of sampled data.

 

Enhanced by Zemanta

Metaphor is my co-pilot: How the literal and metaphorical rely on the same type of knowledge

Share
Classical definition of knowledge, according t...
Image via Wikipedia

“Thanks” to experimental philosophy, we have a bit more evidence confirming, that what many people think about the special epistemological status of metaphor is bunk. We should also note that Gibbs’ and Glucksberg’s teams have been doing a lot of similar research with the same results since the late 1980s.

This is how Joshua Knobe on the Experimental Philosophy blog summarized a forthcoming paper by Mark Phelan (http://pantheon.yale.edu/~mp622/inadequacy.pdf):

In short, it looks like it really is pretty impossible to explain what a metaphor means. But that is not because of anything special about metaphors. It is merely a reflection of the fact that we can’t explain what any sentence means.  Experimental Philosophy: What Metaphors Mean

Phelan went and asked people to paraphrase metaphorical and non-metaphorical statements only to find that the resulting paraphrases were judged equally inadequate for metaphors and literal statements. In fact, paraphrases of metaphorical statements like “Music is the universal language” or “Never give your heart away” were judged as more acceptable than paraphrases of their “literal” counterparts “French is the language Quebec” and “Always count your change”. The result shows something that any good translator will know intuitively – paraphrases are always hard.

So the conclusion (one to which I’m repeatedly drawn) is that there’s nothing special about metaphors when it comes to meaning, understanding and associated activities like paraphrasing. The availability of paraphrase (and understanding in general) is broadly dependent to two factors knowledge and usage. We have to know a lot about the world and how language is used to navigate it. So while we might consider “there’s a chair in the office”, “a chair is in the office” or “how about that chair in the office” as adequate descriptions of a particular configuration of objects in space, the same does not apply to usage. And things get even trickier when we substitute “a cobra” or “an elephant” for “a chair” and then start playing around with definiteness.  We know that chairs in offices are normal and desirable, cobras unlikely and undesirable and elephants impossible and most likely metaphorical. Thinking that we can understand both “there’s an elephant in the office” and “there’s a chair in the office” as simply a combination of the words and the construction “there’s X in Y” is a bad idea. And the same goes for metaphors. We need to know a lot about the world and language to understand them.

One of the pairs of sentences Phelan compared was “God is my co-pilot” and “Bill Thomson is my co-pilot”. Intuitively, we’d say that the “literal” one would be easier to paraphrase and we’d be right but not as radically: 47% of respondents chose “God is helping me get where I want to go” as a good paraphrase and mere 58% went with “I have a copilot named Bill Thomson”. And that goes slightly against intuition. But not if we think about it a bit more carefully. All the same questions we can ask about the meaning of these two sentences demonstrate a significant dependence on knowledge and usage. “In what way is God your co-pilot” makes sense where “In what way is Bill Thomson your co-pilot” doesn’t. But we can certainly ask “What exactly does Bill do when he’s your copilots”, “What do co-pilots do anyway”. And armed with that knowledge and knowledge of the situation we can challenge either statement “no God isn’t really your co-pilot” or “no Bill isn’t really your co-pilot”. Metaphoricity really had no impact – it was knowledge. Most people know relatively little about what co-pilots do so we might even suspect that their understanding of “God is my co-pilot” is greater than of “Bill is my co-pilot”.

This is because the two utterances are not even that different conceptually. They both depend on our ability to create mental mappings between two domains of understanding: the present situation and what we know about co-pilots. We might argue that in the “literal” case, there are fewer more determinate mappings but that is only the case if we have precise and extensive knowledge. If we hear the captain say “Bill is my co-pilot” and we know that “co-pilots sit next to pilots and twiddle with instruments”, we can then conclude “the guy sitting next to the captain and switching toggles is Bill”. If the person sitting next to us said “God is my co-pilot”, we can draw conclusions from our knowledge of usage e.g. “people who say this are also likely to talk to me about God”. It seems a very simple mapping. This would get a lot more complex if the captain said “God is my co-pilot” and the person sitting next to us on the plane would say “Bill is my co-pilot” but it would still be a case of reconciling our knowledge of the world and language usage with the present situation through mappings. So the seeming simplicity of the literal is really just an illusion when it comes to statements of any consequence.

–7 Aug – Edited slightly for coherence and typos

Enhanced by Zemanta

The Tortoise and the Hare: Analogy for Academia in the Digital World?

Share
A mash-up of two images to metaphorically depi...
Image via Wikipedia

Dan Cohen has decided to “crowdsource” (a fascinating blend, by the way) the title of his next book with the following instructions.

The title should be a couplet like “The X and the Y” where X can be “Highbrow Humanities” “Elite Academia” “The Ivory Tower” “Deep/High Thought” [insert your idea] and Y can be “Lowbrow Web” “Common Web” “Vernacular Technology/Web” “Public Web” [insert your idea]. so possible titles are “The Highbrow Humanities and the Lowbrow Web” or “The Ivory Tower and the Wild Web” etc.

via Dan Cohen’s Digital Humanities Blog » Blog Archive » Crowdsourcing the Title of My Next Book.

Before I offer my suggestion, let me pause and wonder how do we know what the book is to be about? Well, we know exactly what it is to be about because what he has in fact done was describe its contents in the form of two cross domain mappings that are then mapped onto each other (a sort of a double-barrel metaphor). And the title, it goes without saying (in a culture that agrees on what titles should be) should as eloquently and entertainingly point to the complex mapping through yet more mappings (if this was a post on blending theory, I’d elaborate on this some more).

We (I mean us the digitized or unanalog) can also roughly guess what Dan Cohen’s stance will be and if he were to be writing it just for us, we’d much rather just get it as a series of blog posts, or perhaps not at all. The paragraph quoted above is enough for us. We know what’s going on.

So aware of the ease with which meaning was co-constructed, I would recommend a more circumspect and ambiguous title. The Tortoise and the Hare with a subtitle:  Who’s Chasing Whom in Digital Scholarship or possibly The Winners and Losers of Digital Academia. Why this title? Well, I believe in challenging preconceptions, starting with our own. The tale of the Tortoise and the Hare (as the excellent Wikipedia entry documents) offers no easy answer. Or rather it offers too many easy answers for comfort. The first comes from the title and a vague awareness of the fact that this is a story about a speed contest between two animals who are stereotypes for the polar opposites of speed. So the first impression is “of course, the hare is the winner” and this is a book about the benefits of digital scholarship, so the digital scholars must be the hare. Also, and also digital equals fast so that means the book is about how the hare of digital scholarship is leaving the tortoise of ivory-tower academia in the dust. And we could come up with a dozen stories illustrating how this is the case.

Then we pause and remember, ah, but didn’t the tortoise win the race because of the hasty overconfidence and carelessness of the hare? So that means that perhaps the traditional academics, moving slowly but deliberately, are the favored ones, after all? Can’t we all also think of too many errors made on blogs, crowdsourced encyclopedic entries and easily make the case that the deliberate approach is superior to moving at breakneck speed? Aren’t hares known for their short and precarious life spans as well as speed while the tortoise is almost proverbial in its longevity?

But the moral of the story is even more complex and less determinate. If we continue further in our deliberations, we might be able to get a few more hints of this. In particular, we must ask, what does this story tell us about speed and wisdom? And the answer must be: absolutely nothing. We knew coming into it that hares were faster than tortoises over any distance that can be traveled by both animals. We’re not exactly clear why the tortoise challenged the hare. Unless it had secret knowledge of its narcolepsy, it couldn’t have possibly known that the hare would take a nap or get distracted (depending on the version of the story) in the middle of the race? So equating the tortoise with wisdom would seem foolish. At best we can see the tortoise as an inveterate gambler whose one-in-a-million bet paid off. We would certainly be foolish (as was noticed by Lord Dunsany cited in the Wikipedia entry) to assume that the hare’s loss makes the tortoise more suitable for a job delivering a swift message over the same journey the following day. So the only possible learning could be that taking nap in the middle of a race and not waking up in time can lead to loosing the race. Conceivably, there could be something about the dangers of overconfidence. But again didn’t we know this already through many much less ambiguous stories?

What does that mean for the digital and traditional scholarship? Very tentatively, I would suggest it is that we cannot predict the results of a single race (i.e. any single academic enterprise) based purely on the known (or inferred) qualities of one approach. There are too many variables. But neither can we discount what we know about the capabilities of one approach in favor of another simply because it proved to be a failure where we would have expected success. In a way, just like with the fable, we already know everything about the situation. For some things hares are better than tortoises and vice versa. Most of the time, our expectations are borne out and sometimes they are not. Sometimes the differences are insignificant, sometimes they matter a lot. In short, life is pretty damn complicated, and hoping a simple contrast of two prejudice-laden images will help us understand it better is perhaps the silliest thing of all. But often it is also the thing without which understanding would be impossible. So perhaps the moral of this story, this blog, and of Dan Cohen’s book really should be: beware of easy understandings.

Enhanced by Zemanta