This week in SOCRMx we’ve been looking at quantitative data is analysis. While I’m not afraid of the maths, it’s a very new area for me, and all the terms can be a little overwhelming. The MOOC has a really nice approach, though, which integrates a historical perspective. From Guinness brewers to statistician wars and blanket bans on methods, it has made for very interesting reading.
The latest area I’ve been looking into is that of statistical significance. We’re prompted:
We can imagine how an over-reliance on p=0.05, or even on other values of p in significance testing, could lead researchers to claim strong support for a particular hypothesis when the evidence is actually weak. Do you know of any examples of such claims? What are the arguments in favour of significance testing? What are the arguments against? How might this debate influence your own research designs?
“The p value is the probability to obtain an effect equal to or more extreme than the one observed presuming the null hypothesis of no effect is true” (Biau, Jolles & Porcher, 2010).
The main advantage of using p-values seems to be that they are simple:
Of primary importance, the test of a null hypothesis is conducted in the context of a simple decision rule and provides a dichotomous outcome (Greenwald et al. 1996, 177). While critics would argue that hypothesis tests provide less information compared to alternative techniques, supporters argue that the binary decisions nevertheless enable scholarly progress and theory testing, which “requires nothing more than a binary decision about the relation between two variables” (Chow 1988, 105; Wainer 1999) (cited in Iacobucci, 2005)
It is also suggested that they can enable researchers to have greater confidence with smaller sample sizes, which might be required in preliminary research (Novella, 2015).
The disadvantages include:
– even when your concern is not replicability, “If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time” (Colquhoun, 2014) / “a P value of 0.05 raises that chance to at least 29%” (Nuzzo, 2014)
– Plausibility of hypothesis has a greater impact on whether an exciting finding is a false alarm than p-values, so a Bayesian framework may be more effective (Nuzzo, 2014).
– P values can “deflect attention from the actual size of an effect”, and “[According to Geoff Cumming, an emeritus psychologist at La Trobe University in Melbourne, Australia] “We should be asking, ‘How much of an effect is there?’, not ‘Is there an effect?’” (Nuzzo, 2014)
– the widespread use (and demand for) p-values can encourage ‘p-hacking’, which Uri Simonsohn (of University of Pennsylvania) describes as “trying multiple things until you get the desired result” — even unconsciously”, which can diminish the criticality and skepticism which exploratory research should be approached with (Nuzzo, 2014).
How might my research be influenced by the debate on significance testing?
– In the presentation of results, I might be inclined to present everything:
- Simonsohn advises researchers to “report how we determined our sample size, all data exclusions (if any), all manipulations and all measures in the study” (Nuzzo, 2014)
- “We also encourage the presentation of frequency or distributional data when this is feasible” (Trafimow & Marks, 2015)
- Cumming [from La Trobe University in Melbourne] recommends reporting effect sizes and confidence intervals (Cumming, 2009; Nuzzo, 2014).
– I’m encouraged to “try multiple methods on the same data set”, and if “the various methods come up with different answers”, to attempt to work out why (Nuzzo, 2014)
– Rather than getting caught on the statistical significance, I might hold on to notions of practical significance:
Many statisticians also advocate replacing the P value with methods that take advantage of Bayes’ rule: an eighteenth-century theorem that describes how to think about probability as the plausibility of an outcome, rather than as the potential frequency of that outcome. This entails a certain subjectivity — something that the statistical pioneers were trying to avoid. But the Bayesian framework makes it comparatively easy for observers to incorporate what they know about the world into their conclusions, and to calculate how probabilities change as new evidence arises (Nuzzo, 2014)
I’m very interested to hear what others who have more experience working with quantitative data than I do see as the implications of the debate.
Throughout the MSc Digital Education (MScDE), I’ve been pushing against my own discomfort with sharing partially formed ideas. In the first course, An Introduction to Digital Environments for Learning (IDEL) this merely involved blogging to an audience of one (my personal tutor) and I think one post that was shared more widely within our cohort. Yet still, the very act writing and pressing ‘publish’ with relative frequency seemed far to concrete and final for what were ‘baby’ ideas – ideas that were still forming, still growing, still so ‘unready’ for any kind of scrutiny. I survived, however, and began to accept that ideas are always like this, perpetually changing, becoming more nuanced, and never complete. It was possible, even, that the early scrutiny which had seemed so invasive might even be helping with idea formation, so with my next course (Digital Education in Global Context), I pushed back harder against the discomfort by participating in and at times leading a group blog, though it remained walled behind the HEI log-in. Later, in Education and Digital Culture, I went public. Initially, it was terrifying. That is, it was terrifying until I realised that I probably didn’t have an audience anyway, and that the nature of the ‘lifestream’ blog that we were doing was somewhat prohibitive in allowing anyone unfiltered access to our musings anyway, due to its (or.. my) often tangential engagement (that sounds like a bad thing: it wasn’t; rather, it is reflective of learning in ‘the age of abundant information’).
Here I am again, starting another course (Social Research Methods, now running through the MOOC #SOCRMX on the EdX platform but with slabs of work within the institutional VLE on either side – a kind of ‘Moodle-MOOC sandwich’, if you like), and once again I’ve been asked to make my thinking public. One would think it would be easy by now – but I have to admit to still having apprehension. I’m very much at the beginning of the dissertation research adventure, and will most probably change direction completely at least 18 more times before settling on a topic. But if this course has taught me nothing, it’s that it’s okay to be tentative. One might even say that ‘finished’, ‘final’, and ‘error-free’ are print literacy values anyway, so here’s to throwing off the shackles with some very provisional thinking.
What kind of topics are you interested in researching?
I’m less interested in the ‘what works’ framing of research questions and more interested in trying to understand messy complexities (fortunately I’m not in search of funding;). I feel that without unpacking some of these complexities, it’s hard to see for what or for whom the ‘what works’ works. More specifically, I’m interested in doing research at the intersection between the field I work in, EAP, and the work I’ve been doing on MScDE, by exploring the relationship between ‘academic literacies’ and digital technologies. This area, of course, has so many potential directions, and my focus will need to be narrowed significantly.
What initial research questions might be starting to emerge for you?
At the moment my questions are not manageable and all a bit ‘woolly’. Possible questions may focus on:
- the actual (digital) literacy practices of (particular group of) students on (particular) undergraduate assignments; and agential roles of technology within this
- the relationship between these literacy practices and the (digital) literacies used by students in their social lives
- student narratives about academic literacies
- lecturers’ narratives: their perceptions of the ‘required’ (privileged) literacies, and their perceptions of the literacies students use
What are you interested in researching – people, groups, communities, documents, images, organisations?
I’m not sure I see these as entirely separate. Rather, I’m drawn to the notion of assemblage and Actor Network Theory, in which ‘the entities that we commonly work with in education – classrooms, teaching, students, knowledge generation, curriculum, policy, standardized testing, inequities, school reform – are in fact assemblies or gatherings of myriad things that order and govern educational practices’ (Fenwick & Edwards, 2012, p. xi). I’m interested in how academic writing practices are assembled. i.e. what they consist of, in terms of human/social and non-human/material elements. I like doing discourse analysis – but I don’t want to privilege the text over the practice of writing.
Do you have an initial ideas for the kinds of methods that might help you to gather useful knowledge in your area of interest?
Whilst the encouragement to ‘craft’ a research approach in this week’s (last week’s) #SOCRMx materials was welcomed, I’m also conscious of the traditions of academic literacies. AcLit principally uses an ethnographic methodology, ‘involving both observation of the practices surrounding the production of texts – rather than focusing solely on written texts – as well as participants’ perspectives on the texts and practices’ (Lillis & Scott, 2007). Breaking this down into methods, Heath and Street (2008, p. 29) suggest that literacy ethnography can include ‘surveys, formal interviews, focus groups, photography, and activity logs along with spatial maps, video recorders, or audio recorders’.
Because I’m interested in both social and material actors in digital writing practice, I was excited to recently come across methods suggested by Ibrar Bhatt and Roberto de Roock (2014) which aim to capture the ‘sociomateriality of literacy events’. Using ‘real-time screen recordings, with embedded video recordings of participants’ movements and vocalisations around the tasks during writing, the result is a multimodal rendition of digital literacy events on- and off-screen’. The analysis, in particular, seems complex – but I’m intrigued to learn more.
What initial questions do you have about those methods? What don’t you understand yet?
There’s still too much to list here..
Do you perceive any potential challenges in your initial ideas: either practical challenges, such as gaining access to the area you want to research, or the time it might take to gather data; or conceptual challenges; such as how the method you are interested in can produce ‘facts’, ‘truths’, or ‘valuable knowledge’ in your chosen area?
The challenges are plentiful – from
- scale and scope being too large to
- local receptiveness to the value of the ‘knowledge’ the study might produce, as historically a study skills approach to academic writing has been supported, and
- access to the equipment to do the recordings if doing a multi-modal recording of students’ literacy practices, and the privacy concerns that might arise from this.
Looking forward to sharing research ideas with others over the next 8+ weeks, and convinced I’ll be more timely with the next post.
Bhatt, I., & de Roock, R. (2014). Capturing the sociomateriality of digital literacy events. Research in Learning Technology, 21. https://doi.org/10.3402/rlt.v21.21281
Fenwick, T. & Edwards, R. (2012). Researching Education Through Actor-Network Theory. Oxford, UK: Wiley-Blackwell
Heath, S. B. & Street, B. V. (2008). On ethnography: approaches to language and literacy research. London: Routledge
Lillis, T. & Scott, M. (2007). Defining academic literacies research: issues of epistemology, ideology and strategy. Journal of Applied Linguistics and Professional Practice, 4(1). http://doi.org/10.1558/japl.v4i1.5
Following a night under canvas we felt properly inducted into the realm of professional campers. Our morning routine, which was pretty much the same for the days that followed, went something like the following:
- I climbed out of a very warm cocoon and sparked up the butane stove and made some fresh coffee (using one of those aeropress devices – absolutely amazing for an on the go quadruple espresso)
- Renee packed up the sleeping area
- We drank said coffee and went into hyperdrive(l). Paired with a bit of honey cake we were on fire.
- Waited for tent to dry a little
- Packed up and
- Walked confidently in the direction that we thought was correct so as to look professional.
Day 2 began beautifully – we filled up our water containers at the font and limbered up to Pic Neoulous (1256m) which was the highest point of the day. What followed was a lovely descent through heathland and forest with lots of birdsong and plenty of hikers coming in our direction.The descent into Le Perthus through an old cork tree plantation with its requisite enormous manor house in a state of disrepair was a reminder of how valuable the trade was. The cork industry is somehow given a nod by Le Perthus, a border town with bizarre tax loopholes that means it sells almost nothing but alcohol and crisps. It was thriving and the kind of destination that would have served us well in other moods. We enquired in the supermarket if any fruit or vegetables were available anywhere in the town. Alas no, only 2km down the road. We settled on two fabada bean stews in cans that would be supper at some point. I have rarely been in a town that I didn’t want to leave in such haste (Ipswich aside). We left and headed towards the Roman ruins which we thought might be a resting place for the day. The temperature at this point was in the mid 30s and we stank.
The Roman ruins, to our minds, would be complete with a water source and probably a feast of some sorts. We found a little cave to rest in for 5 minutes which was very cooling. It seemed to be on some sort of lover’s lane though, so we soldiered on and climbed through Col del Priorat to Mas Nou, then to Col du Figuier. This was about 12km of track walking, where said track appeared to have been engineered to soak up the exact trajectory of the sun without providing much shade. My feet at this point had expanded a size or two and our water stops, or “Sippettes” as we called them, were all too frequent. Thank the mountain gods for hikers crack (basically an energy bar made from solid jam) to push us through. We eventually found the village of Las Illas where there was a free camping spot next to a picnic area. It had running water, a shower, flat ground and a hotel up the road sold beer. The fabada bean stew went down an absolute treat paired with a Catalonian blonde beer.
Temperature: Lucifer’s armpits.
Last year we failed to complete some of the GR10 due in part for a need to enjoy some of the finer elements of French cuisine, wine, beer, hotels, flat horizons – you get the picture. Not carrying a tent meant that our journey was curtailed by the Ariege as the accommodation was less frequent meaning the rambling was larger than we could handle. Being bribed by Renee with fake Twix bars to put in an effort to mount the current ascent became a little bothersome. Twix bars should not be faked.
Last summer we hiked a total of 32.5 days from Hendaye on the Atlantic coast to Gite de Bonac on the NW edge of the Ariege and then skipped most of the region to walk from Merens to Py. All in just shy of 600km from West to East with a cumulative ascent of 30,104 metres (30.1km up) and a cumulative descent of 31,795 metres (do the maths). The Grand Old Duke of York remains happily sedentary in his grave.
This year we aimed to join some of the dots.
Beginning in Barcelona we planned to take the train to Hendaye, just across the border and gazetted as about an hour away. Of course we chose the day when there is a train strike. Two buses, the first to Giron, the second to Perpingnon got us into bierre country. A train to Banyuls sur Mer saw us arrive late evening where line dancing and other celebrations were occuring. If they were for the hikers finishing the walk from West to East then we had made an error of judgement in deciding to go the other way this time. Would Barbara’s Line Dancing and Calling Caberet be awaiting us at the end of this leg?
So. DAY ONE. Saturday 29th July
Temperature: Unbeknownst to us a heatwave was descending upon us. Its name was Lucifer.
From: Banyuls sur Mer
To: a wild camp spot by a font near Pic Neoulous.
Distance: 20.2 km, 1582 metres ascent, 496 metres descent.
Going: Fair to good.
The route wound through vineyards and then steeply up (relatively at this point) through 4 Cols and to Pic de Salifort which gave us great views of the Med. It was properly warm though and our water supplies (about 2.5 litres each) quickly diminished as we looked forward to refilling at Font de Couloumates. Renee disturbed an animal hiding in the undergrowth. I think it was an Ewok. We got our refill but another source was dry. We reached a good place to camp next to another font and spent our first night under ripstop. Dinner was a rehydrated bag of beef bourguignon that was devoured like a dog eating its own sick. We were quite hungry.
Placard at Banyuls enobling the route.
The morning of departure.Banyuls.
Defying gravity. Med behind.
First camp spot approx 1000m above sea level.
Defying gravity again.
Clouds resting in the valleys from whence we came the next morning.
The last week has been spent in the company of friends in Foix, and in the amazing cities of Toulouse and Bordeaux. As you might have gathered, we stopped walking in a place called Py, 5 days on from Merens-les-vals.
Day 1 was dominated by low cloud which obscured the views, but we did find some hot sulphuric springs, where a family were taking an early morning dip: not a bad way to pass a Sunday morning. The other highlight was making the refuge (Besines) before the rain; others were less fortunate, arriving totally drenched an hour later.
Day 2 started with an amazing climb to Col de Coma d’Anyell where the sun created dazzling effects as it burned through the early mist. The Col is officially the boundary of the Ariege, and descending to the lake, Etang de Lanos, is a little surreal as there are frogs everywhere. It was quite beautiful, even though the landscape here feels a little more barren. The end of the day saw us arriving at Lac de Bouillouses which was a honey trap if ever there was. There were buses dropping off day trippers and weekend campers. We could smell the fabric softener -and other markers of cleanliness- before seeing them.. I’m sure they had a slightly different report regarding our scent!
From Lac des Bouillouses (forgot to take pictures!), day 3 began with a long walk out through the Bolquere forest, around another stunning lake, Etang Noir. The afternoon was less enticing, as the temperature rose and the terrain was our least favourite: mostly roads. A reward awaited in Planes, however, as the gite there had a pool and served artisan beers.
Day 4 was a long one, so we set off shortly after first light. After a gradual start through a forest we began a seemingly interminable climb to Coll Mitja (up from 1700m to 2367m over 4km). The descent was similarly steep, but with the added challenge of loose rocks underfoot. We each took quite a few slides, and Renee dove spectacularly off the path and into a soft bush at one point, but no damage done. The afternoon involved another long climb – and with the heat it was almost too much for Will. Were it not for the offer of a secret Twix bar, he may well have sat down on the mountain side and cried… and that was before the descent of 765m (over 6km) into Mantet, closely pursued by a looming thunderstorm. The day was was a tough 26km, with a total ascent of 1759m and descent of 1724m. It gave rise to new discussions about shortening the hike, about saving more for another time.steep… either (the refuge at the bottom is at 1831m).
Our final day hiking took us to Py, which our guide book informed us had the last shop for 3 1/2 days. It was also the last point the trail crossed tarmac for 2 days, so if we were going to stop hiking it had to be here. We asked at a bar about buses – and as there was one coming in less than an hour it seemed fated. Other reasons we stopped (written in the bar we stopped in, hence some verb tense issues – we no longer smell😊):
- Will really wanted to get a tan below his sock line;
- Needed to feed Will 4000 calories at lunch times, as he no longer had enough fat to power the afternoons;
- Couldn’t bare to tell the same story (who we are, our circumstance) for the 30th time in a gite;
- Will couldn’t stand lying, saying he was Australian so as to avoid embarrassment for poor French, for a day longer;
- Despite being surrounded by immense beauty, our energy levels are making the focus the walking: an endurance competition rather than the ppleasure-taking we set out on;
- In the same way that one can reach saturation point when in an art gallery, unable to take in more and/or give the artwork the full appreciation it deserves, our capacity for fully appreciating the stunning landscapes was met;
- We stink. We want to be clean.
- There was a bus. Call it serendipity. We thought about stopping and there was a bus in an hour for just 1€;
- Quite fancy a lie-in.
- 590km felt like a pretty good effort.
We last wrote from Bagnetes de Luchon, missing out the days prior (Germ to lac d’Oo & lac d’Oo to Bagneres de Luchon – more on ‘the missed days’ later), but more on them later. Our announcement of entering the Ariege was misleading in that it was short lived. After a couple of long days and a refuge experience from hell we packed it in and hitched a ride out (babies;).
Day 1 we walked 27km (1524m up & 1630m down) and had a delightful stay in hotel in Fos. The walk was long, but sped-up by the introduction of down hill scampering – not quite running but more than walking and with elements of attempted pole vaulting owing to our newly acquired trecking poles.Day 2 (17.9km) saw us climb 1651m, mostly through cloud and mist, to arrive at spectacular views from Col d’Aueran when the sun finally broke through. Our destination was 245m down: refuge de I’Etang de Araing. There were at least 40 people staying there, and just 1 shower and 2 toilets. This, however, was all managed quite well and was not the problem. The tipping point was somewhere between the snore-chestra of the night (dormitory accommodation and inadequate ear plugs), the rubbish breakfast (crap coffee, powdered milk, nesquick if one wanted it and some scraps of last night’s bread), the hefty bill for the experience (equal to our lovely time in Fos living it up) and the 10.5 hour hike (day 3 in the Arriege) that followed.
Day 3’s hike started quickly: neither of us could wait to get out of there, despite the location’s beauty (and some fantastic elements of the evening before: good company at dinner and watching the international space station whizz by in an otherwise clear sky). After a relatively short climb, we crossed the col (sere d’ araing) and entered old mining territory. It was a hellishly steep descent and a stark, brutal landscape with the detritus of mining gear everywhere. It felt as if the descent was made more unkind by man’s influence. Nature has not had us on our posteriors with such force! [Stats for the day up to lunch: 7.3km, 310m up and 1270m down]. After a quick lunch the ascent began: 800m over 4km. The day was starting to heat up, so we were grateful for the coverage provided by oak and birch trees: more magical forest. At the top we took a variation to the GR10, as the alternative was staying in an unmanned cabin, and we aren’t carrying sleeping mats or bags, or a tent. The signposting indicated 3hrs to our destination: Bonac – but this suggestion must have been for a less warm day or a time when the path was better maintained. We fought our way through shrubs and spikey plants which had reclaimed the path. We squelched through mud down a narrow track sanwiched against an electric fence and rock. A shepherdess slowed our progress as our direction was interfering with her attempts to corral her sheep downhill. Then, another beautiful, but seemingly endless forest, despite the incorporation of the previously mentioned run-walking. 3 hours of descent later (don’t forget the previous eve’s snorechestra) finally Bonac appeared across a river. It was a relief, and also a surprise. As we walked up past the church and towards our gite, we came across something akin to a mefieval market place: stalls selling blueberries from the mountain, hand made leather sandals and flower based lotions and potions spread about before the overflowing bar of our gite. We later learned that the gite and the people selling their wares were part of a collective, or commune, which had been established some 10 or so years prior, or part of a neighbouring collective (ie there are a few in the area). The vibe was welcoming and time to relax was the agenda offered to us. Our original plan had been to continue the next day to Aunac, an 8.5-9 hr hike. It would be like this throughout the Ariege: because we were not carrying a tent, etc., each day would be a long hike between refuges..and because we only had rubbish, airline-issue earplugs..sleep would only be available on chance of missing the snorechestra! What to do? We’d not even been able to confirm the availability of accommodation for most of next week. For one of the days, we had a lead on a different route, into Spain, which might take us to a refuge and eliminate the need to just sleep in all our clothes and Mars blankets for one of the nights.. but as yet we’d not found the route on a map. What to do? The hippies were nice. We stayed for the night. We’d ordered a picnic for the advance. We were also treated to a rock and roll band that evening. The bass player played a tuba through a Korg keyboard and they played Kyuss tracks. There were also travelling Brit hippies. One was called the Bearded Clam and had a youtube channel. We have yet to investigate. He had a pink beard.
We were asked if we’d like a lift the next morning to St Giron, where there is an amazing Saturday market. It had become obvious that the bettet way to cross the Ariege was with a tent and as we don’t have enough time to do the route in entirety, we decided this would be somewhere we’d come back to and jettison ahead via a lift and public transport to Merens Les Vals to pick back up on the trail. We’ve now walked for 2 days from Merens – but one update at a time!
Today we head on an epic hike into the Ariege. Accommodation is less frequent (meaning longer walking days) and more basic. So updates will be infrequent. That said the telecommunication systems may surprise us all. We will finish the Ariage on the 8th so look for a massive update soon after. Big love. Renee and Will x
Distance 13 km
Due to our extra half rest day where we managed 2 four course meals we were feeling pretty strong, though more in smell for me, as that brandy digestif worked some rather bad magic with the sweat pores on the climb up and through some lovely villages, through which the Tour de France passes.
The biggest thrill today was watching paragliders drift past us. Renee counted 17 at one point. The landing spot was Loudenvielle where we stalled for an hour and marvelled at the flowers. Apparently this place is a winner in the best kept flower displays of the region. Very colourful place. After smelling the roses we walked the hour up to Germ. The best part of the day was the brilliant Gite, with pool, quirky art, a lovely bar and just great vibe. Very unexpected. The grub was good and we even snuck into the bar after dinner where a performance artist was in flow. The linguistic element went way over our heads, but it was fabulous to see some theatre in the middle of a village in the middle of a mountain range. By far our best Gite.
Day abandonded at Col de Portet.
We left the refuge this morning hoping that the morning storms had passed. Hail, thunder and lightning woke us. At breakfast there was a lot of planning of new routes from other hikers. We decided to also take a safer route through the ski resort and up to the GR10 via a gravel road. About 15 minutes into the day we huddled in a lee slope as hail and more thunder and lightning returned, our walking poles a safe distance away lest we become lightning conductors.
As it slowed we continued up for about 20 minutes and soon a convoy of work trucks pulled over beside us and told us to get in. They were finished for the day as the weather was too dangerous to be on the mountain. Their tale of the death of a cow from a lightning strike the week before along with the cold water leaking down our backs was enough to persuade us. Officially we will never have walked from the Atlantic to the Mediterranean due to today’s cheat. However the hot shower here at the hotel and not being roasted by lightning was worth it. Oh and the four course lunch with wine and coffee for 13€ each.
Just before our pick up.
Martinez the driver and ski lift engineer, thank you.