Correlations between TESOL Practices and Essay Writing Skills: An Analysis of Learner Essays and Competence Predictors at Classroom Level

Fig. 1 Heuristic model of variables in MEWS.

Doctoral Candidate: Oliver Meyer,
Teaching English for Speakers of Other Languages (TESOL), Writing, Empirical Research in Education, Automated Essay Evaluation (AEE), Multi Level Analysis (MLA)
Prof. Dr. Stefan Keller, Prof. Dr. Heike Behrens, Prof. Dr. Christian Reintjes
Start/Submission Date:
FS 2017 – HS 2019

The proposed dissertation is part of the binational SNF-DFG Lead Agency project “Measuring English Writing at Secondary Level” (MEWS), led by Stefan Keller (FHNW Basel), Olaf Köller (IPN Kiel) and Urs Moser (Uni Zürich). MEWS is the first large-scale empirical study in Germany or Switzerland to focus specifically on critical essay writing in English as a Foreign Language (EFL) at upper secondary level. Previous studies on essay writing skills in either country focussed on the lower secondary level and found that critical essay writing in EFL posed higher challenges than narrative and informative genres (Harsch et al. 2008; Keller-Bolliger 2012). Since English is the lingua franca of academia and business (Keller 2013; Porsch & Köller 2010; Eberle et al. 2008), guidelines for Swiss and German baccalaureate schools state that students should be able to formulate, structure and support arguments with effective rhetorical devices (EDK 1994; KMK 2012; Brupbacher et al. 2009).

MEWS measures English essay writing competences of approx. 2 500 11th/12th grade students in Switzerland (AG, BS, BL, LU, SG, SZ, ZH; N = 1 700) and Germany (Schleswig-Holstein; N = 800) at the beginning (T1) and end (T2) of the penultimate year before learners’ baccalaureate (see fig. 1). Further, MEWS employs a multi-level analysis approach (MLA) to identify individual, classroom and system factors that may contribute to learning outcomes (Keller 2013; Porsch & Köller 2010; Helmke 2004). In order to measure and control individual (level 1) and classroom (level 2) factors, we use different performance tests and questionnaires, which are administered in a computer-based testing environment (laptops, offline). Information about system factors (level 3) is available from educational statistics.

At both measuring points, all of the 2 500 learners respond to two different types of writing assignments (10 000 in total) which closely follow the prompts and evaluation models of the TOEFL test (writing section) by Educational Testing Service (ETS) in Princeton, USA. MEWS entered a collaboration with ETS which provides double blind, holistic human scoring of learner texts in addition to automated essay evaluation (AEE) through e-rater, an electronic scoring engine developed at ETS (Attali & Burstein 2006). E-rater delivers not only another holistic score on the same scoring rubric as the human scores, but several feature scores for various linguistic traits of critical essays.

This dissertation has three main research goals: (1) To give a detailed linguistic description of learner texts from a sub-sample of the entire corpus, providing a fine-grained picture of learners’ competences at different proficiency levels; (2) To identify factors of classroom instruction predicting learners’ writing skills in English; (3) To develop recommendations for selecting TESOL methods and contents at upper secondary level.

For the first research goal, we will review existing models and linguistic descriptions of argumentative texts and identify a set of linguistic features to analyse in a subset of texts at different proficiency levels. The holistic scores will be used to select the learner texts for this subsample. Moreover, we can use e-rater’s feature scores to check for confirmation of our findings or as indications for where to find strengths and weaknesses within the subsample.

For the second research goal, we developed and piloted a classroom questionnaire for T2 based on current pedagogic literature and principles (Hyland 2003, Harmer 2007). The aim is to identify classroom factors which statistically predict learners’ writing skills through regression analyses between the classroom questionnaire data and essay task or feature scores. Additional interviews with volunteering EFL teachers of participating learners will be held after T2 and help us validate our data and understand the rationale and planning of actual ELT curricula.

Finally, our analyses serve to gather potential factors that can explain learners’ essay writing performance and competence gains over one academic year. These help us make an informed and refined assessment of different teaching approaches and materials for teaching English writing in upper secondary schools. Ideally, our data also allows us to make concrete recommendations for teaching writing in an academic context. This will include types of model texts to use, types of linguistic scaffolding to provide and types of feedback and assessment tools to implement.

Attali, Y. & Burstein, J. (2006). Automated essay scoring with e-rater v.2.0. Journal of Technology, Learning, and Assessment, 4 (3), 4-30.

Brupbacher, B., Jucker, A., König, E., Roth, M., & Straumann, B. (2009). Englisch. In: Hochschule und Gymnasium (Eds.): Hochschule und Studierfähigkeit. Zürcher Analysen und Empfehlungen zur Schnittstelle (pp. 89-96).

Eberle, F., Gehrer, K., Jaggi, B., Kottonau, J., Oepke, M., & Pflüger, M. (2008). Evaluation der Maturitätsreform 1995 (EVAMAR II). Schlussbericht zur Phase II. Bern: Staatssekretariat für Bildung und Forschung SBF.

EDK (1994). Rahmenlehrplan für die Maturitätsschulen. Bern, 1997.

Harmer, J. (2007). The Practice of English Language Teaching. Harlow: Pearson.

Harsch, C., Schröder, K., & Neumann, A. (2008). Schreiben Englisch. In Harsch et al. (Eds.), Unterricht und Kompetenzerwerb in Deutsch und Englisch. Ergebnisse der DESI-Studie (pp. 139-148). Weinheim: Beltz.

Helmke, A. (2004). Unterrichtsqualität – erfassen, bewerten, verbessern. Seelze: Kallmeyer.

Hyland, K. (2003). Second language writing. Cambridge: Cambridge University Press.

Keller, S. (2013). Integrative Schreibdidaktik Englisch für die Sekundarstufe. Theorie, Prozessgestaltung, Empirie. Giessener Beiträge zur Fremdsprachendidaktik. Tübingen: Narr Francke Verlag GmbH. (Habilitation).

Keller-Bolliger, R. (2012). Kommunikative Schreibkompetenz in der Fremdsprache erfassen und beurteilen. Berlin: epubli.

KMK (2012). Bildungsstandards für die fortgeführte Fremdsprache (Englisch/Französisch) für die Allgemeine Hochschulreife. Beschluss der Kultusministerkonferenz vom 18.10.2012. Retrieved from beschluesse/ 2012/2012_10_18-Bildungsstandards- Fortgef-FS-Abi.pdf

Porsch, R. & Köller, O. (2010). Standardbasiertes Testen von Schreibkompetenzen im Fach Englisch. In W. Bos, E. Klieme & O. Köller (Eds.), Schulische Lerngelegenheiten und Kompetenzentwicklung. Festschrift für Jürgen Baumert (pp. 85-104). Münster: Waxmann.