Associations between Autonomy-Supportive Teaching, the Use of Non-Academic ICTs, and Student Motivation in English Language Learning
2.1. Sample and Procedure
In our preregistered study (https://osf.io/tvnk2, accessed on 25 November 2023), we used data from three of four waves from a larger longitudinal research project (“Identification of school success factors”, https://osf.io/ucvh5/, accessed on 25 November 2023) that aimed to identify the relative importance of various success factors for grades on the graduation exam in Austrian secondary schools of the highest track (Gymnasium). The project was conducted together with the Austrian Federal Ministry of Education, Science, and Research.
In Austria, secondary education starts after four years of elementary education, usually when students are around the age of 10, and is divided into a lower and an upper phase. Lower secondary education concludes with grade 8, after which students in academic-track schools can decide to continue with upper secondary education or leave the academic-track school to enroll in a vocational school that provides more specialized career preparation. Upper secondary education concludes with a standardized exam (Matura) in grade 12, which, when passed, enables students to enroll at university. About 36% of Austrian students enroll in a Gymnasium after elementary school, and around 29% decide to stay there after grade 8 [41].
As there are approximately 270 public general secondary schools of the highest track in Austria, the 30 Austrian Gymnasium schools recruited for this project provide a representative sample at the school level. Additionally, the sample provides enough power to consider the class level and conduct multilevel SEMs. Schools were recruited by the Austrian Federal Ministry of Education, Science, and Research and received a project report with school-specific results as thanks for their participation. More details about the sampling procedure can be found in the preregistration of the project this study draws its data from. Students were in grade 11 at wave one (May 2022) and in grade 12 in the consecutive waves (September/October 2022, March/April 2023, May/June 2023).
In waves one to three, participants responded to an online survey, which was filled out during a regular classroom lesson, supervised by trained research assistants. On the first page of the online questionnaire, students received information about the voluntariness and anonymity of their participation, provided written informed consent, and were asked for permission for data processing. If this permission was not given, the questionnaire ended. Filling out the questionnaires took on average 25 min in waves one and two, and around 10 min in wave three, depending on the reading speed of the students. This study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of the University of Vienna (00724). In total, 1912 students filled out the questionnaires in at least one data collection wave.
To be considered for the present study, three inclusion criteria had to be met: (1) Students had to have stated the intention to take the written graduation exam (Matura) in English. Participants who did not plan to take the exam or for whom this information was missing were excluded from this study (n = 326). (2) Students who did not state which English class and therefore which teacher they belonged to (n = 39) could not be considered for this study. (3) Students who experienced a change in their learning context (e.g., a change in English group or English teacher) over the course of the data collection waves were excluded from the sample (n = 300), as a change in learning context could have an impact on the outcome variables.
The final sample consisted of 1288 students (mean age at wave 1 = 17.11 years, SD = 0.71, 56.29% female) from 92 English classes. The number of students per English class ranged from 2 to 25 (M = 14, SD = 3.91).
2.3. Analyses
Analyses were conducted using confirmatory factor analyses (CFAs) and structural equation models (SEMs) in Mplus 8.7 [51]. Given the multilevel nature of classroom data, we used the complex design option implemented in Mplus for CFAs while specifying the main models as two-level models. To deal with missing values (between 8.85% and 22.21% on the item level), we applied the full information maximum likelihood approach implemented in Mplus for CFAs. For the main models, we used Bayesian estimation to deal with missing values in all variables.
In preliminary analyses, we calculated CFAs to assess the dimensionality of the autonomy support scales (choices, constructive support), intrinsic motivation, and competence using the robust maximum likelihood estimator (MLR). We assessed the goodness of fit for all models using the comparative fit index (CFI), Tucker–Lewis index (TLI), the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). We followed the guidelines suggested by Hu and Bentler (1999; [52]) regarding cutoff scores for excellent and adequate fit to the data, respectively: CFI and TLI > 0.95 and 0.90; RMSEA and SRMR 39,53].
To answer our research questions, we estimated a multivariate two-level moderated mediator model (see Figure S1). We relied on the common convention to select three values of the moderator, where the mean represents a medium level of ICT use, whereas one standard deviation above and below the mean represent high and low levels of ICT use, respectively [54]. To account for the complexity of calculations and to facilitate the interpretation of results, we calculated five models with a stepwise approach: a mediated model with only the predictor constructive support (M1); a mediated model with only the predictor providing choices (M2); a moderated mediation model with only the predictor providing choices (M3); a mediation model with both predictors, but without the moderator (M4); and the full moderated mediation model (M5). We chose the Bayesian Markov Chain Monte Carlo (MCMC) estimation method to deal with missing data, because bootstrapping in conjunction with multilevel modeling is not available in Mplus 8.7. Eight chains were requested for the Gibbs sampler, which divides the parameters and the latent variables into groups that are conditionally and sequentially generated [51], and we specified a minimum number of 10,000 iterations. Convergence was assessed by carefully examining trace plots for every parameter as well as with the Posterior Scale Reduction criterion, reaching a value less than 1.05. For prior distribution, we used the program’s default settings of non-informative priors. Since we were only interested in the students’ subjective perception of their classrooms and not of more objective classroom climate effects [55], the models were specified solely on the individual level, as our only aim was to control for hierarchical data structure. At the individual level, all variables were group-mean centered.
