Learning . . . Continued

Historically, adult education as a field has been concerned with context. Scholars have long worked to acknowledge and understand how contextual factors affect adult learning and how they often alter the very definition of what is called adult learning. Open any formidable text in the field and you will likely encounter the word “context,” and along with it its synonyms, “background,” “situation,” and “conditions.” Select any manuscript from the field’s research journals (e.g., Adult Education Quarterly) and you’re likely to notice it emerging as a theme. Open the field’s leading specialty journals (e.g., Journal of Transformative Education) and your likely to see context or its referents mentioned in just about every paragraph! Clearly, for adult education researchers, context matters.

The field’s focus on context has emerged as an identity of sorts, fitting nicely with its adoption of constructivism as its principal epistemology. Constructivism is the idea that the meaning ascribed to any object or event is idiosyncratically constructed when a learner integrates it (relates it) with past experience and learning, an idea Fox (2006) has argued comes from the philosophy of descriptive contextualism and its analytic aims, description and understanding. From this perspective, researchers aim to describe and understand the rich contextual details of learning as it occurs in a particular time and place. They are not necessarily interested in predicting or influencing learning in regards to some pre-established aim across a variety of learners (the analytic aims of functional contextualism).

More useful investigations will aim to determine where something works, when something works, and with whom something works best.

This focus on context has been brought to the forefront with the field’s near uniform embrace of qualitative research methods (see Boeren, 2018; Daley, Martin, & Roessger, 2018), methods that dismiss the idea of generalization and what some might call “de-contextualized” truths. The culmination of this is that many of the field’s scholars now see as its mission the emancipation of adult learners rather than the prediction and influence of adult learning. Whereas the former fits easily into a milieu championing context, the latter is often seen as at odds with the idea that context matters. Scientific reports of adult learning, then, commonly come with gentle reminders to remember context.

Consider UNESCO’s decidedly quantitative report on adult and lifelong education (ALE): 3rd Global Report on Adult Learning and Education (UNESCO, 2016). In it the authors take special care to caution that “the plight of the most disadvantaged people may be hidden from view in overall statistics . . . a further reminder of the importance of looking beyond micro-level statistics and considering qualitative accounts of the value of [adult and lifelong education]” (p. 92). The report later concludes its discussion of the impact of ALE on employment by reminding readers of “the need to contextualize the evidence” (p. 100).

Intuitively this feels right (Context matters, doesn’t it?), particularly to researchers with little training in quantitative methods and statistics. But are such reminders warranted? Does a scientific investigation of adult learning necessarily ignore context?

The short answer is no; it doesn’t have to and it shouldn’t. Context matters.

Overly simplified scientific investigations of adult learning may aim to determine whether certain policies or practices work with little regard to their contexts. But more useful investigations will aim to determine where something works, when something works, and with whom something works best. These aims attempt to uncover what I call contextual generalizations; that is, findings that illuminate the contextual nature of adult learning but still provide researchers and practitioners with insights that help predict and influence adult learning in diverse settings.

So how does adult learning science do this and still make generalized causal inferences between interventions/policies and adult learning outcomes? The answer is to use methodologies that account for contextual variables and temporal change. By contextual variables I mean learner background variables (e.g., race/ethnicity, age, gender, SES, education history, cultural preferences, etc.) and learning setting variables (e.g., community SES, country GDP, neighborhood crime rate, school graduation rate, instructor experience, etc). By temporal change, I mean change that occurs in a particular time and place. Ways of accounting for contextual variables while still allowing for generalization are plentiful. I’ll discuss three that I think are particularly useful for experimental, quasi-experimental, and design-based research.

With whom does it work?

To understand with whom something works, scientific investigations of adult learning must consider interactions, a phenomena called moderating effects. Whereas main effects depict overall mean or slope differences between intervention/no-intervention groups and policy/no-policy groups, moderating effects depict how learner background variables influence the effects of practices and policies (i.e., how these variables interact). For example, consider a learning strategy that aims to engage adult learners in reflective dialogue to aid their transfer of course content to real-world settings. Let’s say a researcher randomly assigned learners to one of three professional development workshops. One of these workshops (the experimental group) used reflective activities along with instruction, modeling, practice, and feedback; whereas the other two (the control groups) used solely instruction, modeling, practice, and feedback. The content, instructor, and time of day were the same for each workshop. Transfer to real-world settings was measured by a blinded reviewer who rated each learner’s work products one week later for evidence of transfer. A main effect of reflective dialogue would simply describe the overall mean differences between the experimental and control groups. A significantly higher or lower mean score in any group would illustrate an effect. In this case, a researcher would find this effect using a simple one-way analysis of variance (ANOVA).

The number of learner background variables we could examine is only limited by what theory and research tell us is worth considering.

Although main effects may be interesting to researchers concerned with illustrating reflective dialogue’s overall impact, they largely ignore context. Principally, they don’t tell us with whom the intervention works best. Within the experimental condition, we may find that women show high transfer, whereas men show the same levels of transfer found in the control groups. Overall, this appears as a higher mean transfer score for the experimental group (women drive the average score up), but this effect is misleading; it ignores the differences between men and women. If we were to account for gender, we would likely find an interaction showing how gender moderates the effect of reflective dialogue. Similarly, we could consider things like a learner’s age, rank within the organization, educational level, race/ethnicity, salary, or region of birth. In each case, we could examine these background variables and how they interact with the learning strategy. We could even look at three-way interactions that considered, say, how a person’s age moderates the relationship between gender and reflective activities and its effect on transfer. It could be that older women respond very differently to reflective activities than younger women! For this hypothetical example, it’s complicated.

This analysis would likely take the form of factorial ANOVA, which is just another way of thinking about multiple regression with interaction terms. The number of learner background variables we could examine is only limited by what theory and research tell us is worth considering.

Where does it work?

A second way that scientific investigations of adult learning can consider context is to use multilevel models to examine nested data structures.  Nested data structures occur when data from individual learners are not independent. Instead, they depend upon the contextual unit in which they are situated (i.e., nested). We might say learners are “nested” in classrooms, departments, work groups, organizations, neighborhoods, states, or even countries. Those belonging to a particular unit will tend to perform similarly to others within their unit. As a result, characteristics associated with the contextual unit itself can influence outcomes associated with the learner, and by simply knowing the contextual unit, researchers can make better than chance predictions about a learner’s outcome (often without any knowledge of the learner’s particular background variables). Greenberg and Phillips (2013) explain:

“Two subjects drawn randomly from the same neighborhood are more likely to be similar to one another than two subjects drawn randomly from the entire population of residents in a city, even when controlling for individual traits that are known to be important to the explanation of the outcome” (p. 220).

This is the essence of a nested data structure.

In nested data structures, researchers classify information according to level. In our case, the learner is the level-1 unit and the contextual unit is the level-2 unit. All salient variables pertaining to the learner (e.g., age, race/ethnicity, gender, SES) are level-1 predictors, and all salient variables pertaining to the contextual unit (e.g., group size, group composition, group SES, etc) are level-2 predictors. To analyze these data, researchers use multilevel models. Doing so helps them determine where something works best.

The idea here is that multilevel models can help researchers better understand where something works best and, more importantly, why it works best in that particular setting.

The simplest form of a multilevel model is a random-intercept model. Let’s consider what this would look like in practice. Say we were interested determining whether a GED prep software program helped improve adults’ GED testing scores. The software was distributed to GED study centers across the state. To use the software, learners must be enrolled in an in-person GED prep class, where traditional teaching strategies are also used. Learners are free to use the software as often as they like while in these classrooms. We want to know if there is a relationship between the time a learner spend on the program and the score a learner obtains on the GED. Because all learners must regularly attend a traditional class to have access to the program, we can assume that the time they spend in the classroom is constant.

A traditional approach using multiple regression would simply provide the average relationship between time on the program and GED score across all learners. Even if it incorporated learner background variables into the model (e.g., age, gender, race/ethnicity, SES), it would still ignore the differences in the social context in which these learners learn (e.g., the classroom and community). A random intercept model, though, would measure how learners GED scores differ across classrooms even after controlling for learners’ individual differences. After establishing that GED scores do differ across classrooms, researchers could then use a multilevel model to examine classroom variables that account for this variation (e.g., teacher experience, class size, average educational level in the class, or number of computers in the classroom). The possibilities for examining how level-2 variables affect level-1 outcomes are great, and an extensive discussion of all the ways multi-level modeling can do this is beyond the scope of this blog post. But the idea here is that they can help researchers better understand where something works best and, more importantly, why it works best in that particular setting.

When does it work?

To understand when something works, scientific investigations of adult learning must examine the role time plays in the unfolding of a practice or policy. There are numerous ways of doing this, and the choice of method depends on what serves as the primary unit of analysis (e.g., a person, a group of people, or several groups of people). In general, though, analyses that account for time will present graphical displays of the outcome on the y-axis and time on the x-axis. In this way, readers can visualize how the outcome changes over time, often in relation to the introduction of a practice or policy. Such methods are a considerable departure from cross-sectional designs that examine the effects of a practice or policy at a single moment in time.

Ensuring that researchers have the skill sets to attain contextual generalization will require that adult learning science programs incorporate advanced research design and statistics coursework that goes well beyond the traditional 2 or 3 courses in adult education graduate programs.

When a person is the primary unit of analysis, researchers can use single-case designs. These are very basic illustrations of how a person is performing in regards to some outcome over time. Generalization is accomplished through replication. Single-case designs are primarily analyzed graphically. However, there are ways to incorporate statistical analyses. A commonly used version of a single-case design is called a multiple-baseline design. In this design, measurements of a key outcome are taken repeatedly before and after the introduction of a learning intervention. Measures taken prior to the intervention are part of the baseline phase, whereas those taken after are part of what is often called the treatment phase. The introduction of the practice or policy is staggered across individual learners to minimize the possibility that an event concomitant with the intervention has caused the observed change. To further isolate the effects of the intervention, researchers generally wait until a learner’s baseline has stabilized before introducing it. We could use this design to examine, say, the effects of a book club on the number of pages a learner readers per week. An app like GoodReads could be used to measure the reading habits of a learner prior to joining the book club and during participation in the book club. In this way, we could see visually if participation had changed this learner’s reading habits.

Given the collaborative nature of adult learning, it is often difficult to find authentic applications for single-case designs. Fortunately, there are variations of this design that are used with units of analysis larger than a single person. When an aggregate unit is the primary unit of analysis (e.g., a school, a city, or even a classroom), researchers use time-series designs. Data from time-series designs are generally analyzed statistically—and require advanced statistical knowledge—but often communicated graphically to show aggregate changes over time. A possible application of a time-series design would be if an entire city started a book club and encouraged its citizens to track their progress using data-collection apps. The average number of pages read per week would serve as the repeatedly measured outcome. Researchers would collect a series of aggregate measures before the intervention (participation in the book club) and after it.

Similarly, latent growth models are used when there are multiple aggregate units (e.g., cities, classrooms, schools) participating in the practice or policy, and a researcher also wants to see how between-unit variables (e.g., learning setting variables) affect the outcome of interest. With latent growth models, the researcher can model the degree of change over time, as well as how learning setting variables contribute to the units’ variation in change over time. A comprehensive discussion of this approach is also beyond this post’s scope, but the main idea to take home is that latent growth models are another way researchers can determine when something works to better understand the context surrounding a particular practice or policy.

Conclusion

In this blog post I have tried to show how adult learning science can account for context while still aiming for causal inference and generalization. We do this by determining not only if a practice/policy works, but also where it works, when it works, and with whom it works. I have referred to this as contextual generalization. Ensuring that researchers have the skill sets to attain contextual generalization will require that adult learning science programs incorporate advanced research design and statistics coursework that goes well beyond the traditional 2 or 3 courses in adult education programs.  As the adult learning sciences develops as a field, it will be critical to embrace some of the core values that adult education as a field has established so that the two can work together. I find much shared interest in the idea that context matters.

References

Boeren, E. (2018). The methodological underdog: A review of quantitative research in key adult education journals. Adult Education Quarterly, 68(1), 63-79.

Daley, B. D., Martin, L., & Roessger, K. M. (2018). A call for methodological plurality: Reconsidering research approaches in adult education. Adult Education Quarterly, 68(2), 157-169.

Fox, E. (2006). Constructing a pragmatic science of learning and instruction with functional contextualism. Educational Technology Research and Development, 54(1), 5-36.

Greenberg, D. F., Phillips, J. A. (2013). Hierarchical Linear Modeling of Growth Curve Trajectories Using HLM. In G. D. Garson (Ed.), Hierarchical linear modeling: Guide and applications (pp. 208-225). Thousand Oaks, CA: Sage.

UNESCO. (2016). 3rd global report on adult and lifelong education. Hamburg, Germany: UNESCO Institute for Lifelong Learning. Retrieved from http://unesdoc.unesco.org/images/0024/002459/245917e.pdf

For a while now, I have been advocating for the emergence of a new field, a field unique from what has been traditionally called adult education or adult and lifelong learning. It is a field I call the adult learning sciences. This field shares with the former a focus on adult learning and the practices used to facilitate that process. However, it diverges in some clear ways: it is grounded in scientific epistemology and its analytic aims are primarily functional, aimed at prediction and influence, rather than description and understanding (see Fox, 2006, for a discussion of these different aims). It strives to construct functional models of how adults learn, and to identify the policies and practices that best facilitate learning and the contextual factors that moderate the learning process.

The adult learning sciences take the position that to make scientific advances in adult learning, a community of researchers is needed whose a priori analytic aims are not explicitly emancipatory and thereby functioning as the agenda of a political “interest group.”

Apart from a scientific epistemology, a significant point of departure from adult education is its position on political activism. While it acknowledges that knowledge is never “value-free” and that activism is historically a part of education, it takes the position that to make scientific advances in adult learning, a community of researchers is needed whose a priori analytic aims are not explicitly emancipatory and thereby functioning as the agenda of a political “interest group.” In short, it acknowledges the problems imposed by the naturalistic fallacy: one cannot reason from “what is” to “what ought” to be. And more importantly, it acknowledges the problems for science when one does the reverse (reasoning from an “ought” to an “is”). By uncoupling the adult learning sciences from adult education, the two fields can pursue different analytic aims while working to inform one another. The learning sciences can provide rigorous empirical findings, which adult education can use to pursue its emancipatory aims. And adult education can highlight areas of inequality and social need, which the adult learning sciences can use to direct its scientific inquiry. The key idea here is that they can inform one another, but they don’t have to.

To begin a discussion of what the adult learning sciences might look like, I offer the following suggestions. Some are drawn from the excellent work of Summerhoff et al. (2018), who analyzed over 75 learning sciences programs for commonalities. Importantly, none of these programs identified specifically as the adult learning sciences, but they still offer relevant insights.

From my vantage, the adult learning sciences should be focused on adult learning as an area of inquiry while committing to:

1. Replication and extension of research,
2. Interdisciplinary research teams (psychology, computer science, sociology, economics, educational policy, etc)
3. Measurable outcomes and testable theories,
4. Data-based decision making,
5. Creation of generalizable knowledge,
6. Identification of contextual moderators and nested data structures,
7. Innovative uses of technology to support learning,
8. Functional models of cognition and metacognition,
9. Design of learning environments and scaffolding,
10. Use of experimental, quasi-experimental, and design-based research methods (i.e., testing educational interventions in authentic settings), and
11. Contemporary and advanced uses of statistics and quantitative reasoning.

These are only a start–suggestions meant to further discussion. A field that embraced these ideas while examining adult learning would be very different from the field of adult education as it exists today. As we move into the 21st century and the role of data and science evolves, so too must our field. Its future success depends on it.

References

Fox, E. (2006). Constructing a pragmatic science of learning and instruction with functional contextualism. Educational Technology Research and Development, 54(1), 5-36.

Sommerhoff, D., Szameitat, A., Vogel, F., Chernikova, O., Loderer, K., & Fischer, F. (2018). What do we teach when we teach the learning sciences? A document analysis of 75 graduate programs. Journal of the Learning Sciences, 27(2), 319-351.

So in my previous essay I made the case that adult learning is different from child and adolescent learning. But is there anything in these differences that suggests we should take a different approach to researching and evaluating the field’s practices and policies? If one looks at the historical trends of research in adult education over the past 30 years, one might begin to think so. A tidal wave of qualitative research (interviews, focus groups, storytelling, narrative analysis) has inundated the field, displacing from graduate programs and key journals the once dominant quantitative research paradigm (inferential/descriptive statistics, experimental/quasi-experimental designs, surveys) (Daley, Martin, & Roessger, 2018). In two recent reviews of published studies in adult education journals from Great Britain, Australia, and the U.S, a common conclusion emerged: the field has moved almost entirely toward qualitative research (Boeren, 2018; Nyers & Fylander, 2015).

But other fields that study cognitive and social phenomena involving adults have not followed suit. Psychology, behavioral economics, educational policy, neuroscience, sociology, and the learning sciences all remain primarily scientific fields grounded in quantitative research methodologies. The continued use of quantitative methodologies in disciplines that study adult phenomena suggest that there is nothing inherent in the phenomenon of adult learning that requires our field to adopt a qualitative approach to research–unless, of course, we want to proffer the very tenuous claim that either adult learning shares no common ground with these fields, or these fields just have it plain wrong. Both seem unlikely to me. What is likely is that over the past 30 years (about the length of an academic career) the culture surrounding adult education research has changed because some very influential scholars published some very influential theoretical texts espousing interpretivist, emancipative, and relativist views of learning (mostly normative accounts lacking empirical support). A new generation of researchers were developed in turn under these epistemological and methodological umbrellas, and these new researchers then developed a next generation of scholars who share these ideas. To borrow a metaphor, what started out as a snowball has turned into an avalanche!

Adult education is creating its own island by digging a moat, and its graduate programs are handing the next generation of researchers and practitioners shovels.

The problem is that such a one-dimensional qualitative approach to research is now out of touch with how contemporary organizations make decisions and how other fields are advancing our understanding of human cognition. In a sense, adult education is creating its own island by digging a moat, and its graduate programs are handing the next generation of researchers and practitioners shovels to dig a wider and deeper moat. In an era of data-based decision making, interdisciplinary research, and team-based science it’s hard to see how this ensures the continued success of the field.

Consider for instance the results of a recent survey conducted by the National Association of Colleges and Employers (NACE, 2017). In all, 201 organizations that hire college graduates described the key attributes they sought on an applicant’s resume. The results are telling. Problem-solving skills ranked at the top of the list: 82.9% of respondents stated this was something they looked for. And just five spots down this list were analytic/quantitative skills, cited by over 67.5% of respondents. Seems to me that in addition to attributes like work ethic,  cooperation, leadership, and communication, employers are looking for people who can make data-based decisions to solve problems. Unfortunately, our field is not preparing its practitioners to do this anymore, placing in jeopardy its historic relationship with industry as a developer of skilled training and development specialists, evaluators, recruiters, and instructional designers.

Others have done an excellent job detailing reasons why quantitative reasoning has become such a rarity in the field (see Boeren, 2018; Daley, Martin, & Roessger, 2018), so I won’t do so here. Instead, I’d rather highlight what the shift away from data-based decision making and quantitative reasoning is doing to the field. To illustrate this, let’s first consider the status of the field as one that seeks to improve practice and policy through research, an identity repeatedly championed by the field’s adopted phrase “research to practice.” Contrast this identity with that of an applied field (i.e., a trade) that aims to convey time-tested principles and practices (e.g., cosmetology, woodworking, funeral services). These disciplines are typically found in technical colleges and community colleges, not universities. “Research to practice” disciplines, however, have strong presences in universities, the idea being that the research and practice arms of the field inform one another. Problems of practice direct research, and research directs practice. That’s the idea anyway.

We must adopt pragmatic strategies and research aims that are valued by practitioners and policy makers. In today’s world of big data, this involves quantitative reasoning and data-based decision making.

But I’m not convinced this is happening anymore in our field. An analysis of the field’s research, practice, and policy writings has illustrated a complete disconnect between the language of research and practice/policy (Roessger, 2017). What researchers talk about in our field is different from what practitioners and policy makers talk about. In a research-to-practice field, this is a problem. And when the relationship between research and practice breaks down, so does the perceived value, and thereby presence of the field’s research arm. One needn’t scrutinize too hard the list of research-1 universities in the U.S. to notice the paltry number of those that have adult education programs. Research-1 (R1) universities are doctoral degree-granting institutions with the highest level of research activity among universities. They hire faculty with strong research agendas and expect them to obtain grant funding to support their research. In the U.S. many are land-grant universities that have an explicit mission to do the kind of research that improves the lived experiences of those who live in that university’s home state. By my count, there are 115 R1 universities in the U.S., and only 15 of those have programs that identify specifically as adult education or adult learning. Notable programs have closed over the years, such as those at University of Wisconsin-Madison, University of Michigan, Syracuse, and University of Texas. And none to my knowledge have been created at an R1 during this time. Let’s not also forget R2 universities (Universities with high research activity) and their recent program closings: University of Wyoming, Northern Illinois University, and National Louis University.  Contrast this trend for a moment with the closely related field of K-12 education, which has some semblance of a program in every R1 university. Even more alarming is that when we look at the top 25 ranked Colleges of Education in the country (all of which are R1 universities) only three have programs in adult education. By any measure, this doesn’t bode well for a field that seeks to improve practice and policy through research.

So why is this happening? Well, the answer is complicated and likely involves many factors. But I do think a principal factor is the field’s evolution as a primarily qualitative field concerned with learner perceptions and small-scale studies. Let’s consider what happens in research-driven universities when a discipline decides to uniformly adopt a mode of inquiry that is largely absent from more established–and well funded– disciplines: It becomes siloed. In an era of funded research, this is frankly not a healthy long term strategy. Large funding organizations (e.g., Institute of Educational Sciences, National Science Foundation, and U.S. Department of Education) typically seek to fund interdisciplinary and team-based research. The more expertise listed on a grant application, the less risk the funder takes because they assume there is more “skin in the game,” more checks and balances to ensure the project gets done and pursues attainable goals.  But because most established social science disciplines are comprised primarily of quantitative researchers, and most grants involve large-scale projects, adult education struggles to get involved (although its reach extends to most disciplines and areas of social inquiry). Its researchers struggle to speak the research language of these other disciplines, and often its researchers lack the skills and knowledge to work with large datasets or design studies that minimize threats to things like internal and external validity, two concepts not acknowledged in the qualitative tradition. Gone are the days where Kellogg Foundation grants are exclusively handed out to adult education researchers. Now we must compete with everyone else. And to do so, we must adopt pragmatic strategies and research aims that are valued by practitioners and policy makers. In today’s world of big data, this involves quantitative reasoning and data-based decision making.

Modern scientific approaches to human research are calling for large-N studies to ensure researchers have the statistical power necessary to make valid conclusions. They are calling for pre-registration of studies to ensure that researchers state their research questions and hypotheses before they go digging around in their data and potentially p-hack their results. They are calling for replications of findings to ensure that findings aren’t a fluke, or worse, the result of dishonest research practices. These are the criteria commonly imposed upon funded research proposals and research manuscripts submitted to influential journals with the ability to affect practice and policy. Yet, these are all things that we don’t train adult education researchers and practitioners to do.

It’s time to change course and adopt a more pluralistic approach to research in our field (Daley, Martin, & Roessger, 2018). The future success of our field depends upon developing more practitioners with quantitative reasoning and data-based decision making skills.

References

Boeren, E. (2018). The methodological underdog: A review of quantitative research in key adult education journals. Adult Education Quarterly, 68(1), 63-79.

Fejes, A., & Nylander, E. (2015). How pluralistic is the research field on adult education?: Dominating bibliometrical trends, 2005-2012. European Journal for Research on the Education and Learning of Adults, 6(2), 103-123.

Daley, B. D., Martin, L., & Roessger, K. M. (2018). A call for methodological plurality: Reconsidering research approaches in adult education. Adult Education Quarterly, 68(2), 157-169.

Roessger, K. M. (2017). From theory to practice: A quantitative content analysis of adult education’s language on meaning making. Adult Education Quarterly, 67(3), 209-227.

NACE. (2018). The key attributes employers seek on students’ resumes. Retrieved from https://www.naceweb.org/about-us/press/2017/the-key-attributes-employers-seek-on-students-resumes/