Evaluating energy justice metrics in early-stage science and technology research using the JUST-R metrics framework

1 Introduction

Achieving a rapid and sustainable clean energy transition will require major continued investment in research and development (R&D) of emerging technologies, from basic energy science to eventual demonstration. According to the International Energy Agency, nearly 50% of CO² reductions will need to come from technologies currently under development to meet 2050 net-zero goals (International Energy Agency, 2021). Although decarbonization efforts generally aim to advance societal wellbeing, new technologies may also exacerbate or create social inequities (Carley and Konisky, 2020; International Energy Agency, 2021). A concerted effort is required to center justice and ensure equitable distribution of benefits and burdens across communities as new technologies are designed, deployed, and scaled.

Considering energy justice––which has “the goal of achieving equity in both the social and economic participation in the energy system, while also remediating social, economic, and health burdens on those historically harmed by the energy system” (Baker et al., 2019)––in early stages of energy technology R&D can be particularly impactful. Previous research has shown the importance of early-stage engineering design choices in “locking in” certain costs and environmental impacts associated with buildings (Dunant et al., 2021; Gauch et al., 2022) and other products (Ramani et al., 2010). The scope of these choices, in turn, is determined by research decisions even further upstream.

While it is expected that early-stage R&D decisions will similarly lock in many energy justice impacts of emerging clean energy technologies, few methodologies exist to incorporate energy justice considerations into early-stage energy research, and those that could be applied lack thorough assessment. Here, “early-stage R&D” refers to work falling within Technology Readiness Levels 1-3, as defined by the Department of Energy, which includes work starting from “basic technology research” through “research to prove feasibility” (Office of Environmental Management, 2021). Previous work from our team developed the Justice Underpinning Science and Technology-Research (JUST-R) metrics framework, which can act as a method for assessing potential energy justice considerations of early-stage research (Dutta et al., 2023); however, it remains to be seen how applying this framework during a research project impacts research decisions and processes. Here, we consider use cases in a variety of energy research areas to evaluate the JUST-R metrics framework for its appeal to researchers engaged in early-stage technical R&D and its effectiveness as a method to enable them to incorporate energy justice in their work. We find substantial enthusiasm from researchers around applying the metrics and evidence of the framework’s usability and impact on researcher perspectives. However, researchers find it difficult to change research designs due in large part to structural barriers from both experimental and institutional constraints.

1.2 Background

As energy justice works to change the global energy system to align with principles of justice, it has emerged as a complex, context-dependent undertaking. The scope of energy justice, in its position both as a field of research and a sociopolitical agenda, is inherently broad and entangled with political economy, activism, energy security, and the material realities of climate change (Jenkins et al., 2016). Definitions of energy justice often build on the multipronged structure of procedural, distributional (or distributive), and recognition justice developed in environmental justice scholarship (Fraser, 2001; Schlosberg and Collins, 2014), which have been referred to as the “Triumvirate of Tenets” in energy justice literature (McCauley et al., 2013). In practice, however, the conceptualization and application of energy justice varies dramatically across scales and at different locations in the energy system (Sovacool et al., 2019).

The versatility of the concept of energy justice may be useful in forming a unified understanding of the social implications of energy transitions from a systems perspective, but it comes with its own attendant risks and complications. As Galvin highlights, certain contexts––including institutions of concentrated economic and political power––may invoke the term “energy justice” in ways that negate the radical meanings and historical weight of justice movements (Galvin, 2020). At a larger scale, Ikeme argues that actors in the Global North are likely to work from a different moral position on environmental justice than actors in the Global South (Ikeme, 2003). Ikeme highlights that by focusing exclusively on environmental justice’s distributional elements, Global North perspectives evidenced in policymaking may conflate justice with equity and ignore the full extent of its “preventive, retributive and corrective elements.” Similar contestation over moral claims occurs in the field of energy justice (Galvin, 2019). Integrating energy justice into decision-making processes related to technology research and development therefore depends upon methods for recognizing the scope of energy technology’s global implications and the moral responsibilities of both researchers and institutions (Young, 1990).

Although clean and renewable energy technologies are slated to play one of the most crucial roles in climate mitigation efforts, a gap exists in our understanding of how principles of justice affect technology development, particularly at the earliest stages of R&D. Ravikumar et al. highlight the need for research funders and institutions to reframe equity as integral to energy technology research and enable longer-term structural reform to ensure continued alignment and engagement between researchers and community members (Ravikumar et al., 2023). Calls for a more equitable research enterprise are bolstered by evidence that marginalized community groups tend not to be prioritized in, nor the main beneficiaries of, R&D efforts (Bozeman, 2020; Woodson and Boutilier, 2021; Woodson et al., 2021; Kozlowski et al., 2022). This persistent issue further demonstrates the need for intentional incorporation of energy justice principles that center these communities in energy R&D.

Effective incorporation of equity and justice considerations in science and engineering is not without its challenges, however. Scholars have noted barriers due to differing viewpoints of risk and hazards (Jasanoff, 1998; Flegal and Gupta, 2018); a lack of connections or trust between researchers and community members (Steg et al., 2015; Graves et al., 2022); dwindling time to reach critical decarbonization goals (Partridge et al., 2018; Armstrong McKay et al., 2022; Newell et al., 2022); and limited understanding of community perspectives, values, and methods (Manders-Huits, 2011; Flegal and Gupta, 2018). Additionally, Flegal and Gupta find expert advocates of solar geoengineering research tend to narrowly view equity concerns as analytical problems that warrant feasibility studies and often define visions of equitable futures on behalf of vulnerable communities with whom they are unfamiliar (Flegal and Gupta, 2018).

End-users of energy technologies have typically only been centered when researchers and engineers are preparing for technology implementation and are often understood as more of a barrier to be overcome than an asset to aid in more streamlined, context-specific technology creation (Martin et al., 2020). Earlier consideration of energy justice principles enables researchers to not only think more broadly about the implications and potential contextual impacts of their work, but also enables them to center the needs of communities that have been disproportionately burdened by our existing energy system and its resultant pollution, and that, if concerted efforts are not made, may be left with a newer, cleaner, and yet similarly unjust energy system (Jasanoff, 2021).

In turn, developing methods that enable researchers to embed energy justice principles in clean energy technologies from the earliest stages of R&D requires clearly defining the principles that comprise energy justice, beginning with the core elements of procedural, distributional, and recognition justice. Here, distributional justice considers how benefits and burdens of the energy system are distributed across communities, procedural justice seeks to achieve equitable access to and inclusion in decision-making processes, and recognition justice calls for recognizing and addressing the different experiences and perspectives of communities who have been ignored or misrepresented throughout the creation, deployment, and operation of the energy system (McCauley et al., 2013; Jenkins et al., 2016; Baker et al., 2019). Over time, additional tenets of justice have been introduced in energy justice literature, including restorative justice, which involves recognizing and resolving past injustices; intergenerational justice, which considers impacts to future generations; cosmopolitan justice, which expands the scope of energy justice to investigate impacts associated with the full life cycles of energy technologies and systems (Pellegrini-Masini et al., 2020; Romero-Lankao and Nobler, 2021); and epistemic justice, which draws attention to different forms of knowledge, ways of knowing, experiences, and power dynamics that shape perceptions of knowledge (Tsosie, 2012; Sovacool et al., 2023). Of these tenets, distributional, procedural, recognition, and cosmopolitan justice form the core conceptual pillars of the JUST-R metrics framework, although many metrics overlap with other types of justice such as intergenerational (e.g., hazard metrics described below) or epistemic (e.g., pre-existing knowledge review metrics described below).

1.3 The JUST-R framework

The JUST-R framework consists of 30 metrics from previous energy justice and responsible research literature and 20 new metrics proposed to fill gaps in the literature around applying energy justice to early-stage research (Dutta et al., 2023). The new metrics are organized into five themes.

1. Hidden process costs: Metrics that measure various costs associated with the research life cycle, including costs of managing material inputs from cradle to grave

2. Breadth of pre-existing knowledge review: Metrics that measure the types and sources of knowledge used to inform the research questions, design, and conclusions, such as interdisciplinary sources

3. Distribution of research results: Metrics that measure the sharing of research outcomes, seeking to establish who benefits from the results and knowledge generated

4. Distribution of hazard exposure during the research life cycle: Metrics measuring hazards associated with the research life cycle, including those related to material inputs from cradle to grave and how they may change upon technology scaleup

5. Identification of set vs. flexible parameters: Metrics measuring alternatives explored to status quo practices with potential negative impacts, such as standard materials or processes

Together, the framework prompts researchers to evaluate their current material inputs and outputs, knowledge inputs and outputs, and the solution space of their research discipline to assess the potential energy justice impacts of their specific project.

2 Materials and methods

2.1 Evaluation criteria

This work aimed to evaluate the 20 new metrics in the JUST-R framework, all of which fall into the five themes described above, for their 1) effectiveness and 2) appeal as a method for incorporating energy justice in early-stage technical research. Here, “effective” metrics were defined as being both accessible to researchers and meaningfully influencing, or impacting, their research decision-making and processes. To operationalize effectiveness, the team defined ascending levels of effectiveness using 1) Tiers of Accessibility and 2) Tiers of Impact, as follows.

i. Tiers of Accessibility:

1. Understandable. Do researchers understand the method?

2. Capable of Being Applied. Are researchers able to apply the method?

ii. Tiers of Impact:

1. Awareness. Do researchers know about the method or the issue(s) it seeks to address?

2. Engagement with Material. Are researchers willing to engage with the method?

3. Problem Identification. Can researchers identify problems in their field or work related to energy justice from applying the method?

4. Solution Identification. Can researchers identify solutions to the problems they discovered from applying the method?

5. Articulation of Value of Identified Solution. Can researchers articulate the value of their identified solution in a way that aligns with the intended impact of the method?

6. Solution Application. Do researchers actually carry out the solution(s) they identified?

The Tiers of Accessibility range from whether the method protocol and purpose are generally comprehensible to whether they are straightforward enough for researchers to properly apply the method. They enable assessment of both researchers’ hypothetical understanding of a method (i.e., the method is clear enough to be understood) and the practical implementation of the method (i.e., the method is clear enough to be applied as intended). The ascending Tiers of Impact track the JUST-R metrics framework’s effectiveness in moving researchers from preliminary awareness of the energy justice issues associated with the metrics toward actions aimed at addressing those issues within the research process.

Given the novelty of the JUST-R framework and its goal of broader adoption across the energy research enterprise, the framework’s appeal to researchers was also studied. Here, “appeal” was defined as evoking demonstrated interest in or enthusiasm for engaging with the method. Appeal was operationalized using three Tiers of Appeal, as follows.

iii. Tiers of Appeal:

1. Value. Do researchers see value in applying the method?

2. Continued Usage. Are researchers willing to continue using the method?

3. Recommendation. Would researchers recommend the method to a colleague?

The Tiers of Appeal start with a basic notion of appeal, (i.e., finding value in the method) to a more substantive demonstration of appeal (i.e., being willing to recommend the framework to others).

Note that while here we only evaluate the JUST-R metrics framework, these Tiers of Accessibility, Impact, and Appeal are generalized criteria suitable for evaluating any method and could be used to evaluate other frameworks for incorporating justice, or related principles, into early-stage research.

2.2 Evaluation process

Fourteen completed JUST-R metrics worksheets were collected from research teams working at a national lab during the evaluation period. The research projects they evaluated represent diverse topic areas of renewable energy development, including bioenergy, transportation, energy efficiency and analysis, and energy storage. Ensuring that the evaluation teams worked in a range of different research areas across the laboratory enabled us to test different use cases reflecting a variety of research contexts. The worksheets the researchers used contain the 20 newly developed JUST-R metrics made to be applied from the earliest stages of R&D, with three columns for each metric: “Assessment,” “What could have been done differently?” and “Potential barriers” (Dutta et al., 2023). A blank JUST-R metrics worksheet is included in the Supplementary Material.

Research teams had access to a completed sample JUST-R worksheet as well as optional workshop meetings facilitated by our group. Eight of the fourteen research teams attended the workshop meetings, where they were able to ask facilitators questions about the JUST-R framework and provide immediate feedback while they filled out the JUST-R metrics worksheet. In addition to the workshop meetings, debrief sessions were held for researchers to discuss their experiences applying the JUST-R framework after they had submitted their teams’ worksheets; six of the research teams attended the debrief sessions. Our group gave the research teams a charge code to fund the time they spent completing the JUST-R metrics worksheet and attending the workshop and debrief meetings. Analyses of JUST-R worksheets were used to assess the framework’s effectiveness, as described previously, while analyses of workshop and debrief transcripts and notes were used to assess both the framework’s effectiveness and appeal.

2.3 Analysis

To assess the effectiveness of the JUST-R metrics framework, the completed JUST-R worksheets were evaluated and scored based on whether they showed (a) overwhelming evidence, (b) distinct evidence, (c) slight evidence, or (d) no evidence of an affirmative answer to the questions described previously in Tiers 1-2 of Accessibility and Tiers 1-5 of Impact. Tier 6 of Impact was not included in this analysis of the completed JUST-R worksheets because these worksheets did not ask whether identified solutions were applied; therefore, Tier 6 of Impact was assessed via the debrief sessions.

The JUST-R worksheets, along with transcripts from the facilitated workshops and debrief sessions, were also qualitatively coded, with codes then organized based on their implications for effectiveness and appeal. We employed an iterative thematic analysis process in which two members of the research team separately coded the debrief session transcripts, constructed a preliminary set of codes, then further refined and reconstructed the codes until all identified themes across the workshops and debrief sessions were consistently present when re-coded by a different member of the research team. Finally, these themes were mapped to the Tiers of Accessibility, Impact, and Appeal to better understand the strengths, weaknesses, limitations, and areas of potential improvement of the JUST-R metrics framework.

3 Results

3.1 Scoring

Results of scoring the JUST-R worksheets based on Tiers of Accessibility and Tiers of Impact are summarized in Figure 1. Overall, all worksheets in the assessment showed at least “slight evidence” of meeting Tier 1 of Accessibility (Understandable) and Tiers 1-3 of Impact (Awareness, Engagement with Material, and Problem Identification), with the majority of worksheets showing “distinct” or “overwhelming evidence.”

There was less evidence in the completed JUST-R worksheets of higher Tiers of Accessibility and Impact. While the majority of worksheets still showed “distinct” or “overwhelming evidence” of Tier 2 of Accessibility (Capable of Being Applied) and Tier 4 of Impact (Solution Identification), there were more JUST-R worksheets that showed “no evidence” or “slight evidence” of meeting these tiers when compared to lower tiers. Fewer than one-third of worksheets showed “distinct” or “overwhelming evidence” of meeting the highest assessed Tier of Impact (Articulation of Value of Identified Solution).

3.2 Themes

Four major themes were identified through qualitative coding of the JUST-R worksheets, facilitated JUST-R workshop transcripts, and debrief session transcripts. These four themes and their corresponding subthemes are displayed in Figure 2.

3.2.1 Theme 1: New perspective or viewpoint

Analysis of workshop and debrief session transcripts indicated changes in researcher perspectives after applying the JUST-R framework to their research projects. Researchers reacted positively to the framework’s ability to broaden perspectives and serve as a teaching tool, but also shared concerns regarding the potentially overwhelming scope of impacts the framework prompted them to consider and the difficulty of going against their field’s status quo. These observations are captured in the following four subthemes.

3.2.1.1 Broadening of perspective

Researchers described feeling enabled to undertake a more holistic appraisal of their research and gaining increased awareness of their work’s impacts from implementing JUST-R metrics. For instance, one researcher working with precious metals described being prompted to consider broader implications of mining such materials, rather than only considering their costs. Another researcher was spurred to consider emissions associated with energy consumption of supercomputing: “One thing I did that I’ve never done before is reach out to our supercomputer operations team and ask them, you know, how much emissions [the research] would generate.”

3.2.1.2 Scope creep

Thinking more expansively about their research and its impacts left researchers unsure of which parts of their research process to focus their attention on when attempting to apply the JUST-R framework. The potentially vast scope made it challenging for researchers to allot the necessary time and personnel to fully implement the JUST-R framework; as one researcher put it, “The further we cast our net, the more discussions we would need, the more people we would need to involve…” This interplayed with Theme 2, described below, wherein researchers cited needing more time and resources to fully implement the JUST-R framework.

3.2.1.3 Happy to participate

Applying the method triggered positive emotions like enthusiasm, interest, or realization in the researchers. Researchers reported positive reactions to engaging with their research in new ways, for instance: “It kind of forced me to think outside of my usual box … I don’t normally think that way every day; I feel happy with that.”

3.2.1.4 Method as a teaching tool

Researchers felt the JUST-R framework was useful as a teaching tool, which could be used to introduce energy justice concepts or promote critical thinking about the research process. They suggested the framework would be helpful to show early-career researchers, such as graduate students and postdoctoral fellows, that “these are things we should be thinking about” and to demonstrate a “holistic” approach to research.

3.2.2 Theme 2: Institutional support and incentives

Across technology areas and research projects, researchers indicated a need for improved incentives, resources, and supportive policies at the institutional level to effectively implement the JUST-R framework. This need for further support was represented across three subthemes: time, funding, and incentives; data and information availability; and expertise development.

3.2.2.1 Time, funding, and incentives

A central theme around the discussion of institutional support was the importance of time, funding, and incentive structures at the research institution and in research broadly. It is important to note that the projects selected for evaluation did not plan energy justice funding into their original scope, and only a small amount of funds was allocated to each project team from this study. A prominent portion of the discussion dealing with institutional support centered around aligning incentives for researchers. Energy justice remains a long-term incentive for researchers and research institutions, while shorter-term incentives like publishing, grant awards, and internal promotions do not always align well with integrating energy justice into research. A researcher noted, “I think even the most energy justice-oriented people or people who could be persuaded to be oriented that way have a hard time with that unless they know that that’s part of the incentive structure that they’re supposed to be reporting to, because otherwise their priorities need to be somewhere else.”

3.2.2.2 Available data and information

Researchers highlighted that certain information was not readily available to them to complete the JUST-R framework, especially institutional-level data (e.g., data on energy consumption related to machinery or tools or alternative materials that could be used in the research process). One researcher said, “It’d be great if the lab had summary estimate numbers of the environmental impacts and all the externalities of the different kinds of computing we do. That way can know what those impacts are and try to make them better by sourcing things better.”

Researchers also remarked that certain decisions or relationships are best developed at the program or institutional level. For example, researchers highlighted that to distribute research to non-traditional audiences effectively, including community groups, relationships with those audiences needed to be developed and maintained over time and across projects.

3.2.2.3 Expertise development

Effectively integrating and evaluating social science and non-academic sources requires specific expertise, as does engaging the public. Researchers highlighted that social scientists and individuals with other types of expertise would need to be added to teams using the metrics framework to ensure effective use of information and to respect the capacity of the principal investigator and technical research team. Expertise development also extended to the knowledge required to evaluate the metrics; several teams indicated they did not have the expertise to evaluate the hazard level of various materials or to test alternative parameters.

3.2.3 Theme 3: Energy justice understanding and responsibility

Researchers expressed the importance of more explicitly connecting the JUST-R framework to broader energy justice literature and outcomes throughout the evaluation process. The observations break down into three subthemes: lack of connection to energy justice concepts; energy justice responsibility; and insufficient solutions.

3.2.3.1 Lack of connection to energy justice concepts

Researchers noted that the current model of the JUST-R framework did not provide enough context or examples to apply to a broad base of projects. One researcher said, “When I finished, I was like, I don’t think I have a clear understanding of what energy justice really means. I mean, you made me think differently about my research, a different perspective that I never think about, like waste and how much energy you’re consuming and things like that. It gave me a different perspective.” Similarly, many researchers indicated that they did not feel prepared to evaluate energy justice and that they needed more information linking energy justice to their research. Researchers expressed a desire for the framework to provide more background or education on energy justice.

3.2.3.2 Energy justice responsibility

Similar to comments in Theme 2 highlighting the need for social science and community engagement experts, researchers indicated that they did not think applying energy justice to their research should always be the responsibility of the principal investigator. Although researchers were generally very supportive of the concepts and goals around integrating energy justice into basic research, several researchers questioned the responsibility for evaluating energy justice falling on individual principal investigators.

3.2.3.3 Insufficient solutions

Researchers highlighted that a main barrier to utilizing the framework to drive research outcomes was an insufficient solution space. Solution development was hindered by a lack of viable alternatives to existing research processes or by the presence of technical or ethical trade-offs (e.g., using automation would expose fewer workers to hazard risks but would also take away jobs). Several researchers noted that if they were looking to change the material inputs of their research, there might be very few alternatives. For example, one research team highlighted that they “have few options for where we can source materials due to needing very high purity source materials.”

3.2.4 Theme 4: Method design by technology readiness level and research area

Finally, researchers expressed the desire to see design improvements to the JUST-R framework. In particular, they indicated a need for greater specialization to certain research stages and technical areas to minimize irrelevant information, as well as for more useful—but not overly constrictive—examples and concrete next steps. These desired improvements were captured in six subthemes.

3.2.4.1 Value of examples and facilitation

Resources like completed example frameworks and facilitated workshops made applying the JUST-R framework easier and less intimidating. Researchers reported feeling overwhelmed without these resources, and many of the completed worksheets contained wording clearly inspired by the given example for certain metrics, such as those related to the dissemination of research results.

3.2.4.2 Recommends method for early implementation

Researchers felt that the method was primarily useful for evaluating research in its earliest stages, such as the proposal stage. One researcher referred to the framework as “an exercise for future thinking” and described it as less valuable for researchers close to publication, when there are limited options to “go back and redo.”

3.2.4.3 Desire for a more conclusive ending

Researchers expressed a desire for a more concrete “ending” upon completion of the method, such as a quantitative assessment or suggested next steps. This subtheme interplays with Theme 3A, described previously, with researchers hoping to see a simple and explicit connection between their work and energy justice upon completion of the framework, such as “a score that tells you whether your research is improving energy justice or is harming energy justice.”

3.2.4.4 Sense of redundancy/irrelevancy

Some of the JUST-R metrics seemed redundant or irrelevant to the researchers’ work or field. For instance, researchers felt certain metrics were more suited to assessing deployment-level work than earlier-stage technology R&D. In some cases, this was coupled with an appreciation for the challenge of meeting the needs of diverse research areas: “You’re never going to have a one-size-fits-all with this.”

3.2.4.5 Desire for a more specialized tool

Researchers expressed a desire for the JUST-R framework to be more specialized to their field, research methodology, research stage, and level of seniority within the research institution. While researchers generally felt the framework would be best applied at the earliest stages of a research project, as described in Theme 4 subtheme “Recommends method for early implementation,” they did not see this as entirely consistent across the metrics and suggested the framework could be further tailored for applicability to different points in a research project. They also felt they did not have control over the outcomes of certain metrics due to their standing within their institution (e.g., junior-level or early-career researchers), making these metrics potentially irrelevant, as in Theme 4 subtheme “Sense of redundancy/irrelevancy.”

3.2.4.6 Shrinking perspective

Researchers noted that providing examples sometimes resulted in fixations on certain problems and solutions, to the exclusion of developing other potential problems and solutions. This was also demonstrated by the observation that many completed JUST-R worksheets contained wording identical to, or clearly inspired by, the example worksheet, as discussed in Theme 4 subtheme “Value of examples and facilitation.” One researcher described the conflicting effects of providing examples: “If we hadn’t seen an example, we would have struggled more. But then of course, once you see an example, then you’re not thinking as wide as maybe you would otherwise.”

4 Discussion

The four major themes and sixteen subthemes that emerged from the qualitative analysis of the completed JUST-R worksheets, facilitated workshop transcripts, and debrief session transcripts describe key factors that shaped researchers’ experiences implementing the JUST-R framework and correspond to the evaluation criteria of Accessibility, Impact, and Appeal. Considering these themes and subthemes in conjunction with the evaluation criteria contextualizes the factors that determine how effective and appealing the JUST-R framework was and how those factors operate differently according to the Tier of Accessibility, Impact, or Appeal considered.

4.1 Tiers of accessibility

Analysis of the completed JUST-R worksheets showed evidence of the JUST-R metrics meeting both Tiers of Accessibility. However, analysis of the workshop and debrief transcripts indicates that researchers experienced considerable barriers at both tiers. The subthemes that provide insight into how accessible researchers found the JUST-R metrics framework and which aspects of the framework improved or hampered its accessibility are portrayed in Figure 3.

4.1.1 Accessibility tier 1: understandable

At Tier 1, barriers were primarily conceptual in nature and related to existing understandings of the research process and expectations about the JUST-R framework. As captured under Theme 3 by the subtheme “Lack of Connection to Energy Justice Concepts,” researchers often expressed surprise or disappointment that the JUST-R framework did not provide instruction in energy justice principles, which sometimes left them feeling unsure of how to contextualize the JUST-R metrics. A few researchers also expressed feeling that they lacked the skills or knowledge necessary to comprehend the JUST-R framework, which was reflected by the subtheme “Expertise Development.” These barriers were mitigated through the use of example worksheets and facilitated workshops.

4.1.2 Accessibility tier 2: capable of being applied

While some of the barriers to Tier 2 of Accessibility were also conceptual in nature (including experiences captured by the subthemes “Scope Creep” and “Desire for a More Specialized Tool”), institutional barriers to accessibility were articulated much more frequently with respect to this higher tier. These types of barriers were captured under Theme 2 by the subthemes “Available Data and Information” and “Time, Funding, and Incentives.” Unlike conceptual barriers, which exist only in researchers’ interactions with the JUST-R framework, institutional limitations operate within the triadic relationship between the research team, the JUST-R framework, and the laboratory in which the research team works. Though the design and implementation of the JUST-R framework may address barriers to the second Tier of Accessibility in part, supportive institutional-level policies are also critical and broaden the potential solution space for improving the accessibility of the JUST-R framework.

4.2 Tiers of impact

Despite high levels of engagement with the lower tiers of impact, both the incidence and quality of researcher engagement declined in the higher, more action-oriented tiers (Figure 4).

4.2.1 Impact tier 1: awareness

At Tier 1 of Impact, awareness of the JUST-R metrics was largely implicit (and thus not directly articulated) in the themes and subthemes that emerged from analyzing the worksheets and debrief transcripts, as these documents were produced after researchers had already been introduced to the JUST-R metrics. The scored JUST-R worksheets showed evidence of awareness of both the metrics and the problems that they seek to address, with almost two-thirds of the worksheets demonstrating “overwhelming” or “distinct” awareness. In the debrief and facilitated workshop transcripts, the prevalence of the subtheme “Lack of Connection to Energy Justice Concepts,” which describes researchers’ expressed expectation that the JUST-R framework would include more educational material about energy justice, may indicate that some researchers felt that their level of awareness of energy justice was not sufficient to complete the JUST-R framework.

4.2.2 Impact tier 2: engagement with material

Any initial limitations on researchers’ awareness of energy justice concepts did not appear to dampen their willingness to engage with the material in the JUST-R framework, Tier 2 of Impact. In the scored worksheets, there was more “overwhelming evidence” of researcher engagement with the material than at any other tier of evaluation, indicating a high level of substantial or high-quality engagement with the JUST-R framework. Researchers tended to express positive feelings about the process of completing the JUST-R worksheet, finding that it “made sense,” was “interesting,” and served as “a good reminder” to consider the social implications of their research. Frequently, researchers connected these positive feelings to a sense that the JUST-R framework had enabled them to see their research from a broader, more holistic perspective (as captured in the Theme 1 subtheme “Broadening of Perspective”). Researchers also reiterated the importance of having institutional support to engage with the JUST-R framework, including resources like the facilitated workshops.

4.2.3 Impact tier 3: problem identification

In Tier 3 of Impact, evidence of the effectiveness of the JUST-R framework remained high. More than two-thirds of the scored worksheets showed overwhelming or distinct evidence that researchers had identified energy justice problems within their research processes. Transcripts from the debrief sessions corroborate this finding, as researchers often offered examples of these problems and their effects on how the researchers thought about their research. In conjunction with these examples, however, researchers also described feeling that they were sometimes unable to identify problems in the research process due to the unavailability of certain types of lab-level data (e.g., the energy consumption rates of specific types of equipment). A few researchers also felt that referring to a completed JUST-R framework, which was provided to evaluation teams as an example, had inadvertently “shrunk” their perspective by causing them to fixate on (or even simply copy) the problems listed in the example (these instances were analyzed under the subtheme “Shrinking Perspective”). In addition to enabling the identification of more potential energy justice problems in their research processes, access to contextual, institution-level information may help researchers to think more expansively and creatively about those problems.

4.2.4 Impact tier 4: solution identification

The effectiveness of the JUST-R framework began to decline in Tier 4 of Impact. Although majority of the scored worksheets showed “distinct evidence” that researchers had been able to identify potential solutions to energy justice problems they had identified in their research processes, analysis of the workshop and debrief transcripts indicates that researchers felt uncertain about their ability to develop solutions. They often expressed feeling that they did not have an adequate understanding of energy justice concepts to propose less harmful alternatives to existing research processes (captured by the subthemes “Expertise Development” and “Lack of Connections to Energy Justice Concepts”). In some cases, institution-level data gaps complicated researchers’ attempts to evaluate alternative protocols and materials to use, as information regarding these alternatives was largely unavailable. Researchers also frequently described feeling a lack of control over potential changes to the research process, as factors like materials sourcing and lab protocols were often determined by the institution, rather than individual researchers.

4.2.5 Impact tier 5: articulation of value of identified solution

When researchers were able to identify potential solutions to specific energy justice problems, they often did not feel confident articulating the value that those solutions provided. In some cases, researchers’ feelings of ambivalence related to the solutions themselves, rather than the researchers’ ability to evaluate them: solving one energy justice problem sometimes seemed to create another, leaving researchers feeling uncertain about how (or whether) they could ethically choose one alternative over another. This dilemma sometimes made researchers feel that the solutions they had identified were insufficient (as reflected in the subtheme “Insufficient Solutions”).

Researchers’ concern about the broad implications of potential solutions relates to a general broadening of their perspectives on the effects of their research beyond the lab. Thinking expansively about the research process could be overwhelming, even to the point of discouraging action. Some researchers described not knowing where to end their energy justice evaluations while using the JUST-R framework, as they could conceptualize how their research’s sphere of impact could expand endlessly through a complex global network of supply chains, economic developments, and social effects (a concern captured by the subtheme “Scope Creep”). In such a complex solution space, defining the value of any particular solution can be a daunting task.

4.2.6 Impact tier 6: solution application

At the final Tier of Impact, there was little evidence, either in the scored worksheets or the transcript analysis, that researchers had been able to apply their proposed solutions. The impact of the JUST-R framework in these use cases thus did not amount to changes in the research process. This is likely due to a variety of reasons at multiple levels. At the smallest level, researchers noted difficulties related to the design of the JUST-R framework itself, specifically the lack of a clear “conclusion” to the JUST-R metrics framework that may have offered more guidance on how to implement changes in the research process. At the level of the research team, researchers sometimes described feeling that they did not have the expertise necessary to apply the proposed solutions, or that applying the solutions was outside of their job descriptions (as analyzed in the subtheme “Energy Justice Responsibility”).

Frequently, researchers felt that the most substantial challenges existed at the level of the institution, citing a lack of time, resources, or information about how to change certain research protocols. It should also be noted, however, that the design of the evaluation process may have introduced or exacerbated institution-level barriers in the application of researchers’ proposed solutions. The evaluation occurred during the last quarter of the laboratory’s fiscal year, which is often a particularly busy period for researchers. Higher-than-typical work demands may have allowed for less time to commit to energy justice efforts than during other times of the year.

Figure 4 provides an overview of the factors that influenced the framework’s impact on research processes from Tier 2 (Engagement with Materials) through Tier 6 (Solution Application) given the implicit nature of Tier 1 of Impact (Accessibility) that was previously mentioned.

4.3 Tiers of appeal

By linking the themes identified in the worksheet scoring and the thematic analysis of the transcripts to the Tiers of Appeal as seen in Figure 5, we better exemplify the levels of appeal in applying the JUST-R metrics framework, and similar tools, to technical early-stage research projects. While researchers generally find the framework to be valuable, the decision to continue using the method or recommend it to other researchers depends on institutional factors and the researchers’ topic areas.

4.3.1 Appeal tier 1: value

Tier 1 of Appeal, finding basic value in the framework, mapped to several subthemes, including “Broadening of Perspective” and “Sense of Redundancy/Irrelevancy.” In regard to broadening perspectives, researchers discussed that part of the framework’s value was providing a method to prompt new ways of thinking about scientific problems or using the method as a teaching tool to advance this thinking. However, researchers also highlighted that parts of the JUST-R framework are redundant or irrelevant to their research and position (captured under “Sense of Redundancy/Irrelevancy ”), and thus they did not find the framework as valuable as it could have been if it were better tailored to their work. This was particularly true for researchers who felt that the framework did not accommodate their research discipline.

4.3.2 Appeal tier 2: continued usage

The first consideration for Tier 2 of Appeal particularly related to the subtheme “Importance of Time, Funding, and/or Incentives.” The feedback from researchers highlighted that institutional support and research culture factors played a key role in their willingness to continue using the method. Specifically, researchers noted that their continued use of the method would hinge on whether more time, funding, incentives, and expertise were available to ensure that applying the framework was not an “unfunded mandate.” Researchers also noted that there was a firm desire for a more conclusive ending to the process (reflected under subtheme “Desire for More Conclusive Ending”). In other words, researchers wanted a more concrete outcome, score, or set of suggested next steps. They linked this desire for more explicit guidance directly to their likelihood to continue using the framework. Here, given the potential trade-offs and complexity of incorporating justice into energy R&D and other technological pursuits, it is important to note that a more conclusive outcome or ending may not be possible and researchers may have to grapple with a lack of an inherently “right” or “better” choice, especially when evaluating these choices from multiple social, economic, and technical viewpoints.

4.3.3 Appeal tier 3: recommendation

The final Tier of Appeal mapped most directly to three subthemes: “Method as a Teaching Tool,” “Recommends Method for Early Implementation,” and “Desire for a More Specialized Tool.” Several research teams indicated that they would not only recommend the JUST-R framework as a tool to evaluate energy justice in a given project but also as a tool to teach researchers about energy justice (“Method as a Teaching Tool”). Researchers also suggested they would recommend the framework to colleagues only in the very early stages of research project development and planning. In subtheme “Desire for a More Specialized Tool,” the evaluation highlighted that the type of research a colleague did would dictate their enthusiasm for sharing the framework. They indicated that applicability would drive their recommendation because the framework worked well for some research areas, such as material science and chemistry, and less well for others, such as grid analysis and modeling. This was likely due to the background of the research team that developed JUST-R, which leaned more towards the former areas, and highlights the value of evaluating the framework with researchers from a diverse array of technical areas.

4.4 Evaluation implications and future research

The JUST-R framework provides a method that could alter how scientists approach basic technology research. However, we can only expect widespread use of energy justice frameworks if they are accessible, appealing, and impactful. Importantly, this study highlights that researchers are enthusiastic about methods that can enable the serious consideration of energy justice in basic science. Still, ensuring broad uptake of the JUST-R metrics framework and other energy justice methods will require careful attention to improving method design and creating supportive institutional contexts for implementation.

4.4.1 Future work

Accounting for diverse technologies and research areas will be an important area of focus in the continuing development of the JUST-R metrics framework, allowing for more engagement across scientific disciplines. Throughout the evaluation, the need for the framework to expand to include specialized metrics for diverse technologies was prominent in participant feedback. Addressing this issue through future research is a critical outcome of the evaluation and will require engagement with scientists and researchers across various disciplines in the energy sector to ensure metrics are both helpful and relevant to their work. This specialization will not necessarily require creating a new version of the JUST-R metrics framework for each discipline in the energy sector, but rather leveraging the existing framework as the foundation for creating more domain-specific amendments. Researchers will be more likely to recommend the framework and find its implementation more impactful if their research is well-represented in the JUST-R metrics.

A lack of exposure to energy justice concepts, both generally and as these concepts relate to researchers’ specific fields of expertise, made it more difficult for researchers to understand the broader justice implications of the metrics they were evaluating and the implications of potential solutions identified. This finding suggests more targeted educational interventions either prior to implementing the JUST-R framework or in tandem with its use could bolster the effectiveness of applying the framework. Additionally, further clarifying the energy justice and responsible innovation principles that begot each metric in the framework itself could enable researchers to more concretely make the link between the metrics they are evaluating and these concepts.

Another key focus of future work will be more clearly defining the solution space for energy justice in the research process and better supporting the application of harm-reducing alternatives. In part, this will require refining the structure of the JUST-R framework. As currently designed, the framework promotes critical investigation into the energy justice implications of the research process using a semi-structured approach; it does not offer a checklist or rubric to score the research process’s energy justice “performance.” Feedback indicated that although the JUST-R framework’s current design helped researchers identify potential energy justice issues in their research, it did not help researchers identify solutions or alternatives. The diversity of possible interpretations of the metrics may produce critical and generative thought processes, but figuring out how to enact energy justice-centered improvements in the lab may require more structured guidance. It is crucial that future research account for both the desire to provide a critical, creative lens to evaluate research and the need to provide researchers with concrete next steps. Further, in this evaluation, researchers were only tasked with evaluating EJ in existing projects rather than formulating new questions. Future evaluations should compare results from researchers prompted to change their research questions and orientation at the beginning of their research projects and those who implemented the metrics into existing research.

Finally, solutions to institutional barriers, like incentives and project funding to accommodate energy justice metrics, would need to be in place to account for the increased workload associated with balancing energy justice evaluations and current research incentives. Although not a normative research topic, institutional barriers may be difficult to address. The results of this evaluation suggest that research institutions committed to energy justice will need to prioritize mitigating time, funding, and information barriers to effectively integrate energy justice into basic science research. Additionally, better understanding of the influence of various actors, and thus their responsibilities, in our pursuit of a just energy transition requires a broader view of the interplay among these actors, such as researchers, their various institutions, decision-making processes, and impacts, and the complexity thereof.

Separate from the specific themes identified through the analysis, the study and evaluation process also highlighted the momentum behind integrating Diversity, Equity, and Inclusion (DEI) into research teams and institutions. These initiatives are critical to advancing equity in both research processes and research outcomes. However, we found that there was a broad tendency to conflate DEI and energy justice when discussing interventions in the research process. Researchers were quick to revert to DEI concepts rather than focusing on the outcomes of research and how they impact disadvantaged communities. We hypothesize that the continued use and impact of the energy justice evaluation methods will hinge on the proliferation of energy justice principles and education, including an emphasis on how DEI and energy justice are distinct yet complementary concepts.

4.4.2 Broader implications

Beyond the scope of the JUST-R framework, the findings of this evaluation point to the challenges and opportunities that may arise when attempting to integrate social impact and justice metrics in basic science. Our results indicate that integrating energy justice metrics should be possible when the right incentives and infrastructure are in place and could prompt transformational changes to research culture and eventual technology outcomes. As more frameworks emerge to evaluate social impact and justice metrics in research, this study points to the importance of responsively evaluating and improving these methods to ensure usability in a basic science context.

Data availability statement

The datasets presented in this article are not readily available because of privacy concerns. Requests to access the datasets should be directed to EG. Requests to access the datasets should be directed to EG, elizabeth.gill@nrel.gov.

Ethics statement

Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

Conceptualization, BA; methodology, BA, ND, EG, CH, AC, KA, and KF; investigation, BA, CH, ND, and EG; writing—original draft, BA, CH, ND, and EG; writing—review and editing, ND, EG, BA, CH, KF, AC, and KA; project administration, KA and EG. All authors contributed to the article and approved the submitted version.

Funding

This work was authored in part by the National Renewable Energy Laboratory (NREL), operated by Alliance for Sustainable Energy, LLC, for the US Department of Energy (DOE) under contract No. DEAC36-08GO28308.This work was supported by the Laboratory Directed Research and Development (LDRD) Program at NREL. The views expressed in the article do not necessarily represent the views of the DOE or the US Government. The US Government retains and the publisher, by accepting the article for publication, acknowledges that the US Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for US Government purposes.

Acknowledgments

The authors gratefully acknowledge Wilson Mcneary, Anne Starace, Emily Warren, Michael Griffin, Jaylynn Medina, and Douglas DeVoto for providing feedback on the evaluation process and analysis. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE- 2039655. BA also acknowledges funding support from the Georgia Tech Brooks Byers Institute for Sustainable Systems and Alfred P. Sloan Foundation.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fenvs.2023.1206013/full#supplementary-material

References