Learn more about

Evaluation Frameworks

One of the most significant challenges in implementing equitable strategies is identifying how to measure equity and impact. The outcomes and issues the EDD/EDO is seeking to address with equitable strategies take time to fix, often resulting from generational strategies that either contributed to or perpetuated marginalization across many communities and racial groups. When creating indicators for the region, consider these guideposts for the more significant long-term impact. This approach of explicitly incorporating equity within metrics and indicators can help each CEDS update build upon each other, providing direction for future advisory councils and stakeholders in the region. To create a range of guideposts for measuring the progress of equitable growth, consider three key areas: macro-level, program-level, and qualitative indicators. 

  • Macro-level Indicators: Build on the use of macro-level data throughout the CEDS by identifying key long-term regional equity metrics. 

  • Program-level Indicators: Craft shorter-term metrics for programs and activities related to CEDS priorities to ensure longer-term progress. 

  • Qualitative Indicators: Incorporate community voice and tell a more complete story of the region by incorporating data from interviews, focus groups, and surveys. 

Resources 

  • Data will be incorporated and referred to throughout the entire CEDS, from showcasing trends in the summary background to identifying priorities in the regional analysis. These are typically broad macro-level data points, such as income, poverty, and education across race and geography. These indicators don’t often change dramatically on five-year timelines. This section is the opportunity to outline metrics for longer-term analysis across future CEDS will help benchmark progress further into the future.

    Disaggregate indicators by demographics. As discussed in this guide, disaggregating data is the first step in approaching data equity. Disaggregated data is needed to fully know who is impacted and how systemic barriers may influence outcomes. When measuring an overall poverty reduction, there must be a demographic lens to ensure racial, gender, or other disparities are addressed. When identifying macro-level indicators, ensure that several are demographic-specific, including race, age, gender, or geography.

    When identifying peer regions, incorporate demographics. Some EDDs like to include a peer comparison metric within their CEDS to provide regional context and comparison for the regional economy. These comparisons are only as useful as the methodology used to identify them. In identifying peer metro areas, include equity and demographic metrics to ensure alignment with selected comparison regions, including racial segregation and income inequality.

    Idea: Understanding Migration


    Among the common descriptors that individuals use to describe their region is growing or declining. While growth, regarding population, jobs, or industries, is an important metric to better understand the movement of residents, migration data can illuminate population trends alongside descriptive demographics, including if young adults are staying in the region or Black residents are leaving the region. When broken out with demographics, these trends can provide a fuller picture of equity changes over time. For example, in St. Louis, migration trends have shown that many factors, including historic disinvestment and the rising cost of living, have led to Black residents' displacement from historically Black neighborhoods. Additionally, this data illuminates how federal rural investments may impact youth outmigration. By better understanding migration, steps can be taken to leverage shrink-smart or smart-growth strategies within the region. Explore more through the Census Bureau’s Migration Data searchable tools and maps.

    Example: Sacramento Area Council of Governments | California, Pop. 2,397,382 (6 counties)

    In their 2020 CEDS, the four organizational CEDS partners identified impact metrics across four areas, including regional inclusive economic indicators, innovation, and business growth indicators, workforce indicators, and transportation infrastructure. Several of these metrics focus on equity, including metrics by race, gender, and the use of the Brookings Institution’s Metro Monitor inclusion and prosperity metrics.

    Idea: Potential Indicators


    When crafting indicators for the region, potential equity metrics could include:

    Gini Index: A metric of economic inequality, the Gini coefficient summarizes the spread of income across the population, from equally shared to held by one or a few individuals.

    Theil’s H Index: A metric of racial segregation that indicates how evenly members of racial and ethnic groups are distributed across a region as a whole.

    Living Wage: Understand how cost burdens, such as housing and transportation, impact low-wage workers can be understood by calculating a living wage and exploring metrics related to income and living wage by racial and demographic groups.

    Racial Wealth Gap: Metrics that look at average household wealth gaps across racial groups serve to illustrate inequities. For example, comparing the average wealth gap between households with a white householder against households with a Black householder could serve as a metric to understand racial wealth gaps regionally.

    Resource: Innovative Data

    Explore the Innovative Data section of this guide for a compilation of data equity tools.

  • Building on macro-level indicators, incorporate program-level metrics to better illustrate the regional organization's role and foster transparency and accountability. When launching and leading programs related to identified regional priorities, set shorter-term metrics about these activities to support the longer-term macro-level indicators.

    Consider evaluation metrics in program planning. When translating the CEDS goals into objectives and activities, ensure that all program planning includes evaluation at the outset. Using frameworks like a theory of change or a logic model, begin with the program's intended goal and then work backward to ensure that the intended impact on the community is at the center of the activities. 


    Build research partnerships. Look at local community assets, such as anchor institutions and potential university partners, including EDA-funded University Centers, with resources and insight into designing program evaluation methods. Seek ways to build deeper partnerships with these institutions, both as strong economic drivers and in the insights they can support in developing metrics and indicators for short-term and long-term success. Take steps to formalize these agreements with memorandums of understanding and data-sharing agreements to protect the privacy of any participants.

    Idea: Outputs and Outcomes


    When looking at the intended impact of EDD/EDO programs, give thought to both the short-term and the impact. In programming planning, through tools like logic models, craft both outputs and outcomes for each program led by the EDD/EDO. Be clear about the differentiation of these two items; an output metric may not translate to a broader impact, and outcomes can’t be evidence-based without output metrics. Both metrics should be specific about demographics to ensure that an equity lens is being applied to program evaluation.

    Outputs: Identify the direct evidence of the activity. For example, how many people were trained, how many referrals were made, or how much funding was distributed.

    Outcomes: Identify the result of the activity. For example, how many people found employment, how many people received business licenses, or how many jobs were created. 


    Use this example Economic Development Logic Model from the EDA to identify outputs and outcomes with program partners to ensure consensus, buy-in, and success in reaching broader goals.

    Example: Greater Portland, Inc. | Oregon, Pop. 1,832,634 (4 counties)


    In their 2022 CEDS, the EDD team built an interactive online matrix of their action plan, which included all the objectives, related outcomes, corresponding metrics, and leading organizations. These metrics include tracking program participants by race and metrics for participation in a regional equity summit.

    Resources

  • Qualitative research creates the opportunity to embed community voice directly into the CEDS. Qualitative data is essential in telling the full story of the regional economy, providing deeper context and consideration to the story the larger data tells. Create opportunities for qualitative insights by hosting public forums or listening sessions, conducting targeted focus groups, and collecting broader local community surveys. These efforts should be embedded through the CEDS development process, not just at the outset or after a draft has been completed; there should be opportunities throughout to provide the community with room to provide input, feedback, and proposed changes.

    • Collect demographics. Consider how to collect demographic information for any survey, focus group, interview, or another type of qualitative research. With these demographics, parse out results, insights, and experiences by gender, race, age, and beyond. When seeking to address systemic barriers, use information that is broken out to identify differences in experiences. Consider making these identifiers optional to allow for the comfort level of participants or respondents. Additionally, explore how these questions are worded and the response selections available. For example, regarding gender, consider offering alternative options beyond just male and female to account for a range of identities. 


    • Collaborate with strategic partners for data collection. When identifying strategic partners, consider how they may support the distribution of any surveys or hosting focus groups, interviews, or events. The trusted role and relationship these partners may have would allow for greater feedback and insights for the CEDS. Explore how trusted partner locations, such as public housing facilities, libraries, or public schools, in specific areas in the region could be more accessible to specific audiences. The partnerships formed during the CEDS development process significantly impact the final document and strategies within it.

    Example: Chittenden County Regional Planning Commission | Vermont, Pop. 58,191 (4 counties)


    When compiling the 2022 CEDS for the West Central Vermont region, the commission worked closely with partners to host regional economic development workshops. Early in the process, one of these CEDS Workshops was co-hosted with the Vermont Racial Justice Alliance. This workshop sought to inform leaders and partners across the region about the CEDS development and begin gathering feedback from stakeholders that would inform their SOAR analysis. Since this meeting, the commission has released two drafts of their CEDS for continued community feedback with multiple avenues for feedback virtually and in-person. Learn more here.

    Resources

    Who's Coming? Respectful Audience Surveying Toolkit: To inform survey design, this toolkit from OFBYFOR ALL provides insights to thoughtful collect demographic information.

    Qualitative Research Practice Guide: Code for America shares their qualitative research best practices in this guide, which includes considerations for ethics, interviews, and analyzing data.

    Chittenden County Regional Planning Commission | Vermont, Pop. 58,191 (4 counties)


    When compiling the 2022 CEDS for the West Central Vermont region, the commission worked closely with partners to host regional economic development workshops. Early in the process, one of these CEDS Workshops was co-hosted with the Vermont Racial Justice Alliance. This workshop sought to inform leaders and partners across the region about the CEDS development and begin gathering feedback from stakeholders that would inform their SOAR analysis. Since this meeting, the commission has released two drafts of their CEDS for continued community feedback with multiple avenues for feedback virtually and in-person. Learn more here.

    Resources

    Who's Coming? Respectful Audience Surveying Toolkit: To inform survey design, this toolkit from OFBYFOR ALL provides insights to thoughtful collect demographic information.

    Qualitative Research Practice Guide: Code for America shares their qualitative research best practices in this guide, which includes considerations for ethics, interviews, and analyzing data.

Review your current CEDS

  • What macro-level indicators did you identify? Are any indicators focused on race, gender, ability, or geography within the region? If you identified peer regions, did you include demographic considerations? Have you explored migration data related to the region? What metrics did you use to measure inequality or segregation?

  • What program-level indicators did you identify? Are any indicators focused on race, gender, ability, or geography within the region? How were these indicators integrated in program development and planning of your activities? Did you identify outputs and outcomes for programs internally? Who are your research partners?

  • How did you collect qualitative information? How is this information reflected in the CEDS? Did you collect demographic information at events and in interviews, focus groups, and surveys? Did you include quotes? Did you include stories to illustrate priorities?