Measuring Effectiveness of San Diego County Homelessness Programs

San Diego Taxpayers Educational Foundation, Brandon Miner, Harry Pachchala, Alyssa Escamilla, Diego De La Toba, Ryan Penkala

ABSTRACT

This study evaluates the effectiveness of homelessness intervention programs in San Diego County using a mixed effects modeling approach. By analyzing comprehensive datasets, including annual pro gram expenditures and point-in-time counts of homelessness across cities, the programs most effective in reducing unsheltered homelessness rates were identified. The analysis accounts for city-specific vari ations, such as local policies and economic conditions, by treating ”city” as a random effect. The results indicate that flexible funds and rapid rehousing programs are the most effective at reducing homelessness rates, while transitional housing, housing stability services, and emergency shelters show limited or counterproductive impacts. The study highlights the importance of tailored policy recom mendations to optimize resource allocation and effectively reduce homelessness, despite limitations in data granularity.

1. RESEARCH QUESTION

The San Diego Taxpayers Educational Foundation seeks to determine which homelessness programs in San Diego County are the most and least efficient in reducing the number of people experiencing homelessness.

2. DATA

2.1. Data Sources

The input variable is year to year municipal home lessness program expenditures from 2015 through 2022, which are accessed from each city within San Diego County through public records requests and compiled by SDTEF data scientists.

The response variable is Point-in-Time Count (here inafter ”PIT count”) which is posted annually by the Regional Task Force on Homelessness (hereinafter ”RTFH”). The PIT count is conducted in January each year and is a snapshot measure of the number of people experiencing homelessness in the county of San Diego on that specific day.

2.2. Addressing Bias

Researchers identified a potential source of bias stem ming from significant variations between cities within San Diego County. For example, cities like Coron ado consistently reported lower rates of homelessness, whereas cities like El Cajon consistently had higher rates. During exploratory data analysis, researchers de termined that home values of each city had very little correlation with annual PIT count. This is likely due to the high-cost nature of home ownership within the re gion. That is to say, San Diego County home values in a city with a large homeless population versus those of a city with a small homeless population are very similar. Failing to account for differences such as local govern

ment policies, rental markets, quality of education, and overall economic conditions in these cities could intro duce bias into the analysis.

An additional potential source of bias in the follow ing analysis was the inability to directly compare vari ables of differing scales. Larger-scale variables, such as program funding measured in millions of dollars, could dominate the model, overshadowing smaller-scale vari ables. This imbalance could lead to biased results, as the model might overemphasize the impact of larger vari ables while underestimating the significance of smaller ones.

To address the potential for bias within this study re searchers use a mixed effects model, which allowed an alysts to treat ”city” as a random effect to account for variations between cities across the county. Addition ally, the data is standardized by rescaling all numeric variables to have a mean of zero and a standard devi ation of one. This ensured that all variables were put on the same scale so that they could be accurately and fairly analyzed.

2.3. Limitations

This analysis is limited to using the annual PIT count, which is provided to the public by the RTFH. While the RTFH collects deidentified, individual-specific enroll ment data on people experiencing homelessness (here inafter ”PEH”), they unfortunately do not allow this data to be accessible by the public, and thus researchers are limited to just the annual PIT count.

The PIT count provides a one-day snapshot of the number of people experiencing homelessness in San Diego County. However, because it is an annual mea sure, it limits researchers’ ability to capture fluctuations in homelessness that may occur throughout the year.
2

Figure 1. A box plot showing the distribution of unshel tered homelessness rates across cities.

Factors such as economic challenges or seasonal changes are not reflected in this data. Consequently, the true effectiveness of the various homelessness programs eval uated in this paper remains uncertain.

3. ANALYSIS

3.1. Methodology

This study employs a mixed effects modeling approach to evaluate how homelessness intervention programs im pact unsheltered rates per 100,000 residents across cities in San Diego County. This method allows one to assess the overall effectiveness of various programs while ac counting for differences in homelessness trends unique to each city. The analysis follows a clear sequence of steps, outlined below, to ensure robust and interpretable results.

The first step was to examine how unsheltered home lessness rates differ across cities in San Diego County. As shown in Figure 1, a box plot reveals significant vari ability between cities. Cities like Coronado consistently report lower unsheltered homelessness rates, which are evident in narrow and lower-positioned box-and-whisker plots. In contrast, cities like El Cajon show a much broader range of values and higher median rates. This variation emphasizes the importance of accounting for city-specific factors in the analysis to prevent mistak enly attributing these differences to program effects.

3.2. Standardization

To ensure the analysis accurately reflects program im pacts, the data was normalized. All numeric variables, such as program funding, were rescaled to have a mean of zero and a standard deviation of one. This step en sures comparability between variables measured in dif ferent units. Standardization was particularly crucial because some variables with larger magnitudes, such as
Figure 2. Homelessness rates in each city without impact of programs.

funding, could disproportionately influence the model. Rescaling these variables ensures all predictors con tributed equally to the model.

3.3. Mixed-Effects Model

The mixed effects model was developed to quantify the impact of various programs while accounting for dif ferences among cities. In this approach, fixed effects represented the average effect of each program, such as rapid rehousing or flexible funds, on homelessness rates across all cities. Random effects accounted for variations unique to each city. For example, factors such as local policies or housing market conditions that are not explic itly measured can influence homelessness rates. Treat ing ’city’ as a random effect allows the model to include these unobserved influences.

By incorporating both fixed and random effects, the model distinguishes between general program impacts and city-specific contexts, providing a clearer under standing of what works universally versus what might vary locally. As illustrated in Figure 2, the random effects capture how homelessness rates differ between cities when fixed effects like program impacts are ex cluded. The wide variability among these intercepts un derscores the influence of city-level factors (e.g., local policies and housing markets) and further solidifies the need for mixed-effects modeling in this context. After developing the model, researchers estimate the effects of each intervention program on unsheltered homeless ness rates. These estimates, called fixed-effect coeffi cients, indicate the direction and strength of each pro gram’s impact. A positive coefficient means that the program is associated with an increase in unsheltered rates, which may occur if the program is deployed in areas with greater homelessness needs. Conversely, a negative coefficient suggests that the program is associ-
3

Figure 3. A bar chart of fixed-effect coefficients ranked by their estimated impact on unsheltered homelessness rates.

ated with a reduction in unsheltered rates, highlighting its potential effectiveness.

As shown in Figure 3, flexible funds and rapid re housing programs stand out with the most negative co efficients, demonstrating their strong association with reduced homelessness rates. On the other hand, transi tional housing, housing stability services, and emergency shelters show positive coefficients, suggesting that they may correlate with areas experiencing higher unshel tered rates or are not as impactful in reducing homeless ness. The visualization effectively ranks these programs, making it easier to assess their relative effectiveness.

3.4. Multicollinearity

To address warnings generated by the model, re searchers conduct a Variance Inflation Factor (VIF) analysis to assess multicollinearity. Multicollinearity oc curs when two or more predictors are highly correlated, making it difficult to isolate the individual impact of each program. VIF measures how much the variance of a predictor’s estimated coefficient increases due to cor relations with other predictors.

The analysis revealed that the predictor “work for hope” had an extremely high VIF, indicating it was almost perfectly collinear with other variables in the model. As a result, “work for hope” was removed from the model to resolve this issue and ensure the accuracy of the remaining estimates.

3.5. Residual Analysis

After the model was constructed, residuals (the dif ferences between observed and predicted homelessness rates) were analyzed to assess its fit. As shown in Figure 4, the scatter plot of residuals demonstrates a roughly random pattern around zero, confirming that the model’s assumptions of homoskedasticity (the vari
Figure 4. A residual scatterplot showing the distribution of residuals around zero.

Figure 5. A QQ plot illustrating the normality of residuals.

ance of the error term in the model is constant) and linearity were largely met. However, records with the largest residuals (e.g., outliers appearing far from the zero line) were flagged, suggesting that their homeless ness rates may be influenced by unmeasured factors not captured in the model.

Additionally, Figure 5 shows the QQ plot of residuals, which evaluates whether they follow a normal distribu tion. Most residuals align closely with the diagonal line, indicating a reasonably normal distribution. However, some deviations at the extremes point to a potential underestimation of variability in certain cities. These flagged records prompted further analysis to examine the robustness of the findings when outliers were ex cluded. Based on the residual analysis, cities with residuals exceeding two standard deviations from the mean were identified as outliers. These cities were ex cluded in a secondary analysis to test the robustness of the findings. Figure 6 shows the fixed-effect coefficients after outlier removal. While some magnitudes shifted (e.g., transitional housing coefficients became slightly less positive), the relative rankings of program effective ness remained consistent. This consistency suggests that the primary conclusions of the model are robust even when potential outliers are excluded.
4

Figure 6. Fixed-effect coefficients after excluding outliers.

4. DISCUSSION

4.1. Summary of Findings

The analysis revealed that there was a significant dif ference in the effectiveness of each homelessness preven tion program, which was reflected in their fixed effects coefficients. Researchers found the three most effective programs in reducing homelessness to be flexible funds, rapid rehousing, and family reunification programs. The three programs associated with the highest increases in homelessness rates were housing stability services, emer gency shelter, and transitional housing.

These findings are consistent with prior intraregional research on homelessness, identifying preventative pro grams as being more effective than crisis management programs.

4.2. Effects of Limited Access to Regional Data

The Department of Housing and Urban Development (HUD) requires that each Continuum of Care (CoC) in the country keep and maintain the Homeless Man agement Information System (HMIS). The arbiter for San Diego County’s HMIS is the RTFH. City managers throughout the region have confirmed that this data is public, and should be shared with researchers upon re quest. Other CoCs in southern California openly share HMIS data when requested, indicating that refusal of such data is uncommon. As stated above, the RTFH has expressed disinterest in sharing their data, despite repeated efforts to request such data by the San Diego Taxpayers Educational Foundation.

Because crucial enrollment data for people experienc ing homelessness in San Diego County is unavailable to the public, researchers must rely on incomplete infor mation which heavily limits the accuracy of predictive models.

REFERENCES

1. San Diego Union-Tribune, Why won’t San Diego publicly share detailed data about who’s enrolled in homelessness programs?, 2024. Available at: https://www.sandiegouniontribune.com/2024/05 /25/why-wont-san-diego-publicly-share-detaile d-data-about-whos-enrolled-in-homelessness-pro grams/. Accessed: November 12, 2024.

2. Homelessness Services Analysis Draft, 2024. Available at: https://static1.squarespace.com/ static/5af07fd050a54f8fc370c748/t/66310b91b146 05521ab44c6b/1714490257610/20240501+Homele ssness+Services+Analysis+Draft.pdf. Accessed: November 12, 2024.

3. Regional Task Force on Homelessness San Diego (RTFHSD), Reports and Data, 2024. Available at: https://www.rtfhsd.org/reports-data/. Accessed: November 12, 2024.