Short Description
Last summer, the Illinois Youth Investment Program (IYIP) was established to provide Illinois' at-risk youth ages 16-24, with the opportunity to participate in the employment recovery of the State. The Illinois Youth Investment Program (IYIP) is a multi-faceted approach to youth employment that invests in the future of Illinois' at-risk transition-age youth (16-24). This approach to youth employment accounts for the youth's employment barriers as well as the physical, emotional, social, and mental health needs while helping them to secure and sustain long-term and/or career employment, ensuring a greater likelihood of success and self-sufficiency. Providers could offer programming in one category, or multiple categories.
• Category 1: Short-term, summer employment
• Category 2: Targeted long-term employment
• Category 3: Career Development - Industry-linked long-term employment
• Category 4: Expansion of industry-Linked career development opportunities
IDHS developed performance measures and standards based on the program categories. IYIP providers report these measures monthly, or as needed, to IDHS. The providers report on youth placement and participation in employment, apprenticeships, and post-secondary education. The IYIPE vendor will work with the BPYD to create an evaluation plan to conduct research with a goal to understand IYIP operations, data measures and collection, evaluability, and make recommendations for program improvement and preparation for future evaluation.
Objective
The IYIPE vendor will work with the BPYD to create an evaluation plan is to conduct research with a goal to understand IYIP operations, data measures and collection, evaluability, and make recommendations for program improvement and preparation for future evaluation.
The research goal is to understand IYIP operations, data measures and collection, evaluability, and make recommendations for program improvement and preparation for future evaluation. We seek to collect data to answer the following research questions.
1. What is IYIP and how do program providers operate?
2. To what extent do IYIP providers operate similarly or differently?
3. What are IYIP provider attitudes toward, and readiness for, evaluation?
4. How is data collected, entered, shared, used, and overseen by providers and IDHS?
5. To what extent is the data accurate, complete, and consistently collected by providers?
6. What is needed to conduct an IYIP outcome evaluation?
ICJIA will work closely with IYIP stakeholders, including those from IDHS and individual providers, to establish a leadership team to finalize our research plan. We will hold regular meetings with the leadership team to discuss issues, answer questions, and share progress. Additionally, staff will work closely with data system managers. The project may require the execution of interagency data use agreements (DUAs) or Intergovernmental Agreements (IGAs). We will submit as evaluation, rather than research, to the ICJIA IRB secretary.
ICJIA proposes conducting a process evaluation to describe IYIP operations across provider sites. This will involve creating logic models, reviewing relevant documents, and to the extent possible with accurate and complete data, analysis. We will use Microsoft Excel or SPSS (Statistical Package for the Social Sciences) to run descriptive statistics, and if possible, bivariate statistics and more.
ICJIA will work with the leadership team to establish a logic model for use in the study. Logic models are used to visually depict the dynamics between inputs (e.g. programming components), outputs (program impact), manners of measurement, and goals over time. This model will help best assess readiness for evaluation, as lack of clarity surrounding key program elements to be measured could prevent the inclusion and appropriate treatment of these features (Centers for Disease Control and Prevention, n.d.; Community Toolbox, n.d.).
ICJIA will review and synthesize all relevant program documents, performance measurements and standards, and deliverables. Document review and analysis is a qualitative research method to gather knowledge of a program for evaluation. Sourcing information from multiple points within can guide other data collection (e.g. interviews, surveys, focus groups), as well as validate and understand the data collected (Bowen, 2009).
Therefore, we anticipate reviewing grant NOFOs, applications for funding, and other grant-related documents. We will review the assessments selected for the program for appropriateness (reading level, length), reliability, validity, prior studies of use, completeness, and accuracy. Such as:
• Worksite professional skills assessment (pre and post)
• Illinois' essential employability skills assessment
• Casey life skills assessment
• Career cluster inventory
• Employment 101 pre-assessment
ICJIA expects to review data systems used such as Illinois workNet or other data system as directed by IDHS. We will want to learn how data is entered and to what extent data is reviewed for quality assurance. We will also review and include the relevant peer reviewed and grey literature to provide context, background, and support for any findings or recommendations.
Many programs, especially those newly implemented, may not be prepared for evaluation. Beyond data quality and consistency, performance measures (such as those designated for IYIP) do not always translate to suitable outcomes that can be measured for evaluation. Outcome evaluations need a viable comparison group and data over time to measure impact or change. As such, ICJIA’s inclusion of an evaluability assessment along with a process evaluation will yield the necessary information and data insights for a complete and accurate future evaluation. An evaluability assessment can help programs determine the viability, feasibility and necessity of an evaluation; decide if an evaluation can contribute to improved program administration or management; as well as illuminate the tasks and changes needed for future evaluation (Centers for Disease Control and Prevention, n.d). An evaluability assessment can save time and resources which can be used for a future evaluation.
ICJIA will employ field research in-person to conduct field observations and focus groups with IYIP provider staff.
Field observations. We will employ stratified random sampling to select program sites conduct field observations. We propose observing up to six IYIP sites and since the majority of grantees are in Region 1, and specifically in Chicago, we will conduct more in that region based on the following.
• Region 1: 2 Chicago, 1 non-Chicago
• Region 2: 1
• Region 3: 1
• Region 4: 1
ICJIA will schedule an in-person site visit in their offices. We will observe the office dynamics such as location, design, size, equipment, staffing, and clients. We will ask to review paper or electronic documents on site such as:
• Case plans
• Participant files
• Employment equity plan
• Payroll documentation
• Youth assessments
Focus groups. During the site visit, we will meet with the program director, a program staff person, and a person who enters data for the grant. We will develop semi-structured interview questions to ask each program site. Questions may ask about the provider organization, program operations; program strengths/ weaknesses; performance measures and standards; and data collection and entry. We will record the focus groups and transcribe them for analysis. We will analyze the data using NVivo qualitative software and analyze by site and collectively by theme.
Survey
ICJIA will design, administer, and analyze an online Qualtrics survey to all IYIP providers. This survey will supplement the more in-depth data collection occurring during site visits and focus groups. This survey will gather information from all IYIP providers. We may ask questions about the provider agency; IYIP operations; provider attitudes toward, and readiness for, evaluation; how data is collected, entered, used, shared, and overseen; and data accuracy and completeness.
There will be up to three deliverables. All deliverables will be reviewed by IDHS and ICJIA. Reports will be published on the ICJIA website.
• IYIP evaluability assessment report. This will include a brief summary of related literature, program description, methodology, findings, and recommendations. The recommendations will provide ways the program can improve data collection and set themselves up for future evaluation.
• Process evaluation data report. If existing IYIP data is found to be useable, we will create a data report containing descriptive statistics, data summaries, and any advanced statistics if plausible.
• Final presentation to leadership team and stakeholders. We can present a summary of findings and discuss them with IDHS.