Working with the UDA and MCDS: a comparative experience

Promising developments in the compilation and availability of individual and unit-level public education data as aggregated in State Longitudinal Data Systems (SLDS) has motivated the development of the SDLS profiles for each of the 50 states state SLDS profiles forming the core of the State Longitudinal Data Systems Research3 study and its publicly available website.  The resources resulting from this research stand as objective reports and assessments of the various state of SLDSs as portrayed through National Center for Education Statistics (NCES)4 funding applications and reports, state and agency sponsored public education websites, Data Quality Campaign (DQC)5 reports, and comments offered by respective state agency representatives.  However, these profiles and resources only tell part of the story.  This brief compare and contrasts researcher experiences in working with two of the nation’s highly regarded SLDSs offers a glimpse into some of the benefits and challenges of accessing and relying on SLDS data in support of public education policy research, and decisions.

The Promise of the Nation’s SLDSs

Though policy decisions made throughout public education in the hopes of improving the efficiency and effectiveness of student and household outcomes are made by well-intended administrators, educators, and legislative representatives, the troublesome reality has been that it’s extraordinarily challenging to measure such policy outcomes and their related effects.  Like all of social sciences research, education policy research suffers from substantial endogeneity and self-selection bias, making it difficult to infer a causal link between policy and outcomes.  If only there were an available data resource reflecting the experience of each public education student and institutions within a particular state with such granular detail as to support assessment of the efficacy of program changes and policy initiatives.  Such a resource would be required to link each student records throughout the public education and eventual workforce experience, and be most valuable if it included supporting household data.  Given the coordinating role played by primary, secondary and higher education agencies in each state and the oversite of the federal level public education agencies, such a data repository is not only feasible, but expected in the era of big data.

As a result, the State Longitudinal Data System (SLDS) was born and those states qualifying for federal level development support, 47 states in all, experienced early nourishment from more than one billion dollars of federal grant funding in support of public education’s lofty goals.  Though all data points are linked at the student level via a unique state student identifier, the SLDS data, once compiled, is de-identified and its reporting is subject to state and federal student privacy statutes.

States began receiving federal SLDS development funding in 2005 – now, twelve years later, we’re able to clearly see where these data systems have met their promise and where expectations have been disappointed.  Trends in system architecture, management, ongoing support, completeness, and accessibility are observable, as are the struggles to keep such a common sense effort alive and relevant.

A review of publicly available information regarding two of the nation’s State Longitudinal Data Systems (SLDS) – Utah’s UDA and Missouri’s MCDS – suggests each is well resourced, interoperable between relevant state agencies, and accessible to stakeholder and outside researchers alike.  However, suggestions do not necessarily turn into realities and while there are SLDSs, there are also data systems that do not live up to their promise.

This brief includes a limited overview and anecdotal retrospective of one research teams experience in working with Utah’s SLDS as managed by the Utah Data Alliance (UDA)6 and Missouri’s SLDS, the Missouri Comprehensive Data System (MCDS)7, as managed by the Missouri Department of Elementary and Secondary Education (MO DESE)8.   This includes the experiences of Richard Haskell, PhD,  Peter Seppi, and fellow researchers9 in Westminster College’s10 Bill & Vieve Gore School of Business11 while working with the UDA and MCDS in support of public education related research sponsored by the Ewing Marion Kauffman Foundation12.  This is not intended to be a formal program assessment, and may not be indicative of the experience of others working with these same or other SLDSs, but it does give an interesting perspective on the differentiated nature of the nation’s SLDSs and the fulfillment of their early promises.

Working with the UDA

The Utah Data Alliance (UDA) manages Utah’s SLDS13 under a centralized governance system inclusive of stakeholders representing the Utah System of Higher Education (USHE)14, Utah State Office of Education (USOE)15, Utah College of Applied Technology (UCAT)16, Utah Education Network17, and Utah Department of Workforce Services (UT DWFS)18, and additional data imported from the National Student Clearinghouse (NSC)19.  Utah’s SLDS is one of 47 state longitudinal data systems (SLDS) funded through grants made available by the National Center for Education Statistics (NCES), an institute of the US Department of Education.  UDA funding include federal NCES grants20 funding of $20,677,282 ($4,561,763 [2007], $9,617,736 [2009 ARRA], $6,497,783 [2015]), and an additional $1.8 million provided by the Utah State Legislature as of fiscal year 2015.

The UDA offers a fully operable data system with de-identified student and unit-level state public education data representing the student education experience, including limited socio-economic indicators, and unit-level income and expense, faculty, administration, and staff data.  The Utah SLDS has 9 of 10 state actions and 10 of 10 essential elements as assessed by the Data Quality Campaign (DQC)21.

The data set includes individual-level, public K-12 and higher education data for all Utah public education students beginning with the 2000 high school graduation cohort22 and includes Utah workforce data for those workers who were also participants in Utah higher education.  The system also include a limited amount of NSC data for those students participating in Utah public K-12,  but who did not exclusively participate in Utah higher education.  The system includes an interactive dashboard23 with macro level, descriptive data, and facilitates requests for other macro-level data output requests.

Access to individual level data is highly restricted and accessible only to UDA stakeholder representatives and authorized researchers.  Individual level data access by outside researchers is restricted to a secured workstation inside of the secured UDA data lab in the Sorenson Arts and Education Building on the University of Utah’s main campus.  The most recent academic year for which data has been updated is 2014-201524.

The State Longitudinal Data Systems Research Team (the team) began working with the Utah Alliance in 2013 and became the first outside researcher review of the data, which also meant ours was the first outside data request to be considered by the UDA stakeholder committee, the management board governing the centralized Utah SLDS.  Though the process suffered from the sort of time delays one might expect, the UDA research coordinator managed the process and within several weeks from the formal request’s submission date the request was granted, access to de-identified student and unit-level data was arranged, and data files were culled with the appropriate variables, observations, and time frames.  Though de-identified, it was necessary to access the data at a secure UDA sanctioned location through a secure network with dual layers of data and network security.  The team was given access to data personnel empowered to assist within a well-defined set of parameters, and was required to provide a log of all access dates and times.  Data and outcomes were only allowed to be reported after having been scrutinized by the UDA personnel responsible to maintain strict FERPA and state-level privacy and data security protocols.  As the team had outcomes prepared for release, advance notification to the UDA management committee was required to allow UDA stakeholders sufficient time to comment if any chose to do so, though the team held the right to release data with or without amendment based on committee commentary.

The team’s data examination included primary, secondary and higher education data for those students in the 2008 – 2010 Utah public high school graduation cohorts in an effort to identify the effects of various dual-credit enrollment programs on Utah’s general student population (Haskell 2016[a]) 25 and underrepresented students (Haskell 2016[b])26.  The Utah data was well organized, complete, and included sufficient linkages to infer causal relationships between student participation in dual-credit programs and higher education outcomes for the subject populations.  It also allowed for an analysis of cost savings to student households and the state.

Working with the MCDS

The Missouri Comprehensive Data System (MCDS) is Missouri’s SLDS27 managed by the Missouri Department of Elementary and Secondary Education (DESE) under a federated governance structure28. The MCDS, created for the purpose of collecting and analyzing Missouri public education data at the individual, course, institution, and system levels, aggregates data records from the breadth of the Missouri public education data collection systems, including the Missouri Student Information System (MOSIS)29 and the Enhanced Missouri Student Achievement Study (EMSAS)30.

Missouri received $8,967,686 in 2009 under the NCES Statewide Longitudinal Data Systems Grant Program31 as one of 47 states qualifying to receive federal funding to create a state longitudinal data system (SLDS). The state’s legislature has not allocated additional SLDS funding.

The MCDS receives data from several different sources, including DESE, the Missouri Department of Higher Education (MDHE)32, and the Missouri Department of Economic Development (MDED)33. DESE provides records through its MOSIS data collection system, a system that gathers Missouri public education, K-12, student-level data, assigning a unique identifier variable to each student which stays with them throughout their K-12 education. MDHE provides records through its EMSAS data collection system, a system that gathers higher education student-level data on those attending public higher education institutions in the state of Missouri.  EMSAS obtains the full name of those students for whom it receives data and also requires student’s MOSIS identifier variable if that student attended an elementary or secondary education institution in the state, necessary requirement for the interoperability between the MOSIS and EMSAS databases. MDED provides records on labor market and workforce outcomes, tracking individual and unit-level data using each Missouri resident’s social security number as a unique identifier variable.

Given the system’s federated governance structure, data sharing among MCDS stakeholders occurs under a series of MOUs (memos of understanding) sufficiently broad to allow state agencies to access de-identified data on an as-requested/as-needed basis. While the MCDS has a process through which outside researchers may request data access, there is no record of outside research having been conducted using Missouri’s data.

The team began working with the MCDS in 2016, at the request of the Ewing Marion Kauffman Foundation34, in an effort to establish an MOU allowing for an examination of Missouri public education outcomes resulting from participation in various dual-credit programs, similar to the study performed in Utah using UDA data.  Given the MCDS’s federated governance structure, this required gathering support from representatives with DESE and MDHE and confirming through each agency’s legal team that the MOU the respective agencies have with one another was sufficiently broad to allow each to enter into an MOU with an outside organization.  It also required the express support of one of Missouri’s Commissioners of Public Education.

Through more than eight months of discussion and data structure examination, including the identification of those individual and unit-level variables to be included in the study, it was determined that approval of an MOU would only be permissible in the context of a program evaluation somewhat broader than originally intended.  The team was amenable to this requirement and subsequently received the support of the appropriate commissioner, Dr. Stacey Preis35, who authorized the program evaluation, instructed the respective administrators to prepare the MOU for signature and presentation to the DESE’s legal counsel for final approval, and directed DESE to begin culling the required data.  It was agreed the team would travel to Missouri and would access the de-identified student and unit-level data in a DESE facility and behind two level of data and network security.

During this process MDHE and DESE representatives and others who had dealt with MCDS data collection and use raised concern over whether or not an approved MOU would result in the required data with credible individual and unit linkages across agencies.  These concerns were addressed and the process continued, but ultimately without success.  This was consistent with the MCDS’s DQC rating36 (10 of 10 essential elements and 7 of 10 state actions).  After several additional months and repeated inquiry, Preis reported the MOU had not been approved and declined further comment.  Informal discussions revealed a likely inability to link student-level records across agencies and put into question the viability of federated data system as an SLDS as outlined by NCES.

Missouri public education’s funding proposal for the NCES grant included the development of a fully operable, P20W style SLDS37.  However, after expending more than $9,000,000 in state and federal resources, the Missouri SLDS remains only partially operable and neither includes important inter-agency linkages, but there are anecdotal reports over concerns about the accuracy of the collected data.  The system’s federated governance structure has not led to a robust system with high data quality or flexible user interface.  The MCDS data dashboard38, a focal element of the funding proposal, contains limited aggregated data outcomes and no interactivity.  It does, however, include a snapshot of the status of Missouri’s progress towards its “Top 10 by 20”39 goal adopted prior to the 2016-2017 academic year and reportedly abandoned in April 201740.

While the MCDS is an SLDS as interpreted by Missouri public education stakeholders, it is not an SLDS as based on the interpretation of NCES.  Nor is it an operable SLDS based on the experiences of the research team.

There are SLDSs… and then there are SLDSs

The UDA/MCDS comparison reflects one user group’s experience in working with two differently operable and managed public education data systems. While an examination of the differences in governance structure may suggest a centralized system resulted in a more functional management outcome, it should be noted that many states enjoy high-functioning, federated SLDSs (North Carolina41, Virginia42, Idaho43).  Difficulties in working with the MCDS are more likely correlated with access constraints44, such as those imposed by the Missouri state legislature, the system’s lack of ongoing state financial support, and the state’s lack of willingness to support excellence in its public education system.

In direct contrast to the MCDS, the UDA and its centralized management structure offers a richly resourced and interactive data dashboard, enjoys ongoing funding support from its state legislature, has qualified for multiple rounds of NCES funding, and has been used by researchers from within the stakeholder group as well as those representing outside, independent research examinations.

While Missouri and select other state have observably lessened their support for a robust state longitudinal public education data system, states such as Utah have increased theirs.  At the direction and funded support of the Utah state legislature the UDA will soon be replaced by the Utah Data Research Center (URDC)45 and will include broader access to a more robust set of state education, workforce, and healthcare related data.

Like many SLDSs across the nation, the Utah has responded to ever increasing privacy concerns, but has been prepared to make available all data allowable by state and federal statute.  Such data availability results in better informed public education policy and improved education choice decisions made by students and their supporting households.

1 Richard Haskell, PhD, Associate Professor of Finance, Gore School of Business, Westminster College, Salt Lake City, Utah (2017); http://www.rhaskell.org/haskell

2 Peter Seppi, BS Finance & Accounting, Westminster College, Salt Lake City, Utah (2017): https://www.linkedin.com/in/peter-seppi-11000091

3 State Longitudinal Data Systems Research, Bill & Vieve Gore School of Business, Westminster College, Salt Lake City, Utah; www.slds.rhaskell.org

4 National Center for Education Statistics, US Department of Education; https://nces.ed.gov/programs/slds/

5 Data Quality Campaign; https://dataqualitycampaign.org/

6 Utah Data Alliance; http://www.utahdataalliance.org/

7 Missouri Comprehensive Data System; https://mcds.dese.mo.gov/Pages/default.aspx

8 Missouri Department of Elementary and Secondary Education (DESE); https://dese.mo.gov/

9 State Longitudinal Systems Research Team http://slds.rhaskell.org/research-team

10 Westminster College, Salt Lake City,  Utah; https://www.westminstercollege.edu/

11 Bill & Vieve Gore School of Business, Westminster College, Salt Lake City,  Utah; https://www.westminstercollege.edu/about/academic-schools/undergraduate-schools/bill-and-vieve-gore-school-of-business

12 Ewing Marion Kauffman Foundation; http://www.kauffman.org/

13 UTAH SLDS Profile, State Longitudinal Systems Research; http://slds.rhaskell.org/state-profiles/utah

14 Utah System of  Higher Education (USHE); https://higheredutah.org/

15 Utah State Office of Education (USOE) aka Utah Board of Education; http://business.utah.gov/partners/public-education/

16 Utah College of Applied Technology; www.ucat.edu

17 Utah Education Network; http://www.uen.org/

18 Utah Department of Workforce Services; https://jobs.utah.gov/

19 National Student Clearinghouse; http://www.studentclearinghouse.org/

20 Utah SLDS grants, National Center for Education Statistics (NCES) SLDS Grantee Program; https://nces.ed.gov/programs/slds/state.asp?stateabbr=UT

21 State Progress, Data Quality Campaign; https://dataqualitycampaign.org/why-education-data/state-progress/

22 As of this writing, the earliest high school graduation cohort for which the UDA has complete data is 2002.  Each year the UDA stakeholders add more recent data and pick up data from previous year cohorts.

23 http://www.utahdataalliance.org/dashboards/

24 The most recent academic year for which the data has been updated is the 2014-2015 school year.

25 Haskell, Richard E., 2016 [a], Effects of Dual Credit Enrollment and Early College High School on Utah Public Education, Applied Economics and Finance, Vol. 3, No. 2, May 2016, Redfame Publishing; http://slds.rhaskell.org/documents/2017/07/effects-of-dual-credit-enrollment-and-early-college-high-school-on-utah-public-education.pdf

26 Haskell, Richard E., 2016 [b], The Effects of Dual-Credit Enrollment on Underrepresented Students: The Utah Case, International Journal of Economics and Finance, Vol. 18, No. 1, 2016, Canadian Center of Science and Education; http://slds.rhaskell.org/documents/2017/07/the-effects-of-dual-credit-enrollment-on-underrepresented-students-the-utah-case.pdf

27 Missouri SLDS Profile, State Longitudinal Data Systems Research; http://slds.rhaskell.org/state-profiles/missouri

28 Federated Vs. Centralized Governance Structures; http://slds.rhaskell.org/federated-vs-centralized-slds-governance

29 The Missouri Student Information System is a data collection system managed by the Department of Elementary and Secondary Education, created for the purpose of collecting and analyzing K-12 individual student level data; https://dese.mo.gov/data-system-management/student-information-system

30 The Enhanced Missouri Student Achievement Study is a data collection system managed by the Department of Higher Education, created for the purpose of collecting and analyzing higher education individual student level data; http://dhe.mo.gov/data/emsas/

31 Utah SLDS grants, National Center for Education Statistics (NCES) SLDS Grantee Program;https://nces.ed.gov/programs/slds/state.asp?stateabbr=MO

32 The Missouri Department of Higher Education (MDHE) strives to coordinate higher education policy that fosters a quality postsecondary system, as well as to increase participation in Missouri ‘s public institutions; http://dhe.mo.gov/

33 The Missouri Department of Economic Development’s mission is to create solid, high-paying jobs and to boost economic development across the state to help local communities grow and prosper; https://ded.mo.gov/

34 http://www.kauffman.org/

35 Preis, Stacey, PhD, Missouri Deputy Commissioner of Education, Learning Services;https://dese.mo.gov/learning-services

36 Missouri, Data Quality Campaign rankings; https://dataqualitycampaign.org/action-issues/

37 A P20W style SLDS includes student and unit-level public education and workforce data inclusive of pre-school through 20 years of public education and labor force data (wages, industry classification, etc.); https://nces.ed.gov/programs/slds/p20w.asp

38 MCDS Dashboard; https://dese.mo.gov/top-10-20/data-dashboard

39 Missouri Public Education, Top 10 by 20 Campaign, https://dese.mo.gov/top-10-by-20

40 Delaney, Ryan, 2017, Missouri public education drops goal of having schools among nation’s 10 best, St. Louis Public Radio, April 18, 2017, http://slds.rhaskell.org/documents/2017/07/missouri-education-department-drops-goal-of-having-schools-among-nations-10-best.pdf

41 North Carolina SLDS Profile, State Longitudinal Data Systems Research; http://slds.rhaskell.org/state-profiles/north-carolina

42 Virginia SLDS Profile, State Longitudinal Data Systems Research; http://slds.rhaskell.org/virginia

43 Idaho SLDS Profile, State Longitudinal Data Systems Research; http://slds.rhaskell.org/state-profiles/idaho

44 Legislative Constraints, State Longitudinal Data Systems Research; http://slds.rhaskell.org/legislative-supports

45 By July 2017, the Utah Data Alliance (UDA)  will be replaced by the Utah Data Research Center (UDRC) persuant to UT SB 194; http://www.workforcedqc.org/state-solutions/utah