This article is Part 2 of our ESSA: Evidence of Effectiveness series. Make sure to check out Part 1: Understanding the Difference between Research-based and Evidence-based.
As we shift our focus from research-based to evidence-based, we raise the bar on the level of rigorous support schools need to adopt, support, and scale instructional programming across the classrooms and schools in their district or state. First came the approval of state ESSA plans, and now the next phase of identifying and implementing Evidence of Effectiveness is underway.
But what does that mean? What roles do the State Education Agency (SEA) and the Local Education Agency (LEA) play in establishing and demonstrating evidence? What’s expected of each? And, more importantly, what is at stake in terms of accountability?
Like always, let’s start at the beginning.
THE IMPACT OF EVIDENCE OF EFFECTIVENESS ON SCHOOLS
If you were an administrator and you were eye-level deep in the bylaws of NCLB, back when it was the law of the land, you probably remember the numerous times you saw the words “scientifically-based research.” In fact, this phrase shows up 110 times in the NCLB law!
In a nutshell, NCLB was narrower in focus, was less rigorous, and indicated that a program had merit if it was “based on scientifically-based research,” which left far too much open for interpretation. NCLB tried to implement a data-driven system, but it fell drastically short. Because of this, it became exceptionally difficult for schools to define the evidence needed.
In contrast, ESSA seeks data-driven accountability too, using clearly defined Evidence of Effectiveness criteria. ESSA seeks to hold programs to higher standards by clearly outlining these four levels of effectiveness. It calls for programs to incorporate systemic evaluation, demonstrating considerably higher levels of program effectiveness than we have in the past.
ESSA encourages the use of strategies for evaluating Evidence of Effectiveness of programs schools use. Ultimately, ESSA effectively raises the bar on the level of rigorous support schools need to adopt, support, and scale instructional programming across the classrooms and schools in their district and state.
ESSA incentivizes states to work more closely with their schools and districts to use evidence-based programs. The goal is to attain stronger positive student outcomes and reduce spending on programs that lack research and Evidence of Effectiveness.
ESSA: EVIDENCE OF EFFECTIVENESS STANDARDS
ESSA defines strong, moderate, and promising Evidence of Effectiveness. In addition, a fourth category defines an opportunity for program implementations that use approaches that demonstrate a research-based rationale for effectiveness. The four categories are defined as follows (in brief):
- Tier 1 = Strong: At least one randomized, well-conducted study showing significant positive student outcomes.
- Tier 2 = Moderate: At least one quasi-experimental (i.e., matched), well-conducted study showing significant positive student outcomes.
- Tier 3 = Promising: At least one correlational, well-conducted study with controls for inputs showing significant positive student outcomes.
- Tier 4 = Demonstrates a Rationale: Practices that have a well-defined logic model or theory of action, are supported by research and have some effort underway by a SEA, LEA, or outside research organization to determine their effectiveness.
Based on ESSA’s Evidence of Effectiveness, it is now clear that research-based programs fall under Tier 4 in demonstrating effectiveness.
WHAT DO EVIDENCE REQUIREMENTS MEAN FOR SEAS, LEAS, AND ADMINISTRATORS?
First, and foremost, it’s important to keep in mind that at every level of accountability – school, district, and state – the level of ‘Evidence’ being sought is that of individual student growth. Additionally, keep in mind that all references to evidence eventually filter to effectiveness of use in the classroom.
When ESSA shifted the power from federal to state levels, it essentially allowed states to create their own ESSA accountability plans based on the law. The same is true for how the Department of Education provided guidelines on evidence-based interventions and programs and allowed each State Education Agency (SEA) to interpret the guidance for their specific requirements around rigor and study design. It’s also worth noting that states have yet to announce the full role of evidence in their funding decisions.
Once the SEA has determined its guidance around evidence-based interventions, it is up to the Local Education Agency (or district) and the schools to execute and establishing the evidence of impact.
Under ESSA, district and school leaders are responsible for selecting effective, sustainable, scalable instructional approaches that will meet the unique needs of their students. Local evidence supplemented by evidence of comparable schools is imperative to achieving positive student outcomes.
Think of it as an inverted triangle.
The broad upper level, the Department of Education (DOE), is responsible for making sure that states adhere to their ESSA plans. State Education Agencies ensure they adhere to the plans they laid out and sent to the DOE in 2017 and hold Local Education Agencies (LEAs) accountable. These plans submitted to the DOE also included a definition of guidance to districts regarding levels of effectiveness. Schools then report their data (including that of their Evidence of Effectiveness) to the LEA. School administrators are, therefore, responsible for the data and evidence for their classrooms – and ultimately every student.
IT STARTS AT THE CLASSROOM & SCHOOL LEVEL
Take a moment and really think about your school. Think about the classrooms and each of your students. Think of the demographics, the structure, and your teachers and support staff. Can you think of all the ways your school could be different from another? What kind of school would be similar to yours? How are the groups of students similar and different?
Schools vary in so many different ways. Therefore, seeking evidence that a program can meet needs based on similar demographics, structure, etc., is a step closer to improving outcomes for students. Once the needs have been identified, SEAs, LEAs, schools, and other stakeholders will determine the interventions that best serve their needs, implementing rigorous and relevant Evidence of Effectiveness at the local level.
Implementation planning begins with classroom educators and other school stakeholders understanding their local needs and establishing plans to meet them. We use strategies and programs collectively to put an implementation plan in place and then gather the Evidence of Effectiveness in the form of data.
STRATEGIES, DATA, AND IMPLEMENTATIONS
Do you ever feel like part of your job as an administrator is to be part psychometrician? Who would have thought that there could be so much data? At present, ESSA and SEA guidelines are clear that effectiveness results must be determined by a neutral third party to any vendors or suppliers involved in an implementation. This means that it is up to the schools to implement and evaluate research efforts toward determining intervention or program effectiveness
Rigorous data must be gathered to identify the strategies and tools that best meet the school’s objectives. After all, how will you be able to decide whether or not a strategy or program is effective if you don’t have data to prove it? The rigor of the data collected determines the eligibility for sustaining and scaling effective solutions. What this means is the more effective a program is proven to be in your school (Tier 1–3), the more merit it has to be upheld within the school and with stakeholders and implemented throughout the district.
BUT DATA IS ONLY HALF THE EQUATION.
An effective planning process precedes and defines the data collection process. Like I mentioned, planning starts at the classroom level. First, identify what the needs are for the school, then execute a plan for implementation.
Think of your plan and execution like a formula for success:
[Implementations] = [Strategies] + [Tools/Programs]
Once you’ve put that process in place (implementation), you’ll be able to collect data and determine Evidence of Effectiveness.
WHAT ARE THE STAKES OF ACCOUNTABILITY FOR ADMINISTRATORS?
As I mentioned in the previous blog, the bar has been raised by ESSA. In addition to being the law, there are so many reasons why it is important to put a strong plan in place on how you will meet the different levels of Evidence of Effectiveness:
- The time for Implementing programs that demonstrate strong, moderate, or promising evidence of effectiveness (Tiers 1 -3) is mandatory and is just around the corner (2019).
- The most important funding that is impacted by these provisions is the 7% Title 1, Part A, funds set aside by each state to assist the lowest-performing schools. If schools do not demonstrate that the top three tiers of evidence support the interventions (programs) they use, these funds cannot be made accessible to the school.
- Programs that you’ve proven to be of the highest effectiveness (Tier 1: Strong Evidence) will ultimately demonstrate the most significant, positive outcomes on your students and ultimately be considered effective, cost-effective, and achievement-effective programs.
- Some ESSA programs – including some competitive grants and Title I, Section 1003 – require the use of “evidence-based” programs that meet higher levels of evidence. It is recommended that you look through the “Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments” to understand best practices and recommendations your school must adhere to for ESSA accountability in using these funds.
- Implementing programs that have been proven to be of high Evidence of Effectiveness are more defensible and hold more weight with stakeholders. Additionally, these are more scalable and sustainable within the district.
WHAT ARE THE NEXT STEPS?
First, make sure to read our previous blog about the difference between Research-based and Evidence-based research. Second, I highly recommend exploring your state’s ESSA plans and how your SEA further established outlines for how evidence would apply to local districts and the schools in their state accountability plans.
