10 Things You Always Wanted To Know About Data-Driven Decision Making
Everyone's talking about D3M. Use this guide to help prevent all that data from driving you nuts.
Way back in the 1980s, a new way of doing business evolved as corporations began collecting, combining, and crunching data from sources throughout the enterprise. Their goal was to improve the bottom line by discerning hidden patterns and thereby improving the decision making process. Two decades later, the No Child Left Behind (NCLB) law is pushing school districts to do the same—this time, of course, the goal is increased student achievement.
While data alone won't provide school leaders with the finesse, experience, and intuition needed to get kids on the right learning path, a growing number of administrators are convinced that the data-driven decision-making (D3M) process can fundamentally change education—from our understanding of what really works with kids to administrative processes and professional development.
To assist you in getting your head around the realities and ramifications of data-driven decision making (and, yes, to make the subject a bit livelier), we've compiled this list of 10 truths. Some of these things will entice you; others will surprise you; a couple are sure to momentarily paralyze you. But, at least, now you know.
1. If you're not using data to make decisions, you're flying blind.
Imagine you're piloting a plane at night, in the middle of a storm. Without data from your navigation instruments, you'd be a goner. "Running a school without a data warehouse is like being a pilot without an instrument rating," says Brian Osborne, supervisor of evaluation, assessment, and research at the Plainfield Public Schools in New Jersey. "This is why there's so much policy churn in schools. Administrators simply don't have enough information to make good decisions." The Plainfield schools began using a data warehouse and analysis solution from eScholar three years ago. Although Osborne is a recent arrival in the district, he says he quickly became a big fan of the technology's power to help school leaders find the right flight path toward student achievement.
2. This is all about a process, not a specific technology.
Traditionally, D3M solutions have centered around a data warehouse—a central repository that collects data from many different sources—in combination with high-end decision-support tools that run queries and reports on the data. But in fact, a wide variety of technologies and solutions can be used to support a D3M process. Harry Hayes, superintendent of the Bloomfield, New Mexico, school district, says the D3M process should always begin with these questions: "How's business? How do you know it? And what can you do to improve it?" Hayes is a big proponent of the management-review process, and uses Northwest Evaluation Association (NWEA) test data to help him analyze teaching strategies and make better-informed decisions about professional development. Gregory S. Decker, principal of Lead Mine Elementary School in Raleigh, North Carolina, analyzes school data using Microsoft Access database software. He also uses Pearson's SuccessMaker management system for student assessments and on-demand reports. Decker and his staff have spent the past four years mapping, benchmarking, and making predictions about where each child—and each teacher—should be at every stage, and he says test scores have soared as a result.
3. Get ready to feel threatened.
"The downside to data-driven decision making is that it makes people much more accountable; there are no corners in which to hide," says Robert Ewy, director of planning at Community Consolidated School District 15 in the suburbs of Chicago. Ewy runs MicroStrategy's decision-support tools on an IBM data warehouse. "Before we began using a data warehouse, we thought we were making decisions based on data," Ewy says, "but now the quality has dramatically improved." Of course, the idea behind D3M is improvement, not punishment—and it's not just about the quality of teaching, but about the quality of leadership. "Data-driven decision making succeeds if the educational administration can model the habit of mind involved in data inquiry," Osborne says. "It's a culture shift, a new learning paradigm." He says that rather than having the answers all the time, educators need to come up with the right questions. Still, asking the right questions and interpreting the data are not the same as coming up with solutions, Hayes warns. "Data are simply indicators, tools to be used," he says. "They are a means, not the ends in themselves." Proponents of D3M in schools say this is where the quality issue really comes in: Once there is evidence of problems, do administrators and teachers have the chops to address the problems effectively?
4. You will be spending more money, not less.
Irene Spero, vice president and project director of the D3M initiative at the Consortium for School Networking (CoSN), says administrators need to prepare for a costly long haul when implementing data-driven decision making. "The first year is all about setting goals in the community and district. Year two is about roll-out and implementation, and it's not until years three or four that you can really see the effects," she explains.That investment in time can translate to a hefty price tag. Regardless of whether you've bought an expensive solution or not, the process itself demands a serious commitment to training, data cleanup, and maintenance that is often overlooked. "Every three to six months there must be a review of the data validation processes and cleansing of the data to ensure quality," says Katie E. Lovett, chief information officer of Fulton County Schools, in Atlanta, Georgia. "End users must be trained on the tool, but more important is training on how to use data for analysis and understanding how the data is structured in the data warehouse. And users also must be trained in data-warehouse security."
5. Data-driven decision making does not save time.
Don't start planning a vacation with all the time you think you're going to save having D3M in your district, warns Timothy L. Schaap, data analyst for Township High School District 214 in Arlington Heights, Illinois. "Data-driven decision making refocuses your time," Schaap says, "but it is definitely not a timesaver." Schaap believes D3M is akin to doing action research—a process of simultaneously pursuing change and understanding, continuously refining methods and interpretation in light of what's been learned. Schaap notes that both administrators and teachers have to take ownership in the action-research process—and that means more work, not less.
6. Your data's cleanliness is next to Godliness.
You've heard the expression, "garbage in, garbage out." The transactional systems that supply data to your warehouse—including student information systems, instructional management systems, testing applications, and financial software—are used and updated daily by staff members throughout your district, making data-entry errors a regular fact of life. You may also discover stagnant "pools" of data that have been forgotten and aren't regularly updated. If you don't clean up this "dirty" data, your entire D3M initiative could be worthless. The process of cleaning transactional data and exporting it to a data warehouse can be complicated and time-consuming, says Shawn Bay, founder and CEO of eScholar. The biggest risk in implementing a D3M solution is that people may think it's a lot easier than it really is, he says. "Working with data is very deceptive," Bay says, "and educational data is the most complex I've ever worked with." The lesson: Make sure your selected vendor has expertise in data cleansing, and write it into the contract.
7. Don't shoot first and ask questions later.
You should figure out exactly what questions you want the data to answer before you tackle the issue of whether to buy a turnkey D3M solution or build your own using off-the-shelf components. "It's absolutely critical to know that before you decide," warns Illinois's Ewy. Some districts go it alone, others switch vendors along the way, and still others create "blended" solutions, supplementing products and services as needed. Dick Barkey, executive director of information technology for the Adams 12 Five Star Schools in Thornton, Colorado, says his district decided to build its own data warehouse solution because no commercial applications met the district's needs. Called ScholarsMart, the homegrown solution was coded entirely in-house using standard software development tools such as Microsoft's SQL Server and Visual Basic. "The ScholarsMart system is tightly linked to our SASI student information system from Pearson," Barkey explains. Teachers in his district can access information about current or prior students as well as state and district test results, and schedule changes are updated daily.
8. A good D3M solution is one you can afford to change.
Don't lay out a lot of cash up front, Schaap says. The reason: It's hard to walk away from a lemon when you've invested a lot—and you never know when you may want to chuck the whole thing and start over. District needs and product features change, Schaap explains, so it's important to consider initial costs as well as interoperability. Odds are, you won't have to bag a system that is adaptable, however. The same D3M tools can often be used for very different goals: David Heistad, executive director of testing, evaluation, and student information in the Minneapolis Public Schools, uses the NWEA item banks to create tests aligned to state and local standards. In contrast, Bloomfield, New Mexico, superintendent Harry Hayes uses the NWEA solution for professional development.
9. NLCB is just the beginning of your journey.
Although NCLB is driving the momentum for data-driven decision making, Spero says, school districts should view D3M solutions in the context of sweeping and system-wide school improvement efforts—not as simply a tool to meet federal mandates. "This is an opportunity to go way beyond what's required," Spero says. Heistad agrees: "One must go beyond the current federal model of comparing this year's third graders to last year's third grades and comparing average test scores for one site versus another." He says he believes that schools should be rewarded and supported based on the degree to which every student is making progress towards high standards—in addition to the number of students who have "jumped over" the bar.
10. Word of warning: D3M is highly addictive.
Raymond Yeagley, superintendent of the Rochester (NH) Public Schools, recalls a school board meeting in which a board member asked how students were doing in math. "With the web-based Quality School Portfolio, two minutes later I had the answer," he says. That kind of power is not just addictive—it's contagious. Getting that information to the people who need it—and in a form that they can use—is critical. The Minneapolis schools offer paper reports for anyone who cannot access the reports on-line; these are distributed to school teams, parents, and the community. Barkey advises that regardless of how the information is transmitted, it's important to make sure that it is presented in a simple list, table, or drill-down form with good built-in explanations and definitions. "The more you use a data warehouse," concludes Ewy, "the more you want to use it."
- D3M Buyer's Guide (PDF)
Pamela Wheaton Shorr is editor of The Heller Reports' Educational Sales and Marketing Insider, and is a frequent contributor to Scholastic Administr@tor.