Education software and related products are an $8 billion global industry. The USA Every Student Succeeds Act of 2016 (ESSA) strongly backs "evidence-based" materials. But no federal agency conducts independent evaluations of commercial educational software. School districts usually rely on "insider" statistics provided by the for-profit companies.

Question 1:

As a statistics student, what possible sources of bias might you be alert to when reading reports of "insider" studies? How might such biases show up in the reports?

Question 2:

The Hechinger Report promises to provide "in-depth, fact-based, unbiased reporting on education that is free to all readers." What more might you want to know about this organization as you read its reports?

A Hechinger Report review states that dozens of companies promote the effectiveness of their products, including some that sell very widely-used education software. Many overstate their effectiveness based on very limited evidence or on effects not seen by independent evaluators. Some "tout student growth that their own studies admit is not statistically significant."

Question 3: What is meant by "statistically significant"?

The following table is reproduced from materials produced by a company called Edgenuity in the article quoted above.

Side by side bar graphs showing test results in time series by grade level

Misleading graph - edgenuity Tara-Garcia-Mathewson-EdtechResearch3.jpg

Question 4:

Study this side-by-side bar graph. State at least two things that it appears to show. Then write at least one or two questions that are not answered by the graph but could give more context to the information it presents.

The Hechinger study noted that Edgenuity used this chart to claim that use of its math course supposedly led to an eightfold increase in pass rates on state math tests at Altamont High School in Utah. But the school only introduced Edgenuity in the 2015-2016 school year.

Question 5:

Where do you think Edgenuity got its "eightfold increase" estimate? Are you convinced? Knowing that the school started with Edgenuity in 2015-2016, what would be your estimate of the increase in test scores?

Question 6:

Most students who were in 9th grade in 2015 were probably in 10th grade in 2016 and so forth. How do you think that might affect your interpretation of the graphs, if at all?

Question 7:

What is the measure of success being used in this table? Comment on the pros and the cons of using this measure. What observations do you have, based on this exercise and your own K-12 math experiences?