MasteryTrack: System Overview

(1)

MasteryTrack:

System Overview

March 16, 2015

(2)

Background

The Learning Accelerator (TLA) is a nonprofit whose mission is to accelerate the implementation of high-‐ quality blended learning in districts across the U.S. A key part of blended learning is mastery-‐based progression: enabling students to move forward in their learning at their own pace as they master content rather than based on traditional time structures. This is essential for improving student achievement. While many districts across the country are getting interested in mastery-‐based

progression and starting to implement it, this is very challenging work. One of the key barriers is that the software solutions to enable this are not sufficiently robust and do not address some key issues. TLA is concerned that this problem is not being solved sufficiently quickly or effectively by the marketplace. We also have a point of view about how this should be done.

The Learning Accelerator developed MasteryTrack to demonstrate the key elements of mastery-‐based progression and to show how it can be implemented at scale in districts across the country in a way that maximizes benefits for students and districts. MasteryTrack is a prototype that may be offered at scale in the future, or may be used primarily to demonstrate key concepts and processes that push the field forward.

Objectives

While we expect the list and scope of objectives to grow as MasteryTrack is expanded and enhanced, several initial objectives have guided its development to this point:

• Create integrated mastery-‐based dashboards of student learning for all grade levels and content areas that are available to all teachers and students

• Develop and enable a “plug-‐and-‐play” or “configure-‐to-‐order” architecture for learning and practice resources by making the tracking and reporting of student mastery separate from these resources

• Incorporate ongoing innovations from the marketplace in learning and practice resources as well as learning management systems

• Minimize costs for schools, teachers, and students

Foundational Principles

MasteryTrack is not a complicated system. However, it is based on a few foundational principles that are essential and must be adopted by those who use the system.

Mastery is Binary

Either students have mastered a learning objective and are ready to move on, or they have not. There are not different levels of mastery. Students move forward as they master content, with the result that they will often advance at different rates. This applies to all subjects and for all types of learning objectives. Though it may often look different based on the nature of the objectives, the same framing applies to all.

Demonstrating Mastery

(4)

1. Scope of activity –The scope of activity defines exactly what it is we want students to know or be able to do. The specific knowledge or skills we expect students to master are directly linked to established learning objectives.

2. Level of accuracy – The level of accuracy required is a specific measurable outcome that varies depending on the nature of each learning objective. Although methods and ranges may be different, the level of accuracy must be clearly defined for each objective to be mastered. Examples of level of accuracy could be 9 out of 10 correct on an Algebra quiz or acceptable rankings on a rubric for essay writing.

3. Time – While the amount of time required to successfully complete a task or activity will vary, the allowable amount of time must be clearly defined. There may be cases where a sliding scale of time is needed for students with special needs.

If students can achieve a target level of accuracy for a defined scope within the specified time limit, they have demonstrated mastery and are ready to move on in their learning.

System Elements

There are several elements of a system designed to enable mastery-‐based progression: • Learning objectives

• General approach for demonstrating mastery • Mastery threshold

• Scalable process for demonstrating mastery of each objective -‐ Students • Scalable process for assessing mastery for each objective -‐ Teachers

Each element must be clearly defined for the system to function effectively.

Learning objectives

The starting point for a system based on mastery-‐based progression is the set of learning objectives: what do we want students to learn? What do we want them to know or be able to do when they have “learned” or “mastered” what they are supposed to learn? For learning objectives to effectively enable mastery-‐based progression they must have three defined characteristics: they must be specific, discrete, and demonstrable.

Specific

The learning objectives must be sufficiently specific so that mastery thresholds can be assigned and the process for mastery can be defined for each one.

For example, “multiply” or “multiply two numbers” are not sufficiently specific to enable precise assessment of mastery because they do not make it clear how exactly we would know whether a student had mastered them. On the other hand, “multiply two two-‐digit numbers” or “multiply a whole number of four digits by a one-‐digit number” provide the level of specificity needed. “Understand the periodic table” is not sufficiently specific, whereas “use the periodic table to identify the atomic number of specific elements” is.

(5)

When a learning objective is specific, it is possible to clearly define what kinds of actions a student would need to take in order to demonstrate mastery, and how a teacher might assess the student’s attempt to demonstrate mastery.

Learning objectives are of varying levels of complexity and will require different degrees of expertise (e.g., deeper learning, higher-‐order thinking). For example, these are each specific learning objectives:

1. Find whole-‐number quotients and remainders with up to four-‐digit dividends and one-‐digit divisors.

2. Effectively solve multi-‐step word problems with all four operations. 3. Create an accurate 12-‐month monthly cash flow projection.

4. Write a 5-‐paragraph essay using the tools of persuasive writing to convincingly argue that slavery was (or was not) the primary cause of the American Civil War.

Some learning objectives will be more advanced versions of previous skills (e.g., write a good paragraph vs. write a good 5-‐paragraph essay). Some will be the integration of multiple skills (e.g., solve multi-‐step word problems). But each is specific and binary—there are not “different levels of mastery” of a learning objective. There are different learning objectives and mastery of each of them is binary.

Many existing standards, including many in the Common Core, are not sufficiently specific and must be modified to enable effective mastery-‐based progression. An increased level of clarity is required because teachers must be able to explicitly determine when a student is ready to move on to the next learning objective. In the past this decision has not been required since students moved forward based on time even if they clearly had not mastered the material (this is the meaning of a “C” in today’s grading system, for example).

Discrete

The objectives must be discrete so that each can be clearly identified and mastery can be demonstrated and assessed. This may require existing standards to be modified so those that include multiple

elements are separated.

For example, “Multiply or divide to solve word problems” is not one objective because it requires different skills and should be separated into two separate objectives. The exception is capstone

standards that are specifically established to assess whether students are able to use multiple skills and diagnose which to use. But the specific objectives for the supporting skills should be separate (and probably must be mastered before the capstone skill can be mastered).

Demonstrable

A student’s mastery of a learning objective must be explicitly demonstrable. The key question a teacher must be able to answer is “Has this student mastered this learning objective and is she ready to move on in her learning?” It must be possible for the student to provide the required evidence to demonstrate mastery so the teacher can answer this question.

(6)

Some existing standards are not demonstrable, and therefore must be either modified or excluded. For example, this 4th grade Common Core math standard is not demonstrable:

“Interpret a multiplication equation as a comparison, e.g., interpret 35 = 5 x 7 as a statement that 35 is 5 times as many as 7 and 7 times as many as 5.”

However, the knowledge and skill addressed here is inherent in the ability to multiply effectively, which is covered in other standards that are specific, discrete, and demonstrable. This standard can therefore be safely excluded.

Other Thoughts about Learning Objectives

Specific learning objectives may need to be created for new areas that are increasingly seen as important for students but not explicitly taught or assessed in the typical K-‐12 curriculum. Teamwork, collaboration, negotiations, and interpersonal communications are examples of skills and areas of content knowledge we want students to learn. Specific learning objectives must be established for these areas, each of which is binary and can be mastered.

General Approach for Demonstrating Mastery

Mastery of learning objectives can be demonstrated in different ways. For many math objectives, the clearest way for students to demonstrate mastery is for them to effectively complete problems. For many writing objectives the approach is for students to write papers or essays. In chemistry or biology students may need to effectively complete experiments or lab projects. The general approach for demonstrating mastery must be clear for each learning objective, and the students must be able to execute it effectively. The approaches for demonstrating mastery must also work at scale, meaning that a teacher with an average number of students can manage the process and that a student with an average number of classes can as well.

Threshold for mastery

For each learning objective a clear threshold for mastery must be defined. The nature of the threshold will vary based on the type of objective, but it must always enable a binary decision: has the student mastered this learning objective and is he/she ready to move on in his/her learning?

In some subjects (including math and also probably some sciences), the threshold may consist of three elements:

1. A scope of a skill or content knowledge (e.g., multiplying two one-‐digit numbers) 2. A level of accuracy (e.g., 9 out of 10 correct)

3. A time limit (e.g., 2 minutes)

For this example, if a student could attempt 10 problems multiplying two one-‐digit numbers and get 9 or more correct within 2 minutes, the student would be assessed as having mastered this skill and could move on.

Mastery thresholds are more qualitative for other subjects, but must always be clear and enable a binary assessment.

(7)

Scalable Process for Demonstrating Mastery -‐ Students

For each learning objective the specific mechanics for demonstrating mastery must be clear—what exactly does the student do? MasteryTrack is designed to provide this solution for some parts of math that lend themselves to automated assessment of mastery. For these learning objectives, the scalable process for demonstrating mastery is for students to complete problems in MasteryTrack. For learning objectives in other subjects the process will be for students to write a specific essay or complete a particular project separate from MasteryTrack.

Scalable Process for Assessing Mastery -‐ Teachers

Similarly, it must be clear how teachers will assess student mastery for each learning objective. For learning objectives where assessment of mastery is automated in MasteryTrack, the process is simply that MasteryTrack will make the assessment based on thresholds defined by educators. The automated assessment saves teachers time, which they can re-‐allocate to more valuable educational activities. For many objectives the process will be that the teacher will review the student’s work and determine whether the student has mastered the objective and is ready to move on. While this is similar to the grading teachers often do in the current system, the distinction is that in each case the teacher is assessing student mastery based on a binary determination. There are no A’s, B’s, and C’s. The only question is whether the student has demonstrated mastery of the objective.

(8)

MasteryTrack: How It Works

MasteryTrack consists of two parts: the Teacher View and the Student View.

The Teacher View

The primary element of the teacher view is the dashboard. It shows all students in each class and their learning progress for a set of learning objectives. The teacher can select which class to view and which set of learning objectives to show. The key is that the teacher can see at a glance where each student is in their learning.

The dashboard shows the current status of mastery for each student and for each learning objective. Each column is a learning objective and each row is a student. There are four possible states for each student and each learning objective:

1. Mastered -‐ The student has mastered the learning objective.

2. Unsuccessful -‐ The student tried to master the learning objective but did not succeed. 3. Ready to attempt mastery -‐ The test for mastery for this objective has been released to the

student by the teacher, and now the student can attempt to demonstrate mastery.

4. Not yet attempted -‐ The student has not yet attempted to master the objective and the teacher has not yet released the test to the student.

Each of these states is demonstrated on the dashboard by a different circular icon: • Mastered -‐ green check mark

• Unsuccessful -‐ purple X

• Ready to attempt mastery -‐ blue arrow • Not yet attempted -‐ grey clock

The teacher can select which class to view or to see all of their students at once. The teacher can also select which learning objectives to view. The learning objectives have a nested structure. This hierarchy is similar to that used elsewhere in education (for example, the non-‐profit organization Gooru uses a similar structure). Key levels of the hierarchy are:

(9)

1. Subject (e.g., math, science) 2. Course (e.g., algebra, chemistry)

3. Unit (e.g., elementary foundations, covalent bonds) 4. Topic (e.g., addition, multiplication)

5. Learning objective (e.g., multiply two one-‐digit numbers)

The names of the levels are not particularly important. The key is that the teacher can navigate through the structure to select groups of learning objectives and with a few clicks of the mouse is able to see the desired updated dashboard of student learning progress.

Teachers can release tests to students. In the system’s current design, a student cannot take a test to attempt to demonstrate mastery until the teacher releases the test to the student. If students have tried to demonstrate mastery of an objective but failed, the teacher can review their results and either re-‐ assign the test to the student or mark the student as “sufficient,” meaning that they did not master the objective according to the defined mastery threshold but that the teacher believes they have mastered it sufficiently and are ready to move on in their learning.

The assessment of student mastery is embedded in MasteryTrack for the learning objectives for elementary school math, which means the dashboard is updated automatically since the grading of the tests is done by the system. For other topics the assessment mechanism may require the teacher to grade a paper or review a completed lab project, in which case the teacher will manually update the dashboard and mark students as having mastered content.

The Student View

The student view is similar to the teacher view, except that students can only see their own data. The structure of the dashboard is the same, but it only shows the progress for the individual student rather than the class. The hierarchy and description of the learning objectives are the same as the teacher view.

(10)

Students can see the list of tests that have been assigned to them, and can take the tests. The tests are launched when the student selects the desired test, and the student completes it immediately. The results are displayed when the test is completed, and the dashboards for both the student and the teacher are instantly updated based on the outcome.

Other Elements of the System

There is a “Learning Center” in MasteryTrack that is designed to be a repository for learning and practice resources aligned to each learning objective. No learning resources are embedded in MasteryTrack. However, teachers can store links to resources in the Learning Center to make it easier for both students and teachers to find resources that can help students learn each specific objective.

The dashboard shows only the title of each learning objective, but there are descriptions on a separate screen that both teachers and students can access easily. This approach makes it easy to navigate the dashboards.

(11)

Historical Timeline

TLA CEO Scott Ellis first developed the concept that became MasteryTrack in 2009. Over the course of several years the system took shape, first in excel and PowerPoint, then as a clickable prototype

covering the basic operations of elementary math, and then as a functional prototype covering math but with a structure that can be extrapolated to the entire K-‐12 curriculum. The following timeline provides a historical look at how the system progressed over time.

June 2009 – Initial Concept

Scott Ellis began to develop the concept that became Mastery track: a system to track student mastery based on specific steps of learning and a binary assessment for each.

March 2011 – Initial Mastery-‐Based Progression Materials

Scott began to capture his ideas about the structure of mastery-‐based progression in an Excel file that outlined the process and its initial components. The initial concept included the demonstration of mastery complete with dashboards and output reports to be used by students, teachers, and administrators. He reviewed the Common Core standards for Math for grades 1-‐5 and started to

translate and categorize them to enable mastery and the development of structured tests. Having begun the work months before, by the end of March it was all integrated into the Excel file. This file evolved over time as he added functionality and refined the output reports. He also created a Powerpoint overview and structure of the system with details of its operation, as well as a simple financial model to assess the potential revenue for the system based on a range of potential assumptions.

May 2011 – The PRIDE System

In May 2011 he created descriptions of the concept (called the PRIDE System at that time) in both Word and Powerpoint. The Powerpoint slides described a version for schools as well as an app for phones to be used by parents. He communicated with several potential clients and districts to assess their interest and determine potential viable paths forward. While there was modest interest from a few teachers and parents, there was no clear mandate or pathway on how to proceed.

June 2011 to May 2013 – MasteryTrack as a Possible Solution

As Scott became busy with other activities, he did minimal work to further develop the Excel, Word, and Powerpoint versions of the system, but continued to think about the concept and assess its potential impact. It became increasingly clear that the software required to enable mastery-‐based progression for all 50 million students in the U.S. was not currently being created. Scott determined that his initial concepts for MasteryTrack could provide a much-‐needed solution.

Summer 2013 – 4th_{Grade Math}

In early summer 2013 Scott fully translated 4th_{grade math standards from the Common Core into a}

structure that would enable mastery-‐based progression. His colleague Michela Marini helped by entering the text of the standards for several other grades into the Excel file. She also conducted an initial search for learning software that would be aligned to the learning objectives and captured this in the file.

(12)

September 2013 – Presentation to Gooru

In September 2013 Scott presented his vision of mastery-‐based progression and the overall software architecture concept to the CEO and many members of the staff of Gooru. The presentation was captured as a set of Powerpoint slides describing the mastery-‐based progression concept and links to personalized learning.

Early Fall 2013 – Developing Prototype with Gooru

Scott worked with the Gooru team to create a prototype of the system. He shared the project structure document (Powerpoint slides) that defined the specific learning standard to use (“effectively solving multi-‐step word problems with all four operations”). He also defined its five component elements (addition, subtraction, multiplication, division, and word problems) and the sub-‐elements of each. The team began to create a prototype using the Gooru platform. One of Gooru’s technology leads, Ben Asilung, spent several hours on the prototype. Scott also discussed ideas with one of Gooru’s designers, who shared a few example mock-‐ups. Scott gave feedback on the mock-‐ups, and shared that one of them was somewhat like what he had in mind. Ben created an example quiz based on questions Scott sent him as a first step for envisioning how the program would work.

Late Fall 2013 – Abandoning Prototype with Gooru

Although Scott was very grateful to the Gooru team for their efforts, the prototype development for MasteryTrack did not closely align with their core mission. Therefore Ben and Scott agreed to abandon the prototype effort. Scott got a recommendation from Gooru to connect with ConcentricSky, a software company they had used in the past.

January 2014 – Contract with ConcentricSky

Scott started working with ConcentricSky to create a clickable prototype to enable the concept to be shared with teachers to assess interest and determine whether the next stage of prototype should be created.

May 2014 – Functional Prototype with ConcentricSky

Feedback from educators and other reviewers was very positive, so ConcentricSky and Scott began work on the next phase, creating a functional prototype that would enable it to be piloted in a few districts to test the response from students, teachers, and administrators.

August 2014 – Working Prototype

By fall 2014, a working prototype of MasteryTrack was in place. Scott and the ConcentricSky team conducted ongoing bug checking and improvements, and began looking for teachers to pilot the system.

January 2015 – Beta Testing

In January 2015 Scott started using the system with a group of beta testers (family and friends) to assess the functionality and design, and prepare for use with students and teachers.