25 Years: Gateway Testing at Michigan

[image of small group working in class]
Michigan Calculus Classroom, early 90s

Editors’ note: This is the first of a series of blog posts on the state and history of the University of Michigan’s undergraduate mathematics program. Calculus reform came to Michigan in the early 1990s—closing in on 25 years ago. So this is the first installment of a “25 year retrospective.”

“In the new approach, as you know, the important thing is to understand what you’re doing rather than to get the right answer.”
–Tom Lehrer

Gateway Testing

What’s a gateway test? Our definition is that it is a test of basic skills. These may be skills that are prerequisite for success in a course, or may be skills which every student in a course should develop. For a pre-calculus course, we find that basic algebra skills and function manipulation are prerequisite skills; the canonical example of a skill intrinsic to a course is rule-based differentiation in calculus I. Of course, most calculators, Wolfram|Alpha, and many other tools can easily do the calculations that these skills allow students to perform, and we use calculators and other technology in our classes. So why should we test the skills? We do it because we find that students without prerequisite skills struggle in a course (even in the presence of technology), and we continue to believe that there really are skills (like differentiation in calculus I) that any student who has successfully completed the course should be able to do.

However, while these skills are intrinsic to or essential for success in the courses, they are not our educational focus. Our courses focus on conceptual understanding; e.g., for calculus I, what a derivative tells us, and how we can use this to solve problems—not how to find the derivative of \sin(\cos(x^2) + 2). This conceptual understanding is fundamental to our courses, and it is therefore the subject of the bulk of class and instructor time, and it is what we evaluate on exams. Gateway exams are the mechanism by which we ensure that students also have or acquire the basic skills we expect of them, and we find that they work with minimal investment of time in-class [9].

[testing lab, showing desks and layout]
Gateway Testing Lab

Our gateway tests are 7 or 10 question tests, administered on-line (we use WeBWorK[3] as our homework and testing platform), which students may take multiple times and which they must complete with almost no errors to pass (we allow at most one error). Students may practice the tests as many times as they like, but to get credit for having passed the test they must take the test in a proctored lab where their identity is verified and where they are not allowed to use outside resources. The tests have a time limit of twenty or thirty minutes, depending on the test. Because the skills being evaluated on the gateway tests are those which we expect every student in the course to have or acquire, completion of the test is not a part of the students’ course averages; instead, any student who doesn’t pass a gateway by the specified deadline (students may take the tests in a 2–3 week window during the semester) has her/his final grade reduced by 1/3 to a full letter grade at the end of the semester.

This small carrot/large stick model plays well with student psychology. In general, students in a course will often assume that they can make up missed material later in the course by doing better on the final (or, not uncommonly, pleading with the instructor after the fact to be allowed to do extra credit). Because it is explicitly not possible to “make up” a gateway, or to get back the grade penalty by doing better on the final, there is a strong incentive for students to complete the gateways successfully.

We currently have five gateway tests and are adding a sixth in fall 2015. The “original” four are (1) an entrance gateway (that is, a test of prerequisite skills) for math 105 (Data, Functions and Graphs; a pre-calculus course), (2) a differentiation gateway in math 115 (calculus I), (3) an entrance gateway on differentiation and (4) an integration gateway in math 116 (calculus II). An entrance gateway on integration was added to math 215 (calculus III) in fall 2005. In fall 2015, math 214 (a linear algebra course taken by some engineers) will add a gateway test on matrix operations and row reduction.

Historical Background

Gateway tests as we use them at the University of Michigan appeared in the world of undergraduate mathematics education sometime before the 1980s. They were sometimes referred to as “barrier tests,” though this characterization appears to paint a half-empty rather than half-full glass. We prefer the term “gateway,” suggesting that they lead to something great. Duke introduced them in the 1980s as part of their calculus reform efforts [1], and the U.S. Military Academy was using them by the early 1990s [4]. The University of Michigan was pilot testing them for pre-calculus by 1992.[6] The tests in use in 1992 were slightly different from the test we now give, with five different tests of five problems each, which students were required to complete perfectly. These were pencil-and-paper tests initially generated by hand, and subsequently created with an “enormous” Lotus 1-2-3 [10] spreadsheet.[7,8] The first administration (or, possibly, two) of each test was done in class.[8] It is worth noting that for the first couple of years failure to pass the gateway test resulted in a failing grade for the course, though instructors were told to ensure that this didn’t happen unless the student would fail the course for other reasons anyway. Also when it was first implemented the deadline for completing the test was the end of the semester, which proved less desirable than fixing an earlier deadline.[8]

[students working in the math tutoring center]
Math Tutoring Lab

The pre-calculus gateway was revised, and a derivative gateway added to calculus I, in 1993 as the Department reformed its calculus sequence to create what was then called “New Wave” calculus. This was subsequently renamed “Michigan Calculus,” a moniker that we continue to use today. Gateways for calculus II were added by 1997. The calculus tests had seven problems and students were allowed at most one error, while the pre-calculus gateway had 12 questions and allowed at most two errors. We believe that the tests were generated using a modification of the original Lotus 1-2-3 spreadsheet, which created TeX files from a fairly expansive testbank which appeared larger than it actually was because the problems were numbered non-sequentially (e.g., 11, 12, 13, 14, 21, 22, 35, 36, etc.; problem numbers were printed on the tests that students took). When gateways were being taken, versions of the tests would be created and made available in the Math Learning Center (a.k.a. Math Lab: our free walk-in math tutoring center). At least some instructors would administer the test once in class, and students could take the test again, once per day, in the Math Lab, until the test came due.

The logistics of managing these tests were formidable. In fall semesters, our pre-calculus course had approximately 600 students enrolled; calculus I, approximately 1600; and calculus II, 800. Students taking gateway tests would come to the Math Lab and get a test from the student manning the desk at the front of the lab. The date and time would be stamped on the test when they received it, and they would complete the test at one of the tables in the Math Lab. When they had completed the test they would hand it in at the front desk, where it would be stamped with the date and time again and put in a box for grading. The student manager in the Math Lab, or the (faculty) Math Lab Director or the Introductory Program Director would grade the exams when they were not otherwise busy, and, in any case, not while a student waited. Grading was done with as close to zero tolerance for errors as possible, with a goal of having as fast a turn-around as possible.

This system worked, but in a somewhat qualified manner. Time constraints on grading had the unfortunate consequence that students would not get their graded exams back immediately, which could result in them taking the test again before finding that a previous test had a passing grade. One could imagine a situation in which a student in a class that met Monday, Tuesday and Thursday could take the test after class on Monday, not get it back on Tuesday—and therefore have taken a test on Tuesday, Wednesday, and Thursday (before class) before discovering in class on Thursday that they had in fact passed the test on Monday! And the Math Lab was frequently overwhelmed by the number of students coming in to take a gateway test and for tutoring, and was therefore neither a good testing environment nor easy to monitor. To (try to) address the grading issues, the tests were converted to multiple-choice, but this was done with a heavy heart and the (fulfilled) expectation that it would reduce the efficacy of the tests. The math 115 and 116 gateways were converted back to short-answer when gateway testing was moved on-line.

[screenshot of a calculus gateway]
WeBWorK Gateway Test

To address the issues noted above, the gateway tests were converted to an on-line format in 2001, moving through a number of different testing systems before settling on WeBWorK in 2005. Pilot testing took place in fall 2000 and winter 2001, allowing investigation of different testing software and of logistical issues with the on-line test. Full implementation of the on-line test occurred in fall 2001. Moving to an on-line test prompted three changes to the tests. First, when the test required students to enter their responses as a mathematical formula, we increased the time limit by 10 minutes, allowing 30 minutes for what had been a 20 minute pencil-and-paper test. This was intended to reduce students’ anxiety and complaints about getting problems wrong because of errors in mathematical typography that they might not have made when writing the problem by hand. Second, we allowed students to take the test twice per day rather than once, but required that they go to the Math Lab to review a failing test with a tutor before returning to take it again. Finally, we created a version of the gateway that allowed students to easily practice the test whenever they wanted.

Since the implementation of the on-line system, we have added additional gateway tests. Calculus III (math 215) added an entrance gateway on integration skills in fall 2005, and in fall 2015 a gateway test on matrix operations will be added to one of our linear algebra courses, math 214.

Assessment

Do these tests work? There are a number of possible measures of this. We might first ask if students pass the tests; if not, it is unlikely that they are accomplishing anything productive. If students pass the tests, we might next ask if they appear to be trivial—do all students pass them immediately, or do they require some level of work to pass. And, if students are passing the tests and the tests do appear non-trivial, the larger question of whether students learn the covered skills comes to the fore.

Do students pass the tests? The answer to this is unequivocally “yes”: pass rates on the pre-calculus entrance gateway are about 93%; on the calculus I differentiation gateway 99%; on the calculus II entrance gateway 99%, and on the calculus II integration gateway 92%. The pass rate on the calculus III entrance gateway (which is identical to the math 116 integration gateway) is about 95%. Thus, students do pass the test.

Are the tests trivial? This does not appear to be the case. The average number of proctored tests taken per student taking the pre-calculus entrance gateway is about 2.7; for the calculus I differentiation gateway the number is 2.0; for the calculus II entrance gateway 1.5 and for the integration gateway, 2.9; and for the calculus III entrance gateway, 2.4. This is in addition to the times that students take the practice version of the tests (in fall 2014, calculus I students took approximately 2.8 practice tests each).

Finally, do students learn the material that is being tested? We have some evidence for this. Testing of the gateways when they were implemented in 1993 indicated a strong correlation between students’ having passed the gateway and having acquired the skills being tested on the gateway test.[6] In 2002 we assessed the on-line tests by giving students short pencil-and-paper pre- and post-tests on the gateway material. The pre-test was administered immediately after the tested skills had been covered in class, and the post-test shortly after the finish of the gateway tests. Comparison of pre- and post-test results concluded that students’ skills improved significantly, and as the single biggest component of the course between the pre- and post-tests was the gateway test, we are fairly confident that this improvement may be attributed to students’ efforts on the gateway test.[5] In addition, calculus I students who had not taken calculus before agreed, on average, with the statement that they learned how to take derivatives as a result of having to complete the gateway.[5] These conclusions are similar to those drawn by another assessment of a test similar to our gateways.[2]

Logistics

Running our gateway program is a nontrivial task. Our pre-calculus, calculus I and II courses enroll a total of about 3000 students. In the recent past we’ve had class-sections capped at 32 students, resulting in about 20 class-sections of pre-calculus each fall, 55 sections of calculus I, and 25 of calculus II. For each of these sections we create a separate “course” in our WeBWorK system. Calculus III enrolls slightly over 1000 students in the fall, and the linear algebra course in which we have a gateway has about 240 students. Thus, all told, we create well over 100 WeBWorK courses each semester for over 4000 students, and actively manage their rosters thoughout the add/drop period. Because all of these courses already use web homework, course and roster management is, in some sense, outside of the gateway program. We will discuss the web homework more in a subsequent blog post.

To allow students to take the test for practice as often as they like, two versions of each gateway are created: one that does not require proctor authorization, and one that does. The former may be taken by students as many times as they like, from wherever they like—except that it is not available in the gateway testing lab. WeBWorK supports access restrictions for tests based on computers’ IP addresses, and we therefore prevent the practice test from being taken in the gateway lab to avoid the possibility that a student could take an unproctored test there and then expect that s/he has passed the test. The proctored test has identical content and is drawn from the same testbanks, but requires a proctor’s authorization to start the test and again to grade it. This allows proctors to verify a student’s identity when s/he starts the test, and to simply not authorize a test for grading if a student is not following testing protocol. Our proctors are undergraduate student workers employed through our Math Lab. The proctored test has access restrictions that prevent it from being taken except in the gateway lab. Computers in the gateway lab are configured to allow internet connections to only the University’s login and name servers and the Department’s instructional technology server.

There is a third, mostly invisible, version of each test as well. This also requires a proctor’s authorization to take, and is called the “instructor proctored gateway test.” The goal of this test is to allow instructors the leeway to administer the test in their office, or to give a student an extra attempt at the test on a day when the instructor is working with her/him on the gateway skills.

Our first gateways start in the first full week of classes and the last finishes about two-thirds of the way through the semester. This results in about 12,000 visits to our gateway lab (in the fall semester), which seats approximately 30 simultaneous test takers. Thus, scheduling the open and due dates for the different gateway tests is a delicate matter: our rule of thumb is that we can’t have more than about 1500 students taking tests at the same time, and if there are different tests being taken simultaneously, we must ensure that their deadlines are not too close.

Our gateway lab is open the same hours as our Math Lab, viz., 7–10pm Sunday–Thursday, and 11am–4pm Monday–Friday. We aim to have two or three proctors staffing the lab at all times. Occasionally near deadlines (especially the math 116 integration gateway deadline) we open a second computer lab to increase capacity and reduce wait times. Gateway proctors are managed by the Director of the Math Lab, and they are either off-duty or become tutors in the Math Lab when there are no gateways being offered. Proctors undergo a short training session and are provided with procedural information about the tests.

Accessibility

It is worth including in our discussion of the logistics of our gateway tests some note about accessibility. Overall, our experience with our on-line tests is that they are well-equipped to deal with issues of accessibility. In recent fall semesters slightly under 3% of students in pre-calculus, calculus I, II and III have documentation indicating that they need extended time on exams, which we accomplish easily by updating the gateway test time limits for them, and in general the gateway lab is a space in which there are limited distractions. In the past 15 years of on-line gateway testing, we have also provided accomodation for students with needs that range from use of a mouth-controlled pointing device for all computer input to screen magnification of up to 500%.

Conclusions

We do many things in our Introductory Program, and gateway testing is just one piece of a very big system. In some respects, it’s an odd piece of the puzzle—most of what we do revolves around trying to increase student collaboration in and out of the classroom, increasing students’ engagement with the big ideas of the material we cover, and de-emphasizing basic skills and traditional lecture-centered teaching techniques. Gateways, on the other hand, are only concerned with skills. But by having this focus, gateways fundamentally support our ability to teach the courses as we want to: they make these skills expected and intrinsic, and students acquire them largely on their own, with the result that we are able to spend class and instructor time on the concepts and activities that will have an impact on deep student learning.

References

1. Blake, L. 2016. Personal communication.

2. Fulton, S.R. 2003. Calculus ABCS: A Gateway for Freshman Calculus. PRIMUS 13(4):361–372.

3. WeBWorK. http://webwork.maa.org. Accessed 11 August 2015.

4. Giordano, F.R. 1992. Core Mathematics at the USMA. Department of Mathematical Sciences, USMA. West Point, NY.

5. LaRose, P.G. & R. Megginson. 2003. Implementation and Assessment of On-line Gateway Testing. PRIMUS 13(4):289–307.

6. Megginson, R. 1994. A Gateway Testing Program at the University of Michigan. In Preparing for a New Calculus, A. Solow, ed., pp.85–89. Mathematical Association of America. Washington, D.C.

7. Megginson, R. 2014. Gateway Testing in Mathematics—Past and Present. WeBWorK and Mathematics Support Center Workshop at HKUST 10–11 June 2014. http://www.math.ust.hk/~support/workshop-2.html. Accessed 11 August 2015.

8. Megginson, R. 2015. Personal communication.

9. Speyer, D. 2012. Some thoughts on teaching Michigan calculus. In the Secret Blogging Seminar blog,
https://sbseminar.wordpress.com/2012/02/08/some-thoughts-on-teaching-michigan-calculus/. Modified 11 April, 2013. Accessed 17 August, 2015.

10. Lotus 1-2-3. https://en.wikipedia.org/wiki/Lotus_123. Modified 19 June, 2015. Accessed 11 August, 2015.

Comments are closed.