SYSTEM ALERT (resolved): Scores saving, but not always showing.

Dear JumpRopers,

This afternoon at approximately 2:00pm EST, JumpRope experienced an error with one of it's servers that caused some scores to not display properly within the Grade and Mastery tab of JumpRope.  Especially as the term comes to a close, we know how crucial it is that the system operates reliably.  I'd like to personally apologize for the trouble.

As of 3:28pm EST, the error has been resolved.  Throughout the duration of the issue, scores were saving properly but were not showing up where expected in the Grade tab, causing understandable panic and frustration.  We did our best to correct the error as soon as possible, and as of now things are running smoothly again.  This issue did not affect any scores entered before the problem began around 2pm, and (again, to be very clear) scores were saving correctly the entire time.  In other words, you do not need to re-enter any scores - and hopefully most of you didn't waste too much time trying to re-enter while the issue was ongoing.

During this time, other aspects of JumpRope were operating normally.

If you'd like more details on the technical nature and the solution to the problem, feel free to read on.

Technical Details

JumpRope takes several steps to guard your data and protect it from corruption or accidental deletion.  In fact, every time you save a score it is written to four different hard disks in two different physical locations before you see the "Scores saved" confirmation.  Three of these four could fail at any time and we can still recover your scores.

In addition to this "synchronous" replication of your data, we also host an "asynchronous" replica of the data that we use to improve performance.  In other words, your data is safely written to four locations and you are notified, and then it is later (usually within < 10 milliseconds) written to yet another location that is optimized and cached for quick reading of the data.

This afternoon, the server that hosts the asynchronous replica of your data experienced a failure.  This failure did not crash the machine or raise any obvious flags to our operations team, unfortunately - rather, it simply stopped updating the latest scores that you had entered and kept returning data as if nobody was entering scores after the error first began.  In other words, you were looking at a "stale" version of your data when JumpRope showed you the scores, even though your newly entered scores were safely stored in the other four locations.

It took us about half an hour to notice that there was a problem, and for this we apologize.  We are taking steps to make sure that any time this error (and others) happen again, we will be notified immediately.  Once we became aware of the issue, we had to reboot the machine and make sure that it caught up with the real version of the data.  At that point, it was nearly 3pm and some of you had already contacted us notifying us of the problem.

The server took a little over half an hour to catch up with the hundreds of thousands of scores that had been entered in the meantime, and thus normal operation was restored over the next 30 minutes depending on the order in which scores had been entered.  As of 3:28pm, the server was completely up to date and is now running normally.

Again, I'd like to personally apologize to those of you who were affected by this issue.  We do our best to keep the system running smoothly and reliably, especially at crucial moments such as the end of the term.  Today, we failed at that goal.  Having taught myself for many years, and can very much understand your frustration.

To those of you entering a new term and our NYC folks going through Regents exams, best of luck!  As always, if you need assistance with JumpRope you can always reach out through our support system.

Thanks,

Jesse