Because all-tests.sh was modified locally, I couldn't cleanly
cherry-pick 8672455f  from edx-platform. I took all of Jay's changes
ignoring all changes made on the solutions side.