1. 22 Feb, 2017 1 commit
  2. 17 Feb, 2017 1 commit
  3. 16 Feb, 2017 2 commits
  4. 14 Feb, 2017 4 commits
  5. 13 Feb, 2017 2 commits
  6. 10 Feb, 2017 2 commits
  7. 09 Feb, 2017 1 commit
  8. 08 Feb, 2017 1 commit
  9. 07 Feb, 2017 2 commits
  10. 06 Feb, 2017 1 commit
  11. 05 Feb, 2017 1 commit
  12. 03 Feb, 2017 1 commit
  13. 01 Feb, 2017 3 commits
  14. 31 Jan, 2017 2 commits
  15. 28 Jan, 2017 2 commits
  16. 27 Jan, 2017 1 commit
  17. 26 Jan, 2017 1 commit
    • Disable student state writes for crawlers. · 5ef1e080
      When crawlers like edX-downloader make requests on courseware, they are
      often concurrently loading many units in the same sequence. This causes
      contention for the rows in courseware_studentmodule that store the
      student's state for various XBlocks/XModules, most notably for the
      sequence, chapter, and course -- all of which record and update user
      position information when loaded.
      
      It would be nice if we could actually remove these writes altogether
      and come up with a cleaner way of keeping track of the user's position.
      In general, GETs should be side-effect free. However, any such change
      would break backwards compatibility, and would require close
      coordination with research teams to make sure they weren't negatively
      affected.
      
      This commit identifies crawlers by user agent (CrawlersConfig model),
      and blocks student state writes if a crawler is detected. FieldDataCache
      writes simply become no-ops. It doesn't actually alter the rendering
      of the courseware in any way -- the main impact is that the blocks
      won't record your most recent position, which is meaningless for
      crawlers anyway.
      
      This can also be used as a building block for other policy we want to
      define around crawlers. We just have to be mindful that this only works
      with "nice" crawlers who are honest in their user agents, and that
      significantly more sophisticated (and costly) measures would be
      necessary to prevent crawlers that try to be even trivially sneaky.
      
      [PERF-403]
      David Ormsbee committed
  18. 25 Jan, 2017 3 commits
  19. 24 Jan, 2017 2 commits
  20. 23 Jan, 2017 3 commits
  21. 20 Jan, 2017 1 commit
  22. 19 Jan, 2017 1 commit
  23. 18 Jan, 2017 2 commits