1. 26 Oct, 2017 1 commit
  2. 12 Jun, 2017 1 commit
  3. 18 Apr, 2017 1 commit
  4. 26 Jan, 2017 2 commits
    • Remove select_for_update option for FieldDataCache · 608fceb0
      We've already been ignoring the param for a long time. This just
      removes it entirely.
      David Ormsbee committed
    • Disable student state writes for crawlers. · 5ef1e080
      When crawlers like edX-downloader make requests on courseware, they are
      often concurrently loading many units in the same sequence. This causes
      contention for the rows in courseware_studentmodule that store the
      student's state for various XBlocks/XModules, most notably for the
      sequence, chapter, and course -- all of which record and update user
      position information when loaded.
      
      It would be nice if we could actually remove these writes altogether
      and come up with a cleaner way of keeping track of the user's position.
      In general, GETs should be side-effect free. However, any such change
      would break backwards compatibility, and would require close
      coordination with research teams to make sure they weren't negatively
      affected.
      
      This commit identifies crawlers by user agent (CrawlersConfig model),
      and blocks student state writes if a crawler is detected. FieldDataCache
      writes simply become no-ops. It doesn't actually alter the rendering
      of the courseware in any way -- the main impact is that the blocks
      won't record your most recent position, which is meaningless for
      crawlers anyway.
      
      This can also be used as a building block for other policy we want to
      define around crawlers. We just have to be mindful that this only works
      with "nice" crawlers who are honest in their user agents, and that
      significantly more sophisticated (and costly) measures would be
      necessary to prevent crawlers that try to be even trivially sneaky.
      
      [PERF-403]
      David Ormsbee committed
  5. 09 Dec, 2016 1 commit
  6. 01 Dec, 2016 1 commit
  7. 20 Oct, 2016 1 commit
  8. 24 Jun, 2016 1 commit
  9. 30 Mar, 2016 1 commit
  10. 08 Dec, 2015 1 commit
  11. 26 Aug, 2015 1 commit
  12. 01 Aug, 2015 2 commits
  13. 14 Jul, 2015 1 commit
  14. 09 Jul, 2015 1 commit
    • Optimize grading/progress page to reduce database queries (cache max scores). · 79de77cf
      The progress page did a number of things that make performance terrible for
      courses with large numbers of problems, particularly if those problems are
      customresponse CapaModule problems that need to be executed via codejail.
      
      The grading code takes pains to not instantiate student state and execute the
      problem code. If a student has answered the question, the max score is stored
      in StudentModule. However, if the student hasn't attempted the question yet, we
      have to run the problem code just to call .max_score() on it. This is necessary
      in grade() if the student has answered other problems in the assignment (so we
      can know what to divide by). This is always necessary to know in
      progress_summary() because we list out every problem there. Code execution can
      be especially slow if the problems need to invoke codejail.
      
      To address this, we create a MaxScoresCache that will cache the max raw score
      possible for every problem. We select the cache keys so that it will
      automatically become invalidated when a new version of the course is published.
      
      The fundamental assumption here is that a problem cannot have two different
      max score values for two unscored students. A problem *can* score two students
      differently such that they have different max scores. So Carlos can have 2/3 on
      a problem, while Lyla gets 3/4. But if neither Carlos nor Lyla has ever
      interacted with the problem (i.e. they're just seeing it on their progress
      page), they must both see 0/4 -- it cannot be the case that Carlos sees 0/3 and
      Lyla sees 0/4.
      
      We used to load all student state into two separate FieldDataCache instances,
      after which we do a bunch of individual queries for scored items. Part of this
      split-up was done because of locking problems, but I think we might have gotten
      overzealous with our manual transaction hammer.
      
      In this commit, we consolidate all state access in grade() and progress()
      to use one shared FieldDataCache. We also use a filter so that we only pull
      back StudentModule state for things that might possibly affect the grade --
      items that either have scores or have children.
      
      Because some older XModules do work in their __init__() methods (like Video),
      instantiating them takes time, particularly on large courses. This commit also
      changes the code that fetches the grading_context to filter out children that
      can't possibly affect the grade.
      
      Finally, we introduce a ScoresClient that also tries to fetch score
      information all at once, instead of in separate queries. Technically, we are
      fetching this information redundantly, but that's because the state and score
      interfaces are being teased apart as we move forward. Still, this only
      amounts to one extra SQL query, and has very little impact on performance
      overall.
      
      Much thanks to @adampalay -- his hackathon work in #7168 formed the basis of
      this.
      
      https://openedx.atlassian.net/browse/CSM-17
      David Ormsbee committed
  15. 12 Jun, 2015 1 commit
  16. 04 Jun, 2015 1 commit
  17. 03 Jun, 2015 1 commit
  18. 20 May, 2015 21 commits