Skip to content

E
edx-platform
Commit 136f8d4e by Christine Lytwynec
Options

removing jenkins doc from platform

parent e6e443c7
Showing with 4 additions and 235 deletions
docs/en_us/internal/jenkins/builds.md
## Starting and checking the status of builds
There are three ways that builds start in our jenkins testing infastructure for edx-platform.
##### 1) Automatic builds for Pull Requests
* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people).
If you are not, someone will start a build for you during the pull request review process. You will still be able to
view the build results as described below.
* How it gets started
>* When you submit a pull request to the edx-platform repository, a jenkins build will
> automatically start and run unit and acceptance tests at the most recent commit.
>* When you add a new commit to the PR, a new build will be run for those changes.
>* Sometimes it may take a little while for the build to start. That usually just means that
> jenkins is pretty busy.
* How it is reported
>* You will know a build is started if you see this:
>
> ![Running Tests](jenkins_images/started_tests.png)
>
>* When it is finished you will see either a green checkmark or a red X, indicating that the
> build either passed or failed respectively.
>
> ![Passed Tests](jenkins_images/passed_tests.png)
>
> ![Failed Tests](jenkins_images/failed_tests.png)
>* You can click on 'details' to take you to the jenkins build report.
##### 2) Manually started builds for pull requests
* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people).
* How it gets started
>1. Go to [edx-all-tests-manual-pr](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/)
>2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the
> upper right corner of the page. Else, the 'log in' link will be there.
>3. Click 'Build with Parameters' in the left navigation column.
>
> ![Build with Params](jenkins_images/build_w_params.png)
>
>4. Enter the PR number from edx-platform that you want to test.
>5. Click on 'Build'.
* How it is reported
>* This will be reported the same as Automatic builds for Pull Requests are. (See point 1 of this
> section.)
>* When you start the build, it will redirect you to the log page. You can watch this page for
> results as well.
##### 3) Manually started builds for commits
* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people).
* How it gets started
>1. Go to [edx-all-tests-manual-commit](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/)
>2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the
> upper right corner of the page. Else, the 'log in' link will be there.
>3. Click 'Build with Parameters' in the left navigation column.
>4. Enter the commit hash that you want to test.
>5. Click on 'Build'.
* How it is reported
>* When you start the build, it will redirect you to the log page. You can watch this page for
> results.
>* The results will also be reported to github, and will show up next to your commit on a PR or
> other places that the commit is listed (search results, etc.).
See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide
docs/en_us/internal/jenkins/jenkins.md
## Guides for edx-platform tests on Jenkins
* [Starting and seeing the status of builds](builds.md)
* [Finding reports (coverage, pep8/pylint, test results)](results.md)
* [Troubleshooting failures/errors](troubleshooting.md)
\ No newline at end of file
See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide
docs/en_us/internal/jenkins/results.md
# Looking at build results
_build log_ will refer to something from the following urls:
```
https://jenkins.testeng.edx.org/job/edx-all-tests-auto-pr/[NUMBER]/
https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/[NUMBER]/
https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/[NUMBER]/
```
_build report_ will refer to something with the following urls:
```
https://jenkins.testeng.edx.org/job/edx-platform-report/[NUMBER]/
```
Note that navigating to a build via github (clicking 'details') while it is in a pending or failed state
will take you to a _build log_, but doing so after it has reported back as passed will bring you
to a _build report_. To get to a _build log_ from a _build report_, click on the build number in
the 'Upstream Builds' section:
>![Upstream Builds](jenkins_images/build_link.png)
## Coverage Reports
Note: Coverage reports are only generated when your tests pass.
* How to find:
__From the build report__
> On the left side you will see:
>
> * 'Coverage.py Report' which shows the html reports produced by the coverage
> package.
> * 'Diff Coverage Report' which shows the coverage of _new_ lines of code
>
> ![Coverage Nav Links](jenkins_images/coverage_report_nav_links.png)
__From the build log__
>1. Click on one of the unit test or bok-choy shards.
>
> ![Unit and BokChoy Shards](jenkins_images/unit-bok-choy-shards.png)
>
>2. Click on 'Build Artifacts'.
>3. You can find the generated html in the reports folder.
> - For unit tests:
> + reports/cms/cover
> + reports/common/lib/calc/cover
> + reports/common/lib/capa/cover
> + reports/common/lib/xmodule/cover
> + reports/lms/cover
> - For bok-choy:
> + reports/bok_choy/cover
## Code Quality Reports
These are generated regardless of test restults.
Note that a passing build does __not__ mean that you have not introduced new pep8/pylint violations.
* How to find:
__From the build report__
> On the left side you should see:
> * 'Diff Quality' which shows _new_ violations
> * 'Violations' which shows _all_ violations
>
> ![Quality Nav Links](jenkins_images/quality_report_nav_links.png)
>
> There will also be a graph on the build report page showing trends of pep8 and pylint
> violations.
>
> ![Quality Graph](jenkins_images/pep8_pylint_graph.png)
__From the build log__
> 1. Click on the quality shard to get to the report for the pep8 and pylint results.
>
> ![Quality Shard](jenkins_images/quality_shard.png)
>
> 2. Under 'Build Artifacts', you should see the reports.
>
> ![Quality Reports](jenkins_images/quality_artifacts.png)
>
+ "diff\_quality\_pep8.html" and "diff\_quality\_pylint.html" are probably the most
useful to look at.
- If don't see these here, click on 'Build Artifacts' and check in
'reports/diff_quality'
- These show how many _new_ violations have been introduced in your changes.
+ You can also download the full reports (files ending in '.report')
- These show _all_ of the found violations, not just newly introduced ones.
## Test Results
###### Finding results for individual tests
* If your build fails, the shards that had failures will have a red dot in the chart on the build log.
> ![Failed Unit Tests](jenkins_images/failed_shard.png)
* Click on the failed shard, and at the bottom of the new page you should see something like this:
> ![Test Results Link](jenkins_images/test_results_link.png)
* The 'Test Result' link (above) will take you to a summary of the test results (below).
> ![Test Results Page](jenkins_images/test_results_page.png)
* From this page you can click through the test names to get to the console log for each test.
###### Console output for the shard
Examining the console output of a build can provide more information about the test results. For example,
sometimes a worker will have an error during setup for the tests. If this happens, you will see a red dot for the shard, but no
test failures reported (since the tests did not run). In this case, checking the console log can be helful to determine the
reason for the build failure. It can sometimes be useful to review even if you can find the individual error message for a test.
To find it from the build log, click on the 'Console Output' link in the left navigation column.
> ![Console Log](jenkins_images/console_log.png)
See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide
docs/en_us/internal/jenkins/troubleshooting.md
# Troubleshooting failing jenkins builds
###### Check the logs to find out why your build failed.
* See [here](results.md#console-output-for-the-shard) for how to find the log.
* If the log shows the build erroring out before tests begin running, report this to the test
engineering team. Be sure to share the link to your build log.
###### Run tests on devstack
* If a build fails on jenkins, try to reporoduce and debug the failures in devstack.
* See the [testing guide](../testing.md) for how to run the tests.
- For lettuce acceptance test debugging, see [this section](../testing.md#debugging-acceptance-tests-on-vagrant).
* For more debugging tips, see the [testing FAQ](https://github.com/edx/edx-platform/wiki/Test-engineering-FAQ).
There you'll find tips for:
- setting breakpoints in tests
- visually debugging acceptance tests
- running tests from a single file
- and more..
* "What's devstack?" ... See [here](https://github.com/edx/configuration/wiki/edX-Developer-Stack)
###### Check the 'flaky test' list
* Known flaky tests can be found with filters on the openedx Jira site
- [Known unresolved flaky tests](https://openedx.atlassian.net/issues/?filter=10600)
- [Recently resolved flaky tests](https://openedx.atlassian.net/issues/?filter=11001)
* Remember that a test being listed as flaky doesn't mean that it can't fail for other reasons. Look into
the logs and confirm that it is failing for the same reason as listed in the issue ticket.
* If your build has failure that is a recently resolved flaky test, try rebasing from master. (A new auto-pr
build will start when you do this on an open PR.)
###### Check if the failure is occurring on the master branch
* Tests run for the master branch are [here](https://jenkins.testeng.edx.org/job/edx-all-tests-auto-master/).
You can inspect them the same way you would a PR build.
See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment