task.yml
6.37 KB
-
Add support for templating to Luigi configuration files, redux. (#407) · bad58b51
* Allow multiple secure configurations to be loaded. This change allows a caller to specify multiple configuration files in the secure config repo to be loaded, which enables the caller to specify a "base" configuration, containing special values, which can be interpolated into subsequent configuration files that are loaded. These configuration paths are now passed on from remote-task itself to launch-task when invoked on an EMR cluster. * Weave through paths in a more consistent fashion. We're now properly including 'uid', which is hard-coded to 'automation' in production, to the paths. This should make the code more resilient to cases where no remote name is specified. We've also moved the constants back into remote.py, given that they weren't necessary in local.py and thus did not need to be shared. * Remove old local wildcard import. * Remove our extra command-line arguments before invoking Luigi. Luigi gets angry when it runs a workflow and sees command-line arguments that Luigi itself didn't ask for or that the workflow didn't ask for.. which is precisely what --additional-config is. This change modifies sys.argv in place, removing all key/value pairs for --additional-config, before we invoke luigi.run(). It *seems* to work, but we're also modifying a global in the sys module, so, you know.. * Weave arguments through run(); don't modify sys.argv. * Remove extraneous/old comment.
Toby Lawrence committed