This allows usage of custom ssh configs for remote hosts where
the inventory user does not match the configured user.