Enable horizontal autoscaling for service deployments

[AlexIoannides]

Description
As a Machine Learning Engineer, I would like for the number of replicas standing behind my services, to scale automatically, based on CPU utilisation, so that I do not have to frequently monitor CPU utilisation and manually change the number of replicas.

Tasks

• extend the config schema to allow for a new (optional) scale_out_replicas parameter, that will represent the number of replicas above those specified in replicas, that Kuberentes can scale the deployment up to.
• bodywork.k8s.service_deployments need to be extended to enable CRUD operations for HorizontalPodAutoscaler resources.
• if scale_out_replicas is present, then the bodywork.config.StageConfig object should flag to bodywork.workflow_execution.run_workflow, that a HorizontalPodAutoscaler resource should be created.
• think a lot about how to test this - e.g. monitor a service that will deterministically consume CPU resources and require scale-out to kick-in?
• update docs, where required.

Resources

• refer to section 15.1.2 and listing 15.2 of 'Kubernetes in Action'.