jenkins2-ha-agents does not scale down if there is at least one job running

[Ten0]

On this template:

aws-cf-templates/jenkins/jenkins2-ha-agents.yaml

Lines 2326 to 2332 in e80083b

	BuildActiveQueueEmptyAlarm:
	Type: 'AWS::CloudWatch::Alarm'
	Properties:
	EvaluationPeriods: 1
	Statistic: Maximum # special rule because we scale on build queue length
	Threshold: 0
	AlarmDescription: 'Alarm if Build Queue is empty and no jobs are active.'

The alarm only triggers if there are absolutely no jobs left running.
This means that if e.g. during the day you need on average 20 agents, max 50, but never less than 1, you'll still always pay for 50 agents because as you always have one job running the alarm never triggers.

It looks like this could be fixed by:

• Making this alarm trigger if there are agents that don't do anything
• Using scale-in protection (on ASG) to prevent instances that currently have jobs running from being picked for termination

[michaelwittig]

I agree. The tricky part seems to set scale in protection in a reliable fashion as quickly as possible after a job runs on an agent.