[Other] Recommended way to count messages for auto-scaling when not using prometheus

[mauvilsa]

Community Support Policy

• I have read RabbitMQ's Community Support Policy

RabbitMQ version used

4.0.3

How is RabbitMQ deployed?

Community Docker image

Steps to reproduce the behavior in question

I have seen in the monitoring docs that the management plugin API should only be used for development environments, and for production prometheus is recommended. We want to auto-scale services based on the number of messages in a queue including ready and unacknowledged, but do not plan to use prometheus. For auto-scaling we use Keda (see kedacore/keda#6494).

Counting messages using the amqps protocol doesn't work because it doesn't support counting unacknowledged messages. The only current alternative in keda is to use https via the management plugin. But this conflicts a bit with what we see in the rabbitmq docs. An idea could be to implement counting of messages through the rabbitmq_prometheus plugin. But from what I understand that can't be used to get information about a single queue, making it a bit overkill each call would provide too much non-relevant information.

Questions:

• Are metrics from the management plugin deprecated in the sense that eventually they will be removed?
• What is the recommended way to count number of ready+unacknowledged messages in a single queue when not using prometheus?

[michaelklishin]

@mauvilsa use GET /api/queues/{vhost}/{name}.

DO NOT use GET /api/queues to get a few metrics of one queue, that would be really really wasteful.

[JorTurFer]

Hello @michaelklishin,
Currently, KEDA calls to /api/queues/{vhost}/{name} if user sets a given queue, we only call to /api/queues when user wants to use regex.

But keeping aside the querying endpoint, the important question for us is if metrics from the management plugin are deprecated in the sense that eventually they will be removed or we can continue using them for scaling (although maybe we need to add a note about enabling the feature on RabbitMQ 4.0). I ask because of this post (quite old, tbh)

[michaelklishin]

@JorTurFer at the very least you can probably use GET /api/queues/{vhost} and require the virtual host to be provided for such "queue group metrics". If you can use Prometheus, I'd use Prometheus.

I doubt that management metrics would be removed but Prometheus-based tools are superior to what the management UI has to offer, and it will always be the case.