r/openstack • u/Capital-Pool4987 • 8d ago

HA proxy issue

We have created openstack from scratch. Now we have implemented high availability such that all services must discover each other through a floating ip on ha proxy. Moving forward from ha01 to ha02 is okay , but back to 01 it takes almost 17 minutes. In the horizon logs , every service i.e keystone, neutron is down. Then they come up again later. The only solution i can see is lowering the kernel retries but that doesnt really solve the core issue. https://docs.openstack.org/kolla-ansible/latest/reference/high-availability/haproxy-guide.html -> under failover tuning. Kindly assist.

0 Upvotes

50% Upvoted

7 comments sorted by

2

u/MisterJaerico 8d ago

Can you explain more about your setup?

1

u/iammpizi 7d ago

A floating IP and a VIP are very different things . I assume this is a typo here . Also neutron services do not “discover” each other. I am assuming what you mean here is that if an haproxy backend goes offline the other is not responding quickly enough

1

u/Capital-Pool4987 4d ago

Hello. The design is such everything is in high availabilty. The horizon nodes, identity, compute, database etc. So due to this design everything communicates with each other through the proxies that themselves are in high availability.

So when we kill the first proxy server , the floating ip moves to the second node...now the journey would be such that a browser request hits the floating ip that now is on proxy 2. This goes to one of the three keystone servers...then communication between the keystone server and one of the three database servers is again through the ha proxy and so on.....

Now going forward i.e proxy1 is down and the floating ip moves to proxy2, its okay. But going back from proxy2 to 1 inroduces a downtime where in the haproxy logs everyone is complaining. I.e neutron api is down, placement api, keystone. The after 17 minutes everything comes back up. Although restarting apache manually on horizon and the control node seems to bring things up. Also interestingly the reverse of the forward has no issue by this i mean 1 -> 2 then 2 -> 1 then 1 -> 2 ( this has no issue)

I hope the explanation is better.

1

u/iammpizi 4d ago edited 4d ago

Can you explain to me how you configured this "floating ip" ?
This looks more to me like a VIP as I said. Otherwise you can perhaps show a diagram ?

1

u/redfoobar 6d ago

You will need to put a bit more info here.
I guess you are moving an IP around between 2 machines.
Is the issue with the IP not being reachable? e.g. if you cannot ping it you probably need to do a gratuitous ARP.

Is the IP up but are the services themselves not working? Look at the API logs and HAProxy logs. e.g. what is "down" mean in the logs? Can it not reach the database? Not reach keystone etc..

1

u/Capital-Pool4987 4d ago

Hello. The design is such everything is in high availabilty. The horizon nodes, identity, compute, database etc. So due to this design everything communicates with each other through the proxies that themselves are in high availability.

So when we kill the first proxy server , the floating ip moves to the second node...now the journey would be such that a browser request hits the floating ip that now is on proxy 2. This goes to one of the three keystone servers...then communication between the keystone server and one of the three database servers is again through the ha proxy and so on.....

Now going forward i.e proxy1 is down and the floating ip moves to proxy2, its okay. But going back from proxy2 to 1 inroduces a downtime where in the haproxy logs everyone is complaining. I.e neutron api is down, placement api, keystone. The after 17 minutes everything comes back up. Although restarting apache manually on horizon and the control node seems to bring things up. Also interestingly the reverse of the forward has no issue by this i mean 1 -> 2 then 2 -> 1 then 1 -> 2 ( this has no issue)

I hope the explanation is better.

1

u/redfoobar 4d ago

What is “down”.

eg if you connect directly to one of the backend is it reachable?

eg is the issue IP connectivity or is the service itself crashing/no longer listening.