PHP-FPM
The Instance went down again. Instead of rushing to get the instance back up I check some logs and it appears that when I increased the values I did not increase the pm.max_spare_servers. I allowed more to be generated but did not allow them to exist 😉. Which of course took php-fpm down.
I increased pm.max_spare_server so hopefully this time I have enough to handle the load. As you can see in the screenshot below as soon as more spare servers were added that brought it above 100, the whole service shutdown bringing the site down.
The setting still may need massaging. Time will show.
Radio Free Trumpistan likes this.
Radio Free Trumpistan
in reply to Unus Nemo • •I dug 'way back to 2023 to recall the error messages I was originally getting on QOTO and Dr. Freemo was telling me I was the only one encountering them. It was a Rate Limit error, and the reason for that problem was that I was going straight to the Federated tab instead of the Home tab (the equivalent would be the Friendica "Conversations on this and other servers" circular tab, then choosing Other Servers. It turned out that I was indeed exceeding the Qoto rate limit by accessing the Fediverse. Dr. Freemo fixed that, once we nailed down what it was, but I've been getting the 500 error ever since. Hope this helps.
Unus Nemo likes this.
Unus Nemo
in reply to Radio Free Trumpistan • •@Radio Free Trumpistan
Thank you I appreciate the effort you have put in to help. I am waiting for the next HashTagGame event to see if the Instances crashes or not. I found a logic error in my configuration and I am hoping that my updates fixes the issue. If not I will keep working on it 😉.
Radio Free Trumpistan likes this.
Radio Free Trumpistan
in reply to Unus Nemo • •Unless there's a technical disaster at Paul's end, it's always going to be at 9 pm Eastern...and even when his server goes down as it did recently, he's got a team to start it anyway. 😁
I did play Hashtag Games on Qoto with no problem, though--just sayin'. Just fishin' out possible clues when I find 'em.
Unus Nemo likes this.
Unus Nemo
in reply to Radio Free Trumpistan • •@Radio Free Trumpistan
I am fishing as well. It is the only thing that is new. I have been up for months at a time without a crash and suddenly it starts. Now of course just because two things happen at the same time does not mean it is related. There could be something else I am overlooking. Though I do know:
None of this is definite. And I may be barking up the wrong tree. Though I suspect that surviving the event without a crash will be a good indicator.
Radio Free Trumpistan likes this.
Radio Free Trumpistan
in reply to Unus Nemo • •I guess the think to do is to keep after it until we find "the red X"....that is, narrow it down to the actual culprit. I do know that the thing here and Qoto have in common is that handling traffic from all those other servers in the Fediverse is like trying to drink from a fire hose, so it's not surprising that systems can get overwhelmed.
Default display on this Friendica account is 40 messages and I dropped that down to 20. If you think dropping it down to 10 might be helpful, I'll certainly change that setting.
I also have automatic refresh turned off because there's no way I can read or reply to stuff that comes in as fast as all that. I tried it before and it was frustrating, so I don't want to change that one....but that'll give you an idea of the fire hose effect is experienced at the user end when reading Global Other Servers.
Unus Nemo likes this.
Unus Nemo
in reply to Radio Free Trumpistan • •@Radio Free Trumpistan
40 messages should be fine. Though I agree with you on automatic refresh. The thing is that the instance should be stable with hundreds of users. We have 3 active users. There is no way I should be seeing any hiccups at this point. Unfortunately Friendica comes with an extremely poor default configuration. So I have had to work out fixes and stay on top of issues that other admins report on github. Friendica World has more than a 1000 users and though Ruud (the admin) has his issues it tells me that I should be able to handle more.
I do not expect to handle a thousand users mainly because I cannot afford a larger storage layer. So I am looking at realistically capping out at around 250. Most storage is not permanent but I still have to have enough to handle the flow of messages for the user base.
Though I appreciate the effort you put in I would like you to not take it easy on the server. I
... show more@Radio Free Trumpistan
40 messages should be fine. Though I agree with you on automatic refresh. The thing is that the instance should be stable with hundreds of users. We have 3 active users. There is no way I should be seeing any hiccups at this point. Unfortunately Friendica comes with an extremely poor default configuration. So I have had to work out fixes and stay on top of issues that other admins report on github. Friendica World has more than a 1000 users and though Ruud (the admin) has his issues it tells me that I should be able to handle more.
I do not expect to handle a thousand users mainly because I cannot afford a larger storage layer. So I am looking at realistically capping out at around 250. Most storage is not permanent but I still have to have enough to handle the flow of messages for the user base.
Though I appreciate the effort you put in I would like you to not take it easy on the server. It is important that I get the configuration right. Like I said with 3 active users (and a relay going) there should be no issue at all with resources giving out. PHP-FPM is not running out of resources due to them not being available it is running out of them due to the resources not being allocated to it.
We will eventually hit the X to be sure. It is just a matter of research and implementation. Thanks for your help.
Radio Free Trumpistan likes this.
Radio Free Trumpistan
in reply to Unus Nemo • •Happy to do what I can. Okay, put 'er thru the paces it is, then. Will do!
Unus Nemo likes this.
Unus Nemo
in reply to Radio Free Trumpistan • •@Radio Free Trumpistan
Friendica is supposed to able to operate with far less resources than I have available on my VPS. I have dedicated that VPS to nothing but Friendica and support services for Friendica. I have only had this Instance up for a few months. So in time I will get it right 😉.
Radio Free Trumpistan likes this.
Radio Free Trumpistan
in reply to Unus Nemo • •That's the spirit!
Unus Nemo likes this.