Downtimes Problems

Server flatlined third day in a row

At the moment we don’t have any idea why. As I am in Italy right now I will investigate this after my return home in the middle of the week. Please stay tuned.

Is anybody aware of any current exploits for ejabberd?

Problems Transports

ICQ transport problems

We experienced problems with ICQ transport the last days. According to the developers of Spectrum this is a “glib poll” issue. epoll unfortunally does not work with glib (yet).

Therefore Spectrum ICQ transport now uses libev3 and epoll.

It looks good at the moment, I hope it stays that way. We will see what happens during the next days as our server is the only server which runs this configuration. Not even the Spectrum developers were able to test it, yet. ;-)

Downtimes Problems

Numerous crashes…

…happened the last days. This occured out of nothing. Same symptomatic like some time ago when the Jabber server thread suddenly took all available resources and killed the server itself.

We digged deep into the problem and hope, we found a solution for it – which we implemented about one hour ago. A short service outage took place because of a restart to activate the setting.

Hopefully we got to the problem.

Downtimes Problems

DoS again?

Are we under attack again? It seems – because the server went down yesterday and today. Sorry for the outage, we are looking into the logs as I write this post. Maybe we can find something…

Downtimes Problems

Jabber service outage

There was an outage of our Jabber services the last hours. Unfortunally the hard disk ran completly full because of a faultly backup job.

Everything is back online. Sorry guys!

Downtimes Problems

DoS once again yesterday evening

Once again the server was hit. At around 8pm yesterday evening the services were unreachable for about 30 minutes, the transports for about 60 minutes.

For sure there will be another incident, but this time everything will get logged. From what we then read out of the logs we will choose how we react. If it has something to do with Nimbuzz, we will block Nimbuzz completely.

Downtimes Problems

DoS attack again…

Again we suffered a DoS attack. The Jabber services and website were down from ~10:45pm to 00:30am.

For sure we have to do something about this. We are already digging into this, also recieving feedback from other Jabber server admins. Maybe abusive Nimbuzz accounts are the reason and, if this turns out to be true, we are also thinking about firewalling Nimbuzz.

What would you think about blocking Nimbuzz? Are you having friends from Nimbuzz in your roster? Tell us your opinion.

Downtimes Problems

Server flatlined four times in not even two hours…

Due to until now unknown reasons the server which holds the Jabber services crashed four times in the last not even two hours. From one second to the other the ejabberd processes took every resource they could get, and even more. 8 gigs of RAM and 8 gigs of swap, everything gone. Plus a lot of CPU load. The machine was loaded that “top” refreshed just every 5 minutes and in the end just a hardware reset helped to reboot the machine.

For the tech geeks:

top – 19:56:21 up 31 min,  1 user,  load average: 22.86, 13.11, 8.71
Tasks: 240 total,   3 running, 231 sleeping,   0 stopped,   6 zombie
Cpu(s):  1.4%us,  5.8%sy,  0.0%ni, 12.4%id, 80.3%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   8190900k total,  8138972k used,    51928k free,      796k buffers
Swap:  8393848k total,  7276916k used,  1116932k free,    42404k cached

3239 ejabberd  20   0 15.7g 6.0g 460 S   23 76.7   3:06.65 beam.smp

We are looking into this issue. Maybe a severe bug with ejabberd, maybe a DoS attack. We don’t know, yet.