Incident Timeline

On July 26th, 2025 at approximately 5:15AM PT, the server encountered a bug causing the majority of players to freeze. Logging out and back in, just caused you to get stuck at the character selection screen.

5:30AM - [GM] Cole was made woken up and made aware of the issue.

5:45AM - The investigation was fully in effect.

6:15AM - The cause of the issue was identified and the process for bringing the game up begun.

6:23AM - The phoenix server was brought back online.

6:25AM - [GM] Cole begun work on a permanent solution for this issue to ensure it didn't happen again.

6:35AM - [GM] Cole was made aware of an issue the server issue caused with the Demon Servant boss. Read more in it's own section below.

Issue Root Cause

This issue was caused by the TCP communication layer for the server communicating with each player. Essentially a laggy player could cause the server to lag.

We've enacted a fix that ensure no players connection can negatively affect the server - or other players. Just themselves. (I'm making this sound way easier than it actually was)

This fix will be deployed to the main game via a hotfix patch, relatively quickly after this post goes live.

Demon Servant Issue

Literally as the Demon Servant died, the server issue happened. This caused the DS to die, set its new spawn time, but not give any rewards. No items, money, or exp.

We verified this information with our logs and then verified the guild of the player reporting the issue were the only ones to deal damage to the boss. (No contention)

As no reward was given, spawn time was saved, our options to restore the game to it's rightful integrity were limited. Typically in this case, we'd have drop logs because the items would drop and no one could pick them up, so our resolution would be to simply restore those items.

This situation was not typical, the items were never dropped. Money was never rewarded. Exp was never rewarded. To "Restore" for this situation, the only real option we had was for this kill to happen again. We could not do this in CoT14, because it'd reset the boss timers thinking this was the real kill. It'd also have some pretty weird side effects after 2-3 respawns.

As such, [GM] Cole teleported that guild to GM land, and let them re-kill the DS, getting their rightful entirely random, rewards. If GM land gets bugged after the fact, it doesn't affect anyone but GMs, so, best option.

Typically we wouldn't ever share information from a private support ticket, but as this is a very unique case and the community at large deserves transparency, we're adding a screenshot below.