Update from the Epoch Team

While we are undergoing this large and rapid transition to get the server ready for relaunch we would like to bring clarity and detail to some of the events over the last week and delve into the nitty gritty of the process of Epoch joining Ascension and what it means for both parties.

Recap of the situation up till last week

Internally, up until about one week before launch, we were anticipating a smaller population. Growing pains still seemed likely, but with gradual growth this would afford us time to scale the team and improve infrastructure. That said, there was a hope the realm could hold more players and the performance metrics from the stress test seemed promising at 2k players. This was a naive blunder on our part as we didn’t predict a bottleneck to show up after a few hundred extra players where the database begins to choke.

Fixing the bottlenecks

After spending the next two weeks dealing with the player scaling issues plaguing our launch we finally started seeing positive signs that the server no longer chokes at 3k players. A major culprit in what has been the bottleneck was an optional configuration in TrinityCore itself where we were tracking information about quest completion and abandonment status. Since we’re a custom server with a lot of custom quests, this information is very valuable to us. The problem was that this system in TrinityCore was horrendously implemented, about as unoptimized as it could be. On a technical note: there was no indexing, no primary keys, with 500,000 rows from many players this hung and consumed all worker threads. Once that happens very many things start failing.

During the period of tracking down this issue we communicated various messages about what we believed at the time needed to be addressed and it certainly did help to migrate the server to a well-configured Linux box and improving the DB backend, but at the end of the day, it was a TrinityCore feature itself that biting us the hardest.

It was thanks to some pserver veterans that jumped onboard to help that we were able to pinpoint the exact source. They spent the days adding deep realtime statistics and logging to async db queries which then allowed us to isolate the problem and disable the feature. Without this we had no fast way to isolate this aside from carefully combing over various core systems. We will be going out of our way to ensure that this is fixed and properly optimized in both the open source TS-WoW codebase and hopefully TrinityCore itself so nobody has to suffer from this again.

Leading up to last Friday

At this point we had made changes to the database, server infrastructure and core-level adjustments, as well as, frankly, just disabling the quest tracking system in the config. At the stage of crunching where these fixes were in fatigue was starting to set in and the infrastructure had been migrated very rapidly which introduced friction, especially owing to various pipeline and support tools not being fully updated to work smoothly on the new systems. Looking ahead we knew much of the time in the coming weeks will be spent hotfixing all the potential bugs and other issues that will crop up. While this is to be expected of a launch, the compounding effects of the increased development load as well as new infrastructure friction took its toll on the dev team.

The servers were quietly put back online to test if everything is running fine in a live environment. That would afford us a small window of time to spread player load across zones and verify the character prune works as expected. If the results were positive there would be a follow up update post that the servers are up and will remain so. After a while there were issues with a server crash and a displaced building in the dwarf starting area. Those alone can be hotfixed, but a larger issue looming over us was that the server deployment and development pipeline was now arduous to work with due to a hasty infrastructure migration and because two members on the team were dealing with pressing IRL situations that had cropped up at different points during the launch days. Magic, who had been dealing with loss of a close family member as well as immense stress had to step away for his own health and well-being. Kaytotes was juggling handling the servers and dealing with a demanding IRL situation.

Since pushing forward with the launch at current pace was evidently not sustainable and harmful to the people’s health the team made a decision to take the weekend off. This would buy us time to clear our minds, get some rest and decide what path to take going forward.

Weekend break

Over the weekend there were extensive talks and deliberations on how to manage the situation we were in as well as how to deliver Project Epoch in a way that will be stable to play on and not a frustration for the players. During that period members from the Ascension team that have previously helped Epoch with technical matters reached out and Kaytotes and Dutch from Ascension got in touch.

Talks with the Ascension team

They expressed interest in having Project Epoch as a standalone thing in their broader ecosystem, stating explicitly that the project remains independent and autonomous, retaining full control over how to run things as well as monetization. There would also be a signed contractual agreement where this was laid out in clear detail that Epoch controls the vision and features of the project, including all monetization, and Ascension provides the infrastructure and technical expertise. Their engineering team was ready to crunch to have the Epoch backend integrated into their infrastructure ASAP as well as rapidly patching our current server software so it will have the stability and massive scalability their servers are capable of.

Our team spent the next days weighing the options and contemplating the decision, having clarifying conversations about all sorts of ramifications of joining under the broader Ascension umbrella. After sleeping on it and putting much thought into the matter the team reached full agreement to proceed with this. We see it as having many benefits for us both as a project and as a team that does not infringe upon or diminish what we want our project to be. And while, of course, it was not an easy decision to make, it was not lost on us that there were many, many positives to this. Most importantly of all, it becomes sustainable for us to pursue Epoch further and we know with certainty that the many thousands of hours each of us has put into this project are gonna come to fruition.

What’s in this for Epoch

We enter into a contractual agreement that Epoch retains full integrity and autonomy to execute our vision as well as having full control over anything pertaining to monetization. Epoch enters into the broader umbrella of Ascension servers. We will not operate like, say, the Classless realms which are stylistically different in many ways and have different monetization than we do. We do not deviate from our original vision for what store content on Epoch should look like; no pay-to-win, no pay-for-convenience. Tasteful, high quality cosmetic items that perfectly blends in with and complements the overall S-tier art direction, groundedness and world feel we consider integral to the original World of Warcraft setting. The Ascension team understands that if there was a monetization strategy that jeopardized the vision of the project that would not be in their best interests.

Furthermore, this grants us access to not only massive server upgrades but also major client and modding improvements itself. Including some of the best talent in the whole modding scene. Ambitious ideas that we’ve had to shelf because of technical limitations are now fully back in play and our development team is able to work on Epoch full-time, allowing us to expand the scope and quality of the custom content we produce.

Over time as we begin to integrate this we will elaborate on what this means both for the dev team and as a player. Down the line, there will be significant performance improvements, improved game rendering, expanded addon API, among an enormous list of other improvements that we can utilize in improving the feel of the game as well as pushing our content further.

What’s in this for Ascension

Among the topics discussed with the Ascension team it was of interest to understand the bigger picture and logic of what’s in it for them. While some know Ascension from the Classless realms that have been around for years they have also been working towards becoming a diverse hub for custom WoW content. They provide the technical expertise, infrastructure and game engine aspects and the projects running on their infrastructure can dedicate all their resources to executing their ideas.

Many of the well-known Ascension servers have a very different style from Epoch. They were not going to make Classic+ the way Epoch is doing. They specialize in different gameplay. By having Epoch under the broader umbrella they have an offering for the purist Classic+ audience that is interested in high quality new content that builds on the feel and world build of the original World of Warcraft.

Introducing Epoch into their broader ecosystem is worthwhile for the exposure alone, if a fraction of the Epoch playerbase wants to check out say the Classless style realms or upcoming projects like WORS (known as “World of Runescape”) then this brings new players for other projects. The launcher is designed to make it easy to jump between different Ascension projects, even when they feature radically different content. And them doing a good job at integrating Epoch into their custom WoW hub sends a signal and builds trust which increases the appeal of future potential projects as well.

Dutch elaborated as to why they see a compromise in the values or monetization of the project as against Ascenion’s interests:

Epoch has a specific vision of more pure, unaffected classic+ experience

The appeal and value of having Project Epoch in the ascension ecosystem is actually completely lost if that pure unaffected classic+ experience is jeopardized. It is in Ascension’s best interest if it adheres to their vision.

There are plenty of crazy custom realms at Ascension. The value of having Epoch in the ecosystem is enough, and it isn’t valuable at all if it has monetization that isn’t compatible with the core vision.

Thank you

During the 2 weeks of hard crunch to get the server up and running there were several people from outside that poured a lot of time and effort into helping us get off the ground. The war room thing sounds a bit cringe in hindsight, but it really felt like one, we had numerous people on call with us for most waking hours of the day. We had the people behind TSWoW itself helping in an extraordinary capacity with linux migration, async logging, code reviews, massive restructures to TSWoW itself, and botting framework for gauging player load. Radeghost and Gaga from Whitemane and several other devs assisted with linux migration, db configuration, code reviews, and general advice and guidance. We are enormously grateful for all the help that was received during the first 2 weeks of launch crunch.

Same goes for all who sponsored the project GitHub, much of the money we earned through the sponsors will go back into paying for the launch hardware we bought. When the server is back up we would like to set up a plaque in capital cities memorializing and giving our thanks listing the GitHub names of all the sponsors that supported us.

We must also extend a massive thanks to Magic who has been with us since the start of Epoch, and even before that. He parted ways and did a clean break from the pserver scene as a whole. As a team we are very understanding of how difficult his situation has been for him and are full of gratitude for how much care and effort he has put into Epoch over the years. We wish him the best. And when the server is up, should you ever choose to do the Lost Rams quest in Dun Morogh pour one out for Magic, a great friend of the project who cared for improving the modding scene as a whole. That was one of the earlier quests he did and it was fun for those of us on the team back then to see his satisfaction with getting working. 🐐

And finally, our gratitude to all who has been with us through this. From the old timers back when Epoch was a small comfy community and to everyone who has recently joined and stayed with us through the ups and downs in getting this server out. We hope to make up for all of this when the realms are back up and you get to experience what we’ve been working on.

Onwards!

Now we are done with the talking, there have been too many words and the best way to show commitment and integrity will be to demonstrate it. The coming updates will be to clarify and give exact information about everything you need to know to get set up and ready for the relaunch. In addition there is a direct line of communication from the Ascension team themselves continuously providing updates and addressing frequently asked questions from their side.

A team of engineers are working hard to have the server up and running in an excellent state of performance. In the coming days we will be running stress tests on the systems to push the infrastructure so this project is ready for a proper launch.

Our working relationship with Ascension has been excellent so far with fast communication. While their team is getting the infrastructure ready we are adapting to the new structure and getting ready to handle and respond to the content side of things when the servers are up.

As for the present state of things, the team morale is high and we are looking forward to get cracking again at the roadmap we have laid out before us. While there have been some “we are so back” -> “it is so over” cycles over the last few weeks we’re happy to announce that we are so back. We will not be compromising on what we want Epoch to be, we have a vision and we’re gonna go extremely hard on it.

See you on relaunch!