Archive of Our Own

Since the end of December 2024, AO3 has had numerous periods of slowness, downtime, and related issues such as missing kudos emails and delayed invitations. We've been taking some steps to improve the situation, but we are also working on some highly time-sensitive updates to our infrastructure, so we can't spend as much time as we'd like on performance improvements. We expect some slowness and downtime to continue until our new servers are delivered and installed in a few months.

We first noticed some strain on the servers we use for Elasticsearch (which powers searching and filtering) in the middle of last year. The new servers we wanted weren't available yet, so we repurposed some of our other servers to help with the load on Elasticsearch until we could get the hardware.

Unfortunately, the hardware wasn't available on its October release date, and our temporary fix couldn't hold up to the traffic increase we experience at the end of every year. This has led to periods of noticeable slowness over the last several weeks.

The servers we wanted finally became available in early January, and we completed the process of getting quotes and requisitioning them by January 15. Our purchase was confirmed on January 28, but it will take a few months for the servers to be delivered and installed.

We estimate the new Elasticsearch servers will be in place by early April. Until then, you might run into the following issues, especially during busy periods:

• all pages loading more slowly
• Elasticsearch-powered pages like search results and work and bookmark listings taking longer to update
• error pages
• automated checks from Cloudflare's Under Attack mode
• stricter rate limiting
• issues with services like the Wayback Machine or Tumblr RSS accounts that rely on bots, scrapers, or other automated tools, which we have deprioritized in favor of traffic from users

In addition to new Elasticsearch servers, we'll be purchasing five database servers to improve the capacity and resilience of our database cluster. We don't currently have enough database power to handle increased traffic and do certain types of maintenance at the same time. This means we sometimes have to take AO3 offline to resolve database issues, as we did for our February 7 maintenance. Additional hardware should help us avoid this situation in the future, but it will take some time for the purchase to be completed and the servers to be installed. We do not anticipate any database issues while we wait and there is no risk of data loss.

We're very sorry for the disruptions, and we appreciate your patience and your generous donations, which fund purchases like these.

For updates on slowness, downtime, or other issues, please follow @AO3_Status on Twitter/X or ao3org on Tumblr. We're also in the process of setting up a status account on Bluesky and a status page, but they're still works in progress and might not receive all updates just yet, so please make sure to check Twitter/X or Tumblr for a fully accurate list of updates.

The Archive has seen a marked uptick in traffic during March, with weekly page views increasing from 262 million to 298 million in just two weeks. We expect this trend to continue, and in order to keep the site running, we need to take emergency measures. The quickest, most helpful change we can make is caching the works we serve to logged out users. Unfortunately, this means that starting immediately, logged out users may experience a delay in work updates, and hits from logged out users will no longer be included in works' hit counts.

Why is this happening?

The increased traffic is putting a strain on our database servers, which receive dozens of requests every time someone loads a work. (We plan to order new hardware to help with this strain, but first we need to finish some ongoing server maintenance and determine which hardware to order. Delivery and installation of servers ordinarily takes a few months, and there may be unexpected delays due to the pandemic.)

Serving cached copies of works to logged out users will drastically reduce the number of database requests we make. Caching means we don't have to ask the database for the latest information every time someone visits a certain page in a given period of time. Instead, one of our front end servers gives everyone the exact same copy of that page. After about an hour, that copy is updated.

What changes will I notice?

Starting immediately, you may notice the following changes:

1. When a new chapter is posted, logged out users will only be able to access it by direct link until the cache updates, which will happen about once every 60 minutes. Other changes to the work (e.g., edits made by the creator or new comments or kudos that have been left) may also not be visible to logged out users until the cache is updated.
2. Because work pages need to be identical for all logged out users, we've had to stop automatically filling in guests' names and emails on the comment form. (You can request an invitation and create an account if you'd like the form to remember you!)
3. Logged out users will see the adult content warning on every work rated either Mature, Explicit, or Not Rated. This is temporary and will be fixed as soon as possible.
4. New hits from logged out users will not be added to works' hit counts. (Existing hits will not be lost.) The code that increases hit counts lives on our application servers, so it will not run when the front end servers hand out cached copies of works.

(This section was updated at 00:25 UTC April 1.)

Will hit counts be fixed?

We are exploring options that will allow us to resume counting hits from logged out users, but it may take some time to find and implement a viable long term solution. We'll work as quickly as we can, but we ask for your patience -- our volunteers may need to prioritize additional performance improvements or their own wellbeing in these stressful times.

We will keep you updated on any significant progress or setbacks here on AO3 News and on our @AO3_Status Twitter feed.

Updated 11:00 UTC April 24: We have deployed new code that allowed us to resume counting hits from logged out users, along with some general changes to how hits are being measured.

Over the last few weeks, you may have noticed a few brief periods when the Archive has been slow to load (or refusing to load at all). This is because our Elasticsearch servers are under some strain. We've made some adjustments to our server setup, but because this is our busiest time of year, we expect the problems to continue until we're able to have new servers delivered and installed in a few months.

We've been planning this server purchase for a while now, but the machines we wanted -- AMD's Epyc Rome CPUs, which have an increased core count and are cheaper than the Intel equivalent -- didn't come on the market until August. Now that they've been released, we're working on finding the best price to help us make the most of your generous donations. We expect to order them very soon.

While we're waiting for our new servers, we plan to upgrade the Elasticsearch software to see if the newer version offers any performance improvements. We hope this upgrade and the changes to our server setup will keep things from getting much worse during our end-of-year traffic influx.

Thank you for your patience, and for all the donations that allow us to buy new hardware when these situations arise!

Update 27 November: The servers have been ordered, but it will still be a few months before they are delivered and installed.

Update, April 4: We successfully deployed an improved version of the code referenced in this post on March 29. It now takes considerably less time to add a work to the database.

-

You may have noticed the Archive has been slow or giving 502 errors when posting or editing works, particularly on weekends and during other popular posting times. Our development and Systems teams have been working to address this issue, but our March 17 attempt failed, leading to several hours of downtime and site-wide slowness.

Overview

Whenever a user posts or edits a work, the Archive updates how many times each tag on the work has been used across the site. During this time, the record is locked and the database cannot process other changes to those tags. This can result in slowness or even 502 errors when multiple people are trying to post using the same tag. Because all works are required to use rating and warning tags, works' tags frequently overlap during busy posting times.

Unfortunately, the only workaround currently available is to avoid posting, editing, or adding chapters to works at peak times, particularly Saturdays and Sundays (UTC). We strongly recommend saving your work elsewhere so changes won’t be lost if you receive a 502.

For several weeks, we’ve had temporary measures in place to decrease the number of 502 errors. However, posting is still slow and errors are still occurring, so we’ve been looking for more ways to use hardware and software to speed up the posting process.

Our Friday, March 17, downtime was scheduled so we could deploy a code change we hoped would help. The change would have allowed us to cache tag counts for large tags (e.g. ratings, common genres, and popular fandoms), updating them only periodically rather than every time a work was posted or edited. (We chose to cache only large tags because the difference between 1,456 and 1,464 is less significant than the difference between one and nine.) However, the change led to roughly nine hours of instability and slowness and had to be rolled back.

Fixing this is our top priority, and we are continuing to look for solutions. Meanwhile, we’re updating our version of the Rails framework, which is responsible for the slow counting process. While we don’t believe this upgrade will be a solution by itself, we are optimistic it will give us a slight performance boost.

March 17 incident report

The code deployed on March 17 allowed us to set a caching period for a tag’s use count based on the size of the tag. While the caching period and tag sizes were adjusted throughout the day, the code used the following settings when it was deployed:

• Small tags with less than 1,000 uses would not be cached.
• Medium tags with 1,000-39,999 uses would be cached for 3-40 minutes, depending on the tag’s size.
• Large tags with at least 40,000 uses would be cached for 40-60 minutes, but the cache would be refreshed every 30 minutes. Unlike small and medium tags, the counts for large tags would not update when a work was posted -- they would only update during browsing. Refreshing the cache every 30 minutes would prevent pages from loading slowly.

We chose to deploy at a time of light system load so we would be able to fine tune these settings before the heaviest weekend load. The deploy process itself went smoothly, beginning at 12:00 UTC and ending at 12:14 -- well within the 30 minutes we allotted for downtime.

By 12:40, we were under heavy load and had to restart one of our databases. We also updated the settings for the new code so tags with 250 or more uses would fall into the “medium” range and be cached. We increased the minimum caching period for medium tags from three minutes to 10.

At 12:50, we could see we had too many writes going to the database. To stabilize the site, we made it so only two out of seven servers were writing cache counts to the database.

However, at 13:15, the number of writes overwhelmed MySQL. It was constantly writing, making the service unavailable and eventually crashing. We put the Archive into maintenance mode and began a full MySQL cluster restart. Because the writes had exceeded the databases' capabilities, the databases had become out of sync with each other. Resynchronizing the first two servers by the built-in method took about 65 minutes, starting at 13:25 and completing at 14:30. Using a different method to bring the third recalcitrant server into line allowed us to return the system to use sooner.

By 14:57, we had a working set of two out of three MySQL servers in a cluster and were able to bring the Archive back online. Before bringing the site back, we also updated the code for the tag autocomplete, replacing a call that could write to the database with a simple read instead.

At 17:48, we were able to bring the last MySQL server back and rebalance the load across all three servers. However, the database dealing with writes was sitting at 91% load rather than the more normal 4-6%.

At 18:07, we made it so only one app server wrote tags’ cache values to the database. This dropped the load on the write database to about 50%.

At 19:40, we began implementing a hotfix that significantly reduced writes to the database server, but having all seven systems writing to the database once more put the load up to about 89%.

At 20:30, approximately half an hour after the hotfix was finished, we removed the writes from three of the seven machines. While this reduced the load, the reduction was not significant enough to resolve the issues the Archive was experiencing. Nevertheless, we let the system run for 30 minutes so we could monitor its performance.

Finally, at 21:07, we decided to take the Archive offline and revert the release. The Archive was back up and running the old code by 21:25.

We believe the issues with this caching change were caused by underestimating the number of small tags on the Archive and overestimating the accuracy of their existing counts. With the new code in place, the Archive began correcting the inaccurate counts for small tags, leading to many more writes than we anticipated. If we're able to get these writes under control, we believe this code might still be a viable solution. Unfortunately, this is made difficult by the fact we can’t simulate production-level load on our testing environment.

Going forward

We are currently considering five possible ways to improve posting speed going forward, although other options might present themselves as we continue to study the situation.

1. Continue with the caching approach from our March 17 deploy. Although we chose to revert the code due to the downtime it had already caused, we believe we were close to resolving the issue with database writes. We discovered that the writes overwhelming our database were largely secondary writes caused by our tag sweeper. These secondary writes could likely be reduced by putting checks in the sweeper to prevent unnecessary updates to tag counts.
2. Use the rollout gem to alternate between the current code and the code from our March 17 deploy. This would allow us to deploy and troubleshoot the new caching code with minimal interruption to normal Archive function. We would be able to study the load caused by the new code while being able to switch back to the old code before problems arose. However, it would also make the new code much more complex. This means the code would not only be more error-prone, but would also take a while to write, and users would have to put up with the 502 errors longer.
3. Monkey patch the Rails code that updates tag counts. We could modify the default Rails code so it would still update the count for small tags, but not even try to update the count on large tags. We could then add a task that would periodically update the count on larger tags.
4. Break work posting into smaller transactions. The current slowness comes from large transactions that are live for too long. Breaking the posting process into smaller parts would resolve that, but we would then run the risk of creating inconsistencies in the database. In other words, if something went wrong while a user was updating their work, only some of their changes might be saved.
5. Completely redesign work posting. We currently have about 19,000 drafts and 95,000 works created in a month, and moving drafts to a separate table would allow us to only update the tag counts when a work was finally posted. We could then make posting from a draft the only option. Pressing the "Post" button on a draft would set a flag on the entry in the draft table and add a Resque job to post the work, allowing us to serialize updates to tag counts. Because the user would only be making a minor change in the database, the web page would return instantly. However, there would be a wait before the work was actually posted.
The unexpected downtime that occurred around noon UTC on Tuesday, March 21, was caused by an unusually high number of requests to Elasticsearch and is unrelated to the issues discussed in this post. A temporary fix is in currently in place and we are looking for long term solutions.

AO3 and Fanlore users take note: both sites will be down for scheduled maintenance this Friday, 20th March. The maintenance will take place from 17:30 to approximately 20:45 UTC. What time is that for me?

Note that maintenance may also take slightly longer than expected, so there is no need to contact us if you cannot access the site at exactly 20:45 UTC.

For site status updates about AO3 and Fanlore please follow @AO3_Status and @fanlore_news on Twitter.

If you're a regular Archive visitor or if you follow our AO3_Status Twitter account, you may have noticed that we've experienced a number of short downtime incidents over the last few weeks. Here's a brief explanation of what's happening and what we're doing to fix things.

The issue

Every now and then, the volume of traffic we get and the amount of data we're hosting starts to hit the ceiling of what our existing infrastructure can support. We try to plan ahead and start making improvements in advance, but sometimes things simply catch up to us a little too quickly, which is what's happening now.

The good news is that we do have fixes in the works: we've ordered some new servers, and we hope to have them up and running soon. We're making plans to upgrade our database system to a cluster setup that will handle failures better and support more traffic; however, this will take a little longer. And we're working on a number of significant code fixes to improve bottlenecks and reduce server load - we hope to have the first of those out within the next two weeks.

One area that's affected are the number of hits, kudos, comments, and bookmarks on works, so you may see delays in those updating, which will also result in slightly inaccurate search and sort results. Issues with the "Date Updated" sorting on bookmark pages will persist until a larger code rewrite has been deployed.

Behind the scenes

We apologize to everyone who's been affected by these sudden outages, and we'll do our best to minimize the disruption as we work on making things better! We do have an all-volunteer staff, so while we try to respond to server problems quickly, sometimes they happen when we're all either at work or asleep, so we can't always fix things as soon as we'd like to.

While we appreciate how patient and supportive most Archive users are, please keep in mind that tweets and support requests go to real people who may find threats of violence or repeated expletives aimed at them upsetting. Definitely let us know about problems, but try to keep it to language you wouldn't mind seeing in your own inbox, and please understand if we can't predict immediately how long a sudden downtime might take.

The future

Ultimately, we need to keep growing and making things work better because more and more people are using AO3 each year, and that's something to be excited about. December and January tend to bring a lot of activity to the site - holiday gift exchanges are posted or revealed, people are on vacation, and a number of fandoms have new source material.

We're looking forward to seeing all the new fanworks that people create this year, and we'll do our best to keep up with you! And if you're able to donate or volunteer your time, that's a huge help, and we're always thrilled to hear from you.

Update December 14, 18:00 UTC: As of this week, all systems should be back to normal. We're still working on optimizing our server settings, so very brief downtimes for maintenance should be expected. If bookmarks still won't sort correctly for you - we're working on a more permanent fix to the underlying issue, but it might be a short while yet. As always, we're keeping an eye on Support tickets and messages to our Twitter account, and will react as quickly as possible if anything seems off. Thank you all for your patience.

Update December 3, 16:00 UTC: We have re-enabled the sort and filter sidebar on work listings only. Bookmark filtering and sorting is still turned off and will likely be off for a few more days. (The filters are the sidebar that allows you to narrow down a list of works or bookmarks by character, rating, etc.) We will continue to work on the underlying issue. In the meantime, we suggest using the Works Search to help find what you’re looking for.

All works and bookmarks should be showing up normally. Work re-indexing is complete, so we hope to be able to turn on filtering for works again in the next day or two.

Bookmark re-indexing is still ongoing, so it will be several days before we can turn bookmark filtering back on.

Please follow the @AO3_Status Twitter feed or check back here for further updates.

Update 2 Dec: Listings for works, bookmarks, tags, and pseuds are unavailable due to issues with our search index. Our coding and systems volunteers are currently looking into it, and we will keep you updated on our progress. Our Support team is working on a back log, so there might be delays in getting back to users individually. Please consider checking the @AO3_Status Twitter feed or our banner alerts instead.

Update 30 Nov: All bookmarks have been re-indexed and should show up correctly again. Any issues that might still be lingering will be sorted out when we upgrade Elasticsearch, which we're planning for mid-December. Downloads should be working without the need for any workarounds now. Thank you for your patience!

The Good

We recently deployed new code, which fixed a couple of very old bugs and introduced improvements to the kudos feature. Behind the scenes, we've been working on setting up new servers and tweaking server settings to make everything run a little more smoothly during peak times. The end of the year (holiday season in many parts of the world) usually means more people with more free time to participate in more challenges, read more fic, or post more fanart, resulting in more site usage.

One way to measure site usage is looking at page views. This number tells us how many pages (a single work, a list of search results, a set of bookmarks in a collection, a user profile, etc. etc.) were served to users during a certain time frame. Some of these pages can contain a lot of information that has to be retrieved from the database - and a lot of information being retrieved from the database at the same time can result in site slowness and server woes. During the first week of January we had 27.6 million page views. As of November 17 we registered 42.9 million page views for the preceeding week.

We've watched our traffic stats grow dramatically over the years, and we've been doing our best to keep up with our users! Buying and installing more servers is one part of the solution, and we can't thank our all-volunteer Systems team enough for all their hard work behind the scenes. On the other hand, our code needs to be constantly reviewed and updated to match new demands.

Writing code that "scales" - that works well even as the site grows - is a complicated and neverending task that requires a thorough understanding of how all parts of the Archive work together, not just right now, but in six months, or a year, or two years. As we're all volunteers who work on the Archive in our free time (or during lunch breaks), and there are only a handful of us with the experience to really dig deep into the code, this is less straightforward than a server acquisition and will take a little more time.

The Bad

As such, we've been battling some site slowness, sudden downtimes (thankfully brief due to our awesome Systems team) and an uptick in error pages. We can only ask for your patience as we investigate likely causes and discuss possible fixes.

For the time being, we have asked our intrepid tag wranglers to refrain from wrangling on Sundays, as this is our busiest day and moving a lot of tags around sadly adds to the strain on the current servers. We sincerely apologize to all wrangling volunteers who have to catch up with new tags on Monday, and to users who might notice delays (e.g. a new fandom tag that's not marked as canonical right away). From what we've seen so far, this move has helped in keeping the site stable on weekends.

The Ugly

We are aware of an issue with seemingly "vanishing" bookmarks, in which the correct number of bookmarks is displayed in the sidebar, but not all are actually shown. The most likely culprit is our search index, powered by a framework called elasticsearch. All our information (work content, tags, bookmarks, users, kudos, etc. etc.) is stored in a database, and elasticsearch provides a quicker, neater access to some of this data. This allows for fast searches, and lets us build lists of works and bookmarks (e.g. by tag) without having to ask the database to give us every single scrap of info each time.

It appears now that elasticsearch has become slightly out of sync with the database. We are looking into possible fixes and are planning an elasticsearch software upgrade; however, we must carefully test it first to assure data safety.

This problem also affects bookmark sorting, which has been broken for several weeks now. We are very sorry! If you want to know if a particular work has been updated, please consider subscribing to the work (look for the "Subscribe" button at the top of the page). This will send you a notification when a new chapter has been posted.

(Note: Since we're sending out a lot of notifications about kudos, comments and subscriptions every day, some email providers are shoving our messages into the junk folder, or outright deny them passage to your account. Please add our email address [email protected] to your contacts, create a filter to never send our emails to spam, or check the new "Social" tab in Gmail if you're waiting for notifications.)

A problem with file downloads only cropped up fairly recently. We don't think this is related to the most recent deploy, and will investigate possible causes. In the meantime, if a .pdf or .mobi file gives you an error 500, try downloading the HTML version first, then give it another shot. This should help until we've fixed the underlying problem.

What You Can Do

If you have not already done so, consider subscribing to our twitter feed @AO3_Status or following us on Tumblr. You can also visit the AO3 News page for updates in the coming weeks or subscribe to the feed.

We thank everyone who has written in about their experiences, and will keep you all updated on our progress. Thank you for your patience as we work on this!

It's the season of giving! So, we're pleased to announce that invitation requests are back on the AO3!

Once upon a time (i.e. six months ago), users with Archive accounts could request a few invitations to give out, allowing them to share the Archive with friends and help form communities of like-minded fans.

Unfortunately, earlier this year, as many of you may remember, the Archive was having serious performance issues (we saw the sad 502 page far too often). While our coders and systems team hurried to implement emergency fixes, it was decided that we needed stricter control of the number of accounts being created to reduce the likelihood of unexpected overload. (Generally, people browsing the site without being logged in put a certain amount of stress on the servers, but it's the account perks like bookmarking, subscribing, and accessing a full reading history that contribute to server load to a larger degree.) Back then only 100 invitations were issued to people in the queue each day, so additional user requests could have a serious impact! So, in June, the difficult decision was made to stop giving invitations to existing users. You can read more about what was going on then in our post, Update on AO3 performance issues.

Over the next five months our software upgrades and code improvements caught up with the demand. The queue rate was increased several times, most recently to 750 invitations per day. Given that, we've wanted to go back to giving out invitations to existing users, but there were a few issues to be resolved before we could start.

First, the request form had to be altered to set a maximum number of invitations that a user can request at once. Second, the 1,200 user requests that were in the list when it was shut down had to be addressed. Since we had no limits on how many invitations could be requested back then, we had quite a few requests for very large numbers. Due to limitations in the software, individually lowering those numbers now would require manually editing each request, as would granting only some of the requests at once rather than the whole list.

So, two decisions were made:

1) Everyone with a pending request will receive 1 invitation, just to clear out the backlog.

2) User requests are being re-opened! You can now request a maximum number of 10 invitations at one time. Even with this hard limit in place, we ask that everyone ask for only what they need at a time. Once we've hit the figurative switch and re-enabled this feature later today you will be able to request invitations from your Invite a friend page.

We very much appreciate all of our users, and we are proud of our growth this year, even through the bumpy times. We are glad that once again we can enable you to bring more people on board!

We've been talking a lot recently about how much the AO3 has expanded over the last few months. One easy statistic for us to lay our hands on is the number of registered accounts, but this only represents a tiny portion of site activity. Our awesome sys-admin James_ has been doing some number crunching with our server logs to establish just how much we've grown, and provided us with the following stats (numbers for June not yet available). Thanks to hele for making them into pretty graphs!

Visitors to the AO3

The number of unique visitors to the site has increased almost every month since December 2010 (each unique IP address is counted as one visitor). There are a few points where the rate of increase gets more dramatic: there was a jump of 244,587 across December 2011 and January 2012, compared to one of 137,917 over the two months before that. This can probably be accounted for by the fact that during December and January, holiday challenges such as Yuletide bring more people to the site. This theory is borne out by the fact there was a slight dip in the number of visitors during February 2012, indicating that some of the extra traffic in the previous two months were 'drive by' visitors who didn't stick around.

May 2012 saw a steep increase in the number of visitors: there were 211,628 more visitors to the site than there had been the month before! The rapid increase in visitors was not without its price: this was the month of many 502 errors!

Traffic to the AO3

The increase in the number of visitors to the site has also been accompanied by an increase in overall site traffic (how much data we're serving up). Again, there's a significant spike during December/January. Interestingly, there's no dip in traffic for February 2012, showing that even though there were some 'one time' visitors over the holiday period, there were also plenty of people who stayed and continued to enjoy fanworks on the site.

The increase in traffic to the site clearly accelerated in 2012. Between January and May 2011 traffic increased by just 159.92 GB; the same period in 2012 saw an increase of 1,870.26 GB! In fact, with an increase of 566 GB during May 2012, that month alone saw almost as big a jump in traffic as the whole of the previous year (595.63GB)!

And the other stuff

With these kinds of numbers, it's not surprising that there've been a few bumps along the way. For information on how we're dealing with the growth in the site you can check out our posts on performance and growth and accounts and invitations.

Many thanks to our dedicated volunteers for their hard work dealing with the growth of the site, and to our fabulous users for their patience with our growing pains - and for creating the awesome fanworks so many people are flocking here to see!

The demand for AO3 accounts has recently exploded! Unfortunately, the rapid increase in users has also created some site performance issues (you can read more about these and what we're doing about them in our post on Performance and Growth).

We use an invitation system in order to help manage the expansion of the site, and to help guard against spam accounts. Until recently, demand for invitations was low enough that the system didn't result in people waiting for a long time. However, because so many people signed up at the same time, the queue is now really long, and the waiting times are months rather than days or weeks. We know this really sucks for the people waiting for accounts (especially those who are concerned that their work may be deleted from other sites).

We really really want to cut down on waiting times, but we also need to ensure the site remains stable. It's a bit difficult to tell exactly how much difference more registered users will make, because we know that many of the people waiting for an account are already using the site as logged-out users, so to some extent we're already dealing with their extra load, However, every time someone new creates an account, it is a little bit more load on the servers, both because account holders have access to more features (personalisation, subscriptions, history, etc) and because if they're posting works, they also tend to attract more new people to the site who want to access their fanworks! Logged-in users also have more personalised pages, which makes it harder for us to serve them pages from the cache (which puts less load on the site.) At the moment of writing, there are 56,203 registered users on the site and 28,863 requests in the queue: this means we're looking at adding more than half as many users again. That's a pretty massive potential increase, so much as we'd love to, we can't issue invitations to everyone who wants one right away.

What's the plan for issuing more invitations?

Once we got our big performance issues under control, we cautiously increased the number of new invitations being issued every day from 100 to 150. We didn't make an announcement about this right away in case we needed to decrease it again (although lots of people noticed their wait time had decreased!). However, we've been keeping on eye on the site and it seems to be coping happily with the increase.

We need to install some more RAM (see our post on performance) which should be happening very shortly. Once we've done that we'll increase the numbers being issued to the queue again - by another 50 per day initially, and possibly more after that if we don't see any warning signs that it's causing problems.

Even if we up the number of invitations to 300 per day, it's still going to mean that some people will have to wait up to three months. Unfortunately, because we're looking at so much demand, there's a limit to how far we can tackle this. :(

What about invitations for friends?

We used to allow users to request invitations to give out to their friends (just to clear up one source of confusion - user accounts never came with a 'friend invitation' by default, although we did give out some unsolicited ones on our first birthday). However, as the demand increased, we were getting more and more requests of this nature, and it became difficult to keep track of it in a fair way. We decided that in order to make sure we knew exactly how many invitations were being issued, and in order to make it as fair as possible, we'd restrict invitations to the queue only for now. This means that it's 'first come, first served', whether or not you know anyone on the AO3. We know people would really like to build their communities on the site, and we will reenable the option in the future, but only when we're sure the performance situation will allow it.

Can I pay to get an account quicker?

No, not at this time. The AO3 is funded by donations to our parent, the Organization for Transformative Works, but donating doesn't give you an account on the Archive.

When we started the AO3, we decided not to have paid accounts for a few reasons. First of all, we wanted to make it something that you could use whether or not you had any financial resources: we know many fans can't afford paid services or don't have a way of paying for them. Secondly, we wanted to add a layer of protection for fans' real life identities: if you pay for your fan name account with a real name credit card, there is an increased chance that the two identities could be linked (either by accident or via a legal demand for our records; we are committed to fighting for your privacy but can't guarantee that we'd win such a battle). Finally, adding paid features to the Archive itself would have increased the complexity of what our coders had to code, especially if we had some features available only to paid accounts. For all these reasons, we decided to fund the AO3 indirectly via donations to the OTW, which couldn't be linked to your account on the Archive, allowed us to provide the site to everyone whether or not they could pay, and made it easier for us to use existing payment systems which we didn't have to code ourselves.

It's possible that in the future we may consider some form of paid service, if we developed other ways of dealing with the above concerns. However, it's not something we're considering right now - if it does become a more real possibility in the future, we'll post and ask for discussions about it.

I'm worried my works on another site are at risk! What can I do?

We've recently had a lot of direct requests for invitations from users who are worried about their works being deleted from Fanfiction.net. This creates a dilemma for us, because protecting at-risk fanworks is a fundamental part of what we do. However, right now the volume of those requests is so high that there's simply no fair way to prioritise them, which is why we're only issuing invitations via the queue. We're very sorry about this. If you're worried your work is at risk, we recommend you back it up to your own computer (Fanfiction.net has a 'download as HTML' option, or you may wish to use a a tool such as Fanfiction Downloader or Flagfic), so that you can upload it to the AO3 or any other site at a later date. You may also wish to manually back up your reviews and comments - sadly these can't be transferred to the AO3 even if you have an account, but you may want to keep a record of this important part of your fan life for yourself.

We're pleased that so many people want to be here, but very sorry we can't accommodate everyone right away. Thank you for your patience and support.

Everyone at the Archive of Our Own has been working hard dealing with the recent site expansion and performance problems. Now that we've been able to deal with the immediate issues, we wanted to give everyone a bit more detail on what's happening and what we're working on.

The basics

Our recent performance problems hit when a big increase in users happened, putting pressure on all the bits of the site not optimised for lots of users. We were able to make some emergency fixes which targeted the most problematic points and thus fixed the performance problems for now. However, we know we need to do quite a bit more work to make sure the site is scalable. The good news is there are lots of things we know we can work on, and we have resources to help us do it.

Some users have been concerned that the recent performance problems mean that the site is in serious trouble. However, we've got lots of plans in place to tackle the growth of the site, and we're also currently comfortable about our financial prospects (we'll be posting about this separately). As long as we are careful and don't rush to increase the number of users too fast, the site should remain stable.

The tl;dr details

What level of growth are we experiencing?

The easiest aspect of site growth for us to measure is the number of user accounts. This has definitely grown significantly: since May 1 almost 12,000 new user accounts have been created, which means a 25% increase in user numbers in the past two months. However, the number of new accounts created is only a small proportion of the overall increase in traffic.

We know that lots more people are using the site without an account. There are currently almost 30,000 people waiting for an invitation, but even that is a very, very partial picture of how many people are actually visiting the site. In fact, we now have approximately one and a half million unique visitors per month. That's a lot of users (even if we assume that some of those visitors represent the same users accessing the site from different locations)!

A bit about scalability

The recent problems we've been experiencing were related to the increase in the number of people accessing the site. This is a problem of scalability: the requirements of a site serving a small number of users can be quite different to those of a site with a large userbase. When more users are accessing a site, any weak points in the code will also become more of a problem: something which is just a little bit slow when you have 20,000 users may grind to a halt entirely by the time you hit 60,000.

The slightly counterintuitive thing about scalability is that the difference between a happy site and an overwhelmed one can be one user. Problems tend to arise when the site hits a particular break point - for example, a database table getting one more record than it can handle - and so performance problems can appear suddenly and dramatically.

When coding and designing a site, you try to ensure it is scalable: that is, you set up the hardware so that it's easy to add more capacity, you design the code so it will work for more users than you have right now, etc. However, this is always a balancing act: you want to ensure the site can grow, but you also need to ensure there's not too much redundancy and you're not paying for more things than you need. Some solutions simply don't make any sense when you have a smaller number of users, even if you think you'll need them one day in the future. In addition, there are lots of factors which can result in code which isn't very scalable: sometimes it makes sense to implement code which works now and revise it when you see how people are using the site, sometimes things progress in unexpected ways (and testing for scalability can be tricky), sometimes you simply don't know enough to detect problem areas in the code. All of these factors have been at work for the AO3 at one time or another (as for most other sites).

Emergency fixes for scalability

When lots and lots of new users arrived at the Archive at once, all the bits of the site which were not very scalable began to creak. This happened more suddenly than we were anticipating, largely because changes at the biggest multifandom archive, Fanfiction.net, meant that lots of users from there were coming over to us en masse. So, we had to make some emergency fixes to make the site more able to cope with lots more users.

In our case, we already knew we had one bit of code that was extremely UNscalable - the tag filters used to browse lists of works. These were fine and dandy when we had a very small number of works on the Archive, but they had a big flaw - they were built on demand from the list of works returned when a user accessed a particular page. This made them up-to-the-minute and detailed, but was a big problem once the list of works returned for a given fandom were numbering in the thousands - a problem we were working around while we designed a new system by limiting the number of returned works to 1000. It was also a problem because building the filters on demand meant that our servers had to redo the work every time someone hit a page with filters on it. When thousands of people were hitting the site every minute, that put the servers under a lot of strain. Fortunately, the filters happen to be a bit of code that's relatively easy to disable without hitting anything else, so we were able to remove them as an emergency measure to deal with the performance problems. Because they were such a big part of the problem, doing this had a dramatic effect on the many 502s and slowdowns.

We also did some other work to help the site cope with more users: largely this involved implementing a lot more caching and tuning our servers so they manage their workload slightly differently. All these changes were enough to deal with the short-term issues, but we need to do some more, and more sustained work to ensure that the site can grow and meet the demands of its users.

Scalability work we're doing right now

We've got a bunch of plans for things which will help scalability and thus ensure good site performance. In the short term (approximate timescales included below) we are:

• Installing more RAM - within the next week. This will allow us to run more server processes at once so we can serve more users at the same time. This is a priority right now because our servers are running out of memory: they're regularly going over 95% of usage, which is not ideal! We have purchased new RAM and it will be installed as soon as we can book a maintenance slot with our server hosts.
• Changing our version of MySQL to Percona - within the next week. This will give us more information about what our server is doing, helping us identify problem spots in the site which we need to work on. It should also work a bit faster. We've currently installed Percona on our Test Archive and have been checking to see it doesn't cause any unexpected problems - we'll be putting it on the main site in the next week or so. Percona is an open source version of MySQL which has additional facilities which will help us look at our problems. In addition we hope to draw on the support of the company who produce it (also called Percona).
• Completing the work on our new tag filters - within the next month. These will (we hope!) be much, much more scalable than the old ones. They'll use a system called Elasticsearch, which is built on Solr/Lucene. These are solutions which don't use the MySQL database, so they cut down on a lot of database calls.

Scalability stuff we're doing going forward

We want to continue working on scalability going forward. We've reached a point where the site is only going to get bigger, so we need to be ready to accommodate that. This involves some complex work, so there are a bunch of conversations ongoing. However, this will involve some of the following:

• Analysis of our systems and code to identify problem spots. We've installed a system called New Relic which can be used to analyse what's going on in the site, how scalable it is, and where problems are occurring. Percona also provides more tools to help us analyse the site. In addition, Mark from Dreamwidth has kindly offered to work with us to take a look at our Systems setup - Mark runs the Systems side of things at Dreamwidth and has lots of experience in scalability issues, so having his fresh eyes on the performance site will help us figure out the work we need to do.
• Caching, caching and more caching. We've been working on implementing more caching for some time, and we added a lot more caching as part of our emergency fixes. However, there is still a LOT more caching we can do. Caching essentially saves a copy of a page and delivers it up to the next person who wants to see the page, instead of creating it fresh each time. Obviously, this is really helpful if you have a lot of page views: we now have over 16 million page views per week, so caching is essential. We'll be looking to implement three types:
  • Whole page caching. This is the type we implemented as an emergency fix during the recent performance issues. It uses something called Squid, and it's the best performance saver because it can just grab the whole page with no extra processing. Unfortunately, this can also cause some problems, since we have a lot of personalised pages on the site - for example, when we first implemented it, some people were getting cached pages with skins on they hadn't chosen to use. There are ways around this, however, which allow you to serve a cached page and then personalise it, so we'll be working on implementing those.
  • Partial page caching. This is something we already do a lot of - if there are bits that repeat a lot, you can cache them so that everything isn't generated fresh each time. For example, the 'work blurbs' (the information about individual works in a list of search results) are all cached. This uses a system called memcached. We'll be looking to do more, and better, partial caching.
  • Database caching. This would mean we use a secondary server to do complex queries and then put the results on the primary server, so all the primary server is doing is grabbing them.
• Adding more servers. We’re definitely going to need more database servers to manage site growth, and we’re currently finalising some decisions on that. At the moment, it looks like the way we’re going to go is to add a new machine which would be dedicated to read requests (which is most of our traffic – people looking at works rather than posting them) while one of our older machines will be dedicated to write requests (posting, commenting, etc). Once we've confirmed the finer details (hopefully this week), we expect it to take about two months for the new server to be purchased and installed.

Resources: finances

We'll be posting separately about the financial setup for the AO3, but the key thing to say is that we're currently in a healthy financial state. :D However, as the site gets bigger its financial needs will also get bigger, and we always welcome donations - if you want to donate and you can afford to do so, then donating to the OTW will help us stay on good financial footing. We really appreciate the immense generosity of the fannish community for the support already you've shown us. <3

Resources: people

A lot of supporting the site and dealing with scalability is down to the people. As we grow, we need to ensure we have the people and expertise to keep things running. We are a volunteer-run site and as such our staff have varying levels of time, expertise, and so on. One important part of expanding slowly is ensuring that we don't get into crisis situations which not only suck for our users (like when the 502s were making the site inaccessible) but also cause massive stress for the people working to fix the problems. So, we're proceeding cautiously to try to avoid those situations.

We've been working hard over the last year or so to make it easier for people to get involved with coding and working on the site. We're happy to say this is definitely paying off: we've had eight new coders come on board during the last few months who have already started contributing code. Our code is public on github, and we welcome 'drive by' code contributions: one thing we'd like to do is make that a bit easier by providing more extensive setup instructions so people who want to try running the code on their own machines can do so.

If you'd like to get more involved in our coding teams, then you can volunteer via our technical recruitment form. Please note that at the moment, we're only taking on fairly experienced people - normally we very much welcome absolute beginners as well, but we're taking a brief break while our established team get some of the performance problems under control so that we don't wind up taking on more people than we can support. We love helping people to acquire brand-new skills, but we want to be sure we can mentor and train them when they join us.

Lots of people have asked whether we'd consider having paid employees. It's unlikely that we'll have permanent employees in the foreseeable future, for a number of reasons (taxes, insurance, etc), but we are considering areas where we would benefit from paid expertise for particular tasks. Ideally, this would enable us to offer more training to our volunteers while targeting particularly sticky sections of code. Paying for help has a lot of implications (most obviously, it would add to our financial burden) and we want to think carefully about what makes sense for us. However, the OTW Board are discussing those options.

We're incredibly grateful to the hard-working volunteers who give their time and energy to all aspects of running the AO3. They are our most precious resource and we would like to take the opportunity to say thanks to all our volunteers, past, present and future. <3

Yet another update from your tireless archive volunteers! James from our Systems Committee has been making adjustments behind the scenes to stabilize the servers and get the most out of our caching, and we've seen some good improvements there. At the same time, we've been working on improving or scaling back the areas of our code where changes will give us the biggest gains.

Filtering

In order to improve performance further, tag filtering on work listing pages is disabled for the time being, until we roll out our new system. You can read more about this change in our post on disabling filters. We know this is an inconvenience for many users, but the filters are really the 800-pound gorilla sitting on top of our database - the works pages are both the most popular and the slowest on the site, which is a bad combination. We've had plans to fix them for a while, and that's underway. However, we need a few more weeks to finish and deploy the upgrade, since it also affects our search engine and quite a lot of our code. Our top priority is to make sure works remain accessible to users, and that new works and feedback can be posted and accessed. Looking carefully at our code and our stats, we concluded that removing filtering was the best way to ensure these goals in the short-term.

You'll still be able to view all the works for a particular tag, view the works for a user or collection in a particular fandom, and use our search feature to refine your results. Our post on disabling filters includes some handy tips to help you find what you're looking for. We hope to have full functionality restored to you soon! As a bonus side effect of this change, we've been able to remove the 1000 work limit on lists of works. This is because without the filters we can rely on the pagination system to limit the amount that we retrieve from the database at one time. So, while you can't filter your results any more, you CAN go through and read every work posted in your fandom! We hope this will compensate a little for the inconvenience.

Work Stats Caching

We've also done more caching of work stats (all the counts of comments, bookmarks, hits, etc.), so you may notice that these update more slowly on index pages now. The information is all still being recorded; we're just waiting a little longer to go get the counts for each work to spread out the load.

People Listings

The alphabetical people listings on the People page weren't actually that useful for finding users, and they were another performance drain.

We've replaced the full alphabetical listing with a listing of 10 random users, and added emphasis on the search. Note you can use wildcards in the search, so if you're not sure of someone's name you can enter part of it followed by an asterisk to get similar names. For example, entering Steve* would get Steve_Rogers_lover, SteveMcGarrettsGirl, stevecarrellrocks, etc.

Invitation Requests

We've suspended user requests for additional invitations for now as well. If you need invitations urgently for a challenge or for an archive rescue project, please contact Support. We also fixed an issue that potentially allowed users to snoop for other emails in the waiting queue.

Thank you!

Thanks to everyone who has been working hard on these issues, especially James, who has put in lots of hours tweaking the servers, and Elz, who has been doing the heavy lifting on code changes. Thanks also to all of you for your patience and understanding while we work!

And finally...

The great news is that so far, this emergency measure does seem to be having a noticeable effect. Our server load has diminished dramatically since we deployed this change:

Since last month, we've been experiencing frequent and worsening performance problems on the Archive of Our Own as the site has expanded suddenly and dramatically. The number of new users joining the site doubled between April and May, and we currently have over 17,000 users waiting for an invitation. We've been working hard to deal with the 502 errors and site slowdowns, and we've implemented a number of emergency fixes which have slightly alleviated the issues, but these haven't been as effective as we'd hoped. We're confident that we will be able to fix the problems, but unfortunately we expect the next round of fixes to take at least two weeks to implement.

We know that it's really frustrating for users when the site is inaccessible, and we're sorry that we're not able to fix the problems more quickly. We wanted to give you an update on what's going on and what we're doing to fix it: see below for some more details on the problems. While we work on these issues, you should get better performance (and alleviate the load on the servers) by browsing logged-out where possible (more details below).

Why so many problems?

As we mentioned in our previous post on performance issues, the biggest reason for the site slowdowns is that site usage has increased dramatically! We've almost doubled our traffic since January, and since the beginning of May the pace of expansion has accelerated rapidly. In the last month, more than 8,000 new user accounts were created, and more than 31,000 new works were posted. This is a massive increase: April saw just 4,000 new users and 19,000 new works. In addition to the growing number of registered users, we know we've had a LOT more people visiting the site: between 10 May and 9 June we had over 3,498.622 GB of traffic. In the past week, there were over 12.2 million page views - this number only includes the ones where the page loaded successfully, so it represents a lot of site usage!

This sudden and dramatic expansion has come about largely as a result of changes on Fanfiction.net, who have recently introduced more stringent enforcement of their policies relating to explicit fanworks which have resulted in some fans no longer being able to host their works there. One of the primary reasons the AO3 was created was in order to provide a home for fanworks which were at risk of deletion elsewhere, so we're very keen to welcome these new users, but in the short term this does present us with some challenges!

We'd already been preparing for site expansion and identifying areas of the site which needed work in order to ensure that we could grow. This means some important performance work has been ongoing; however, we weren't expecting quite such a rapid increase, so we've had to implement some changes on an emergency basis. This has sometimes meant a few additional unexpected problems: we're sorry if you ran into bugs while our maintenance was in progress.

What we've done so far

Our sys-admins and coders have implemented a number of things designed to reduce the load on the site over the last week:

• Implemented Squid caching for a number of the most performance intensive places on the site, including work index pages. For the biggest impact, we focused on caching the pages which are delivered to logged-out users. This is because all logged-out users usually see the same things, whereas logged in users might have set preferences (e.g. to hide warnings) which can't be respected by the cache. We initially implemented Squid caching for individual works, but this caused quite a few bugs, so we've suspended that for now while we figure out ways of making it work right. (You can read more about what Squid is and what it does in Release Notes 0.8.17.
• Redistributed and recalibrated our unicorns (which deliver requests to the server and retrieve the data) to make sure they're focused on the areas where we need them most. This included setting priorities on posting actions (so that you're less likely to lose data when posting or commenting), increasing the numbers of unicorns, and adjusting the time they wait for an answer.
• Simplified bookmark listings, which were using lots of processing power. We'll be looking into revamping these in the future, but right now we've stripped them back to the basics to try to reduce the load on the site.
• Cached the listing of guest kudos so the number doesn't have to be fetched from the database every time there are new kudos (which caused a big strain on the servers)

Implementing these changes has involved sustained work on the part of our sys-admins, coders and testers; in particular, the Squid caching involved a great deal of hard work in order to set up and test. Several members of the team worked through the night in the days leading up to the weekend (when we knew we would have lots of visitors) in order to implement the performance fixes. So, we're disappointed that the changes so far haven't done as much as we'd hoped to get rid of the performance problems - we were hoping to be able to restore site functionality quickly for our users, but that hasn't been possible.

What we're going to do next

Although the emergency fixes we've implemented haven't had as much impact as we'd hoped, we're confident that there are lots of things we can do to address the performance problems. We're now working on the following:

• New search and browse code. As we announced in our previous post on performance issues, we've been working for some time on refactoring our search and browse code, which is used on some of the most popular pages and needs to be more efficient. This is almost ready to go -- in fact, we delayed putting it onto our test archive in order to test and implement some of the emergency fixes -- so as soon as we have been able to test it and verify that it's working as it should, then we will deploy this code.
• More Squid caching. We weren't able to cache as many things as we'd initially hoped because the Squid caching threw up some really tricky bugs. We're continuing to work on that and we'll implement more caching across the site once we've tested it more thoroughly.
• More servers. We're currently looking at purchasing a more robust database server and moving our old database server (aka 'the Beast') into an application slot, giving us three app servers. We'll also be upgrading the database software we use so that we can make the most of this server power.

When we'll be able to implement the fixes

We're working as fast as we can to address the problems -- we poured all our resources into the emergency fixes this week to try to get things up and running again quickly. Now that we've implemented those emergency fixes, we think that we need to focus on making some really substantive changes. This means we will have to slow down a little bit in order to make the bigger changes and test them thoroughly (to minimise the chances of introducing new bugs while we fix the existing problems). Buying servers will also take us some time because we need to identify the right machines, order them and install them. For this reason, we expect it to take at least two weeks for us to implement the next round of major fixes.

We're sorry that we're not able to promise that we'll fix these problems right away. We're working as hard as we can, but we think it's better to take the time to fix the problems properly rather than experimenting with lots of emergency fixes that may not help. Since the AO3 is run entirely by volunteers, we also need to make sure we don't burn out our staff, who have been working many hours while also managing their day jobs. So, for the long term health of the site as a whole, we need to ensure we're spending time and resources on really effective fixes.

Invitations and the queue

As a result of the increasing demand for the site, we're experiencing a massive increase in requests for invitations: our invitations queue now stands at over 17,000. We know that people are very disappointed at having to wait a long time for an invitation, and we'd love to be able to issue them faster. However, the main reason we have an invitations system for creating accounts is to help manage the growth of the site -- if the 16,000 people currently waiting for an invitation all signed up and started posting works on the same day the site would definitely collapse. So, we're not able to speed up issuing invitations at this time: right now we're continuing to issue 100 invitations to the queue each day, but we'll be monitoring this closely and we may consider temporarily suspending issuing invitations if we need to.

Until recently, we were releasing some invitations to existing users who requested them. However, we've taken the decision to suspend issuing invitations this way for the present, to enable us to better monitor site usage. We know that this will be a disappointment to many users who want to be able to invite friends to the site, but we feel that the fairest and most manageable way to manage account creation at present is via the queue alone.

What can users do?

We've been really moved by the amount of support our users have given us while we've been working on these issues. We know that it's incredibly annoying when you arrive at the Archive full of excitement about the latest work in your fandom, only to be greeted by the 502 error. We appreciate the way our users have reached out to ask if they can help. We've had lots of questions about whether we need donations to pay for our servers. We always appreciate donations to our parent Organization for Transformative Works, but thanks to the enormous generosity fandom showed in the last OTW membership drive, we aren't in immediate need of donations for new servers. In fact, thanks to your kindness in donating during the last drive, we're in good financial shape and we're able to buy the new server we need just as soon as we've done all the necessary work.

As we've mentioned a few times over the weekend, we can always use additional volunteers who are willing to code and test. If this is you or anyone you know, stop by Github or our IRC chat room #otw-dev!

There are a few things users can do when browsing which will make the most of the performance fixes we've implemented so far. Doing the following should ease the pressure on the site and also get you to the works you want to see faster:

• Browse while logged out, and only log in when you need to (e.g. to leave comments, subscribe to a work, etc). Most of our caching is currently working for logged-out users, as those pages are easier to cache, so this will mean you get the saved copies which come up faster.
• Go direct to works when you can - for example, follow the feeds for your favourite fandoms to keep up with new works without browsing the AO3 directly, so you can click straight into the works you like the sound of.

Support form

Our server problems have caused some problems accessing our support form. If you have an urgent query, you can reach our Support team via the backup Support form. It's a little more difficult to manage queries coming through this route, so we'd appreciate it if you'd avoid submitting feature requests through this form, to enable us to keep on top of bug reports. Thanks!

Thank you

We'd like to say a big, big thank you to all our staff who have been working really hard to address these problems. A particular shoutout to James, Elz, Naomi and Arrow, who have been doing most of the high level work and have barely slept in the last few days! We're also incredibly grateful to all our coders and testers who have been working on fixing issues and testing them, to our Support team, who have done an amazing job of keeping up with the many support tickets, and to our Communications folk who've done their best to keep our users updated on what's going on.

We'd also like to say a massive thank you to all our users for your incredible patience and support. It means so much to us to hear people sending us kind words while we work on these issues, and we hope we can repay you by restoring the site to full health soon.

A note on comments: We've crossposted this notice to multiple OTW news sites in order to ensure that as many people see it as possible. We'll do our best to keep up with comments and questions; however, it may be difficult for us to answer quickly (and on the AO3, the performance issues may also inhibit our responses). We're also getting lots of traffic on our AO3_Status Twitter! Thanks for your patience if we don't respond immediately.

Welcome to our third Release in this week! Elz, James, and Naomi contributed code to this release, and Ariana, bingeling, Enigel, Jenn, and Kylie from our testing teams worked it over. Our sysadmins and coders have done more work to address the performance issues that have been affecting the archive as well as several other bugfixes.

PLEASE NOTE: in the name of drastically improving performance, this deploy may have a few side effects that appear at first to be errors or confusing! Please do read over these release notes and make sure that they don't cover a problem you are experiencing before you contact support.

Further efforts to battle the 502 errors!

This release includes caching of most pages for guests using Squid! Squid will serve up saved versions of pages without hitting our database or application, which increases speed and decreases server load for everyone.

The tricky part is making this work with all of the dynamic elements of the site: skins, content that gets updated by users, personalized messages, etc. We have decided to turn squid on quickly to keep the Archive running smoothly but we'll be working on finding the right balance between customization and performance as we go forward, so you may see some tweaks to different aspects of the site as we fine-tune this.

Current issues related to the caching:

• Site skins have been disabled for logged out users for the time being - if you rely on this feature for accessibility needs, please contact support and we will get you an account ASAP so you can use the skins again.
• Comments and kudos from guests may not show up at once for other guests. When a guest leaves kudos or posts a comment, they will see the comment/kudos added. If another guest then visits that same page (or the same guest reloads the page), however, they will see the most-recently-cached version, which may not yet show their comment/kudos count.
• Guests may occasionally see a stray error message or notice appearing at the top of a page that does not appear to be related to anything you've done. We are working to track all of these errors down but it is hard to be sure we've gotten them all. The messages should not affect using the archive.
• Hits that are handled by Squid (most hits from guests) will not appear in the hit count immediately. The hit counts will be updated once a day from the squid logs.
• Duplicate hits from the log files (for instance on page reloads by the same guest) will no longer be removed because of technical limitations, so hit counts may increase more quickly in some cases.

Squid will be enabled after we update the code, so you may not notice any changes right away.

For those interested in knowing more about Squid, see the detailed explanation below!

Changes to Subscription emails

We've gotten feedback about how people use their subscription emails and in response we have adjust the subject lines and message content to allow people to identify the content more easily. Emails will now contain subs of one type (author, series, or work) and the name/title of the first one in the subject together with the number of other updates.

Details

• Subscriptions:
• Email subjects will now say [AO3] instead of [Archive Of Our Own].
• Subscriptions will be bundled by type with subject lines of the form [authors] posted [first item] and [#] more, where first item will be one of: [Work Title], [Chapter Title] of [Work Title], [Work Title] in [Series Name].

• Performance:
• Skin chooser is turned off for logged out users.
• Nearly all pages will be cached for logged out users.
• Comment forms and other forms that are getting data for logged out users will have their details remembered in cookies and filled in by Javascript rather than remembered in the page.

• Bug Fixes:
• 500 errors were appearing on some work listings because of an interaction between caching and time zone conversion - this should be fixed now.

Details About Squid

Senior coder Ana has written up some helpful information about Squid for those who are curious:

"Squid is a really powerful tool that does a lot of things, but we’re using it primarily as one thing: a reverse-proxy cache. A reverse-proxy cache is a system designed to cache (that is, store copies of) web pages. It sits between users’ requests and the rest of the site and stores the responses to some requests so that instead of making the server build the page from scratch again, Squid can check to see if someone’s looked at that page recently and pass on the cached version. This is really useful when you want to send the same page to lots and lots of users because it means that instead of forcing the servers to generate the pages over and over, we can store a copy and give that copy out to everyone.

Of course, sometimes pages change: an author edits a story, or someone leaves kudos, so you don’t want to let Squid keep those copies around forever. Right now we let Squid keep copies for 20 minutes, and then it throws them away and gets a new one. This feels like the right balance between keeping things up to date, but not overloading the servers.

In addition, logged in users get customization on every page, in the form of the user bar at the top of the page if nothing else, which means that we don’t want Squid to store or give pages to logged in users. If it did, then every user would see the user bar for whoever made the request that Squid saved, and it would only change every twenty minutes.

This same principle holds true for all on-page customization (such as the skin-chooser), and finding the right balance between customization and cacheability (how suitable a page is for storing and giving out to everyone) is going to be an ongoing project as we try to weigh site performance against nifty features and information."

These release notes written and compiled by Ana, Claudia, Elz, Enigel, Jenn, Lucy, and Naomi.

Mini-release notes: Battling the 502s errors and responding to user feedback

Our coders and sys-admins have been working hard to deal with the performance issues we've been experiencing over the last few weeks. Releases 0.8.15 and 0.8.16, only a few days after deploying 0.8.14, see the introduction of several tweaks to our code and server setup that should help alleviate the site slowness we addressed in our recent post, AO3 performance issues.

New code by andreja, Ariana, Elz, Enigel, and Naomi. Tested by Elz, Enigel, Jenn, Kylie, Sarken.

Performance fixes

As noted on our Known Issues page, one of the major bottlenecks were tag listings for very popular fandoms, which would either load very slowly or throw up 502 errors. We've added caching for the first five pages of results - these will expire when a work using that tag is posted or revised, so the listings will still be up to date.

Another significant bottleneck were the bookmarks listings, both the main one and the bookmarks on tags. We've simplified the functionality, as we look into ways of reworking the bookmarks for a much better performance.

With the help of New Relic, a web service that monitors and analyzes site performance in great detail, we found another source of slowness: works with a large amount of (guest) kudos. Showing an updated kudos count for every work as it was accessed was putting an undue strain on the server, so for now the number of kudos on a work will be cached (i.e. not fetched from the database in real time) and only updated every five minutes.

As per a user suggestion (thank you, fydyan), we also looked into ways to prioritize certain user actions over others, so that trying to post a work or a comment would be less likely to throw up an error (potentially taking all input with it) than, say, accessing a user profile or browsing the site. Many thanks to Sidra from Systems for implementing this!

Invitation emails

We had a bug which was preventing notification emails for invitations being sent. We've fixed this bug, and the emails which were affected have been resent. This may result in some people receiving their invitations twice - invitations can only be used once, so please take note of the invitation code and don't pass the extra email along to a friend if it's a code you've already used!

Plans for our next few deploys

We're continuing to work hard on performance fixes and will soon be implementing much more caching across the site. We'll be posting with more details about this shortly.

We've had lots of feedback about our recent changes to notification emails. Unfortunately, we cannot roll back these changes for performance reasons (for more details see our post on Email changes and USER STATS!). However, based on feedback we've received so far, we will be adjusted the way subscription emails are batched and labeled, hopefully in the next deploy. Thanks for bearing with us while we work to improve this!

Details

• Bug fixes:
• Invite emails were not being sent; this has now been fixed and the delayed ones resent.
• The work count on the front page used to vary depending on whether the user was logged in; it now shows the total of all published works.
• Previewing a work before posting made the wordcount not show; this has been fixed.
• Improvements:
• Kudos counts on works are now cached, but will change when new kudos is added by logged in users. Guest kudos counts will update every five minutes. Thus it will cause less of a load on the database.
• Works listings under tags are now cached up to the fifth page. The listings will update when a work with that tag is posted or revised.
• We've removed the grouping of bookmarks by work. Also, before it was trying to get the bookmarks using tag synonyms, or bookmarks that were not using the tag directly. The main bookmarks listing now grabs the most recent public bookmarks on the site, and the tag listings show the bookmarks that are tagged directly with the requested tag.
• Text on the collections page has been changed to clarify what characters are allowed in collection names.
• Small change to text on the tag edit pages: the label for the synonym field was changed to "Choose an existing tag or add a new tag name here to create a new canonical and make this tag its synonym." for the sake of clarity.
These release notes written and compiled by mumble, Enigel, Jenn and Lucy.

As pretty much all of our users have no doubt noticed, we've been experiencing some problems with Archive loads: slowdowns and the appearance of the dreaded 502 page have become a regular occurrence. We're working on addressing these issues, but it's taking longer than we'd like, so we wanted to update you on what's going on.

Why the slowdowns?

Mostly because there's so much demand! The number of people reading and posting now is overwhelming - we're glad so many people want to be here, but sorry that the rapid expansion of the site is making it less functional than it should be.

We now get over a million and a half pageviews on an average day, often clustered at peak times in the evening (particularly when folks in the Western Hemisphere are home from work and school) - we were using a self-hosted analytics system to monitor site traffic, and we had to disable it because it was too overloaded to keep up. The traffic places high demands on our servers, and you see the 502 errors when the systems are getting more requests than they can handle. Ultimately we'll need to buy more servers to cope with rising demand, but there's ongoing work that we've done and need to continue to do to make our code more efficient. We've been working on long-term plans to improve our work and bookmark searching and browsing, since those are the pages that get the most traffic; right now, they present some challenges because they were designed and built when the site was much smaller. We've learned a lot about scaling over the years, but rewriting different areas of the code takes some time!

What are you doing to fix it?

Our Systems team are making some adjustments to our server setup and databases. Their first action was to increase the amount of tmp space for our MySQL database on the server - this has alleviated some of the worst problems, but doesn't really get at the underlying issues. They're continuing to investigate to see if there are additional adjustments we can make to the servers to help with the problems.

We're also actively working on the searching and browsing code: that's been a big project, and it will hopefully make a significant impact. Because it affects a lot of crucial areas of the site, we want to make sure we get everything right and do as much testing as we can to ensure that performance is where it needs to be before we release it. We're switching from the Sphinx search engine to elasticsearch, which can index new records more rapidly, allowing us to use that for filtering. That will offer us more flexibility, get rid of some of our slower SQL queries, and take some pressure off our main database, and it also has some nice sharding/scaling capabilities built in.

We also try to cache as much data as we can, and that's something we're always looking to improve on. Systems and AD&T have discussed different options there, and we'll be continuing to work on small improvements and see what larger ones we may be able to incorporate.

When will it be fixed?

It's going to take us a few weeks to get through all the changes that we need to make. Our next code deploy will probably be within the next week - that will include email bundling of subscription and kudos notifications, so that we can scale our sending of emails better as well. After that, we'll be able to dedicate our resources to testing the search and browsing changes, and we're hoping to have that out to everyone by the end of June. We rely on volunteer time for coding and testing, so we need to schedule our work for evenings and weekends for the most part, but we're highly motivated to resolve the current problems, and we'll do our best to get the changes out to you as soon as we can.

Improving the Archive is an ongoing task, and after we’ve made the changes to search and browse we’ll be continuing to work on other areas of the site to enable better scalability. We’re currently investigating the best options for developing the site going forward, including the possibility of paying for some training and/or expert advice to cover areas our existing volunteers don’t have much experience with. (If you have experience in these areas and time to work closely with our teams, we’d also welcome more volunteers!)

Thanks for your patience!

We know it's really annoying and frustrating when the site isn't working properly. We are working hard to fix it! We really appreciate the support of all our users. ♥

As we hope users have noticed, the recent performance issues on the AO3 have been alleviated! \0/ The fix came courtesy of our lovely Systems team, who tweaked our server settings to add more unicorns! Unicorns are the workers which actually make the requests of the servers - if there are too many, then things slow down because the servers are overloaded, but if there are too few, then things slow down because we are refusing requests we could potentially serve. (It's a whole unicorn balancing act!)

We weren't optimistic about how much difference changing our unicorn settings would make, because we know we have some underlying application issues which can bog down the database. So, we're really pleased that Sidra was able to find a new balance for the unicorns which has produced noticeable improvements in performance.

Meanwhile, behind the scenes, many of our other fine staff have been working on the performance issues. Sysadmin James_ has been working on partitioning the database: data is stored in tables, and in the new version of MySQL we're using, there's a way to split these tables into parts without affecting anything else. This means that you can search within a much smaller table rather than the whole big block of data, which makes everything faster. With the help of Systems staff, our coders have been going through the slow query database, looking for places where the code runs slowly and needs rewriting to be more efficient. James_ has also been looking into caching options for us - we'd like to use Squid or something similar (the unicorns need some tentacly friends), but we have a lot of different options which affect exactly what is displayed on the page, so we need to figure out the best ways of dealing with those.

Senior coder Elz is also continuing her work on rewriting our browsing filters, which are one of the biggest problem areas in terms of performance - she hopes to have the first version of this ready for testing soon.

We're really happy that the unicorns have helped us address the immediate issue, but we know this doesn't get us off the hook - we still have lots of work to do! While much of this work goes on behind the scenes, we wanted our users to have a glimpse of what's happening when we say 'we're working on it'. :)

We'd like to say a big thank you to all the coders and sysadmins who've been working hard on dealing with these issues. We'd also like to say thank you to all the users who have sent us messages of support and encouragement - we really appreciate it and it's a big boost when we're struggling with some tricky issues. Thanks for your patience while we continue to improve the site. <3

As many users will no doubt have noticed, the AO3 has been experiencing some performance issues since the start of the year. When we posted on 5th January, we were expecting those problems to ease once the holiday rush was over. However, that hasn't turned out to be the case. We're working on ways of dealing with the performance issues, but we wanted to keep you updated with what's going on while we do that.

Why the slowdowns?

In the past month, over 2000 new users have created accounts on the Archive. At the same time, the number of people reading on the Archive - with or without accounts - has been steadily growing. This has been part of a general trend, as you can see if you look at the graph showing number of visits to the Archive since November:

We're always much busier on Sundays, but the number of visits has been gradually going up each week since November (and the same holds true for the preceding months). However, before December we were hovering around the 135,000 level for visitor numbers at peak times. You can see that the visitor numbers began to climb more dramatically in December, peaking on 2nd January when we had 182,958 visitors. Crucially, after that spike it didn't drop back down to anything like the levels it had been at previously: we're now at more than 150,000 visits on a regular day, and more than 165,000 on Sundays, our busiest day. Wow!

We were expecting a big spike over the holidays, when there are lots of challenges and lots of people with a little spare time for reading and creating. However, we hadn't expected site usage to remain quite so high after the holidays were over! The increases mean that the site is now under a holiday load every day, which is one reason things have been running a little slowly.

The other reason for the slowdowns is that the increase in our number of registered users, and the holiday challenge season, has produced a big increase in the number of works. In fact, 11,516 new works have been posted since the end of December already! More data in our databases means more work for things like sorting, searching, etc - this means that sometimes the database just doesn't serve up the result you need in time, and the unicorn which is waiting to get that result gives up and goes away (yes, really - our servers are assisted by unicorns :D).

We've been expecting this general effect for a while now, and we've been working towards implementing things to deal with it; however, we weren't expecting quite such a big jump in site usage in the past month!

What are you doing about this?

The Accessibility, Design & Technology and Systems Committees had a special meeting on Saturday to discuss ways of dealing with the immediate problem, as well as longer term plans. It can be tricky to test for high load situations before they actually occur, but once they do occur there's lots of data we can gather to help us address the most crucial issues. (We're also working on implementing more tools which will help us test this stuff before it comes up.)

Short term

More caching: We already cache pages (or sections of pages) across the site - this means we store a copy which we can serve up directly, instead of creating the page every time someone wants to use it. If something changes, then the cache is expired and a new, updated copy is created. Hitherto, we've focused on caching chunks of information which are unlikely to change rapidly: for example, on any works index the 'blurbs' which show the information about each work are cached. However, some of the heaviest load is caused by rapidly changing pages like the works index. We're moving towards more caching of whole pages, so that a new copy of the works index (for example) will be created every five minutes rather than generated each time someone asks for it. This means things like works indexes will be a little slower to update - when you add a new work, it won't appear on the list until the cache expires - but that five minute delay will massively reduce the weight on our servers.

More indexes: We have a few places in our databases - for example the tables for the skins - which could use more indexes. Indexes speed things up because the server can just search through those rather than the whole table. So, we're hunting out places where more indexes are needed, and implementing them. :)

Medium term

Bad queries must die: We have a few queries which are very long and complicated, and take a long time to run. We need to rewrite these bits of the code to make them simpler and faster! In many cases this will be quite complicated (or else we would have done it already), but it's a priority to help us speed things up.

New filters for great justice: The filters that are implemented on our index pages are not really optimal considering the size of the site now - the limitations of that code are the reason we have to have a 1000 work cap on the number of works returned. We have been working on this for a long time - we need to completely throw out what we have and implement a system which works better for the site as it is now. Again, this is really complicated, which is why it's taken us a long time to achieve it even though we knew it was important - the good news is that we have now done quite a lot of work on this area and the first round of changes should be out in the next few months.

Long term

Long term, we're going to be moving to a setup which allows us to distribute our site load across more servers. This will involve database sharding - putting different bits of the database on separate servers - so it will take quite a lot of planning and expertise. If you're a user of Livejournal or Dreamwidth, you might be aware that your journal is hosted on a certain 'cluster' - we'd be moving to a similar system. We want to make sure we do this right, but based on the way the site is growing we think this is now high priority, and our Systems team are working to figure out the right ways forward.

Summary

We know it's really frustrating when the site runs slow or is timing out on you: many apologies. We really appreciate users' patience while we deal with the issues. As you'll see from the above, there are some immediate things we can do to ease the problems, and we also have a good sense of where we need to go from here. So, while these changes need to be implemented as a matter of urgency, we feel confident we will be able to tackle the problems. If you have expertise in the areas of performance, scalability and database management, we would very much welcome additional volunteers.

As we move forward on dealing with problem spots on the site, we may implement some changes which are visible to users: the caching on the index pages and the changes to browsing and searching are two of the most obvious. We'll let you know about this as we go along - we think the effect will be beneficial for everyone, but do be prepared for a few changes! You can keep up with status and deploy news on our Twitter @AO3_Status.

While the growth in the site means we're facing some problems a little sooner than we expected, we're really excited about the fact so many people want to read and post to the AO3. Thanks to everyone for your fannish energy - and apologies for the fact we sometimes slow you down a little.

As many of you will have noticed, we had some site slowdowns and 502 errors over the first couple of days of the year. Apologies for the inconvenience! If you've run into this problem and been wondering what was going on, you might be interested in this:

Yes, it looks like lots of fans decided to celebrate the New Year with some delicious fanworks. On Monday 2nd January we had 182,958 visits, and over 1,066,216 pageviews! Furthermore, an octopus swam off with our servers - volta_arovet's Texts From Cephalopods has had 46,301 hits at the time of writing! So, our servers had plenty of work to do!

Over 2000 new users have joined the Archive in the last couple of weeks, and we have hosted several great challenges, including Yuletide (2598 works!), Due South Secret Santa (a more modest 34 works), and Homestuck Ladyfest Exchange (124 works). So, while we're sorry to have had some slowdowns, overall we are super pleased with how well our shiny new servers have held up - those of you who were with us during the holiday season in previous years will remember that the high traffic of holiday challenges made our old servers very sad.

Looking forward, we're not too worried about performance in the immediate future - there are some code improvements we know we need to make which will improve matters a lot, so those will be high priority. If the AO3 continues to expand at the same rate as this year, we will be looking at more servers sooner rather than later. But in light of the graph above, we're pleased that while we certainly slowed down, we didn't grind to a halt! Thanks to all the coders and sysadmins who did the work to make this possible, and thank you to all the OTW members whose donations helped us buy those hardworking servers (we are always grateful for volunteers or donations)! And, of course, thanks to everyone who reads and posts on the AO3 - we're excited to welcome so many of you!

Once again, apologies to those of you who have been affected by the slowdowns - but hurray for so much beautiful fannish activity!

As most people are sadly only too aware, the Archive of our Own has been experiencing some performance issues recently. The sad 502 page is showing up increasingly frequently, to the frustration of everyone concerned! We have also had a couple of instances of downtime. We are working on the problem, but we know that our users are wondering what is going on, so we want to give a bit more information on what is going on behind the scenes.

Why is this happening?!

The main reason for 502 errors is the sheer load on the site. We've seen a massive increase in user numbers over the past few months - on Sunday 25th September we had 105,000 visits, of which 46,000 were unique visitors. These visitors racked up an impressive 575,000 page views! This is now a pretty average day for us - we're thrilled that so many people are enjoying and using the site, but we've expanded a bit more rapidly than we expected, so it's a little bit challenging for our servers.

Our recent server outages were caused by a problem with the servers themselves; our Systems team are tracking this to its source and making some changes to fix it.

Why do I sometimes get a 502 as soon as I click on a page?

Since 502 errors are associated with site slowness, it can be a bit unexpected to get a 502 as soon as you click. The reason this happens is that our servers are set up to keep an eye on how many requests are in the queue for them to handle. If there are more than a certain number, then the chances that the page will time out before it can be delivered are high. So, the server gives the 502 page right away instead so that you don't wait a long time only to be disappointed.

Didn't you just buy new servers recently?

Yes, pretty recently! Thanks to the generosity of fans who donated to our parent Organization for Transformative Works, we were able to purchase 5 new servers at the beginning of the year. These are actually doing a great job (if we were still on our original two servers, the site would have keeled over completely by now). However, the speed of our expansion means that we will need to purchase more servers sooner rather than later. When we do this, we'll also have to make some big changes to the underlying infrastructure - one option is shard the database, which means we'd split it up into separate chunks so each server only has to deal with a bit of it (if you're interested in the problem of scaleability and how sites deal with lots of users, this post on how LiveJournal handled it is a good read). We're actively researching now to figure out the best way of doing this and the kind of hardware we'll need.

What are you doing to fix the problem?

One reason the high loads on the site are having such a drastic effect is that some of our code is really optimised for fewer users (since we weren't expecting to have so many this soon!). So we're currently making a number of changes in the code to make it more efficient and reduce the number of database reads/writes (these are the things that tend to cause slowdowns) and updating the application software to one which should be more efficient. Our Systems team are also investigating updating the server operating system which might be necessary to take care of the issue which caused the server outages (it's hard to reproduce the conditions on the real site in our test site). The current set of new code has required intensive testing, which means we haven't been able to roll out the changes as quickly as we would like, but we hope to make these updates in early October. Longer term, we'll be doing some sustained work on scalability so that we can restructure things and buy more servers.

What can I do to help?

We really appreciate how understanding users have been about the problems - it helps our team a lot! If you want to do more and you have some financial resources, then a donation to the OTW will help ensure that we are able to continue expanding our servers (money donated to the OTW supports all OTW projects, but the vast majority is spent on servers and other running costs for the AO3). Finally, if you happen to be an experienced sys-admin or database expert - especially one with experience in performance tuning and scalability - and you would be willing to donate some time, we would welcome the additional expertise and support. (If you're interested, get in touch with our Volunteers and Recruitment Committee.)

And finally...

We'd like to say thanks to everyone who has supported the site in various different ways - while the 502 errors are HUGELY annoying, they do show that people enjoy using the AO3. Thanks also to all our users for your patience while we get to grips with our new success and deal with the performance problems. Finally, an HUGE thank you to our Systems team, who bear the brunt of the work on these issues - the wonderful Sidra has been woken in the night more than once to deal with server issues, and we appreciate her hard work and dedication more than we can say. ♥

We are working hard to resolve the performance issues, and we'll keep updating users as we have more news. The latest site status updates can be found on our Twitter AO3_Status.