e-petitions: handling traffic
Since I last blogged about e-petitions we had what conservatively could be described as “something of a traffic spike”. The amount of interest surrounding the site massively exceeded all our expectations.
Given the time available to us, we had rated the site for about 10 requests a second, basing our expectations on 50% more usage than the original e-petitions site. However, during peak times we were seeing non-cached bursts of traffic through to the site of between 70-120 requests a second: we’d load tested up to about 40-50. This caused the site to intermittently produce 500 errors; this in turn producing headlines that we really didn’t want to see.
Most perplexingly, we were still seeing intermittent failure messages in the logs when the site was getting about 20-30 requests a second, even though that had worked fine in testing. During this time none of the servers were under a huge amount of load, so we struggled to find the bottleneck.
Eventually we discovered that the hardware firewall we’d put in place to help improve security wasn’t able to handle the number of network connections required of it, and was randomly dropping network connections. This included connections on the internal network, which caused connectivity problems to the seach and database servers. This caused most of the intermittent failures people saw. The firewall had been set up after we had done our load testing and we’d not re-run our testing since then, so we hadn’t spotted the problem .
Once this had been fixed, we were on to more familiar territory. The dedicated solr server we were running for search was really struggling with only 4 CPUs, so we rebooted it using 8 CPUs and it started working much better.
We also made a number of other changes to the site to make it more robust:
-
We brought an application server down, cloned the disk and set up a third application server within about half an hour. It’s not as quick as running on Amazon EC2, but it’s not a bad turnaround for a more traditionally hosted site.
-
We set up monitoring on the site using Munin, which is a brilliant server montoring tool. This helped us discover the solr issues much more quickly.
-
We went right through the code and turned on caching everywhere we hadn’t yet thought of, including caching of more pages surrounding the signing step.
-
We worked around a sunspot issue which was causing a petition to reindex after every signature, stressing the search server further.
Lessons learnt
- Run your load tests again after any configuration change, even if it shouldn’t make a difference. If we’d done this, we’d have spotted the firewall configuration issue before the public did.
- Set up proper measuring tools before the event. It took us a while to find the best cause of action with the search server because we were relying on New Relic to monitor the application servers only. Once we had Munin running we could more easily make more CPUs available to the search server.
The vast majority of the above changes were made within one (very late) night: from a customer point of view, being agile isn’t just about how flexible you are during development, but how responsive you can be when there are problems. Our technology and stack choice really helped us out here, and I also particularly wanted to thank the Alpha.gov guys (Ben Griffiths, James Stewart and Matt Patterson) and Alex Tomlins from Unboxed for stepping in and helping Jolyon and myself out that evening.
Jolyon and I are giving a talk about e-petitions at LRUG next month if you’d like to hear more.
Share this article
Read more
How to Build a Robust LLM Application
Last month at Cherrypick we launched a brand new meal generator that uses LLMs to create personalized meal plans.
It has been a great success and we are pleased with the results. Customers are changing their plans 30% less and using their plans in their baskets 14% more.
However, getting to this point was not straightforward, and we learned many things that can go wrong when building these types of systems.
Here is what we learned about building an LLM-based product that actually works, and ends up in production rather than languishing in an investor deck as a cool tech demo.
Read moreYour Code Is A Liability
Every chunk of code you commit is more for someone else to read, digest and understand.
Every complex “clever” expression requires another few minutes of effort for each of your team. They must now interpret what you wrote and why you wrote it.
Every line you add limits your project’s responsiveness to change.
Your code is a liability. Never forget this.
Read moreThe Sol Trader Christmas Eve update: moddable missions
The relative radio silence from Sol Trader Towers is for a reason: I’ve been working hard on a flexible and moddable mission structure, that allows players to take a variety of interesting quests in-game.
This build is now available on the forums should you have access (there’s still time if you don’t.)
I’ve built a few missions to start with, including delivering parcels for business or personal reasons, taking characters on business trips and making other characters disappear. It’s great fun to have a variety of things to do for characters now and adds yet more colour to the game. Because it’s completely moddable, I’m also excited to see what storylines other people come up with!
Under the hood
The full details of how to create your own missions are available as a lengthy forum post, which will be kept up to date with changes and clarifications. Here’s an overview:
The missions are organised into packs, which exists under the data/missions
subfolder. If you have access to the beta builds, you’ll see there’s one pack there already: these are the missions that are built in to the game.
There are several csv files in each mission folder:
requirements.csv
: This file details the cases in which this mission might be triggered. Each character in the game has a chance of picking this mission (and becoming the ‘giver’ of the mission), based on the conditions imposed by this file.conversation_player.csv
: The extra conversation options available to the player because of this mission.conversation_ai_response.csv
: The extra options the AI can choose from as conversation responses.opinions.csv
: The extra opinion triggers, used for reactions to the generation and completion of these missions.strings.csv
: The new strings needed for the previous CSV files.
The possibilities for you to build your own missions are expanding all the time, as I add new missions triggers and possible goals for the AI.
What’s next?
At the moment it’s possible to take on any mission from any person, which isn’t very realistic. I need to allow players to gain other character’s trust, so that they will only give you sensitive missions in certain cases. Additionally it will soon be possible to start a career with an organisation, which will give you a rank, a certain amount of built in trust, and access to more senior characters.
I’m also going to be working on the in-space AI very soon. At the moment only freelance traders fly around between planets: it’s time we had passenger ships, military guards and pirates thrown into the mix.
Have a fantastic Christmas and I’ll see you all in the new year with some more updates.
Read moreNew Sol Trader beta: the science of blame and unforgiveness
Previously I wrote about how I’m modelling opinions and prejudice in Sol Trader. It’s time to put some of that information to use.
The opinions a character has of other people, based on the partial events that they know about them, will now directly affect the things that happen in the history generation. This creates new events, which will in turn feed more character opinions.
There’s a new beta available on the forums if you have insider access.
Dudley and Meredith
In the example on the left, we can see that an acrimonious divorce of Meredith’s parents has left an indelible mark on her childhood. She now has a very low opinion of her father, Dudley.
When characters are adults, they can then generate a series of ‘favours’ (or ‘missions’) that they want completed. This is a source of work for the players, although completing certain missions does have real consequences on your relationships with the target of the mission. If they find out you’ve taken a mission against them, then they won’t be happy with you.
To continue our example, Meredith, whom we are now married to, wants us to find out some potentially incriminating information about our own father-in-law, Dudley. It’s up to us whether we take it or not. If he finds out, we’ll make an enemy of him.
As the game goes on, the player will get embroiled in these relationships between the various characters and be able to directly affect their stories. Choosing what to take on and who to ally yourself with forms a major part of Sol Trader’s gameplay.
Sarina’s spiral of doom
Another example: the sad tale of Sarina, our older half sister. I picked Dagny and Warren in history generation to be my character’s parents, knowing that Dagny was cheating on her husband Hayden, mostly to see what happened. Little did I know how much it would affect Sarina, Dagny and Hayden’s eight year old daughter. When she found out about my birth, she got very upset.
She didn’t blame me, thankfully, although she never thought much of me. However, she never really spoke to our mother again, especially since her beloved father Hayden died soon after we were born.
She left home at a young age, and became a political assistant, but she didn’t make too many friends. She was doing ok for a time, only to find out that the love of her life, Richard Ruhr, had been having an affair behind her back all along.
She divorced him, got depressed, quit her job and by the time I grew to adulthood at the start of the game, she was living in a hippie commune somewhere on Mercury, trying desperately to get some gossip on her ex-husband.
New beta out now
This new beta is now available from the forum if you have purchased insider access (if you haven’t there’s still time.) Let me know if you find any other interesting stories such as these!
Read moreModelling opinions and prejudices in Sol Trader
I’ve been working hard on the Sol Trader core gameplay mechanics in the last two weeks. High up on my list was a way of generating more interesting missions for the characters to complete.
In order to have a reason to gather dirt, find locations or desire an early end for an enemy, our characters need to feel strongly about other people they know. This is where their opinions and prejudices come in.
Characters already keep track of the events they know about for each other character in the game. Now they can form an opinion of a character based on the partial set of info they know about someone else’s past.
The plan is to use these thoughts about each other to make decisions about who they’re friends with, deal with relationship breakdown, blame and prejudice.
Here’s an example of how we configure this under the hood for an occasion where a character is caught and reported for taking bribes:
Anyone knowing about this event will think the character is less deserving of sympathy and assume the character is less moral. If we’re the one catching them take the bribes, then the briber becomes much less influential over us. If we’re the one being caught, then the one catching us is definitely no longer our friend. Depending on our profession, we will brief against them or possibly try to take them out.
Now characters have opinions about others, we can use these to guide their conversation choices, who they’re likely to target, give us gossip on, etc. It’s all game design fuel for other behaviours in the game, and will combine to form interesting unexpected effects and tell original stories each time.
Next time I’ll discuss about the new events that get created in the history generation because of these new opinions. Our stylised formulaic view of history is about to become, well, a lot more messed up. Rather like real history…
Read more