Space Clouds

It’s… The End.

Alright guys, I have some terrible news.

We’ve changed servers, of course. This enabled us to do a lot more, performance wise. We’ve been looking around for extra stuff we could do. Jey’s been doing an awesome job at trying to keep up with us messing around, but we went too far.

I pushed the server hamster too far, and it’s gone berserk. It’s now chanting about the end of times, and somehow I think it’s right. We have 2 weeks left, on the 16th of May the hamster will turn into a destructive force we have never encountered before. I tried to save us, but… Our universe is ending. The team’s done the best they can, but we can’t fix it. I’m sorry, everybody. It’s been a nice universe, and I hope we’ll survive the blast and perhaps find civilization in a next dimension.

 

Alright, now to make sure everyone gets it:

Universe reset Saturday the 16th of May at 1PM Eastern Daylight Savings Time. The server downtime will hopefully be kept to a minimum.

Weekly Dev Notes – 29th April

Bloggy

Wow, the third week already!

So, who’s been doing what?

Jeff

Jeff has renewed the server payment. It looks good so far!

Our game page hasn’t been reviewed yet by Steam.

Blue Dwarf

Blue Dwarf has been cleaning up the code for BobboBot, which is our automated IRC bot.

Jey

Jey’s nearly done with setting up the livetest on the website.

Due to a lot of crash fixes, he hasn’t been able to push the update through yet. It’ll be on livetest asap, after which it can be done on the live server.

Suit

I got lots of feedback on the previous weekly dev post regarding the palace bug.

The problem being, with such a widespread use of this unintended feature, it’s nearly impossible to find everyone who has used this.

Ryan

Ryan has been fixing website issues related to the server migration. He’s trying to fix the leaderboard updating.

Other stuff

Steam beta after live patch! Yay!

4.23.2015 Server Migration downtime

**Update** The biggest part of the migration is now done, the web and game should be fully functional, but the migration is still in progress in parallel so expect a few hiccups here and here.

the new irc is accessible using irc.starsonata.com port 7777
or
kiwi irc
or
mibbit irc

As mentioned in http://www.starsonata.com/blog/new-servers-migration-soon/ we have been working on new server hardware / location to increase performance on the webserver/database/test servers and in doing so found upgrade for all our machines. The downtime will begin at 11h00 pm ET (server time) and will last about 8 hours

I have been doing just about everything that could be done without causing downtime to shorten / facilitate the final transition. But a final downtime is still needed to move the dynamic data (forum data, game database), to update the dns and tie everything together on the new servers.

During that downtime, the website will be switched over to a maintenance page. Forum, wiki and blog will all be disabled to allow for a clean transfer. The game server will also be stopped to allow for the database and website communication interface to be transfered without problems.

Website and other secondary services will come back up as they become available. The game server will be the last thing to come back up and in all likelyhood won’t happen before those 8 hours are elapsed.

Weekly Dev Notes – 22nd April

BloggyAnother week, another blog!

So, who’s done what?

Blue Dwarf

He’s been swamped in coursework and other tasks from university this week. Sadly, he couldn’t do much in Star Sonata.

Jey

Jey has been busy fixing crashes on our live and livetest server.

Also, for our new server some updates! A lot of technical talk, but it comes down to this:

More automated stuff for backups/restarts, simpler commands and a better watchscript for restarts. Also, a new test server which  automatically updates/compiles.

Ryan

Ryan has been working on the webserver support for Steam checkout. There was also a bug with the Emperor’s Palace which has been fixed now. Furthermore, Ryan’s been discussing an interesting new concept with Yclepticon. I’m sorry, it’s a surprise! More about this in another blog.

Suit

I’ve been looking at possible droprates for the GB/NF player versions. What’s the droprate? A mission? A ship? I can’t say! You’ll see it when it’s implemented.

Jeff

Busy with Steam. Our Steam page is up for review at this moment.

Any other discussions?

We’ve had a few interesting discussions this week. I’ve summarized them for you guys.

Team score and BvB

Since the team scores are quite high to be able to BvB, we’ve had a discussion on lowering the team score required to be able to BvB. We’ve been thinking of a team score of 50 to be required.

Team skills

Similar issue as with the team score system. Since there are many strong teams, new and lower teams are having difficulty climbing up the ranks to become stronger and better. We’ve been thinking of the possibility of a new system which is less linear than the current one.

And we’ve had a short discussion on building indestructible bases and running for Emperor as devs. But I don’t think I should put that in here, I guess.. Oh well.

 

That’s it for this week! I’ll be back next week with more interesting news and discussions. This weekend I might be able to high five any Irish players around, since I’ll be visiting there friday-monday.

See you next week!

Weekly Dev Meeting – 15th April

Bloggy

Hey, everyone!

As I love informing you guys on everything, I’ve asked the others from the dev team if I could post information from our weekly meetings (yes, we have those!)

Whenever possible, I’ll make some notes on the weekly meetings and I’ll update you guys on what’s been done, and what’s being planned in the near future (except for surprises, because, you know… surprises).

Anyway, we’re off!

So, who’s done what?

Blue Dwarf

Blue Dwarf’s stopped easter. (Boo!)

He’s also worked on some achievement bugs for Steam. He’s working on implementing achievements such as building a base, etc.

Ryan

Ryan has been working on the payments through Steam, to allow you guys to buy both spacepoints and subscriptions through Steam. He also figured out a bug with subspace spawners that were causing hundreds of ai to spawn an hours and fixed it. Before the fix there were over 1606 ai ships in a gal!

Jey

Jey’s been working on the integration of the new server, doing a lot of testing of it all. It looks very positive. Quoting him, the database is “swimming in ram”, the server should hold our live + test servers quite easily.

He’s going to do a full patch, hopefully friday or monday (in the best case scenario. If he encounters any issues, this could be delayed). If all goes well, the server will be migrated the following day.

Jeff

Jeff has been working on banners for Steam. Here’s a few of them! These are some first versions, we’ve been discussing on which ships we should use for this. They should be interesting, but not too high tech (so players can pilot the “banner” ships quite quickly in the game). They also shouldn’t be too dark, since it’s often against a dark background so the ships would get lost in it.

Suit

I’ve been… Making this blog, I guess? I have no idea.

I’ve been playing around ingame, spawning stuff for people in the Colosseum, which was really fun. I got to talk to a lot of you guys, you all gave me some really nice input and/or feedback which I’ve also discussed with the team. I’ll definitely try and be ingame more often.

 

So, that’s it for this week! As I’ve said ingame, after the server migration, we’ll be able to hand out some Steam beta keys. I’ll keep you updated on this.

See you next week!

New servers / migration soon.

As a few may have heard, we have been investigating the possibility of changing provider for our servers. This did not come from our current ones being insufficient, although the webserver was rather overloaded.
But instead came from Jeff mentioning he saw some Hacker News post praising a datacenter company in Quebec called OVH.

After that point, I had a look at what they offered, prices / specs and multiple reviews I could find on the internet.
The price for the specs were… pretty darn impressive, to get machine a bit stronger than what we currently have, we end paying about half our current price but, specs arent everything and the price felt too good to be true.




None the less, we ordered a machine for a week so that we could test the quality of their network and hardware when compared to our current machines.

on liquid we currently have

  • www:
    • AMD Phenom(tm) II X6 1055T Processor (6×2.8GHz)
    • 16 gb ram ddr3 1333MHz
    • 300gb sas raid1 (15k rpm) for the database/webserver
    • 1Tb 7200rpm sata drive for the immediate backups (theres 2 other levels of backups hosted externaly, this is just for the quick every 30minutes backups)
    • 100mbit connection
  • liberty:
    • AMD Opteron(tm) Processor 6128 x4 (32x2GHz)
    • 16 gb ram ddr3 1333MHz
    • raid 1 sata 7200rpm for the main os / source code
    • 1x ssd drive for the universe save (reduce the lag impact from the save to <2ms instead of the old 130ms that often occured on regular harddrive).
    • 1gbit connection (to ensure minimal latency to the game at all time even with a lot of players)

the machines we are getting on ovh are

  • www:
    • Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz (6cores with H-T 12threads)
    • 128 gb ram ddr3 1600MHz
    • 300gb ssd raid 1 for the database/webserver
    • 2tb sata raid1 7200rpm for the immediate backups
    • 500gb of external backup storage (free with most dedicated servers at ovh)
    • 1gbit/500mbit out unlimited connection (1gbit burst up to 10secs)
    • ddos hardware protection from 100gbit/sec to 500gbit/sec depending on the specific
  • liberty:
    • Intel(R) Xeon(R) CPU E5-2670 v2 (20×2.5ghz with hyperthread so 40logical threads, turbo 2.9ghz and maintained at low temperature)
    • 256 gb ram ddr3 1600MHz
    • 3x ssd 300gb in raid1 for the os install and a partition in raid0 for higher performance uni save
    • 500gb of external backup storage (free with most dedicated servers at ovh)
    • 1gbit/500mbit out unlimited connection (1gbit burst up to 10secs)
    • ddos hardware protection from 100gbit/sec to 500gbit/sec depending on the specific





Now, on paper this mean that the website should be a lot more responsive once we switch.
The database will have so much ram available for caching that it should be a world apart from our current configuration, not to mention the cpu being by far superior and so is the storage on which the database will be.
What this also mean for the webserver, is that we will be able to run test servers with full universe without impacting performance all that much.

Currently test server is rather limited on full uni testing, it cannot have both livetest and test running full uni at the same time or the memory drops too low and the database ends up affected.
With this new server, we should in theory have no problem with 2 or maybe even 3 -4 test servers (private test servers for the content devs to run their own separate player related tests etc).
Offcourse cpu is limited, but servers with 1-2 players on them use very little cpu so theres generally no issue here for a test server.

For liberty we are looking at an improvement of a little over 250% performance all around the board. There is a slight drop in the number of physical cores (from 32 to 20) but the performance per core is quite a bit faster.
Both in lag scenario (1500ai fighting in a single galaxy) and the current uni without players, the performance was always 2-3 times faster on the new server compared to the current liberty.




Now for the part the specs and paper rarely properly cover, the connection reliability. As some of you know, it’s been a constant issue over the years.
Some people unable to connect even tho others can, part of the world losing access and need to rely on proxy / gateway temporarily, frequent disconnection even tho their internet never lost the connection as far as they can tell etc.

It is extremely hard to properly test for those things due to the apparent randomness involved, so I came up with a test that I think should catch it as best as possible to allow for comparison.
No server is perfect and our network code is extremely fragile (its been a problem for a long time, the slightest corruption or interruption is often enough to cause a full disconection), therefor having a network that experience less of those interruption sound like a good improvement

Now for the specifics, I split the tests in 3 categories, each with their own frequency. All tests were ran on both liberty and the test machine we took on ovh for 5 days.

  1. A ping test: Every second, each server pinged 49 locations (7 per test regions) around the world looking for packetloss / disconnection as well as latency.
  2. A route test: Every 5 minutes, each server took a snapshot of the route used to reach all 49 locations to compare with the previous and note route change.
  3. A bandwith test: Every 2 hours, each server ran a speed test using the speedtest.net command line tool to 21 (3 per test regions) different speedtest nodes spread around the world.

The ping test has up to 1 second error per grouped packet loss (1 packet loss is 0-1 sec, 2 packet loss in a row is 1-2 sec, 3 is between 2-3 secs etc.). Downtime that were identical to both liberty and the ovh server were removed (the target was likely down and not the route to it)
the new server had a total all around disconnections between 397 and 1489 seconds out of 432000 seconds tested (0.0009% to 0.0034% off the time when not all the international routes were functional)

  • Africa Central: 193-835 seconds down (249ms)
  • Asia: 112-154 seconds down (195ms)
  • Australia: 32-157 seconds down (203ms)
  • England: 27-117 seconds down (97ms)
  • Germany: 23-106 seconds down (85ms)
  • US south west: 6-78 seconds down (72ms)
  • US east: 4-39 seconds down (19ms)

current liberty had a total all around disconnections between 866 and 2309 seconds out of 432000 seconds tested (0.002% to 0.0053% off the time when not all the international routes were functional)

  • Africa Central: 393-1135 seconds down (280ms)
  • Asia: 212-468 seconds down (195ms)
  • Australia: 74-235 seconds down (207ms)
  • England: 87-178 seconds down (120ms)
  • Germany: 73-162 seconds down (105ms)
  • US south west: 12-81 seconds down (60ms)
  • US east: 15-50 seconds down (23ms)

Interestingly enough, the ping on the server based in Quebec, is lower on average (with the exception of the worst case scenario and that would be the US south west), this was not expected.
But it can be explained due to the effort OVH put on their network, they have a lot of private routes / deals with other providers to ensure good connections whenever possible (which is along other things, needed for their impressive ddos protection).
All in all, the downtimes were a lot shorter on the OVH network than on liquid’s.




the second test, theres a little too much data to be put in a way that would remain clear in a post (most routes tested are international, and imply 10-20 gateways), but there was a pretty clear difference in the routes used by each server.

The current liberty changed route on average twice per hour, rarely major change, just a different gateway here and here. Those change often happened at the same time a disconnection was found during a ping, but not always.

On the new server, there were very few actual route changes, 4-5 per day with the exception of central Africa, which saw about 2 every hours as well.




The third test was the bandwith test. While bandwith doesn’t equate to performance when you’re looking at a video game (you dont need 30mbit to play ss, especially not on download) it does give a somewhat good representation to the link quality and load, as a rule of thumb, if a link let you transfer faster, it mean its able to take a whole lot more load.
Keep in mind that while this is the average bandwith speed obtained using speedtest cli, a lot of those ended up thottled at one point or another along the line because it was only 1 source. When doing multiple different tests in similar location at the same time, the total bandwith nearly always approached 700/400 DL/UL

The current liberty network average for each regions DL/UL MBit/sec

  • Africa Central: 97/38
  • Asia: 84/24
  • Australia: 138/96
  • England: 157/43
  • Germany: 165/52
  • US south west: 443/319
  • US east: 409/294

The new server network average for each regions DL/UL MBit/sec

  • Africa Central: 122/44
  • Asia: 124/74
  • Australia: 238/126
  • England: 357/213
  • Germany: 365/252
  • US south west: 343/189
  • US east: 705/494





In a nutshell, trying as I may to find a problem with the new server location, I could not. All I could find were more reasons on why we should switch over. Altho liquid do have their incredible support team going for them (never had problems fixed as fast as with them).
OVH gives us quite a fair bit more power over the hardware which should normally greatly reduce the need to rely on the support (beyond hardware repairs).
Funnily enough this is something quite a few of the review on the internet actually complain about, their control panel is compared to most other services I tried, complex and dangerous.

The OVH server end up being way cheaper than our current ones, an order of magnitude faster and have a more reliable connection as far as I could test.

Have no fear, Easter’s here!

It’s that time of the year again, with all kinds of fun easter stuff!

Until the 13th of April, Easter Warren’s in Sol for a lot of fun missions to do.

The (Zombie) Easter Bunny is back as well, with a vengeance! Can you manage to kill him?

We also have the roaming rats and egg crates, as well as some Easter drops which drop… Everywhere!

Are you ready to hunt some eggs?

As an added bonus, we’ve added a +40% XP bonus until the end of the event. Enjoy, everyone!

Suit’s update on ALL THE THINGS! AI ships! Steam!

Hey there guys, girls, and other people!

I wanted to update you guys on a few things. I’ve been quite busy on working on stats on my 2 new ships. The missions are done, I just have to think of one last thing, which I’m not going to spoil already.

The last time I made the stats, they were incredibly broken. Then I made them underpowered. Hopefully, this time I’ve got it just right! We’ve thought of some nice inbuilt extra’s for you guys. If I’ve done it correctly, they’ll be mid-range in their tech level, not too expensive and quite useful for everybody. After much thinking, I’ve also decided to keep their AI appearance, seeing how you could always add your own funky texture to it. It’d kind of ruin the AI-ship feeling if it would have another skin.

Now, what we’ve been waiting for. STEAM!

We’re working very hard on finishing up stuff, but look. LOOK AT THIS BEAUTY!Steam

 

That’s right. I have it. Deal with it.

We’re still doing some finishing touches, like how it interacts with Steam. Since we don’t know what issues could pop up, I can’t give a definite date. Sorry guys! I’ll try and update you as soon as possible. I poke the others a lot of information, to try and keep you guys up to date as much as I am.

And finally, uni reset! It’s going to be this saturday, we have to change some stuff for Steam!

Just kidding.

Or not, who knows?

(I’m kidding, don’t worry. There won’t be a reset this saturday. A uni reset is always announced at least 2 weeks in advance, and with a special post about it.)

Server Patch 3.18.2015

Server Patch 3.18.2015
Lots of bug fixes here today!

* Fixed a bug with /mc where certain shops would not show up in the list even though they had the best price
* Fixed a server crash
* Small adjustments to Ultimte Art of Healing Augmenter and Ultimate Art of Commanding Augmenter
* Fixed an issue where the Factories in Kalthi Stronghold didn’t use the right commodities
* Fixed a typing error that resulted in a malfunctioning endmission in Strontium-90
* Fixed some typing errors in inline replacement variables used in some missions (such as $startstation)
* Fixed an issue with Primal Leopard Scratch Blueprint requiring a Leopard Will
* Removed the last bits of the monthly event

Monthly Event: Perilous Perilous XP!

This weekend, we’re doing a bonus XP event focuses on the new Perilous Perilous area! AI there will be dropping global +5% xp boosts like crazy! We’ll also have some powerup spawners like the last event going at the same time. Like last time though each account can only have 1 boost active at a time, so the more people in perilous perilous the higher the boost will be. Unlike our last event though, these aren’t tied to limited time spawners. So the XP limit will be based only by how many people participate! If everyone on at the time of the post were to go scoop an XP buff, we’d have +500% xp! We’re also introducing a limited time item that will be a rarer drop that will give a level to any character that uses it (up to level 9000). Like the global xp boosts, it’ll be dropping from AI in Perilous Perilous as well as dungeons there.

epiph

We hope everyone enjoys the event!