Space Clouds

Weekly Dev Notes – 27th May

Bloggy

 

I didn’t even make the meeting. But hey, here’s another update! It’s a very short one, since it was a quiet meeting.

So, who’s done what?

Blue Dwarf

Blue Dwarf’s added a special tag to galaxies, if warp beacons are allowed or not.

The Vert

Vert’s discovered a bug with warp beacons. It’s been tested, and Blue Dwarf fixed that. There’s also some other minor bug fixes.

The big UrQa Capship boss had some tiny bugs, those fixes should be in soon.

Any other news?

No.

Well, kind of. I’ve had some hectic times, which caused me to be unable to make any meetings or do any work. If everything continues like this, I should be fully back by the end of this week. At least I made this blog again!

See you next week!

Weekly Dev Notes – 20th May

Bloggy

Hijacked again! I’m getting good at this. Also the first meeting of the new universe, which went very smoothly I think we can all agree.

The updates

Jey

Jey has been working tirelessly tracking down the crash that appeared around 10PM server time a few days ago and was effecting us again around the same time yesterday. He found the source which involved slaves that used fueled energies.

Jey has also fully enabled that backup system, which allows us, as in other developers, to retrieve saves of the universe, useful for tracking down bugs and suchforth.

The Vert

The Vert has created superitems for some of our newer player ships, the Kalthi ships, Harrier, Mercant Navy Contriver, UrQa’qa Qu’ishi Qa and Divine Behemoth.

He’s now looking into designing some new overloaders for stations.

Blue Dwarf

A player had reported that a Subspace shortcut had been generated between a Wild Space galaxy and Perilous Perilous Space galaxy, having decided that this is not intentional I’ve moved the exit wormhole in Perilous Space and fixed the issue for future universes.

The balance god, yclepticon, requested that the base series of drones, that is Apollo Drones through to Andaman Drones, had their Transference Resistance removed and replaced with +100% Transference Efficiency so that they would not be such a liability to use in galaxies. Also for Ambrosia and Andaman Drones specifically I’ve reduced the number of pellets (projectiles) the Gourd weapons shoot to 1, less projectiles mean less lag in those intense battles, such as when fighting in Anatolia.

A note about Suit

There is a high chance that I’ll also be covering next weeks blog post.
Unfortunately Suit’s partner got involved in an severe incident and is currently in hospital and our thoughts are with them.

Weekly Dev Notes – 13th May

Bloggy

Suit isn’t here this week, so I’ve taken over his lovely weekly blog!
Don’t forget that universe reset is this weekend guys.

So, who’s done what?

Jey

Jey’s been tracking down and fixing a bug involving high CPU usage when alt-tabbing.

Once he’s got that fixed, he’s preparing a patch to fix Vacuum scopes, increasing the timer from 30 seconds to 120 seconds and the pulling. Additionally any other fixes other devs need in.

Ryan

Ryan implemented the changes to Emperor’s tithe as well as added the additional Imperial Chest drops that we discussed last week. He’s been fixing the leader-board issues on the website and its mostly back in order but there is still a bug with the player rankings. He’s is currently looking at a bizarre slave healing bug that causes aggression.

Jeff

Jeff has handed out some reward skin items to players which ergosphere has produced for us and has been taking care of business stuff.

Blue Dwarf

I’ve done the scaling spawn times on spawners that we mentioned last week. If the time between kills is short then the spawn time will be reduced, specifically for The Nexus to help deal with potential heavy influx of players.
I’ve made it so that peons will be able to abandon/transfer their stations now, just like their lonely-teamless friends.
More error checking is performed when setting up mining slaves.
Slave ‘Show Status’ will contain the state of durability of the slave’s gear
I’ve fixed the wild AI XP reward capping bug and a bug during universe generation occasionally causing small asteroid belts having a spawn rate that is too high.

Weekly Dev Notes – 6th May

BloggyWell, the universe might be ending soon, but we decided it was time for another meeting.

Important notes

First off, there was a lot of attention regarding bugs and exploits. One player has been banned for 3 months. More on this later.

We have taken action for the Imperial Seal bug abuse. We can’t punish the exploiters, so instead we’re rewarding the honest players. Imperial Seals will drop from all t21 and t22 bosses, starting next uni.

So, who’s done what?

Jey

Jey’s been cleaning up our new servers and finalizing the patch preparations.

Some people might have noticed some issues regarding their e-mail addresses. This was because of our new server (but not our fault! Yay!). Because we have a new IP, Microsoft didn’t recognize it and it was automatically blocked from their mailing lists. Setting our server to match their policies exactly should stop this from happening.

He’s also been running extra tests on his private server, so we could skip having the patch on our livetest server before going live.

The Vert

Fixed a bug regarding the Divine Prophet. People got stuff they weren’t supposed to, naughty naughty!

Superitems will be added to t22 ships in Perilous Perilous Space.

Suit

I’ve finally finished the player versions of the Nightfury and Green Battleship! They’ll be in on the uni reset.

Speaking of uni reset, I noticed that we were missing our guide. Click here for the guide for the universe reset!

Ryan

People used a bug to get AI-only items. A player has been banned for this for exploiting it. Ryan’s going to add an AI-only tag to a bunch of items, so players can’t get it anymore.

He’s also committing the client-side things for Space Point purchases.

Now for a sadder announcement… Ryan’s removing some of our explicit AI names. No more laughing at R-rated AI!

Jeff

Jeff has been charging PayPal subs which didn’t go through because of the blocked mailing list.

Still no review from Steam! Gah!

Blue Dwarf

Blue Dwarf’s done a few things that’s already been said above. He’s helped me with my finishing touches on the ships and he’s fixed the AI-only stuff you guys weren’t supposed to get.

He’s also finished working on BobboBot with some fancy new commands (like !countdown), you can find out about that in our IRC. Our updated IRC rules are now included in the main rules! Click here to see the updated IRC rules.

As a change to the Nexus, if AI are killed very quickly, they will respawn quicker. This is to prevent people needing to wait a long time for certain AI to spawn. Hopefully this will be in by uni reset.

Other stuff

We discussed a new reward for being Emperor. The emperor’s team could take a tithe from all AI-base sales, all scooped credits and all Colony profits.

It’s… The End.

Alright guys, I have some terrible news.

We’ve changed servers, of course. This enabled us to do a lot more, performance wise. We’ve been looking around for extra stuff we could do. Jey’s been doing an awesome job at trying to keep up with us messing around, but we went too far.

I pushed the server hamster too far, and it’s gone berserk. It’s now chanting about the end of times, and somehow I think it’s right. We have 2 weeks left, on the 16th of May the hamster will turn into a destructive force we have never encountered before. I tried to save us, but… Our universe is ending. The team’s done the best they can, but we can’t fix it. I’m sorry, everybody. It’s been a nice universe, and I hope we’ll survive the blast and perhaps find civilization in a next dimension.

 

Alright, now to make sure everyone gets it:

Universe reset Saturday the 16th of May at 1PM Eastern Daylight Savings Time. The server downtime will hopefully be kept to a minimum.

Weekly Dev Notes – 29th April

Bloggy

Wow, the third week already!

So, who’s been doing what?

Jeff

Jeff has renewed the server payment. It looks good so far!

Our game page hasn’t been reviewed yet by Steam.

Blue Dwarf

Blue Dwarf has been cleaning up the code for BobboBot, which is our automated IRC bot.

Jey

Jey’s nearly done with setting up the livetest on the website.

Due to a lot of crash fixes, he hasn’t been able to push the update through yet. It’ll be on livetest asap, after which it can be done on the live server.

Suit

I got lots of feedback on the previous weekly dev post regarding the palace bug.

The problem being, with such a widespread use of this unintended feature, it’s nearly impossible to find everyone who has used this.

Ryan

Ryan has been fixing website issues related to the server migration. He’s trying to fix the leaderboard updating.

Other stuff

Steam beta after live patch! Yay!

4.23.2015 Server Migration downtime

**Update** The biggest part of the migration is now done, the web and game should be fully functional, but the migration is still in progress in parallel so expect a few hiccups here and here.

the new irc is accessible using irc.starsonata.com port 7777
or
kiwi irc
or
mibbit irc

As mentioned in http://www.starsonata.com/blog/new-servers-migration-soon/ we have been working on new server hardware / location to increase performance on the webserver/database/test servers and in doing so found upgrade for all our machines. The downtime will begin at 11h00 pm ET (server time) and will last about 8 hours

I have been doing just about everything that could be done without causing downtime to shorten / facilitate the final transition. But a final downtime is still needed to move the dynamic data (forum data, game database), to update the dns and tie everything together on the new servers.

During that downtime, the website will be switched over to a maintenance page. Forum, wiki and blog will all be disabled to allow for a clean transfer. The game server will also be stopped to allow for the database and website communication interface to be transfered without problems.

Website and other secondary services will come back up as they become available. The game server will be the last thing to come back up and in all likelyhood won’t happen before those 8 hours are elapsed.

Weekly Dev Notes – 22nd April

BloggyAnother week, another blog!

So, who’s done what?

Blue Dwarf

He’s been swamped in coursework and other tasks from university this week. Sadly, he couldn’t do much in Star Sonata.

Jey

Jey has been busy fixing crashes on our live and livetest server.

Also, for our new server some updates! A lot of technical talk, but it comes down to this:

More automated stuff for backups/restarts, simpler commands and a better watchscript for restarts. Also, a new test server which  automatically updates/compiles.

Ryan

Ryan has been working on the webserver support for Steam checkout. There was also a bug with the Emperor’s Palace which has been fixed now. Furthermore, Ryan’s been discussing an interesting new concept with Yclepticon. I’m sorry, it’s a surprise! More about this in another blog.

Suit

I’ve been looking at possible droprates for the GB/NF player versions. What’s the droprate? A mission? A ship? I can’t say! You’ll see it when it’s implemented.

Jeff

Busy with Steam. Our Steam page is up for review at this moment.

Any other discussions?

We’ve had a few interesting discussions this week. I’ve summarized them for you guys.

Team score and BvB

Since the team scores are quite high to be able to BvB, we’ve had a discussion on lowering the team score required to be able to BvB. We’ve been thinking of a team score of 50 to be required.

Team skills

Similar issue as with the team score system. Since there are many strong teams, new and lower teams are having difficulty climbing up the ranks to become stronger and better. We’ve been thinking of the possibility of a new system which is less linear than the current one.

And we’ve had a short discussion on building indestructible bases and running for Emperor as devs. But I don’t think I should put that in here, I guess.. Oh well.

 

That’s it for this week! I’ll be back next week with more interesting news and discussions. This weekend I might be able to high five any Irish players around, since I’ll be visiting there friday-monday.

See you next week!

Weekly Dev Meeting – 15th April

Bloggy

Hey, everyone!

As I love informing you guys on everything, I’ve asked the others from the dev team if I could post information from our weekly meetings (yes, we have those!)

Whenever possible, I’ll make some notes on the weekly meetings and I’ll update you guys on what’s been done, and what’s being planned in the near future (except for surprises, because, you know… surprises).

Anyway, we’re off!

So, who’s done what?

Blue Dwarf

Blue Dwarf’s stopped easter. (Boo!)

He’s also worked on some achievement bugs for Steam. He’s working on implementing achievements such as building a base, etc.

Ryan

Ryan has been working on the payments through Steam, to allow you guys to buy both spacepoints and subscriptions through Steam. He also figured out a bug with subspace spawners that were causing hundreds of ai to spawn an hours and fixed it. Before the fix there were over 1606 ai ships in a gal!

Jey

Jey’s been working on the integration of the new server, doing a lot of testing of it all. It looks very positive. Quoting him, the database is “swimming in ram”, the server should hold our live + test servers quite easily.

He’s going to do a full patch, hopefully friday or monday (in the best case scenario. If he encounters any issues, this could be delayed). If all goes well, the server will be migrated the following day.

Jeff

Jeff has been working on banners for Steam. Here’s a few of them! These are some first versions, we’ve been discussing on which ships we should use for this. They should be interesting, but not too high tech (so players can pilot the “banner” ships quite quickly in the game). They also shouldn’t be too dark, since it’s often against a dark background so the ships would get lost in it.

Suit

I’ve been… Making this blog, I guess? I have no idea.

I’ve been playing around ingame, spawning stuff for people in the Colosseum, which was really fun. I got to talk to a lot of you guys, you all gave me some really nice input and/or feedback which I’ve also discussed with the team. I’ll definitely try and be ingame more often.

 

So, that’s it for this week! As I’ve said ingame, after the server migration, we’ll be able to hand out some Steam beta keys. I’ll keep you updated on this.

See you next week!

New servers / migration soon.

As a few may have heard, we have been investigating the possibility of changing provider for our servers. This did not come from our current ones being insufficient, although the webserver was rather overloaded.
But instead came from Jeff mentioning he saw some Hacker News post praising a datacenter company in Quebec called OVH.

After that point, I had a look at what they offered, prices / specs and multiple reviews I could find on the internet.
The price for the specs were… pretty darn impressive, to get machine a bit stronger than what we currently have, we end paying about half our current price but, specs arent everything and the price felt too good to be true.




None the less, we ordered a machine for a week so that we could test the quality of their network and hardware when compared to our current machines.

on liquid we currently have

  • www:
    • AMD Phenom(tm) II X6 1055T Processor (6×2.8GHz)
    • 16 gb ram ddr3 1333MHz
    • 300gb sas raid1 (15k rpm) for the database/webserver
    • 1Tb 7200rpm sata drive for the immediate backups (theres 2 other levels of backups hosted externaly, this is just for the quick every 30minutes backups)
    • 100mbit connection
  • liberty:
    • AMD Opteron(tm) Processor 6128 x4 (32x2GHz)
    • 16 gb ram ddr3 1333MHz
    • raid 1 sata 7200rpm for the main os / source code
    • 1x ssd drive for the universe save (reduce the lag impact from the save to <2ms instead of the old 130ms that often occured on regular harddrive).
    • 1gbit connection (to ensure minimal latency to the game at all time even with a lot of players)

the machines we are getting on ovh are

  • www:
    • Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz (6cores with H-T 12threads)
    • 128 gb ram ddr3 1600MHz
    • 300gb ssd raid 1 for the database/webserver
    • 2tb sata raid1 7200rpm for the immediate backups
    • 500gb of external backup storage (free with most dedicated servers at ovh)
    • 1gbit/500mbit out unlimited connection (1gbit burst up to 10secs)
    • ddos hardware protection from 100gbit/sec to 500gbit/sec depending on the specific
  • liberty:
    • Intel(R) Xeon(R) CPU E5-2670 v2 (20×2.5ghz with hyperthread so 40logical threads, turbo 2.9ghz and maintained at low temperature)
    • 256 gb ram ddr3 1600MHz
    • 3x ssd 300gb in raid1 for the os install and a partition in raid0 for higher performance uni save
    • 500gb of external backup storage (free with most dedicated servers at ovh)
    • 1gbit/500mbit out unlimited connection (1gbit burst up to 10secs)
    • ddos hardware protection from 100gbit/sec to 500gbit/sec depending on the specific





Now, on paper this mean that the website should be a lot more responsive once we switch.
The database will have so much ram available for caching that it should be a world apart from our current configuration, not to mention the cpu being by far superior and so is the storage on which the database will be.
What this also mean for the webserver, is that we will be able to run test servers with full universe without impacting performance all that much.

Currently test server is rather limited on full uni testing, it cannot have both livetest and test running full uni at the same time or the memory drops too low and the database ends up affected.
With this new server, we should in theory have no problem with 2 or maybe even 3 -4 test servers (private test servers for the content devs to run their own separate player related tests etc).
Offcourse cpu is limited, but servers with 1-2 players on them use very little cpu so theres generally no issue here for a test server.

For liberty we are looking at an improvement of a little over 250% performance all around the board. There is a slight drop in the number of physical cores (from 32 to 20) but the performance per core is quite a bit faster.
Both in lag scenario (1500ai fighting in a single galaxy) and the current uni without players, the performance was always 2-3 times faster on the new server compared to the current liberty.




Now for the part the specs and paper rarely properly cover, the connection reliability. As some of you know, it’s been a constant issue over the years.
Some people unable to connect even tho others can, part of the world losing access and need to rely on proxy / gateway temporarily, frequent disconnection even tho their internet never lost the connection as far as they can tell etc.

It is extremely hard to properly test for those things due to the apparent randomness involved, so I came up with a test that I think should catch it as best as possible to allow for comparison.
No server is perfect and our network code is extremely fragile (its been a problem for a long time, the slightest corruption or interruption is often enough to cause a full disconection), therefor having a network that experience less of those interruption sound like a good improvement

Now for the specifics, I split the tests in 3 categories, each with their own frequency. All tests were ran on both liberty and the test machine we took on ovh for 5 days.

  1. A ping test: Every second, each server pinged 49 locations (7 per test regions) around the world looking for packetloss / disconnection as well as latency.
  2. A route test: Every 5 minutes, each server took a snapshot of the route used to reach all 49 locations to compare with the previous and note route change.
  3. A bandwith test: Every 2 hours, each server ran a speed test using the speedtest.net command line tool to 21 (3 per test regions) different speedtest nodes spread around the world.

The ping test has up to 1 second error per grouped packet loss (1 packet loss is 0-1 sec, 2 packet loss in a row is 1-2 sec, 3 is between 2-3 secs etc.). Downtime that were identical to both liberty and the ovh server were removed (the target was likely down and not the route to it)
the new server had a total all around disconnections between 397 and 1489 seconds out of 432000 seconds tested (0.0009% to 0.0034% off the time when not all the international routes were functional)

  • Africa Central: 193-835 seconds down (249ms)
  • Asia: 112-154 seconds down (195ms)
  • Australia: 32-157 seconds down (203ms)
  • England: 27-117 seconds down (97ms)
  • Germany: 23-106 seconds down (85ms)
  • US south west: 6-78 seconds down (72ms)
  • US east: 4-39 seconds down (19ms)

current liberty had a total all around disconnections between 866 and 2309 seconds out of 432000 seconds tested (0.002% to 0.0053% off the time when not all the international routes were functional)

  • Africa Central: 393-1135 seconds down (280ms)
  • Asia: 212-468 seconds down (195ms)
  • Australia: 74-235 seconds down (207ms)
  • England: 87-178 seconds down (120ms)
  • Germany: 73-162 seconds down (105ms)
  • US south west: 12-81 seconds down (60ms)
  • US east: 15-50 seconds down (23ms)

Interestingly enough, the ping on the server based in Quebec, is lower on average (with the exception of the worst case scenario and that would be the US south west), this was not expected.
But it can be explained due to the effort OVH put on their network, they have a lot of private routes / deals with other providers to ensure good connections whenever possible (which is along other things, needed for their impressive ddos protection).
All in all, the downtimes were a lot shorter on the OVH network than on liquid’s.




the second test, theres a little too much data to be put in a way that would remain clear in a post (most routes tested are international, and imply 10-20 gateways), but there was a pretty clear difference in the routes used by each server.

The current liberty changed route on average twice per hour, rarely major change, just a different gateway here and here. Those change often happened at the same time a disconnection was found during a ping, but not always.

On the new server, there were very few actual route changes, 4-5 per day with the exception of central Africa, which saw about 2 every hours as well.




The third test was the bandwith test. While bandwith doesn’t equate to performance when you’re looking at a video game (you dont need 30mbit to play ss, especially not on download) it does give a somewhat good representation to the link quality and load, as a rule of thumb, if a link let you transfer faster, it mean its able to take a whole lot more load.
Keep in mind that while this is the average bandwith speed obtained using speedtest cli, a lot of those ended up thottled at one point or another along the line because it was only 1 source. When doing multiple different tests in similar location at the same time, the total bandwith nearly always approached 700/400 DL/UL

The current liberty network average for each regions DL/UL MBit/sec

  • Africa Central: 97/38
  • Asia: 84/24
  • Australia: 138/96
  • England: 157/43
  • Germany: 165/52
  • US south west: 443/319
  • US east: 409/294

The new server network average for each regions DL/UL MBit/sec

  • Africa Central: 122/44
  • Asia: 124/74
  • Australia: 238/126
  • England: 357/213
  • Germany: 365/252
  • US south west: 343/189
  • US east: 705/494





In a nutshell, trying as I may to find a problem with the new server location, I could not. All I could find were more reasons on why we should switch over. Altho liquid do have their incredible support team going for them (never had problems fixed as fast as with them).
OVH gives us quite a fair bit more power over the hardware which should normally greatly reduce the need to rely on the support (beyond hardware repairs).
Funnily enough this is something quite a few of the review on the internet actually complain about, their control panel is compared to most other services I tried, complex and dangerous.

The OVH server end up being way cheaper than our current ones, an order of magnitude faster and have a more reliable connection as far as I could test.