Lots and lots
*** dev blog october 2nd 2014 ***
This is more of a post-action debriefing, let me just note down some facts about the current things going on @ wiki central.
While I was absolutely happy about the new players development I adressed in my last blog post and was down to implementing it in the upload process, another incident crossed my plans:
On the well visited sunday night of september 24th, the wikiloops server went down the hard way and didnt come back up for full 45 minutes - so I started rotating and calling for the server side support team to find out what had happened.
Now, thing is, we are up to day using a pretty cheap so called "shared hosting" account, so server support is limited and I actually can not access all of the backend myself, so anytime I need some insight on what is going on on the database side of the loops, I have to beg for some information.
This time, there was little insight in the support themes reply - it basicly read: "The server overloads because of too many database requests. We suggest you move your project to a more advanced server, which we will be glad to offer to you. Pricing starts at eight times your current monthly cost."
Now, that wasn't helpfull really, and such a cost explosion would endanger the future f this project right away.
So I dropped all fancy waveform inclusion and launched a quest for the cause of the overload. There hadn't been that many more visitors on the 21th than usual, so I didnt really believe it was a "too many users" issue, but obviously something struck the databse with so heavy requests that it stood still at some point.
So I started reducing load, first by throttling the shoutbox and notification-update requests, later by applying a "sleep mode" that will disable some functions the moment you switch to another browser tab. Days went by in the process, and while the server support guys rather told me to either find the cause or move to the expensive server, I was really started to dread the next coming weekend.
But for some reason, all of the mentioned load reduction concepts didnt bring the effect I was looking for. The server didnt go down again, but it still had very slow response times every now and then, so I had to assume the load issue was not resolved by any of my actions.
In the next step, I moved a huge part of the files called within any page load (like pictures) away from the wikiloops server and onto a Network of global servers - that way the number of file requests to the wikiloops server is reduced to pure code and the music files, and the avatars you are looking at may be served from around 16 other machines located somewhere between Frankfurt, Germany and LosAngeles, USA (there was an "+Asia/Autralia" package as well, but I didnt dare to spent the additional 200$ at this point).
Good thing to do, but ... it didnt resolve the issue either.
After a streak of twelve days of taking code apart and looking at all kinds of functions over and over again to make sure no unnecessary database querys were taking place, I finally found it - the search results page was causing so much database load, it took 30 seconds to show me the latest jams on my local development server, and since the latest jams are a quite frequently visited page, chances of this overloading the server are pretty good.
So I applied a quick fix, and shwoops, it only takes 600ms to show the very same results now.
Even tho I have no access to the database load log, I am pretty confident we should be back to an acceptable behavior, even on a shared server.
So, yes, we seem to be back in the safe zone (if next sunday wont prove me wrong), but I will need at least another week to implement all of the other potential load-lowering concepts I spottet on my quest.
No, thats not all.
I had to consult a tax counselor today, because the german tax authorities are getting more and more interested in what it is the "wikiloops media UG" does. I'm no trained accountant at all, so I really need help with all that paperwork (which is filling whole folders by now, even tho I have never actually made one $ in three years!). To make a long story short, the pressing issue could be solved as well, but after all I have been told today, I feel there is no way around giving the accounting into more experienced hands in the long run, because I really have to lower my work load at some point and am lacking the needed knowledge (german taxation is the most complicated worldwide as far as I have been told).
It is a relief to let someone else do that.
It will cost another 150 $ a month, tho. Thanks to the help of the supporting members, this is coverable as long as we dont need the dreaded server upgrade, which is a matter of time, really.
Yes, thats whats going on, so sorry to all of those who may have been awaiting my response to messages or the great initiatives going on around the wikiloops forum lately - it will be some more time untill I will be back at those things myself.
If anyone reading this ends up down here thinking "wow, and I didnt notice ANY of this taking place, and I have been to wikiloops quite often in the last two weeks!" - well, its my pleasure to be at your service, if you feel I should get a free coffee, you know where the donation buttons are...
I'm optimistic the next dev blog will feature some more positive news, heres todays stats for record keepings sake:
17.014 members,
21.445 tracks online
This is more of a post-action debriefing, let me just note down some facts about the current things going on @ wiki central.
While I was absolutely happy about the new players development I adressed in my last blog post and was down to implementing it in the upload process, another incident crossed my plans:
On the well visited sunday night of september 24th, the wikiloops server went down the hard way and didnt come back up for full 45 minutes - so I started rotating and calling for the server side support team to find out what had happened.
Now, thing is, we are up to day using a pretty cheap so called "shared hosting" account, so server support is limited and I actually can not access all of the backend myself, so anytime I need some insight on what is going on on the database side of the loops, I have to beg for some information.
This time, there was little insight in the support themes reply - it basicly read: "The server overloads because of too many database requests. We suggest you move your project to a more advanced server, which we will be glad to offer to you. Pricing starts at eight times your current monthly cost."
Now, that wasn't helpfull really, and such a cost explosion would endanger the future f this project right away.
So I dropped all fancy waveform inclusion and launched a quest for the cause of the overload. There hadn't been that many more visitors on the 21th than usual, so I didnt really believe it was a "too many users" issue, but obviously something struck the databse with so heavy requests that it stood still at some point.
So I started reducing load, first by throttling the shoutbox and notification-update requests, later by applying a "sleep mode" that will disable some functions the moment you switch to another browser tab. Days went by in the process, and while the server support guys rather told me to either find the cause or move to the expensive server, I was really started to dread the next coming weekend.
But for some reason, all of the mentioned load reduction concepts didnt bring the effect I was looking for. The server didnt go down again, but it still had very slow response times every now and then, so I had to assume the load issue was not resolved by any of my actions.
In the next step, I moved a huge part of the files called within any page load (like pictures) away from the wikiloops server and onto a Network of global servers - that way the number of file requests to the wikiloops server is reduced to pure code and the music files, and the avatars you are looking at may be served from around 16 other machines located somewhere between Frankfurt, Germany and LosAngeles, USA (there was an "+Asia/Autralia" package as well, but I didnt dare to spent the additional 200$ at this point).
Good thing to do, but ... it didnt resolve the issue either.
After a streak of twelve days of taking code apart and looking at all kinds of functions over and over again to make sure no unnecessary database querys were taking place, I finally found it - the search results page was causing so much database load, it took 30 seconds to show me the latest jams on my local development server, and since the latest jams are a quite frequently visited page, chances of this overloading the server are pretty good.
So I applied a quick fix, and shwoops, it only takes 600ms to show the very same results now.
Even tho I have no access to the database load log, I am pretty confident we should be back to an acceptable behavior, even on a shared server.
So, yes, we seem to be back in the safe zone (if next sunday wont prove me wrong), but I will need at least another week to implement all of the other potential load-lowering concepts I spottet on my quest.
No, thats not all.
I had to consult a tax counselor today, because the german tax authorities are getting more and more interested in what it is the "wikiloops media UG" does. I'm no trained accountant at all, so I really need help with all that paperwork (which is filling whole folders by now, even tho I have never actually made one $ in three years!). To make a long story short, the pressing issue could be solved as well, but after all I have been told today, I feel there is no way around giving the accounting into more experienced hands in the long run, because I really have to lower my work load at some point and am lacking the needed knowledge (german taxation is the most complicated worldwide as far as I have been told).
It is a relief to let someone else do that.
It will cost another 150 $ a month, tho. Thanks to the help of the supporting members, this is coverable as long as we dont need the dreaded server upgrade, which is a matter of time, really.
Yes, thats whats going on, so sorry to all of those who may have been awaiting my response to messages or the great initiatives going on around the wikiloops forum lately - it will be some more time untill I will be back at those things myself.
If anyone reading this ends up down here thinking "wow, and I didnt notice ANY of this taking place, and I have been to wikiloops quite often in the last two weeks!" - well, its my pleasure to be at your service, if you feel I should get a free coffee, you know where the donation buttons are...
I'm optimistic the next dev blog will feature some more positive news, heres todays stats for record keepings sake:
17.014 members,
21.445 tracks online
nuno1959
I couldn't agree more with Lutz here !
ANYTHING BUT allow Wikiloops to run ''ion the edge'' - am also curious about donation flow to make any necessary adjustment
Pity i REALLY know NOTHING about coding/programming, etc…. otherwise i'd gladly give a bunch of hours per week or so +0
ANYTHING BUT allow Wikiloops to run ''ion the edge'' - am also curious about donation flow to make any necessary adjustment
Pity i REALLY know NOTHING about coding/programming, etc…. otherwise i'd gladly give a bunch of hours per week or so +0
Lutz
I will never really imagine all the issues that come up with a project like this!
Hope you find a way to keep Wikiloops slim enough - the new features, may they be fantastic for the user - should never put the whole in danger. Better to abandon an element that seems to be too costly.
Most important : TAKE CARE for yourself, Dick, you are the heart of it.
Also I'd like to know how the "supporting member" thing is working and if we do approach the goal. We should know, be it now or at least at the end of the year, if our regular donations do what is needed and if the amount that flows to Wikiloops is sufficient or not. We can then adjust our commitment.
Take care (of yourself!) +0
Hope you find a way to keep Wikiloops slim enough - the new features, may they be fantastic for the user - should never put the whole in danger. Better to abandon an element that seems to be too costly.
Most important : TAKE CARE for yourself, Dick, you are the heart of it.
Also I'd like to know how the "supporting member" thing is working and if we do approach the goal. We should know, be it now or at least at the end of the year, if our regular donations do what is needed and if the amount that flows to Wikiloops is sufficient or not. We can then adjust our commitment.
Take care (of yourself!) +0
OliVBee
i'm really glad you were able to find and fix the server load issue ... this really is a crucial topic in my books because i don't think wikiloops is ready yet to go into the next step of its development : a costy web server.
Thus i couldn't encourage you more to track down and apply all the fixes you can come up with on this topic ! wikiloops still needs the low cost hosting until the financial side of this project can absorb any other kind of technical plateform !
On the paperwork side i can only hope a well born soul knowing german accounting could help ... could probably spare a fair amount of $ which could probably go into anything but the pocket of this guy ... but again i very much agree that this side of things has to be taken care of as well ! and you already have so much on your shoulders ... i am trusting your judgement anyway ! (and if its a relief for you then be it !!)
Relax for a bit buddy !! give yourself some room to breath ;) DON'T FORGET TO HAVE FUN !! +0
Thus i couldn't encourage you more to track down and apply all the fixes you can come up with on this topic ! wikiloops still needs the low cost hosting until the financial side of this project can absorb any other kind of technical plateform !
On the paperwork side i can only hope a well born soul knowing german accounting could help ... could probably spare a fair amount of $ which could probably go into anything but the pocket of this guy ... but again i very much agree that this side of things has to be taken care of as well ! and you already have so much on your shoulders ... i am trusting your judgement anyway ! (and if its a relief for you then be it !!)
Relax for a bit buddy !! give yourself some room to breath ;) DON'T FORGET TO HAVE FUN !! +0
wikiloops online jamsessions are brought to you with friendly
support by:
Wikiloops brings musicians from around the world to my home. How would we otherwise ever have the opportunity to play with all of these great musicians?
Wade