Below is a real-time overview of our servers where you can check if there's any known issues.
| Server Name | HTTP | FTP | POP3 | PHP Info | Server Load | Uptime |
|---|---|---|---|---|---|---|
| wtscpanel.whattheserver.com | PHP Info |
Affecting Server - coby
The server is scheduled to have its CPU replaced for an upgrade and we will update this once completed.
Update:
CPU upgrade completed successfully :)
Affecting Server - coby
We are currently aware of the current VPS issue with the VPS on Coby. The server is having its CPU replaced currently and we will update this once completed.
Update:
Server CPU upgrade was unsuccessful so the old CPU was put back in for now.
All services should be back online now.
Affecting Server - coby
We are currently experiencing issues with the Coby VPS node in NL and are working with the Datacenter to get the server back online.
We will provide another update once the server is back online.
Affecting Server - wtscpanel.whattheserver.com
We are updating cPanel/WHM to v78 and do not expect any service interruptions except a possible quick reboot afterwards if needed.
Affecting Server - coby
We're currently investigating a hardware issue with the main VPS node Raid Controller.
We apologize for the inconvenience and are working with the Datacenter to get the hardware checked and replaced.
Update:
After the Raid card was reseated and 2 reboots looks like everything is coming back online without issues
Affecting Server - coby
We are currently investigating a DC outage in NL which affects the VPS nodes and Shared hosting Services. We will update as soon as we know more.
Update: DC is performing emergency Maintenance on core router and should be fixed soon. We apologize about the inconvenience.
Update: Connectivity has been restored and were doing a quick reboot to ensure Proxmox hosts nodes are booted into the latest kernel which we have been putting off for awhile.
Once rebooted they should all be back to normal. Shared hosting should be back online already
Â
Update everything should be online
Affecting Server - coby
Noticing some kernel panics repeating on the VPS's. Rebooting node to prevent any further issues with the kernel since it has been awhile since its been rebooted. We use kernel care but looks like it does require a graceful reboot.
Affecting System - Affects VPS servers shared cpanel and vpn1,vpn2 nodes
As mentioned in the 5 emails i sent starting weeks ago we are moving servers from private rackspace to the public rackspace and upgrading the bare metal OS to latest proxmox and overhauling the system. Hopefully you have taken backups as advised as most likely your going to need a fresh install.
be working on auto dhcp for vps containers so that hopefully when installing from isos in future your network settings will be assigned via dhcp vs needing to be inputted manually
do not expect any issues but putting up notice just in case anyone notices an issue so you know were aware
Affecting Server - Backup FreeRADIUS
We will be upgrading all vpn servers today so each will go down for about 20 mins. We will be starting at vpn01 and working towards vpn09 in that order.
Just a heads up so you know to try another node if you happen to be connecting/reconnecting to one and it fails.
Mik Ram
What The Server CEO/Sys Admin
were aware and investigating crashed node
Were investigating a raid failure with this node no eta when it will be back up.
trying to recover/repair froze and were having trouble getting node booted up
Affecting System - codyl vps node 5.255.87.133
We will be taking this node offline for hardware upgrades(ram+ sdd) and then redeploying it with proxmox.
We have sent out many notifications over last few weeks encouragin clients to take and maintain there own backups as well to be on the safe side in the event the backups we took do not work for whatever reason.
Do not have an exact eta on length of time this will take as of yet will update here as we know more.
Currently looks like the cpanel server is unrecoverable and we will have to reinstall the OS from scratch if we decide to resetup the service. As it was a very old and since discontinued service we will be phasing it out entirely now vs how we had it with signups that were just disabled for last year or so.
We do have some backups for some accounts albeit most are pretty old so we will be sending people an email with there backups as applicable.
Affecting Other - VPN AUTH SERVER
Tomorrow morning there will be a period of 1HR where you will not be able to authenticate to login so we advise to not switch nodes during tomorrow morning till the afternoon to prevent any interruptions.
We are making some changes to the auth system due to a recent failure with our provider who had our failover so were cloning the auth server and setting it up again in a geographically different area to provide better redundancy
Thanks
We are moving are hardware to a new Datacenter at 11am CET time and this will last about 4-6 hours(Hopefully less).
All NL based services will be affected with exception of those whose VPS's are on Sasha node.
This will affect shared/vps/vpn clients.
If your a VPN client just use our non NL based nodes as they will be working fine without issues during this time.
If your VPS client and not on Sasha node you will be offline during this time and will be getting a new IP. Your new IP is already assigned to your container and we mentioned where to see that in our earlier emails to all VPS clients this affected. Once we bring the nodes online at new DC we will have to change the new IP as your primary ip and just remove the old ones and reboot your container after that all should be working again on new IP.
If your curious where were moving to > http://serverius.com/services/ip-transit-services
If you have any questions please submit a ticket and we will answer your questions as fast as we can keep in mind it will be busy so might be slightly delayed compared to our usual response times.
Mik Ram
Were working on correcting some issues on our Codyl Node.
Datacenter is going to be starting work on the RAID array here shortly.
Will post updates as we know more.
Update:
Awaiting hardware support technician to compete diagnostics and repair of the RAID array.
We are very sorry for the incoveniences this has caused and assure you we are doing our best to resolve the issue ASAP.
Update:
Appreciate the patience for the long wait. Raid issue has been resolved with no dataloss thankfully.
Please let us know if you are experiencing any issues.
Affecting System - NL based Services
Were looking into this. Will update as we know more.
We have been working on this and will update this when its been corrected all vps are currently online are unaffected this just affects your ability to manage them via the control panel.
Affecting Other - Affecting all NL based services
Checking into this. :/
Still unable to reach the Datacenter there support panel is offline. :(
Seems multiple other providers experiencing the same issues at this time.
http://www.webhostingtalk.com/showpost.php?p=8723948&postcount=1
They have updated it saying
"Ecatel Offline 6-12-13
We are sorry for not posting earlier in this topic. There is a problem with fiber equipment at this moment. Because of that a big part of our betwork is offline. Techs are working on a solution. Our wel mean apologies for this matter. It will be solved ASAP."
Update: ALL seems to be online at this time.
We had to do emergency patch on host kernels so servers are being rebooted.
Affecting Other - All NL Based Services
Another Emergency Maintenance Message from our Datacenter :/
"Dear Customer,
As a valued customer, Ecatel is committed to keeping you informed about any
changes in network status.
Date: 30/05/2013 - 31/05/2013
Start time (CET): 21:30
End time(CET): 03:30
Network impact: Some interruptions
INFORMATION
Last performed maintenance (28-29-05) was not completed succesfully due to broken hardware. Because of that
we had some delays and longer downtime during the network maintenace. Our well mean apologies for this.
Because last maintenance did not complete succesfully we are forced to perform one FINAL maintenance again on the network and
complete our work. This maintenance will cause a interruption on your line again. We do not expect any problems this time!
After this planned work we do not expect any (un)planned work or issues for a long time.
Our apologies for this late notification but we had to schedule this maintnence
at the earliest to keep our network stable.
If you have any customers running on your servers, please make them aware of
this maintenence because there will be some downtime.
If you have any questions regarding this e-mail, please contact noc at
ecatel.net or info at ecatel.net and we are happy to assist you."
UPDATE: Should be done but not seeing services online yet they must be running overschedule. :/
UPDATE:Services look to be coming online now. Let us know if your experiencing any further issues.
Time converter at worldtimebuddy.com
Affecting System - All NL Based Services
From our Datacenter
"
Date: 28/05/2013 - 29/05/2013
Start time (CET): 21:45
End time(CET): 03:00
Network impact: Some interruptions (some short and some longer).
INFORMATION:
In our last maintenance we have not yet completely solved some (partial) network issues
with our fiber equipment. Because of that we have to replace equipment again with a
different configuration.
This maintenance will take some time and cause some downtime for dedicated and colocation customers.
Our apologies for this late notification but we had to schedule this maintnence
at the earliest to keep our network stable.
If you have any customers running on your servers, please make them aware of
this maintenence because there will be some downtime.
If you have any questions regarding this e-mail, please contact noc at
ecatel.net or info at ecatel.net and we are happy to assist you."
Update: The period they scheduled this for is over but not all services are online at this time yet. Awaiting there reply on updates why its gone over the alloted time. :(
Time converter at worldtimebuddy.com
Solusvm control panel severe packet loss.
Waiting on dc to answer regarding this issue
This is email i received from our Datacenter wanted to let you know as well about the maintenance in datacenter tomorrow.
"As a valued customer, Ecatel is committed to keeping you informed about any changes in network status.
Date: 17/04/2013 - 18/04/2013
Start time (CET): 23:00
End time(CET): 03:00
Network impact: Some interruptions, max 180 minutes.
INFORMATION:
Because of some urgent (partial) network issues we have to replace fiber equipment and one of our corerouters with a new management blade, a new configuration will be deployed on the corerouter aswell. This maintenance will take some time and cause some downtime for dedicated and colocation customers.
Our apologies for this late notification but we had to schedule this maintnence at the earliest to keep our network stable.
If you have any customers running on your servers, please make them aware of this maintenence because there will be some downtime."
Heres a time clock that should show you that timeblock for maintenance in your native timezone for easy reference.
http://bit.ly/15e2sac
This would also effect our NL based VPN and Perfect Dark VPN nodes so use our SE RU RO nodes till the maintenance window is complete
Let us know if you have any questions or concerns.
Time converter at worldtimebuddy.com
Affecting Other - BIlling
The license server for the module is offline and so until they bring it back online we will need to accept manual bitcoin payments.
If you need to pay via coins just send in a ticket and well get u a custom addy to send too.
Affecting Server - Backup FreeRADIUS
Were updating VPN aka vpn1 so please utilize the other nodes till the maintenance is complete.
Were also looking into reported issues with VPN2 will be rebuilding them both this weekend.
Please continue using vpns 3,4,5,6,7 in meantime.
Update:
Was able to correct the issues without a rebuild for nodes 1/2 and both are back online now and fully operational.
Were currently investigating this.
Affecting Server - Backup FreeRADIUS
VPN Auth server is undergoing short maintenance window. Intermittent over next few hours. Please leave current sessions connected to prevent not being able to resign on issues during this time.
Thanks
Affecting Server - Backup FreeRADIUS
VPS node 6/PD3 is experiencing connectivity issues and is being investigated and possibly rebuilt if need be.
Affecting Server - Backup FreeRADIUS
VPN Auth Server Emergency Maintenance Please Standby
Raid Array for node auth server resides on is still rebuilding atm progress 53%
Shared hosting is offline and it does appear there will need to be a server rebuild.
Standby for updates.
Server rebuilt. Online
Were having IP routing connectivity issues with a few ranges of our IP's. Please Standby while we investigate this.
DC corrected the issue
Server Issues were due to ongoing DDoS have been mitigated via black-holing
Clients responsible have been suspended.
I apologize for the inconveniece but our newer node Codyl is temporarily offline for Emergency Mainteance.
Please Standby,
We have noticed some clients having slight timeout issues and high pings on our VPS nodes over last few days. We have contacted the Data Center and they have isolated the issue and working to resolve it ASAP.
Hello all we once again apologize our VPS and Shared Hosting and NL VPN nodes are offline due to Ecatel(Datacenter) core-router being down.
http://noc.ecatel.net/ecatel-network-status.php
|
Below is information regarding the short scheduled network upgrade/maintenance
Date: 27/09/2012 - 27/09/2012
Start time (CET): 00:00
End time(CET): 00:05
Network impact: Short interruption
INFORMATION:
Because
of a MAJOR growth in network traffic we need to perform a very short
network upgrade. One of our coreblades will be replaced and this will
affect all services for a very short time ( max 4 minutes ).
We apologies for this very late notification. To keep the network running stable this upgrades needs to be performed urgently.
If
you have any questions regarding this e-mail, please contact noc at
ecatel.net or support at ecatel.net and we are happy to assist you.
Best regards,
Network Operations
ECATEL (NETHERLANDS) LIMITED.
Hello due to an issue with Data-center core switches corrupted memory failure on Saturday night we needed to reboot our new VPS node and then we have discovered quite a few people are unable to reach there VPS via ping or ssh. Please first attempt to reboot your VPS via your control panel and retry if that does not fix the connectivity issue please file a support ticket with your VPS ip so we can add it to the list of affected ips.
Thanks for your patience
So far here is a list of affected VPS ips and there status. (Obviously the first parts of ip are obscured by *)
*.163 not working
*.164 working
*.165 working
*.166 working
*.167 not working
*.168 not working
*.169 not working
*.170 not working
*.171 not working
*.172 not working
*.173 working
*.174 not working
*.175 not working
*.177 working
*.178 not working
*.180 working
*.181 not working
*.182 not working
*.183 not working
*.184 not working
*.185 not working
*.187 not working
*.188 not working
*.189 not working
*.190 not working
*.191 not working
*.192 working
*.193 not working
*.194 working
*.195 working
Update:
We apologize for the outages connectivity issues etc over the weekend they were not caused by us or our hardware or faulty configurations.
Over the weekend our datacenter (Ecatel) was attacked by group who spawned over 6mil mac addresses and flooded there core switches and corrupted the memory and as a precaution they set a 10 mac address limit on every server. This arbitrary 10 mac address limit affected all hosts like us who have sometimes 50+ VPS with each having its own unique mac address and static IP. We still are unsure why they set an arbitrary number like 10 when they should have just set the exact same number of mac addresses as the servers allotted ip's. This was a very unthought through decision on there part. Due to the limitation this is why the VPS are up but unconnectable as whatever vps came online first got one of the 10 allowed mac address slots and no one else’s could call out or in through the switch.
We are still working with datacenter to get the limit adjusted so this does not happen again and we can get you guys back online.
Seems to be some kind of connectivity issue with our Ecatel Based VPS Nodes and services. Were investigating.
Ecatel support sites are unreachable too atm.
http://ecatel.co.uk/?p=network
We apologize for the inconvenience.
This is a Data center outage not a failure on our end.
VPN Users Please Use nodes 3(Russian) & 5(Sweden) for now.
Update from Ecatel regarding the Brief Outage in there network> http://www.webhostingtalk.com/showpost.php?p=8348336&postcount=10
"hello,
issue has been solved, broken memory has been replaced.
We will perform some major network upgrades within the next month to
prevent future problems and optimize redundancy. Customers will be
informed per email about this.
Our apologies for the inconvenience caused by this matter."
Affecting Server - Backup FreeRADIUS
We understand the importance of minimal downtime when doing maintenance
of any type and for that reason will do what we can to minimize the
overall downtime. As it stands now, we have a 5 hour maintenance window
during which we only expect around 10 minutes of actual downtime for the
switch to reboot and for our VPN Authentication server to become responsive.
Scheduled Date and Time:
September 20, 2012
beginning at 7 PM EDT (5 hour maintenance window)Convert time zones with worldtimebuddy.com
thanks
Mik Ram
What The Server Admin Team
PS:If
you do not disconnect your vpn tunnel or reauthorize by switching to
another node your tunnel should stay open and useable during this time
as it only affects the back end freeradius authentication server for our
VPN nodes when you connect in and it authorizes your login initially
Affecting System - ALL Systems/Services
Our services are currently offline as our Datacenter we have our hardware collocated had outages due to both switch failure of datacenter and fibercuts of upstream providers and due to some attacks by group called "Anonymous" over the last few days.
Datacenter Network Status page for reference: http://noc.ecatel.net/ecatel-network-status.php
Update:(9:19PM -6 GMT Chicago) seems service is restored for the moment.
Official statement from Datacenter
"After 6 months 100% network update we had some major problems.
We very sorry for this matter, 2 days ago a coreswitch failure and now 96 broken fibers because of roadworks.
A
big part of our network went down because of this broken fiber and our
fiber supplier worked day and night to have it fixed ASAP.
The whole network is back online again and running stable, we very sorry for this major inconvenience caused by these 2 errors.
We expect everything will stay stable for a very long time now again!"
Node2 VPS's are having connectivity issues all ips ending with last 3 between this range .163-225
Were looking into it.
Cause
When Ecatel Fixed Switch it apparently forgot to reassign ips correctly and fixed this for us
Affecting System - Affecting all Systems
Were investigating an issue with Datacenter Affecting all Nodes. Unsure of the issue. Trying to reach Datacenter.
Update: Still trying to reach Datacenter(Ecatel) all support sites they have are down and not receiving responses via email. Nothing we can do till they respond.
Update Final: Ecatel core switch failure was cause of the issue Datacenter Outage. Below is the info they sent us.
"Dear customer,
Around 19:00 hours last night the chassis of one of our main core switches failed hopelessly.
We have replaced the hardware and our technical staff have done a
tremendous job getting the situation as it was before this failure. We
offer our sincere apologies for any inconvenience this may have caused
you.
In case you are still experiencing any problems with your server(s) we suggest you reboot your system.
If this does not solve your problem please feel free to contact us at support@ecatel.net
Technical team
Ecatel Ltd.
Best regards,
ECATEL (NETHERLANDS) LIMITED."
Affecting Server - Backup FreeRADIUS
Nodes 2(NL) is under maintenance. Please use node 1& 3 for now.
Affecting System - All Server Nodes
Possible Outage of our nodes were trying to reach Datacenter concerning the issue.
Update: This issue is also Affecting Datacenter's support site so its possible could be Datacenter issue and not ddos attack, but connectivity is hit and miss for both nodes atm.
Second Update: ALL services back and accessible still waiting on information from Datacenter about what happened.
Affecting System - Our system will be temporarily down for a hardware upgrade on July 10, 2012 between 12:00 AM and 14:
Our system will be temporarily down for a hardware upgrade on July 10, 2012 between 12:00 AM and 16:00 PM CET.
Please pardon our progress,
Your What The Server Admin Team
Our system will be temporarily down for a hardware upgrade on July 10, 2012 between 12:00 AM and 16:00Â CET.
Please pardon our progress,
Your What The Server Admin Team
Â
Â
Please Be patient as were working on investigating the issue with datacenter.
update: We are still waiting on Datacenter to adequately fix and respond to out ticket/s. Our server is online but only accessible via IPMI and unconnectable via all outside ips unable to reach gateway were waiting on them to fix the connectivity issues.
second update:We are online now and all back and fixed were working on issuing credit to all
Server Updates and Maintenance to Core OS.
