DNS Resolution Incident
Posted 22 Apr 2014, by Jason Harris
Update: April 22nd 2014 All live-tracking services are back up! Once Easter was over and companies were answering their phones, the first thing in the morning we managed to transfer our domain from our old Christchurch hosting provider to one of the primary domain registrars in New Zealand. Happily, all problems have now been resolved and live tracking has been restored.
April 21st 2014
Well, late last night our automated monitoring software detected a problem with the resolution of the domain trackabus.co.nz. This morning, after getting up early we found out about this problem and started investigating it. It turns out our upstream service provider in Christchurch is no longer resolving our domain name. Even though we can log into the billing area of our hosting provider and we are totally up to date with all payments it looks like there is a DNS problem. Unfortunately, it is Easter Monday and the provider is not responding to either phone calls or the support ticket we have sent them. We are scrambling to find an alternate solution but this problem may persist for the next day or even two.
We had plans to do fail over DNS handling within the next 6 months, but this has now moved up to an extreme priority for us. Since the TrackABus locator units are trying to talk to trackabus.co.nz and the name is not being resolved to our server at 188.8.131.52, it means the units effectively can't talk to our server. In the worst case, we might have to physically visit the locator units on the buses and update the software on them manually.
In any case, we wanted to keep everyone apprised of the situation. Although it is of course bad that these things happen, it is good for us to find these weak points in our overall solution and solve them early on so we can avoid future DNS problems.
In summary, the live trial tracking is temporarily offline due to errors upstream from us. We hope to have our live trials back online within the next day or two.
Jason Harris and Chris Mein