Mitchell Bingemann and Fran Foo | August 07, 2008
FLAWED software patches from Nokia Siemens Networks have crippled Optus's 3G mobile network in Brisbane, the telco said.
Optus says Nokia Siemens provided the telco with faulty software
The upgrade, which has been applied to all Optus 3G mobile switches nationwide, was supposed to provide efficiency gains in speed and capacity load. Instead, it triggered three separate network failures which combined to cause chaos for Optus customers in Queensland, NSW, the ACT and Victoria last week.
As a result subscribers were left without mobile phone services for about 10 hours.
While the upgrade was successfully rolled back in Melbourne and Sydney, the problem again reared its ugly head in Brisbane.
"We had a new patch put in place in the affected areas by midday last Friday but we now have a re-emergence of a similar problem in the Brisbane area," Optus spokesperson Maha Krishnapillai said.
The network in Brisbane was online at 7am but went offline an hour later. At 4pm systems went down again.
"We're not sure how long it will take to fix this problem again but we hope soon," Mr Krishnapillai said.
He said Optus could offer no guarantees that the problem would not recur as it was related to a Nokia Siemens software fault.
"Nokia's latest version of the software was in effect 'contaminated'. We had a bug in the software so we have had to roll it back.
"The issue for us is that we have put the upgrade in other areas and have had no problems whatsoever - so it’s a bit of a lottery."
Engineers from Optus and Nokia Siemens were both responsible for the patch installation.
He said Optus was unsure why the patch had worked in some areas but not others.
"There was extensive testing on both our parts and we were also assured by Nokia Siemens that the software would work in different (network) environments," he said. "But when we went live it was a different story."
"Now we’re just hoping the patch will hold. We can’t say for sure whether or not we will have more issues over the next week in Brisbane," he said.
Nokia Siemens declined to directly address Mr Krishnapillai's claims, saying it valued its close relationship with Optus.
"We're proud of both companies' shared goal of providing a superior mobile service in Australia. Experts from both companies are working in partnership to resolve any remaining network issues, as is always the case when any outage occurs," Nokia Siemens spokesman Ben Roome said.
In the meantime Optus has requested network engineers from Nokia Siemens' Finland headquarters to rectify the problem on-site.
"We have told Nokia to fly out some engineers because we certainly won't be flying them here," Mr Krishnapillai said.
Mr Krishnapillai suggested Optus would be reviewing the service level guarantees it had in place with Nokia Siemens.
"We do have some service level guarantees with them and we will clearly be looking at whether in fact the software they have given us is up to scratch," he said.
Mr Krishnapillai ruled out Optus pursuing legal action against Nokia Siemens or seeking compensation.
"We are not looking to sue them on this, what we are doing is looking to work with them to fix it.
“As I’m sure Nokia would tell you, we are their most important customer in Australia and a very important customer throughout the SingTel group ... they will clearly want to make sure they work with us to fix this problem," he said.
Your Comments:
32 Comment(s)
There are some interesting questions here - which go to the heart of business continuity capability taking a distant second place to haste.
However let's wait for the full story and let's not fall for "love em hate em brand name" traps. Ericsson have always made great black boxes - but the competency of their continuity management culture was hinge in their having to become Sony Ericsson (after the Phillips' New Mexico chip factory fire). The same event that saw Nokia' proactive nimbleness grab global market share. For the sake of Optus's already tarnished reputation (for failure and blame) let us also hope that Nokia do fly in some good thinking and the culture rubs off.
The people who attack Optus can't be genuine, if people think Telstra is expensive now, imagine a Telstra with no competition ! Be afraid, be very afraid !
the broadband in australia is a real big joke! in singapore, i can get 1mbit/s download for S$40 per mth with no cap! and here, having a 4GB cap with 512kb/s with SLA of 50% @$60/mth subscription with 2 year contract is a damn great deal? it is time to implement law to punish these telco, if they cant meet the SLA to the consumer, fine them and fine them Heavily.
'yes' - a company that takes shortcuts 'yes' - network management run by kids with no idea 'yes' - we will retrench staff within 2 months 'yes' - networks staff have low morale 'yes' - thank god Qantas is in the news as well, now I dont feel so bad.
GuruBoy wonder has a point, The thing about Carrier Class or Telco grade services is the 'carrier ' (telco), where's the diligence, value add, carrier good eng practice & rigour ? "It's not our fault" doesn't cut it when you (Optus) are taking the recurring revenue.
Agree with Susan Wong on the Ericsson bit, Optus has tried to cut the bottom line by cutting core quality of our infrastructure, and now has seriously compromised their entire network. This is something thats not going to be fixed easily and I hope the telco ombudsman steps on their necks hard. They not only impact their customers but threaten the competitive market by eliminating themselves as a serious network provider.
Wait.... You mean no one from work can call me on my mobile. Hooray for Optus I'm off home to watch the footy!
Optus is not at fault, yes there have been issues, but at least they were communicating this to customer's rather than not saying anything. Telstra is no better.
I am amazed that all the negative comments here give the impression that no other provider has ever experienced problems when upgrading or day to day operation of a network. How about telstra when they where going through their upgrade, whole towns lost service for days, but because it was in rural areas no mass media beat up. What about Vodafones lack of coverage, again just drive out of the city and see how far you get. Lets cut the hysteria, I agree that the communication from Optus has been rubbish but when Foxtel has gone down for hours or Big Pond cable slows to a walk because of congestion (which I have now) for days these do not make a dent on the news because it is supplied by Telstra and everyone says "thats telstra"
As a small business owner, I am heavily reliant on the Optus mobile and wireless broadband network. The service Sydney on both mobile and HSDPA/WBB has been abysmal - the data service drops out 8-9 times daily, mobile connections fail and I cannot make connection again for 5-6 minutes. Of course, you pay the flagfall for the reconnection when you finally get through. When you have clients waiting on the other end for the call or documents, this does not look good. I will be terminating my contracts and move my business. It is just not worth the time lost.
Optus has a different sort of arrogance in it's management than Telstra. The two should be measured by management arrogance rather than management style. I am so frustrated by both of these organisations and their total lack of concern for what should be their primary purpose - essential services for the nation. Alas no; profit first product third.
Try having no mobile service for FIVE days. No replies to service complaint emails. 'Yes' Optus, Telstra are doing great business this week in our area.
Pretty poor form diverting the blame directly onto a vendor software load. Where is the accountability here? What Tier 1 telco would not test the software themselves and then do a controlled first release, before a general release?. Sounds like they are in a hurry and taking shortcuts
My professional coment for the problem in Optus, again is lack of the Supervision, is not only installations the equipment, also of the conector's, plus the technician test the final equipment and networks.
No complaints here. Always get 400kbps - 600kbps - Virgin
Have had first hand experience of Optus IT budgets being squeezed beyond sensible levels by Singtel, with dramatic loss of morale, key people and stable processes in the business. These performance issues are not a surprise at all.
Go Optus Go! We need you strong and competitive, because what we need is more competition.(and less bickering)
I think they should call Mr Krishnapillai the Teflon Man. At which point will he take accountability for Optus' poor performance and not blame suppliers that they chose. They weren't forced to use Nokia. It might explain why over 50% of the top 60 directors and general managers have left in the last 24 months. Something of pandemic nature is happening in their North Ryde headoffice to cause all the senior staff to leave. Mr Krishnapillai comments seem as confused and floundering as the customers left in the wake of the problems. The new 'voice of Optus' is either not well briefed by his team, doesn't understand the gravity of the problem, or just doesn't understand. Bring back Paul Fletcher. He didn't say much but when he did he at least understood it.
It should be called Droptus not Optus
Is Mr Maha Krishnapillai a complete idiot? Things don't get discussed and open accusations thrown around like this due to potential breaches of the Trade Practices Act. Stating things like this compromises Optus's relationship with companies such as Nokia and Siemens. A software upgrade botched on 1 of 3 of Optus mobile 3G switches after supposedly thorough testing. The outage certainly caused a major inconveinience for the network but 2 switches were successfully upgraded. That's the nature of software programming, it always has a potential scenario where things can still fail despite all the unit testing and debugging process to avoid issues.
Meanwhile us customers just sit and wait it out and pay them for a full months usage.
Disgruntled telco users...call the Telecommunications Industry Ombudsman...don't take bad service. Hand the contract back if it's not happening.
Telstra's network is so much superior then Optus network. Main reason for that is that Telstra knows how to select their suppliers. Ericsson supplies and maintains Telstra's 3G network, and the quality is incomparable. No one can match Ericsson's strength and quality in mobility and you can see that all over the world in both 2G and 3G networks.
If Optus was serious about competing against Telstra they would have selected Ericsson as their supplier. Yeah, I too hope that Optus is not granted the build of Australia's new National Broadband Network.
Well Im still getting dial up speed on my moblie wireless broadband for my laptop.Optus Did the checks and confirmed I was in 3g zone ,but not to be.Thanks Optus.Ill just have to wait 23 months for contract to end....great .
I am a Virgin data subscriber. The service has been so poor they have reimbursed me my subscription (data) fees since February. Pull your finger out Virgin!
Rome is burning while you guys are toying around. Talk about going from bad to worse. The games are in China not Australia. As you are charging like the big boys get your act together. You are losing us (valuable customers) at an ever increasing rate of knots.
If Optus wasn't around their would be less competition period and I doubt that they could use Ericson without Telstra causing them problems and in term upsetting the telco market here!
Ericsson builds and maintains Three's, Telstra's and Vodafone's mobile networks in Australia. Ericsson has an amazing reputation for what they do. Optus older infrastructure is Nokia Siemens and their new buildout is being done by Huawei. It's a bit of a mess at the moment.
If Optus had done their homework properly they would know that to go with Nokia's fairly untested 3G software includes taking risks. The best 3G software is provided by Sweden based Ericsson, which Telstra is using.
Spare us your excuses Optus.
Virgin Broadband which uses the Optus 3G network has been hopeless all year (see whirlpool.net.au for more coverage). This is mainly due to poor planning and agressive sales overloading the network. Optus needs to stop pointing the finger and start fixing their network.
Wont Australia have some fun if Optus is granted the build of Australia's new National Broadband Network.
Post A Comment
We welcome your comments on this story. Comments are submitted for possible publication on the condition that they may be edited. Please provide a screen name and suburb/location - these will be published. We also require a working email address - not for publication, but for verification. Read our publication guidelines.
Share This Article
From here you can use the Social Web links to save Buggy software sends Optus offline to a social bookmarking site.
Email To A Friend