By Herbert Lash
NEW YORK Tue Nov 26, 2013 1:46pm EST
Commuters pass by the NASDAQ Marketsite in New York, May 21, 2012.
Credit: Reuters/Brendan McDermid
NEW YORK (Reuters) – U.S. security exchanges are feverishly working in the wake of August’s Nasdaq trading halt to reinforce a market that too frequently seems to succumb to technology glitches, but these efforts are unlikely to rectify all the weak spots.
What appeared to be a workable solution two months ago to create a back-up for the exchanges’ “securities information processors,” or SIPS, like the one that got clogged with reams of quotes in Nasdaq-listed stocks on August 22, now looks too complicated, according to sources familiar with the talks.
Making the SIPs more resilient highlights a problem for a market that generally operates exceptionally well, despite a number of high-profile technology failures since early 2012, including Nasdaq’s botched handling of Facebook’s IPO that May.
The more than 50 equity trading venues and their links to hundreds of brokerages, with differing software and numerous order interfaces with the market, have created a system so complex it is vulnerable to failure in countless ways.
To combat this, broker-dealers and market makers have put a premium on enhancing their early warning detection of bugs and other anomalies, and swift damage control when problems occur.
Monitoring for potential glitches or other application failures has taken on greater importance with the market now geared to trade large volumes of securities in microseconds, said Donal Byrne, chief executive of Corvil, which monitors the performance of trading platform operating systems.
“The risk systems that are largely in place today were built for human time. They weren’t built for machine time,” Byrne said.
“Machine time happens in microseconds, milliseconds and seconds. If you can’t detect, alert and react within that time frame, there’s a pretty good chance you can’t address the risk that you’re exposed to. Within a second you could have traded yourself out of business.”
SCANT DETAILS ON EXCHANGE PLANS
The Securities and Exchange Commission told all U.S. stock and options exchanges in September to draw up plans for buttressing the market after a software glitch forced Nasdaq OMX Group Inc (NDAQ.O) to halt trading for three hours in August.
The SEC ordered the exchanges to examine five broad issues, including the SIPs, and proposed kill switches to shut down trading when exchange systems malfunction.
The exchanges said they had established a “pathway” for identifying contingencies for critical infrastructure, or the “single points of failure” that scare national security experts. They targeted trading at the open and close and during initial public offerings, key regulatory messaging during trading, outages at the clearing houses for stocks and options and connections to the exchanges’ disaster recovery facilities.
Details were scant. SEC Chair Mary Jo White acknowledged that “work remains to be done.” The exchanges indicated after their 60-day review that a months-long process lies ahead as they amend their rules and open their proposals for public comment.
“It takes the industry time to digest. You need to see stuff in writing, and you need to see the details,” said a source close to talks between the exchanges and broker-dealers.
For example, discussions among exchanges on measures to fortify the SIPs have had mixed results. An idea that seemed feasible two months ago, to reroute data to Nasdaq’s SIP if a processor for the New York Stock Exchange were to fail, and vice versa, has been canned.
The SIPs operated by Nasdaq and NYSE, now owned by IntercontinentalExchange Group (ICE.N), have different fields and data feed formats, making the rerouting plan a technical challenge, said a second source with knowledge of the talks.
“We’ve talked to the industry and they don’t want to move in that direction. They think it’s overly complicated,” the source said.
A front-line defense has been agreed to involving hardware, software and testing procedures to bolster the SIPs, along with system and process enhancements, the second source said. Existing back-up sites will be used if a failure occurs.
“We need to improve the existing back-up facilities and make it so you can quickly fall over to the back-up site if need be and you’re comfortable doing it,” the source said.
Most of the downtime during the three-hour Nasdaq trading halt was to ensure no hiccups would occur when trading resumed.
Some bemoan the lack of post-mortem reports that would provide insight into mishaps. The content of reports on outages filed with the SEC after such incidents are kept confidential.
STANDARDIZATION VS INNOVATION
Current proposals fail to address other shortcomings in the securities industry, especially aberrant software that often has been the culprit behind the vexing market disruptions.
Nasdaq Chief Executive Robert Greifeld has flagged software as an issue, urging standardized measures and a certification process to help reduce the number of glitches, similar to what exists in the electrical and telecommunications industries.
“There has to be some way where the central body that you test against, they certify you and then say you’re good to go,” Greifeld told reporters last month. “The telco industry works that way, obviously the power industry would work that way.”
Yet many in the market are wary of standardization, in effect seeing it as an affront to capitalism.
“The lack of fully standardized systems and technology is what has made our markets (among)the most efficient on the planet,” said Chris Concannon, a partner at Virtu Financial LLC, a leading market maker in more than 200 markets worldwide.
“Standardization is the enemy of innovation,” he said.
Common standards do exist in electronic trading, such as FIX Protocol, a 20-year-old messaging language to facilitate equity trading. Also, knowledge of how each exchange operates has spread as technology officers switch jobs. Still, some have urged increased standardization to reduce glitches.
The SEC earlier this year sought industry comment on creating standards for computer security, capacity and audits.
Systems testing has increased to ensure they function properly and tasks have been automated to reduce “fat finger” mistakes and error introduction when software is updated.
Exchanges go through daily checks before trading begins, and when changes are made to an exchange’s technology. Automation ensures such repetition is conducted the exact same way to eliminate glitches, said Rob Cornish, chief technology officer at the International Securities Exchange, an options trading platform of Deutsche Boerse Group AG (DB1Gn.DE).
“We call it software deployment automation, and we use newly developed technology to help us with managing code and deploying it,” Cornish said. “We also go beyond software. We’re treating our infrastructure as code, with a goal to automate as much of the change process as possible to reduce human error.”
Concannon said damage control is crucial because technology misfires and testing only goes so far. Virtu conducts regular fire drills with many of its trading venues and runs in-house simulations, often with its personnel unaware of the tests.
“Despite lots of testing, it still fails. Hardware fails, switches break, boxes crash and lines go down,” Concannon said, adding that mitigating damage from technology failures is his firm’s top priority, and should be for the industry as well.
“We spend more time on mitigation and early detection than we do on testing,” he said. “More firms should approach their technology that way.”
(Reporting by Herbert Lash; Editing by Dan Grebler)
- Link this
- Share this
- Digg this
- Email
- Print
- Reprints
