While Carl Sagan’s famous formulation introduced a generation to the vastness of the cosmos, more recent history suggests that his memorable term might now be more aptly applied to financial extents: our deficits and debts, perhaps, to the economically or politically minded.  But for those of us with the markets on our mind, the term has to evoke the enormity of the data we create and must manage every day.  We’ve recently been working with the NYSE’s TAQ data in an effort to integrate it into StratBox’s back-testing and optimization capabilities.  And the enormity of the data is really just staggering.

Each day, the NYSE publishes all of the day’s quotes and trades as well as some reference data.  Compressed, the data will just about fit onto a DVD.  For one day.  A DVD.  Compressed.  It’s really mind-boggling.  A year of the stuff, uncompressed, will require over a petabyte of storage.  Over 1,125,899,906,842,624 bytes.  And that’s just the US Equities markets.  You want options data, too?  I hope your uncle is named EMC, because just managing the data is going to be a challenge…

“Information about money has become almost as important as money itself.” — Walter Wriston, former Chairman of Citicorp

The enormity and profile of market data far exceeds the capacity of traditional RDBMSes. While RDBMSes continue to expand their usable capacity - we have used partitioned tables with nearly a billion rows of market data which have performed  astonishingly well - they simply can’t deal with the kinds of quote volumes modern markets are generating daily.  This has spawned a host of specialized timeseries database products, like the grandaddy: Sungard’s FAME which I’d used back in the 90’s to write programs to calculate bond indices at JPM, to more recent offerings like Vhayu and Kdb+.   These timeseries oriented data products undoubtedly have many distinguishing characteristics and features, but they share one immutable characteristic: they are unbelievably expensive - in some cases a single developer seat costs in the high 6-figures for an annual license.

Thus, while no doubt missing out on some of their high-end features and niceties, we’ve decided to seek solutions from some of the original purveyors of petabyte-scaled data: NASA and the NCSA through their HDF5 system.  Designed to support vast scientific data stores and boasting sophisticated capabilities in support of parallel computing environments, it should be possible to get comparable performance to some of the high-end specialized finance products without the sticker shock.  Indeed, it’s potentially another example of free software providing a meaningful contribution to finance.

In researching cost-efficient and highly parallel hardware solutions to pair with our emergent data solution, I’ve come to realize that open-source is expanding its reach into the hardware sphere.

This guy shares his experience and “recipe” for building a powerful and unique rendering cluster inside an IKEA filing cabinet.  It’s admittedly on the funky side for even a SOHO operation, but it’s no joke - it’s more powerful than a lot of production blade servers used on wall st and it cost him less than $4K.  He also includes a (very loosely described) spec for a more powerful next-generation version with some 50-Teraflops of capacity!  So, while the data we’re having to deal with is growing at an incredible rate, the tools we have to manage it are growing proportionately for those who know how to leverage the work so many smart people are producing and freely sharing.

As my dad told me years ago:

“Good programmers write good programs.  Great programmers steal good programs.”

At this point, we’re still in the “discovery” stage of our development of TAQ+HDF5 for StratBox, but as we progress I’ll periodically post some of our experiences.

—

UPDATE

Speaking of my dad, he saw this posting and pointed me to some class notes he’s been working on which describe the Google approach to massive information processing and fault-tolerance.  Interesting and full of great links to both academic and industrial papers/sites on the topic.

I’ve been trading an increasing amount at the open and close of the equity markets using market-on-open (MOO) and market-on-close (MOC) order types and have found that the quality of executions varies enormously between the two types and have spent a bit of time analyzing the differences which I share below.

The quick scoop is that MOC orders almost invariably fill at the exchange’s published closing price, while MOOs vary very substantially from the published open price.  Below I quantify my findings in a bit greater depth.

I looked at 846 recent MOO and MOC equity trades made over the past two months.  Of all of these trades, only one MOC trade didn’t execute at the published close and the price I got was only off by one penny.  Across all of the trades, I received the open or close price 55% of the time.

The remaining 45% of the time I varied from the open by an average of +.04%  This means that I actually saw a slight price improvement in the average case.  That is, if I was shorting then I executed at a price above the listed open and vice-versa for longs.  I’ll take it!

In the below chart I plot the trades against their variance, positive or negative, from the listed open or close.

The biggest difference was a whopping 8.56% but at least it went in my favor.  The stdev across all of the trades was .87% so we’re not looking at too disperse a grouping.

This data is a bit skewed as the majority of the MOO orders are going short.  This is also a pretty limited universe of trades, so I’ll continue to look at the execution quality I’m getting on these order types and will revisit it if I see any interesting changes.

My interpretation is that my broker is making a best-effort to get a fair open price and they’re doing a creditable job of it.  The exchanges are doing a nearly perfect job with MOC orders.

What impact does this have on my strategies?  I’m not sure yet, but my first blush impression is that it might be worthwhile to try to get some price improvement over the posted open price as a means of both improving the results and extending the capacity of such strategies.  It’s a favorable result as it means that strategies which back-test well on open/close data have a pretty good chance of executing well in reality.

A related issue, which I’m still researching, concerns the capacity of such strategies and may be the topic of a future post…

While doing some research on the quality and volume of executions at the open and close of US equity markets, I came across two topical research reports by Celent, a finance consultancy.  The first report is a detailed look at execution quality on nasdaq issues while the second addresses the same topic for the nyse.  An abstract of the first report can be found here and of the second here.   Both are interesting enough on their own, though I’ve yet to acquire the full reports.

Apart the story told by these two graphics, the articles highlight a few further trends.  One is that execution quality is dramatically improving across the board in terms of both speed and price; since 2006, NYSE execution speeds have dropped a remarkable 92%.  At the same time, the difference in price quality across venues has diverged, such that everyone will get you filled faster and better than before, but some do so with much more consistently good prices.

If I can get a hold of the original reports, I’ll write a more detailed review of their contents, but for now I just include the links, the catchy speed-charts and these few high-points.  For shops dealing with smart order routing algorithms, the reports might be well worth purchasing and studying directly.

I mentioned several weeks ago that I’ve been developing and trading a strategy that’s proven to be quite interesting and profitable.  In that post, I described how I’d tried to improve the strategy through the use of a dynamic hedge.  The results of that crude hedge were quite good, but just as no worthwhile software project is ever really complete, trading strategies demand constant iterative development.

Below I describe some of the steps I’ve taken to incrementally improve this strategy, discarding the relatively expensive hedge I’d developed earlier in favor of a complementary strategy.  We see that when you combine two positive and uncorrelated results, you end up with a product that is literally better than the sum of its parts.

The core strategy has remained the same, though the universe of instruments it trades has been both expanded - through the inclusion of a broader set of equities & ETFs - and contracted - through the application of some filters which prevent trading of some of the instruments under various conditions.  These change have yielded a few more points of annualized return with minimal impact on volatility.

As I’d mentioned, the core strategy builds a portfolio of shorts which is sold at the open and bought back at the close.  One of the problems with the original hedge is that it had a cost - literally.  In order to hedge my portfolio, I needed to cut-back the size of my portfolio to accommodate the cost of the hedge.  I mitigated this issue to some degree by employing futures instead of a broad market ETF, but this still reduced my usable capital by approximately 10-15%.  I also had my money sitting idle overnight which seemed a particularly profligate behavior.

This led to the study I described last time in which I observed the relative out-performance of the broad US equity markets overnight.  If I could somehow find a way to capture some of this overnight alpha, I’d be able to both hedge my main strategy and better utilize my capital.

My first effort at such a strategy happily achieves both aims by assembling a long portfolio which is held overnight.  In the chart below, I capture the returns of the two strategies independently and combined.  The core daytime strategy is denoted as “OpenClose” or OC whilst the night-time strategy is  “CountingSheep” or CS.  Both assume an initial capitalization of $1M, employ no leverage and do not reinvest returns - each day or night they assemble a portfolio with the same $1M and profits are put aside and don’t generate interest.

The chart is nice and certainly a big improvement over the earlier hedged approach, but the real power of combining these two strategies is revealed in the two tables below.  The first characterizes their risk-adjusted performance independently and then when combined.  Both have a Sharpe ratio of around 2.0, but when combined they yield a new strategy which is about 25% better on a risk-adjusted basis.  These cells are highlighted.

The key to their compatibility is their correlation.  Or, actually, their absence of correlation.  In the below table, you can see the correlations of their returns to one another and broad market ETFs.  The short daytime strategy is, not surprisingly, negatively correlated with the broad market while the long night time strategy is positively correlated.  The beauty of their combination lies in the lack of correlation between the two of them (highlighted) - they’re essentially uncorrelated.

I hope this post illustrates a couple of different vectors along which strategies can be evolved; in this case, to better manage risk and utilize capital.

One of the key remaining limitations of this particular strategy is its capacity.  Increasing the capacity of a relatively short-term strategy like this one requires optimization of the trade executions which is its own black art but one that plenty of smart people are constantly addressing.  Perhaps in a future post I’ll review some of the techniques applied to this problem for another perspective on the iterative development/evolution of trading strategies.

In particular, I’ve been looking at various market-breadth ETFs recently as possible fodder for the little dynamic hedger I’ve described before, and I’ve noticed an interesting behavior among several of them…

Like the majority of traders, they do better when they’re not trading!

That is, they actually  display better performance at night than they do during the regular trading day; there’s more profit to be had in their gaps between sessions than there is during trading sessions.  Below I quantify this observation more thoroughly…

I looked at the performance of a variety of ETFs across three super-simple “strategies”. The first of them trades overnight - buying the close and selling the following morning’s open.  The next buys at the open and sells at the close.  And the final one is, essentially, buy and hold though it’s implemented as a buy at close, sell and buy at following close.  No costs or slippage are included.  The table below details the results for the SPY (S&P 500), IWM (Russell 2000) and QQQQ (Nasdaq 100).

In all cases, the ETFs perform better overnight - they make more and have almost precisely half the risk, yielding very substantially better risk-adjusted performance measures (here sharpe is calculated with a constant 5% risk-free rate). These cases are highlighted in yellow in the table.  Likewise, the worst performer in all cases was the “day-trader” who bought on the open and sold on the close.  One thing to note about the below summary is that all data is taken from inception for each of the ETFs, so their duration varies!  (For the SPY, since 1/29/1993; for the IWM, since 5/26/200 and for the QQQQ since 3/10/1999).

I tried the same analysis across a few other ETFs to see if the effect was a global one or a US-specific phenomenon.  The results are mixed but likely negative for Europe (EWG), Japan (EWJ) and broad foreign markets (EFA).  I’m not going to summarize that data here, but you can grab the excel workbook I used for this exercise here, though I warn you in advance that it’s about 2.3M.  (If you find errors - please point them out!)

[UPDATE: .XLS 97-2003 version available here is ~3M]

For the US markets summarized in the table, the conclusion is pretty explicit - the great majority of the market’s profits are attained overnight. Even worse: in spite of the positive returns across all of these markets over the relevant periods, all of them show significantly negative intraday performance.

For long-only day-traders, this is the kind of result that should cause nightmares!

I came across and had to share this excellent vignette by Freeman Dyson on the perils of excess model parameterization…

In desperation I asked Fermi whether he was not impressed by the agreement between our calculated numbers and his measured numbers. He replied, “How many arbitrary parameters did you use for your calculations?” I thought for a moment about our cut-off procedures and said, “Four.” He said, “I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.”

One of these days I’m going to give an overview of all the excellent open-source software I use on a daily basis. Until that day comes, I’ll observe that finance remains one of the big areas where open-source software has made relatively limited inroads.

Two production-quality packages fight that unhappy state: QuantLib - a comprehensive framework for quantitative finance - and QuickFix - a full-featured FIX engine. Both are C++ libraries and both provide very nice interfaces to facilitate integration with other languages, including Java. QuantLib is a big and complicated library and integrating it with Java is not totally obvious. Below, I’ll describe how to build and use QuantLib from Java.

These instructions are based on a unix installation. I’m not really a windows developer and don’t have all the shiny tools that windows developers use, so it’s not an area of focus for me. That said, I have managed to build QuantLib under windows by using MinGW+MSYS but it wasn’t terribly easy and I don’t currently have a working installation, so I won’t cover that here. If this is your aim, don’t be dismayed as it is possible and it had all the functionality I enjoy under linux.

Using QuantLib from Java (on linux)

• Build QuantLib
• Requires a working version of Boost. This may prove to be the hardest step of all and you’ll need to use the ample documentation provided by the Boost team.
• Once you have a working copy of Boost, building QuantLib should require little more than

sh autogen.sh
./configure
make
sudo make install

• Build QuantLib-SWIG
• Requires a working copy of SWIG. Again, look to the SWIG instructions, but it should be easy.
• Once SWIG is available, building the QuantLib/SWIG interfaces should only require:

sh autogen.sh
./configure \
--with-jdk-include=${JAVA_HOME}/include \
--with-jdk-system-include=${JAVA_HOME}/include/linux
make -C Java
sudo make install

• Now you’ll have a Jar file with all of the SWIG/JNI stubs in it available in /usr/local/lib/QuantLib.jar. Add this to your classpath.
• Programs which call QuantLib functionality will need to have the LD_LIBRARY_PATH set. This can be done by invoking the vm with something like:

-Djava.library.path=/usr/local/lib

• Programs which call QuantLib functionality will also need to explicitly load the QuantLib libraries. This can be done with something like the following static block appearing before your main method:

static { // Load QuantLib
try { System.loadLibrary("QuantLibJNI"); }
catch (RuntimeException e) { e.printStackTrace(); }
}

• That’s it. Now test your configuration by running the examples in Quantlib-SWIG/Java/examples.

It’s worth understanding how Quantlib is being used from java. SWIG is creating a JNI interface into those methods within Quantlib which have been exposed through their declaration in the swig *.i files. These files are found in Quantlib-SWIG/SWIG and they determine what functionality from Quantlib will be available to you. You’ll likely need to get familiar with a subset of those files that you care about. If you find that some functionality you care about isn’t exposed in those files, you may need to expose it yourself.

There’s a learning curve, but it’s worth traversing so you can get at all the rich functionality so many smart people have put together.