Commodity index funds have become important players in commodity markets. Led by the S&P GSCI and the Dow-Jones UBS (formerly Dow-Jones AIG) indices, these funds allow investors to gain exposure to a basket of commodity futures. The funds, or intermediaries, tend to hedge their exposure in individual commodity markets, and so are major holders of futures contracts.

The size of these funds has never been known with complete accuracy - the companies themselves only sporadically give estimates on funds linked to their indices. By far the best data has been the weekly supplementary Commodity Index Trader (CiT) report to the well-known Commitment of Traders report from the Commodity Futures Trading Commission (CFTC). Since January 2007 this has (and with data back-calculated to January 2006) separated out the long and short positions for traders that predominantly undertake trading related to commodity indices. However, the report only covers the 12 agricommodities most affected by index trading. Our estimates for other commodities, including metals, have had to be made through extrapolation, based on what we know about each commodity index's weightings.

This clearly has its limitations, as there exists nowadays single or sub-basket commodity funds, so the weightings of the general S&P GSCI/Dow-Jones UBS indices are not necessarily a good guide to commodity’s relative share. Another limitation of the CiT report is that the CFTC classification is done on entities, not their positions; so when traders are classified as 'index traders', all their positions are so classified. This may over- or under-estimate the actual figure, if those traders either also have other futures business, or have already netted off commodity exposure before using the futures market.

More light shone on index funds

These potential deficiencies are addressed in a new quarterly report by the CFTC, called 'Index Investment Data'. This takes a different approach to estimating the size of index-fund related activity, by requesting details of their commodity index exposure from 43 entities, all of which are known to be significant users of US futures markets for index-related activity. Importantly, this was not just in the 12 agricommodities covered by the CiT report, but seven others, including gold, silver, copper, heating oil, natural gas, unleaded (RBOB) gasoline, and WTI crude oil, plus an estimate of non-US based index investment (mainly Brent crude, gas oil and LME base metals).

The entities were asked both for a dollar value of their long and short exposure, and the notional amount of futures contracts that would imply. The amount is notional, because the entities need not necessarily have gone long or short that amount of futures contracts - they might have internally netted off index positions against similar contracts held for non-index fund purposes. So for example, if they have sold funds linked to indices but also have short positions with commercial users, they can net these off without recourse to the futures market. The CFTC believes the new report offers a more comprehensive gauge of index-fund activity, at the cost of less timeliness and regularity.

So what does it tell us? First, it reveals that the total value (hereafter AUM, assets under management) of commodity index fund investment on a net basis at end Q2 2009 was $117.2bn. The peak AUM (using this quarterly data) was at the end of Q2 2008, when it totaled just over $200bn; the low (since Q4 2007) was in Q4 2008, when it was just $82.2bn (see chart on the following page). But these great swings in overall value in fact reflect price movements more than changes in holdings. The second chart shows the value of the index fund positions compared with the number of contracts held (again both on a net basis); both are rebased to end Q4 2007 = 100. While the value rose by nearly 40% then fell by nearly 60%, the number of contracts held grew much more modestly (just 8% higher from the start of 2008 to the peak in Q2 2008) and fell back much less, down just 24% from peak to trough (in Q1 2009). Since the trough, the volume of contracts held has risen by 19% and the value by 42%. The third chart makes an estimate of what proportion of the change in AUM (in $bn) was due to valuation changes, and what was due to net inflows - usually they have worked in the same direction, but in Q2 2008 and Q1 2009 they worked in opposite directions. Notably, the strongest month for inflows has been Q2 2009, with $12.3bn.

Drilling down to look at the metals, we see that gold is by far the largest in terms of AUM, at $6.7bn at end Q2 2009, ahead of copper, at $3.6bn, and silver, on $1.8bn. Of course this does not include the LME base metals that will be in the 'Non-US' category, which was worth $22.5bn at end Q2 2009, and unfortunately is not broken down any further (much of that is ICE Brent Crude - see later in this article). The value of the three metals has swung considerably, with copper in particular suffering as its price crashed in Q4 2008. If we look at contracts (net) held, instead of AUM, so excluding the impact of price movements, then we see that all three metals had a record net long by Q2 2009, something which cannot be said for commodity indices as a whole. Copper in particular saw a strong recovery in Q1 2009 and Q2 2009.

We have focused on the net position, as it is the most representative of market impact, but we can also look at the data by splitting out longs and shorts. Here we find that, at end Q2 2009, over all commodities there were 3,051,000 long contracts held and 912,000 short contracts. This made shorts 30% of longs, the highest they have been and sharply up from 18.5% at end Q4 2007.

To what extent this reflects greater pessimism on the part of index investors, or the greater availability of 'short index funds' (which is not unrelated to the demand for them), or even less internal netting of contracts by traders, is hard to say. The individual commodity with the greatest short position as a percentage of the long position (at end Q2 2009) was cotton, at 45%, followed by cocoa, 42%, and coffee, 38%. Unleaded gasoline had the smallest, at 20%. Looking at the metals, gold was 35%, silver 29%, and copper 24%. For the metals, as for other commodities, over time the percentage of shorts has risen. While it seems intuitive that this (at least until Q4 2008) was because of worsening prospects for at least copper and silver, it might have been due to other factors, as outlined above.

How does the report compare to what we already knew? Prior to this we had two sources of information. First, the CIT report, which - as noted above - was available for 12 agricommodities. The differences between the two reports are that the CIT report classifies traders by their predominant activity, and reports directly the contracts held in each commodity on the future's markets by those traders seen as predominantly 'index traders'.

The new quarterly investment report asks entities involved in index business to estimate their exposure in each commodity, regardless of whether they are hedged in the futures market. In practice the difference is small; the chart "CiT and quarterly investment report" on the next page shows the net long position at the end of each quarter for all 12 agricommodities according to the CiT report, and the new quarterly investment report.

Interestingly, the two reports have become more similar over time. In Q4 2007 and for the next three quarters the new report had a markedly lower net long for the first four quarters, from which point the two totals have been very close. We are not sure why this might have been, but it may have something to do with internal netting.  Continued...