G, I got you on this one.
You can either own the gold directly or you can buy a paper claim convertible into the gold itself. There are some hybrid programs where you supposedly own a direct fractional amount of a large gold bar, for example. I say supposedly because the only way you know is through their ledgering system. If you want to trade gold, there's ETFs out there for it like GLD, but that's a separate matter altogether.
The latter has lower transaction and maintenance/storage costs, but the most paranoid of people tend to own the stuff directly. If you want to own the stuff directly, then you're looking at annual storage fees in addition to transaction costs when you buy/sell.
For example, if you own a 1 kilo bar of gold, it's gonna be expensive to store it every year. And it will be expensive to buy and sell each time. And if you wanted to purchase it in smaller increments due to the fluctuating price of gold, you're shit out of luck. On the upside, the bar belongs to you. Thing with gold is that it forces you to buy an asset, store it and pay for it, and watch as the price either goes up or down over the long-term. Like any other asset, it has experienced both LT price increases and declines.
If you're personally comfortable with a paper claim on gold, go here:
http://www.perthmint.com.au/investment_certificate.aspx
If you're interested in the more direct fractional model, this is a cool place to check out:
http://www.goldmoney.com/
If you want to own the stuff directly and abroad, there's tons of options. Non-US banks are full service - they often trade and store the gold for you. Question is which country. Truth is this though. If you're going offline to prevent Uncle Sam from getting his filthy paws on your gold, it's gonna be a losing battle. All these countries would fold like a cheap suit if Uncle Sam wanted to get at your gold. The country that would put up the biggest fight is probably Switzerland, but the transaction/storage fees there are high.
** For the record, I own a few gold coins that I store personally. This information is entirely the result of the exhaustive research my brother and I did on this.