Well, I've finally had time to finish up a set of test hexes and I've learned a few things. My original ideas were OK but there were some problems.
The hexes shown here are 1.5" from flat to flat. My idea was to vary the height for a more natural look but leave several points the same level so a miniature could stand on the hex. The main problem for me arises when you group a large number of these together. I tried to imagine what a large surface would look like if covered with these hexes and I didn't like it. It would end up looking like a quarry heap - nothing but a jumble of loose rock. It also disguises the hex pattern a little too well
While it didn't look terrible, it wasn't what I was after. Also I tried to imagine placing a building or machinery on top of this rock. Most buildings would cover more than one hex and you wouldn't have a level surface to plant it on. I originally thought of having folks put a small blob of air-drying clay down before setting the building. This would end up looking like a mound of dirt that the building would be raised on. However, you could not stack these hexes on top of each other for pillars of rock.
The trick is to make a surface that looks like rock which you can stack up, easily mount buildings on but still have somewhat of a natural look to it (at least more natural than using Heroscape tiles). Here's what I ended up with:
These hexes are exactly 1/2" tall, you can stack them together, mount buildings on them, mechs can stand up and they have a fairly natural look to them. I also made some hexes where one edge tapered down, two edges tapered down, three edges tapered and so on. These would be placed around the outside edges to hide the straight hex rock edges. If I end up making squares (instead of hexes) then there would be fewer sides to taper down. If you wanted some blocks with a more natural varying height you can always fill up the mold only part-way for thinner tiles.
Here is the complete set of hexes that will come out of the mold. I included half hexes (split both ways) and a left and right quarter hex. These will allow you to make flat edges on large sections. You can also just glue the halves together to make full hexes. I had considered making these tiles only 1/4" thick instead so you would not have quite so much heavy plaster. However, these are for building rock hills and rock hills have height. If I end up making other types of ground cover, I imagine those hexes would be thinner.
I'm going to make the test master mold today. Then I'll make some duplicates, cast them and see if these pieces act the way I hope they will. This mold will be large, twice the size (and twice the cost) of my regular molds. I don't like making molds this large because it's more difficult to make and folks don't like to spend twice as much for a mold. However, I would have ended up having to make three regular molds to fit all the pieces on with a lot of wasted space and that would have been even more cost prohibitive to those trying to buy them. Let me know what you think.