factory_girl is great for DRYing up test code and making tests isolated and maintainable. What it’s not so great at is any association more complicated than has_one/belongs_to. I found a nice trick on Stack Overflow for has_many associations.
The Factory.after_ hooks appear to be the only way to do this successfully.
It’s a shame, because it makes it really painful to use factories. What used to be a one-line call to create whatever you need becomes writing your own factories that call factory_girl factories.