As a lawyer, though based in the UK, and a starting publisher myself, I can offer some advice, but remember this does not count as legal counsel.
If you write an RPG book, you can copyright it, just like any other creative work. Copyrighting takes some paperwork, but isn't expensive. You'd also need to acquire ISBN's but those can be bought online for a few tenners each. The copyright protects the entirety of your work, with the exception of the math in your system. Taking d20 (most famous game around atm) The idea of having 6 attributes with scores from 3-18 + bonuses, or the idea of saving throws is common rights. Flat out copying the description of these rules is breach of copyright, but can be seen as fair use by some jurisdictions. Copying elements from your setting would be theft of creative property, and as such illegal. No matter if it is from the same book.
Now if you want people to use part of your work, but not all, you could give out an OGL type license, but I'd suggest getting a really good IP lawyer before doing that.