While at first it may make your skin crawl to think of all those creepy crawlies being so close to you, it is perfectly fine. They are actually helping by breaking down all of the organic matter into the beneficial rich dark brown soil we all know and love. Without these little critters it would be up to just bacteria and microorganisms to do the heavy lifting, and composting would take much longer. What you do not want are cockroaches, potato bugs, flies and other large scavenger insects; these can spread nasty diseases. Worms, pill bugs and all those other tiny, hard-working soil critters are perfectly OK and, in fact, should be encouraged.
Bonus Tip: Give your worms a morning jolt Coffee grounds create an acidic environment to help speed along the decomposition process… and worms love them.