I found the book "USB Complete" by Jan Axelson helpful to understand the basic workings of USB, and for hardware to toy with i think sparkfun.com might have some microcontroller boards with a USB port. There are other sources, which unfortunately the margin of my brain is too small to contain the names of.
the 'simplest' would be to plug in a device that is supported by your operating system, such as a HID device.
the next step would be to implementing the protocol to emmulate that device, within a microcontroller.
the next step would be to controll that device with a host capable micro.