Although the use of color is a common semantic method for conveying information to sighted readers, it should never be used as the sole means of imparting information. Semantic markup should always underlie any use of border colors and shading so that the equal access is provided to readers using non-visual playback methods.
But color is problematic for more than just the problems that arise conveying semantic meaning. The contrast between foregrounds and backgrounds can make reading difficult. Green text on red backgrounds (and vice-versa) can be unreadable for some people, and can generally cause what's known as visual vibrations. Red and black can also be indistinguishable to many readers. Black text on bright white backgrounds can lead to eye strain and headaches.
Likewise, placing one shade of a color on another is also going to make reading difficult, as will placing two light or dark colors on top of each other (luminosity). A reader may be able to fix some of these issues with a configurable reading system, but it is best to avoid them in the first place. Strictly adhering to the color guidelines outlined in WCAG 2.0 is the best first step.
Also avoid placing text over noisy (multi-colored) backgrounds. The less uniform the background is the more difficult it can be for any reader to follow.
More generally, though, be sure to always cleanly separate styles from content. Separation of styling ensures that readers can apply their own style sheets and/or use and built-in color, brightness and contrast controls in their reading system to tailor the display to their needs.