Before we start, I want to point you towards the glossaries at FanGraphs, The Hardball Times, and Baseball Prospectus. Many, or maybe all, of the stats that I use a lot at WHYGAVS originated on one of those three sites, so they can often explain them better than I can. I'll point you back to each site for their respective stats, but they all need acknowledged up front because they are the sites that are changing the way people think about baseball. Without their analysis and hard work we're here talking about why Andy LaRoche sucks because of his .258 batting average, not debating his UZR, WAR, positional relavance, ability to adapt to second base, etc. I am far from an expert on these things; if you really want to learn about them you should go to the source.
Finally, there's Baseball-Reference.com. Greatest site in the universe. All of the less advanced stuff comes from them unless otherwise noted.