If they're lowering the performance of the card, shouldn't the price go down too?!
Yields have nothing to do with performance. Yields are how much of each wafer they are able to use. You see a wafer is a very large chip that has tons of little chips embedded in it, for ease of explanation let's say that each wafer has a 100 chips on it. Due to the manufacturing processes only a percentage of those chips will be viable to actually use in production. Each chip powers one graphics card. So out of each wafer they are manufacturing they are only able to use 50 chips or make 50 cards out of it. Since the wafers are expensive, the higher the yields are on it the better the profit is (or the cheaper the card is able to be passed on to consumers at). Hopefully this makes since.
It's the same thing is true for CPU chips...
Here's a couple pictures...

[Fud] Fermi yields still under 50 percent

[Fud] Fermi yields still under 50 percent

[Fud] Fermi yields still under 50 percent

[Fud] Fermi yields still under 50 percent