I know that this can be a touchy subject, and in reality, everyone needs to make their own decisions. I make my own decisions based on what I feel the FACTORED risk is. Factored risk takes into account both the probablility of an undesireable event, as well as the severity of the results. Many people here have mentioned similar rationale... either they determine that the probability of capsizing is very small, OR that the results won't be so bad (such as shallow water where they can touch).
That being said, I have put over 200 miles on my kayak so far this year, and only about 10 miles have been while wearing a PFD. Conversely, I've put thousands of miles on my motorcycle, and not a single one has been without a helmet. When I get on my motorcycle and ride on public roads, I know I'm taking a risk. No matter how confident I am in my own abilities, I know there are a lot of idiots on the roads, and a lot more people who are just plain distracted... not to mention the possibily of deer running out in the road in front of me and other happenings that are beyond my control. All of these possibilities (and the understanding that my skills are not infallible) combined with the potential (or even likely) severity of the outcomes, cause me to ALWAYS ride with a helmet. After riding motorcycles for 10 years without ever so much as dropping any of my bikes in the driveway, I still wouldn't relent my helmet policy, and when I had my accident, I was very glad of that... I walked away uninjured. When it comes to kayaking, I feel that the likelyhood of an accident is far less... I kayak in lakes, with no current to deal with... when I'm not in an "electric only" or "no wake" lake, I avoid speed zones and weekends/holidays... and I've never tipped my kayak over on accident. I also feel that the severity of an accident in the kayak would tend to be minimal... by avoiding where power boaters and jet-skiers play, I don't feel I'm likely to encounter anything beyond an accidental tipover.
Still, I have practiced my deepwater re-entry as I think every kayaker should. I've tipped the kayak intentionally in deep water to make sure I could 1.) flip the yak upright while treading water 2.) crawl back up onto the kayak 3.) access and put on the PFD from the water 4.) crawl back up onto the kayak while wearing the PFD 5.) swim to shore without leaving the kayak 6.) and crawl back up into the kayak while fatigued.
I won't urge people to wear a PFD, when I myslef do not... I won't tell anyone that there is no need to wear a PFD and they should just leave it off... I WILL urge all kayakers to practice deepwater re-entry, and to evaluate the risks for themselves... AND to wear that PFD when conditions dictate (weather, water temp, boat traffic, whatever). We all take risks everyday of our lives. I feel safer in my kayak on a quiet lake (without wearing my PFD) than I do commuting to work along I-76 in my truck.