I would say the vast majority of anglers I encounter while on my kayak who are wearing PFD's is comforting.
Heck, even most of the recreational watercraft users wear them as well.
The key for me was getting the right type for me and one that was comfortable for paddling and casting.
Oh, and I wanted one with pockets to put my personal stuff (keys, phone etc...)
Once I found one I liked, I would wear it EVERY time I went out, regardless of the time of year.