I'm fortunate that the owner of the company I work for now knows that I podcast and he has even listened to the one that ended my previous career (which, BTW, he said he found nothing in it that he would have fired me for and his father absolutely loved it and wanted to know when I was going to do more like it - but that's him and his father and not my previous employer).
He told me the only thing he would have an issue with is if I criticized him or the company or demeaned any of the employees in a podcast, but my other viewpoints are my own and to share them however I felt. I can completely understand that position. It also helps that he's my age and a) knows what a podcast is and b) has been listening to podcasts for years.
But, yes -- podcast at your own risk if you aren't doing it as a full time job. That's not meant to scare anyone, but it's just a word of caution.