Bald fisherman on Zelda-ocarina of time?

i caught the hat, but i said no way when he asked for it back. then i said ok after that, but he puts it back on. i want him to leave it off. what do i do???