So I decided to cancel my plan ($15/mo)
I went to my audible profile and clicked on Cancel service link.
On the next page I got an offer to keep my membership for half off for next 3 month
I declined again and on the next page I got $9.95 / year offer. Plus I was able to keep my existing points and keep collecting free point (free book) every month.
I think this is very good deal. So you can try and possibly you can drop your membership fee by 90%
Go to Account Settings and proceed to cancellation.