DRUGS AND EX HUSBAND...FORGIVE AND FORGET?!?

My ex husband was heavily involved in drugs which lead to an affair when i was pregnant. Subsequently he left me when my daughter was 3 months old leaving me with her and a little boy of 2, no money and a reprosetion order on the house. He did not provide financially leaving me to fend for myself and the children while he indulged himself in his pleasures. I waited 6 years before i moved on to another relationship and divorced him. He barely contacted us for years had another child with his girlfriend, has now left her and the child and has re-married. After 15 years he now wants to resume proper relations with his children but I begrudge this because of the hardship he put us through... aswell as taking me to court for custody of the children and for a share of the house that he did not put any money towards ! He says if i do not let him see the children he will force me out of the house with a quick sale. Should i let him see the kids freely or is he just bad news for them too ?! He says when they're 16 they'll look for him anyway but I know he will turn them against me as he only has rotten things to say about me anyway to justify the crap he put me through! Should I let the past go and give him freedom with the children?!