Do you think I should trust him now?

Found my husband cheating 4 years ago. We have managed to move on since then. I have stuck by him mainly for my daughter's sake and perhaps for mine too because I am petrified of being on my own. We have been married for 22 years. In the last year we have managed to move our relationship to a better place. He is no longer aggressive not just towards me, but he is more sociable towards my family too. Since my daughter went to university there is just the two of us and we have managed to create a more loving relationship. At this point you must be asking yourselves why I am writing here. The reason is because after a period when we were always making love and showing pion for each other, we have just gone cold. I have started to suspect that he still sees the other woman, he says he doesn't feel well and has been to see a doctor who says there is nothing wrong. I know he does take medication for high cholesterol which may be effecting him but I just have issues trusting anything he says. For example, I find it uncomfortable when he shows me affection after he comes home late from work. My immediate instinct is to have a confrontation but because I don't want us to go back to where we were 4 years ago, I just pretend to be indifferent which makes him more determined to break me and in a way I suppose condone whatever he may be up to. I realise he may not be up to anything, but because he insists which is something he would never have done in the old days I am confused as to whether he wants to fool me into a false sense of security or whether he is genuine. He spends weekends with me, its just when he is at work I don't know what goes on there. I know I can't control his every move and I shouldn't have to I just wish I could get over this.