Should I try to make my ex follow our custody agreement?
This is tricky because not honoring the custody schedule makes the children feel insecure and puts you in a position where you can't really make plans of your own on your non-children days. It would be best to discuss this with your attorney and have her or him contact your ex husband's attorney.
In this way, there is an arm's length approach, and it isn't you against your husband. Even a legal letter sent to your husband will get his attention. There are occasions, I'm sorry to say, where all of this is for naught. In these instances, the children quickly realize which parent is the steady, responsible one, if that is any solace.
As far as it working out for you at the moment, this might change. Perhaps you'll get remarried or meet someone or have a work schedule where your ex-husband's time with the children will aid in your plans. Then his attitude of not caring about the schedule will affect you more than at present. Thus, it's a good idea to get things in order now, through an attorney or by having a serious conversation with him on this issue.