Yeah, that's what I assumed was the reason for Jim's absence in the final stretch of the movie (even when I didn't know Olivia left him as well). He was pretty inconsequential to Mason's coming of age process, therefore reduced to a mere footnote.
I love that the movie actually reflects the truth of people who seem irreplaceable or pretty significant at one point of time in your life become a complete no show later on despite affecting you in a big way (Bill and to a lesser extent, his kids), some people never have much of an impact on you and their presence in your life diminishes entirely (Jim) and there are some people who may not seem to have much of a visible impact on your life and who may seem to be just passing acquaintances but they will always come through for you when you need them the most or even when you least expect it (Olivia's friend Carol who not only provided the family shelter and protection from Bill but also arrived with her daughter to Mason's graduation party years later, much to his pleasant surprise). It just further adds to the authenticity of the movie in an unnoticeable and seamless manner.