Expectations. That's the point. By being accepted by the masses as the custom, the "holidays" create the expectations. So when the socialization (from the cradle) is that at least one someone ~should~ care at a particular time, but then no one does, then the person thus forgotten feels defective, even if they didn't expect any acknowledgment. This is the event at fault, and not the person who knows they don't mean enough to anyone to deserve being remembered on that Special Day set aside for it.

If there were no special day, a person could just let themselves believe that everyone who knows them is just too busy, doesn't go in for that kind of stuff, or hasn't got around to it yet.