If they are on the team, then yes.
Retrospectives are a critical element for the team's continuous improvement. Obviously you would want to educate the team on the value of the meeting and encourage them to attend. However if they resist citing other "urgent" work, then take a cue from Shu, Ha, and Ri. The student must first practice the kata as is. Following the rules. In time, as they master the concepts, they can decide to run experiments with attendance. However, at the beginning, the practice must be taught. The value must be clarified. The experience must happen with everyone from the team.