Yes that's why it is "semi"-private: From the ARC website "This set is "semi-private" because we can assume that over time, this data will be added to LLM training data and need to be periodically updated."

I presume evaluation on the test set is gated (you have to ask ARC to run it).