The subtitles of Kaguya said "It has been four years in a row", but what was meant is four of these final exams in a row, of which we learned that there is five in a year.
Note: in the manga these two exams episodes was for two different of these exams, but in the anime they were combined into one and the same.
So it isn't like Kaguya only losed to Shirogane since she spent time on helping Ishigami even if it can seem like that in the anime~