You want to count values in the range of cells containing the raw data (A4:A32). You want to count the number of values that are equal to the value in cell C4. The correct Excel expression is:
=COUNTIF(A$4:A$32, "="&C4)
The range uses absolute references because the raw data is always in the same range of cells for every row in the frequency distribution table. The condition, however, is different for each row so a relative reference is used there.