It depends on what the subject was. Comey has (or did have) a security clearance, so he was expected to know what was or was not classified. What determines classified is something the government itself does (emphasis mine)
Classified national security information is information created or received by an agency of the federal government or a government contractor that would damage national security if improperly released. Since 1940, the President has managed the system of classifying information by executive order (E.O.); the most recent order concerning classified national security information is E.O. 13526, signed by President Obama on December 29, 2009.
Information can only be classified if an official determination is made that its unauthorized release would damage the national security. Levels of classification correspond to levels of supposed damage. E.O. 13526 specifies that information whose release would cause “exceptionally grave damage to the national security” is classified TOP SECRET; information whose release would cause “serious damage” is classified SECRET; CONFIDENTIAL is the lowest category of classified information currently in use. RESTRICTED is an obsolete category that was discontinued in 1953.
Classified information may take any form. Though paper documents are most common, there are classified photographs, maps, motion pictures, videotapes, databases, microfilms, hard drives, CDs, etc. Regardless of medium, classified information requires protection until it is formally declassified.
So, even thought Comey was being asked questions about his notes, it was the subject itself that kept it classified.
Let's say that he was asked about the Russia investigation. Under the Classification system it would likely be classified under
(c) intelligence activities (including covert action), intelligence sources or methods, or cryptology;
given that the FBI was handling a counterintelligence investigation (only under the newly appointed special prosecutor has it turned criminal). That investigation would still be classified, meaning any memos about that investigation would also be classified.