According to the notice of contest filed by Rita Hart, the evidence for the 22 ballots comes from a combination of poll workers, county recount board records, county auditor records, and affirmations by the affected voters. As for asking the 22 people who they voted for - that has taken place - voters have waived their right to a secret ballot, and the affirmations under oath are included in the notice of contest.
Under the heading "Wrongfully Excluded Ballots", there is a helpful breakdown of each of the votes being contested, with a summary of the evidence supporting Hart's contest. These consist of:
Scott County Curbside Ballots (2)
This relates to two ballots that voters attempted to cast from their vehicles. The evidence for these votes comes in the form of notes made by poll workers at the time, as well as a later determination by the Scott County recount board that the votes were "mistakenly excluded from the initial canvass".
Marion County Absentee Ballots (9)
This relates to nine ballots that were erroneously uncounted in the initial canvass, and as a result were not counted in the subsequent recount. Specifically, the notice mentions a memorialized joint statement made by the Marion County Recount Board acknowledging that a box of votes labelled as holding 457 votes actually held 466 votes, and that as a result, 9 votes went uncounted which were lawfully cast.
The Recount Board decided that it lacked the authority to determine that the ballots were lawfully cast, and so did not include the 9 extra ballots in its recount. The notice of contest, however, makes the point that the ballots had been removed from their secrecy envelopes, which can only take place after a ballot has been determined to have been lawfully cast.
Johnson County Cured Provisional Ballot (1)
This relates to a single provisional ballot cast in Johnson County, which the notice of contest says went uncounted because of election worker error. The evidence it presents to support this claim is an apology letter to the affected voter from the Johnson County auditor which explains as much, in particular the quote "We are very sorry this happened, especially since you did everything you needed to do and should have had your vote counted".
Johnson County Signed Absentee Ballot (1)
This relates to an absentee ballot which was signed by the voter outside of the correct box. The notice of contest suggests that it is entirely reasonable for a voter to sign their name in such a way, and that as Iowa law does not entitle election officials to disenfranchise a voter based on the location of their signature, only its absence, the vote should be counted. The evidence presented for this is an affirmation by the affected voter that by signing as he did, he was affirming that he was legally entitled to cast said vote as required by Iowa statute.
Johnson County Voters With Pre-Sealed Ballot Envelopes (2)
This relates to two voters who received a sealed envelope with which post their absentee ballot back, as opposed to an unsealed envelope. As the voters unsealed the envelope to post their ballot, their votes were not counted as they looked as if they had been tampered with. The notice of contest makes the argument that as the voters were acting in good faith, their votes should be counted. For evidence of this good faith, they refer to various measures the two voters took at the time to try to ensure that they voted within the law, as well as affirmations from the affected voters.
Johnson and Scott County Voters Who Affirm They Sealed Their Ballots (5)
This relates to five ballots that were not counted due to not being securely sealed. The evidence for contesting this is in the form of affirmations from the affected voters attesting to the fact that they did seal them to the best of their ability.
Absentee Ballots Timely Returned to Auditor's Office (2)
This relates to two absentee ballots that were marked as received by election day - evidence for this is presented in the form of a scan of the ballot envelope - but were rejected as untimely because the Linn County auditor did not deliver the ballots to the correct county officials for counting before election day.