Do poor people, people of color, and elderly have difficulty in obtaining ID?
Yes.
For example, here are some facts (citations to supporting authorities are found at this link) (the omission of paragraphs from the source related to subjects not pertinent to the question are not expressly noted in the quoted material below).
Anecdotally, I worked with programs involving unemployment, child
support, TANF, SNAP, Medicaid, Medicare and a few others. At no point
in 5 years working with those programs did I hear of ID being an
issue. I know anecdotes are no substitute for data, but I haven't ever
seen this claim supported with any real data.
The real data quite overwhelmingly establishes that your anecdotal impressions are inaccurate.
11% of U.S. citizens – or more than 21 million Americans – do not have government-issued photo identification.
Note that the source for this statistics clarifies that this is 11% of voting-age American citizens.
A demographic breakdown of adult U.S. citizens with and without photo IDs can be found here. It shows that non-whites are much more likely to not have photo ID than whites (even for high income non-whites and non-whites who are not young adults), that not having a photo ID is strongly correlated with having a lower income (especially among whites), that young adults are particularly likely to not have photo ID (especially among whites). Another source has a convenient chart breaking down some key data points:

Even if ID is offered for free, voters must incur numerous costs (such as paying for birth certificates) to apply for a
government-issued ID. Underlying documents required to obtain ID cost
money, a significant expense for lower-income Americans. The combined
cost of document fees, travel expenses and waiting time are estimated
to range from $75 to $175.
The economic burden and delay make it difficult for poor people to obtain ID, and this issue empirically, is much bigger than most middle class people intuitively expect as illustrated by the large percentage of adult U.S. citizens who don't have IDs.
Young married women whose names have changed are particularly heavily impacted by these costs, and even if one can obtain a suitable ID eventually, it may be challenging to do so in time to vote in the next election (particularly, for example, if you were born and/or married outside the state where you live).
The travel required is often a major burden on people with disabilities, the elderly, or those in rural areas without access to a
car or public transportation. In Texas, some people in rural areas
must travel approximately 170 miles to reach the nearest ID office.
These factors make it more difficult for elderly and disabled people to obtain IDs.
For example, I have a wheelchair dependent client in a major urban area for whom I make house calls as a lawyer, because it takes him four hours roundtrip by bus, and about an hour and a half (at a much higher cost than an Uber or taxi trip) by a specialty wheel chair transportation vehicle that is often not on time for pickups, to travel from his suburban home to my central city office which is near government offices.
Nationally, up to 25% of African-American citizens of voting age lack government-issued photo ID, compared to only 8% of whites.
The difficulty in obtaining government-photo IDs for African-Americans is strongly suggested by this difference in percentages. Most of the difference is mediated by differences in affluence, including a much higher rate of not owning a motor vehicle, and thus not getting a driver's license (the most common form of photo ID).
As evidence from a 2014 court case showed:
In May 2014 federal Judge Lynn Adelman found Wisconsin's state photo
ID law unconstitutional given its adverse impact on many Wisconsin
citizens. In his 90-page decision . . . [he notes] that only 47% of
black adults and 43% of Hispanic adults compared to 73% of white
adults in Milwaukee County hold valid driver's licenses as do 85% of
white adults in the rest of Wisconsin compared to 53% of black adults
and 52% of Hispanic adults.
Some of this is due to the common discriminatory practice of not locating government offices that provide these services in or near neighborhoods where most African-Americans level, or of understaffing those offices so that delays are longer.
An NPR story explores some of the reasons for the differences:
I think the first thing to look at is to look at who actually drives.
The most common form of government-issued ID are driver's licenses and
so the people who are most unlikely to drive, as it is, is elderly,
the poor, people who live in big cities, like African-Americans,
especially young people, too, especially if they attend college. They
may not have need for a car at the moment.
And then people who are in rural areas. The other challenge for them
is they are not near the Department of Motor Vehicles offices, etc.,
etc. where you would get these IDs. . . .
if you're someone trying to get a voter ID, you need that type of
documentation. In order to get an ID, you often need an ID, so it
becomes a Catch 22.
MARTIN: And what about - are the states making any provision to help
voters like this who have not previously had IDs...
DADE: Yeah.
MARTIN: ...to get them? Or are interest groups doing that?
DADE: Well, they are. The interest groups are trying to fill in the
void of information, where to get the IDs, what kind of documents you
need to get them when you show up so you're not caught unawares.
As far as the states go, they are issuing non-driver voter IDs. Many
of them are for free, but the problem is, when you go to Department of
Motor Vehicle offices, the waits are very long, they're time
consuming. The governor of Tennessee, who's a Republican, by the way -
he has expressed concern about the average wait times there, which
extend well beyond an hour. And if you're talking about thousands of
elderly, in particular, they can't wait that long.
A study of Texas voters who tried to vote without photo IDs found some of the following issues came up:
Texans offered various reasons why they didn’t have a qualifying ID
for the 2016 general election. Nearly 30 percent said their IDs had
been lost or stolen. About 11.5 percent cited work obligations while
another 4 percent said family obligations prevented them from getting
one.
Nearly 36 percent of individuals without IDs checked the “other” box,
many of whom indicated they had moved so their current address didn’t
match what was listed on their IDs.
Of those who selected the “other” option, 1.4 percent — 82 people —
cited cost as the reason they didn’t have the appropriate ID.
This isn't a representative sample and doesn't include the lion's share of voters without photo IDs who don't try to vote, but it is nonetheless hard data that provides some insight.
A meta-analysis of 31 other studies on the topic found some key factors. The main reason people reported for not having identification was that it had been either lost or stolen. Other factors included lack of money for fees, lack of knowledge of application process, lack of competency with bureaucratic forms, the requirement of an address, and the requirements of an existing piece of identification in order to apply for additional identification.
For people who are economically marginalized even seemingly minor fees constitute a financial hardship that makes the acquisition of PID prohibitive, and sometimes they are funneled though like of sophistication to commercial fee charging services that facilitate services that the government doesn't charge for or charges much less than the service does for.
[F]ees are charged for replacement birth certificates, and if people
go through “third party” providers rather than state agencies to
obtain this form of ID, additional service fees are incurred. This
means that people who have little or no money and who are likely to
lose or have their PID stolen . . . are further burdened with higher
replacement fees. Ultimately, people regularly prioritize the
immediate needs of food, transportation, or rent rather than the costs
of replacing a lost or stolen document. Furthermore, additional costs
are required if individuals must take public transportation or live in
rural or remote locations and have to travel to service centers.
Additional Burdens For Proof of Citizenship
Also, notably, proof of citizenship is much more challenging to marshal (usually requiring a birth certificate that many people are not currently in possession of, marriage certificates to document change of name, naturalization certificates, and/or passports), then the proof of identity needed for a photo-ID without citizenship verification such as driver's licenses or state IDs that do not require proof of citizenship:
A federal district court that heard a challenge to the Kansas law
noted that “the sheer number” of people blocked “evidences the
difficulty of complying with the law as it is currently enforced.” The
court found that more than 18,000 people who tried to register at the
state’s motor vehicle offices were prevented from registering between
2013 and 2016 “as a direct result” of the state’s documentary proof of
citizenship requirement.
This is because getting a birth certificate and linking it to your identity, or having a passport, can be cumbersome taking time, money and even legal assistance.
A story from the Washington Post puts flesh on the bones of the raw statistics and illustrates in personal and compelling examples, how these factors play out in the lives of individual people without photo IDs. One such story is this one from Texas:
Many of the residents struggling to obtain a valid photo ID are
elderly and poor and were born in homes rather than hospitals. As a
result, birth certificates were often lost or names were misspelled in
official city records.
Hargie Randall, 72, was born in his family’s home in Huntsville, Tex.,
and has lived in the state his entire life. Randall, now living in
Houston’s low-income Fifth Ward neighborhood, has several health
problems and such poor eyesight that he is legally blind. He can’t
drive and has to ask others for rides.
After Texas implemented its new law, Randall went to the Department of
Public Safety (the Texas agency that handles driver’s licenses and
identification cards) three times to try to get a photo ID to vote.
Each time Randall was told he needed different items. First, he was
told he needed three forms of identification. He came back and brought
his Medicaid card, bills and a current voter registration card from
voting in past elections.
“I thought that because I was on record for voting, I could vote
again,” Randall said.
But he was told he still needed more documentation, such as a
certified copy of his birth certificate.
Records of births before 1950, such as Randall’s, are not on a central
computer and are located only in the county clerk’s office where the
person was born.
For Randall, that meant an hour-long drive to Huntsville, where his
lawyers found a copy of his birth certificate.
But that wasn’t enough. With his birth certificate in hand, Randall
went to the DPS office in Houston with all the necessary documents.
But, DPS officials still would not issue him a photo ID because of a
clerical mistake on his birth certificate. One letter was off in his
last name — “Randell” instead of “Randall” — so his last name was
spelled slightly different than on all his other documents.
Kamin, the lawyer, asked the DPS official if they could pull up
Randall’s prior driver’s-license information, as he once had a
state-issued ID. The official told her that the state doesn’t keep
records of prior identification after five years, and there was
nothing they could do to pull up that information.
Kamin was finally able to prove to a DPS supervisor that there was a
clerical error and was able to verify Randall’s identity by showing
other documents.
Is Lack Of A Photo ID Indicative Of Being Ineligible To Vote?
It also bears noting, while slightly off the narrow scope of the question, that the likelihood that an African-American who speaks English with an American accent (a group that makes up the vast majority, probably 98% or more of African-Americans who do not have IDs in all but a handful of counties in the United States) not being a U.S. citizen is vanishingly small.
Put another way, virtually all of the 25% of African-Americans adults without IDs in the United States are U.S. citizens who are eligible to vote (absent felony disqualification which is present mostly in states that also impose other barriers to voting like voter ID laws and also have a long history of racial discrimination). The share of African, Afro-Caribbean, and black Latin American legal immigration to the U.S. is modest, many of those immigrants are naturalized citizens, and those immigrants overwhelmingly have passports. About 6-7% of undocumented immigrants in the United States are black (see here and here) and those undocumented immigrants are overwhelming recent immigrants.
Voters not infrequently have some form of ID but not one of the forms of ID required by Voter ID laws. The types of IDs allowed are tailored to political demographics:
States exclude forms of ID in a discriminatory manner. Texas allows concealed weapons permits for voting, but does not accept student ID
cards. Until its voter ID law was struck down, North Carolina
prohibited public assistance IDs and state employee ID cards, which
are disproportionately held by Black voters. And until recently,
Wisconsin permitted active duty military ID cards, but prohibited
Veterans Affairs ID cards for voting.
One particularly notorious case involves South Dakota's unwillingness to accept Indian tribe IDs, an especially big factor for people who live on Indian reservations that are remote from non-tribal government offices, who are often poor, and are a group of people that almost by definition is made up 100% of U.S. citizens.
Selected Sources
The sources for the fact cited above include the Brennan Center Brennan Center for Justice, Citizens without Proof: A Survey of Americans’ Possession of Documentary Proof of Citizenship and Photo Identification (New York: Brennan Center for Justice, 2006), a report from the federal government's General Accountability Office (GAO), a published academic journal article with three University of California at San Diego professors as authors, and a Harvard law professor's published academic article, Richard Sobel, The High Cost of ‘Free’ Photo Voter Identification Cards (Cambridge: Charles Hamilton Houston Institute for Race and Justice at Harvard Law School, 2014).