Davene Roseborough still can’t believe the number of foster homes she lived in that had locks on the refrigerators and cabinets. At one spot, Roseborough was barred from eating dinner if she wasn’t home by 6 p.m. That place didn’t last long, though. She was bounced from home to home from the time she entered foster care at the age of eight to the time she aged out at 18. Those years were characterized by rampant physical and sexual abuse, for which her foster parents received a $1,200 monthly stipend. By the time she had her own child, she exemplified the kind of mother scrutinized by the very family policing system that raised her—Black, former system involvement, survivor of abuse, mental health struggles. Sure enough, her son was eventually taken from her and placed in the foster system for five years, where he also experienced abuse.
The Administration for Children’s Services, the New York City agency tasked with investigating allegations of child abuse and neglect, has been algorithmically encoding this kind of profiling since 2018. Before DOGE lived and died, before ICE contracted Palantir to supercharge mass deportation, before ChatGPT even launched, ACS was already using AI. What started with a single predictive analytics score meant to identify children at a high risk of severe harm has since grown into a six-algorithm suite of surveillance, which operates with very little public oversight.
Yearly AI disclosures required by city law don’t include detailed information about what data the algorithms are trained on, or how exactly they are used by caseworkers. Despite the lack of transparency, what is evident is that the algorithms function as an expensive way to perpetuate the status quo—encoding extant and extreme human biases in the family policing system into algorithmic ones, and enabling a policing agency to claim it has the technical capacity to oversee the welfare of New York’s nearly 2 million children.
ACS says there’s no need for public oversight of the algorithms because they simply prioritize cases for ACS employees to review, rather than make final decisions. It was only last summer that an investigation by The Markup revealed that the originating predictive algorithm uses 279 variables to assign families a risk score, based on cases from 2013 and 2014 that ended in a child being severely harmed. Though there are some ostensibly reasonable factors, like if a child tested positive for drugs or alcohol, many are completely outside a parent’s control, nakedly use proxies for race, or otherwise indicate poverty rather than abuse. The mother’s age, the neighborhood the family lives in, and whether the parent was abused as a child are all factored in. So is the state of the family’s home, the family’s “financial resource management,” and whether a parent is “unable/unwilling to meet child’s basic needs for food, clothing, shelter and/or medical care,” which quite literally equates conditions of poverty with neglectful or abusive parenting choices. Past involvement with ACS, including involvement that did not result in any sort of finding of abuse or neglect, is also a factor, meaning that ACS subjecting a parent to surveillance serves as its own justification.
When it comes to the family policing system, the problem is obvious: Parents are punished for being poor, Black, or both—now with the help of algorithms. Stevie Glaberson, a former family defense attorney at Brooklyn Defender Services and the current Director of Research & Advocacy at the Center on Privacy & Technology at Georgetown Law, said inputs that penalize poverty and race emphasize the way AI takes in human prejudice and spits it right back out. “Data doesn't just exist in the world. Choices are made at every point. Who are we watching? What are we collecting? What are we noting? And what societal conditions does it reflect?” Glaberson said. “With child welfare, it always, always will reflect racial, poverty, and disability-based biases.”
ACS’s own internal technical audit of the original algorithm “found substantial variation across race/ethnic groups,” which the agency says reflects genuine differences among racial groups and also “some implicit and systemic biases.” The “some” in question: A 2020 racial equity audit commissioned by ACS described “a predatory system that specifically targets Black and brown parents” where “symptoms of poverty are frequently punished as signs of neglect.” Neglect, as defined by ACS, includes things like a lack of adequate food, clothing, shelter, and medical care. The audit surveyed dozens of ACS’s own workers, who consistently reported that allegations of neglect permanently entangle poor families with the family policing system.
ACS is a relatively recent development in New York City’s history. Then-Mayor Rudy Giuliani created the agency in 1996, following public outcry after a 6-year-old girl was beaten to death by her mother despite attempts by her relatives, neighbors, and teachers to warn officials that her life was in danger. Fear of child abuse and death, the beating heart of the agency’s origins, still motivates much of the justification for the status quo. The New York Post meets its annual quota for filth by hammering ACS for failing to prevent one or two shocking cases of abuse every year. But 30 years after the agency was created, abuse cases are not the focus of most of its investigations. In 2023, 75 percent of ACS cases alleged neglect alone. ACS found almost 80 percent of cases to be unsubstantiated that year. Year after year, neglect cases and unsubstantiated cases make up the vast majority of the family policing system.
As mayor, Giuliani cut more than $1.6 billion from public schools in his first three budgets. He cut subsidies to city hospitals, which have a mandate to serve poor New Yorkers, by nearly 85 percent. At the same time, Giuliani oversaw a more than $1 billion increase in the NYPD’s budget to help oversee and enact broken windows policing. By creating ACS while he decimated the city’s social safety net, Giuliani contorted children from a collective responsibility to an individual one. The negative consequences of social abandonment were transmuted into child neglect, a crime for which individual parents could be blamed.
But the question remains: When a parent does not have the resources to care for a child, what should happen to that child? For Black and Latino children, the city’s answer is usually intervention by ACS. While only 48 percent of NYC’s population is Black or Latino, 82 percent of ACS investigations were of Black or Latino families in 2024. The same year, more than 86 percent of the children in New York’s ACS-run foster care system were Black or Latino. Poverty rates also correlate heavily with ACS investigation rates; the poorer the district, the higher the ACS investigation rate. These disparities existed before ACS began using AI, and they still exist, years later.
Richard Wexler, the executive director of the National Coalition for Child Protection Reform, once described risk algorithms as self-fulfilling prophecies rather than predictions. “Proponents say the model is successful if it flags cases that are more likely to result in substantiation or more likely to result in placement in foster care. But if a family is poor, and the poverty is confused with neglect, and a year later the family is still poor, of course there’s a good chance the poverty will be confused with neglect again,” he said.
Still, ACS argues, “predictive risk models have the potential to improve child welfare outcomes by improving resource allocation.” Every year, ACS receives more than 65,000 calls to the city’s child protection hotline. At any given time, the agency has roughly 50,000 active cases open. This enormous mandate is why even the most clear-eyed reckonings with the biases of the family policing system are often presented alongside the plight of the ACS worker.
In a New Yorker article published in 2017, Larissa MacFarquhar initially writes from the perspective of a parent forced to open the door for an ACS caseworker, who proceeds to strip your kids to search them for bruises, and who may take them away immediately and tell them false things about you, forever traumatizing them and subjecting you to terrifying and tedious humiliations to get them back. MacFarquhar follows this scene by writing from the point of view of an ACS caseworker, faced with the impossible challenge of making a home visit and quickly deciding if a parent is neglectful or simply poor. If you make a mistake, you might risk a child’s life, and if you make the difficult choice to take a child away, you might witness unimaginable anguish.
To be fair, it is impossible for a worker to do this job. ACS, as a policing agency, is not especially capable of fostering social health, well-being, or even safety. That’s not what it was made to do. For Joyce McMillan, the executive director of Just Making A Change for Families and a parent who has faced ACS investigations, the outcomes speak for themselves.
“Children age out into homelessness. There are a gazillion lawsuits for children being beaten and battered and sexually abused, and all of these other things while in the foster system,” McMillan told me over Zoom. “Then we have children who don't graduate high school because they change homes regularly, and every time they change homes, they change schools, caregivers, their entire surroundings, they’re completely disoriented. Many of these children, over half of them, end up involved in either the juvenile justice system or the adult prison industrial complex.”
Those inclined to accept these outcomes as necessary evils to prevent the death of even one child don’t have much evidence to justify the trade. Last year, a major study of 3.4 million records of children in foster care found no relationship between removing kids from allegedly abusive or neglectful homes and child mortality. It didn’t go down when more kids were put in foster homes, and it didn’t go up when more families were kept together.
The city doesn’t have the money to support families. That’s why it has to spend billions of dollars on policing them. But the city also doesn’t have the money to police all the families it needs to, so it has to spend even more money on AI to decide which families need policing the most. Austerity, disaster, then AI acting as austerity by another name, on and on in a perfect loop.
“The allure of using AI and algorithmic decision making in government services is very much a product of the assault on government over the last couple of political generations and accompanying hollowing out of government's ability to serve people,” said Kevin De Liban, a former Legal Aid attorney and the founder of TechTonic Justice, a nonprofit focused on how AI shapes the lives of low-income Americans. A cobwebbed social safety net results in policing agencies like ACS improperly and impossibly tasked with social welfare. It’s only too easy for enterprising AI adventists to promise to use algorithms to ease the burden.
Erin Dalton, New York’s current head of Social Services, pioneered predictive algorithms in family policing while serving as the director of the Allegheny County Department of Human Services in Pennsylvania. Allegheny’s algorithm is infamous for targeting low-income and Black families. In its first iteration, the Allegheny family screening tool discriminated against people for using the kind of public benefits that Dalton is now in charge of administering to New Yorkers. When defending the algorithm to the Associated Press, Dalton said, “Workers, whoever they are, shouldn’t be asked to make, in a given year, 14, 15, 16,000 of these kinds of decisions with incredibly imperfect information.”
Concern about workload is so great that one of the six predictive algorithms used by ACS “estimates weekly caseloads by area for the next 12 months” to "identify how busy certain neighborhoods are likely to be” and “optimize workforce distribution across different boroughs.” There will never be an algorithm that can reverse the societal-level impact of state abandonment. But we will do anything—including profile entire neighborhoods into literal heat maps of alleged future child abuse—other than fund universal social goods.
Despite growing recognition of the system’s failures, there is still far greater political willpower to pursue punishment for families than soaring anti-poverty measures. It’s somehow easier to look at the mess of family policing as it exists and promise to use AI to make it slightly better, for the workers who run it. In May, the Administration for Children and Families at the U.S. Department of Health and Human Services announced a $6 million investment “to expand states’ use of technology solutions, including predictive analytics.”
This may seem too grim a diagnosis, given the significant reduction in child removals and increased emphasis on ostensibly voluntary preventive services. The same year ACS began using AI, Congress passed the Family First Prevention Services Act, which prioritizes keeping families together and restricting foster placements. In recent years, ACS has spent as much on preventive services as it has on investigations, investing more than $300 million a year on things like parenting classes and therapeutic mental healthcare. Mayor Zohran Mamdani appointed a new ACS commissioner who lived in foster care herself and champions preventive services that avoid hastily accusing families of child abuse or neglect and separating them.
But families and legal advocates say that just because there are fewer child removals doesn’t mean there’s less family surveillance, or that “support” means material support. “What often happens is when a family comes to the attention of [ACS], a service plan gets generated, not often with their input, and they’re often these cookie cutter service plans,” Glaberson said. “Things like take a parenting class, do a mental health evaluation, go to therapy, and comply. Compliance is a huge thing, and drug testing, even if there’s no allegations of drug use. And so families are sent to run around town to comply with all these services, without regard to whether they have jobs and whether they have the time for stuff that is not relevant at all to their needs or experience.”
The hundreds of millions that ACS spends on preventive services focus heavily on mental health rather than material needs. Ashleigh Washington, a social worker with a doctorate in social welfare from the City University of New York, points to a preventive service that addresses domestic violence as illustrative of the difference. "You're working with the harming party, the non-harming party, and the child. There's caseworkers assigned to all aspects of that. You have a clinical supervisor, you have a deputy director, an assisted deputy director, a team of case planners, a clinical supervisor, social workers,” Washington said. “There is probably very little in those budget lines for if the person needs to move out of the state to get away from an abusive partner.”
Kiara Harris, a mother of a daughter with autism, came to the attention of ACS because of domestic violence, and she agreed to preventive services for fear that saying no might lead ACS to take her child away. She also hoped the services might help her secure diapers and other basic necessities. Instead, they took the form of frequent unannounced visits to her home and her daughter’s school and doctor’s office. This is the kind of support ACS is using algorithms to flag families for. Harris compared the preventive services to living with her abuser. “That's like second nature, learning how having to mask how you feel or act like something isn't bothering you. Being in that relationship, that became survival,” Harris said. “I remember being really, really depressed and being extra mindful to not show any type of weaknesses in front of them when they would come and do the visits.”
ACS claims to use AI to meet some material needs. Shruti Pandey, the Director of Predictive Analytics at ACS, describes the agency’s algorithms as tools to “flag high-risk cases for support,” emphasizing a certain service sensibility. One of the algorithms, Pandey says, predicts which families are most likely to become unhoused and allows ACS to prioritize them for housing vouchers. But ACS only has 100 of these to distribute. The algorithm serves as a means-testing triage system for suffering. Which families deserve to win the lottery and secure one of these 100 vouchers? Is it too grotesque for a human to decide? Would it be easier for everyone if an algorithm could tell us who is more or less worthy of being left behind? Algorithmic prediction helps obscure the political choice at play, painting over indefensible abandonment with the figment of mathematical rigor.
Though AI marks the latest update in this trend, we’ve long ignored the fundamental contradiction between policing and support. We’ve banged our heads against the wall of this lesson for decades, with succeeding waves of policy choices that do anything but address the root cause of the vast majority of suffering among children: poverty. Davene Roseborough has waited almost her whole life for something to change. “I'm almost 50 years old. This happened to me when I was eight, so that's about 40 years ago,” she said. “And children are still not getting what they need. They're still not getting the support that they need.”
