Criticising 'worldbuilding' is pretty popular these days. In an age of megacorp-owned franchises pumping out vast quantities of sludge and the increasing visibility of dysfunctional fandoms filled with obsessive misanthropes, 'worldbuilding' is decried as anathema to good works of storytelling, which tastefully and respectfully give the reader's imagination breathing room to fill in gaps and flesh out things only hinted at. And allowing the reader room to imagine is a good thing! But is it the fault of 'worldbuilding' when this doesn't happen? Sometimes, but I don't think this is an indictment of 'worldbuilding' as a whole.
First of all, what is 'worldbuilding'? Why am I always surrounding it in quotes? Well, like many other things around which it is nearly impossible to have a reasoned, productive discussion, 'worldbuilding' is always vaguely defined in a way that is always, coincidentally, most useful for making whatever point about it someone wants to make. At its core, 'worldbuilding' is setting-building—setting being one of a triad of top-level story ingredients, along with the characters and the plot. Speculative fiction gets hit by this the worst both because it, by its nature, needs its creators to expend effort constructing the unreal setting in which their fantastical story may take place, and because the lesser works of the genre have accrued a reputation for focusing on setting to the detriment of compelling characters or interesting plot. But bad setting-design does not condemn all setting-design, and a properly erected, verisimilitudinous setting is in any case a crucial component of just about any narrative work, whether it takes place in 1990s Brooklyn or the fantastic reaches of West Meoley.
So maybe 'worldbuilding' isn't synonymous to setting-building, but we can narrow it down to unnecessary setting-building. Maybe it's unnecessary because its excessive detail bogs down the reader and makes them lose track of the plot, or because it's mostly unrelated to the characters and plot the reader cares about and confuses them with spurious information. But it hardly follows that just because a story can be ruined by adding too much detail to its setting, that it must necessarily be improved by cutting out detail. What might be superfluous to one reader might prove critical to another reader's ability to envision the progression of the story. The people who complain loudest about 'worldbuilding' per se are rarely, if ever, complaining about 'too much' worldbuilding or pointing out works that would have benefited from 'more worldbuilding'. In any case it seems ridiculous to use 'worldbuilding' to refer to badly written internally consistent fictional settings, as if the act of building a world was in and of itself characteristic of authorial incompetence.
So what now? Here are some forms that 'worldbuilding' may take, and what I think about them, for what little it matters.
First of all are the people who construct a fictional world recreationally by defining its parameters (its cultures, peoples, history, geography, languages) as they see fit. Many of these people never intend to turn their efforts into any sort of story; they build their fictional world for its own sake. I don't think a literary criticism is appropriate in this case—you cannot justly criticize something for doing something that you think leads to bad stories, if they're not doing it for the sake of story to begin with. And it's difficult to see how this hobby, if strange, is any more morally deficient than any of the other strange-but-harmless things people spend their free time doing.
(There are, unfortunately, those who do try to ascribe some sort of moral value to the hobby of worldbuilding, calling it the sign of an authoritarian personality, the hobby of an incipient fascist, etc. This transparently stupid sort of criticism does not merit any further discussion except to point out that it shows up elsewhere as well—I had the misfortune of reading a screed leveling the same charges against programmers who preferred statically typed languages over dynamically typed ones. From an empirical standpoint, the actual would-be tyrants out there are not, for the most part, drawing up tediously detailed timelines of fictional medieval kingdoms or writing burrito-based monad tutorials, although a few of them must be given how large the groups in question are.)
Next, you have the misguided people who construct a world, write a story that is lacking in artistic merit, and then try to justify their incompetence by pointing to the detailed, intricate world that they've built. This is the curse of bad hard science fiction, whose defenders try to excuse paper-cutout characters and implausible social interactions on the basis of the veracity of the science (in practice, just physics, computer science, or aerospace engineering). If someone thinks that bad writing can be saved by excessive 'worldbuilding', by all means call them out for it. No amount of fictional backstory will make a boring character interesting or a nonsensical plot start to make sense, and only here can I concede the point. "The plot sucks, the prose is awful, but at least the world is awesome!" Who cares? The world can't rescue the plot or the prose. But on the other hand, would making the world worse make the story better in any way? Perhaps you can argue that this writer should spend less time worldbuilding and more time honing their craft in other ways, and your advice would be sound. But that isn't a flaw intrinsic to worldbuilding as a concept.
Proceeding on, you have those who are able to write a decent story populated with interesting characters carried along by an interesting plot, and they set their story in a universe that they have detailed so much that parts of it do not relate to the plot or characters at all. My favorite example of this is Mass Effect, a game which has indeed received some degree of acclaim for how deeply realized the fictional setting it takes place is. The 'world' (galaxy) of Mass Effect is built not just by things you learn by talking to different characters, but also by the snippets of text describing all the completely optional planets you can visit, and by an in-game 'Codex' containing dozens of encyclopedia articles describing different aspects of the Mass Effect setting. This is classic worldbuilding, and I would argue worldbuilding done right (at least in the first two games). You can play the game without touching the Codex and have a great time. Or you can read about how starship radiators use drops of liquid metal to shed waste heat, or how ground armies are supported by large groups of robotic drones (neither topic of which ever becomes relevant to the story), but in doing so you probably feel more invested in the world, not less invested. It helps that Mass Effect falls within that genre of space-opera-cum-firm-science-fiction where the audience can reasonably be assumed to have a passing interest in how all the various wondrous machines and gadgets function, and that some of this knowledge can reward a player by providing additional context to missions or plot events that are still intelligible on their own. But if Mass Effect does 'worldbuilding' right, then we can't easily say that 'worldbuilding' itself is the problem.
Finally, we come to what I think most people are actually complaining about when they complain about worldbuilding, and in this case it's intimately tied into the nature of megacorp-owned entertainment in contemporary society. Large entertainment corporations love large, sprawling franchises that share a common setting and often a common ensemble cast of characters. They love them because they can accomplish some sort of entertainment equivalent of 'lock-in', investing consumers in these sprawling worlds so they are more inclined to consume new media set in a familiar setting. And they reduce risk—rather than having to take the chance with a new IP, they can simply mine an existing IP (and its vast troves of nostalgia, cultural import, and fan favorites) for increasingly derivative content until the end of time.
So when people complain about worldbuilding, they are often complaining about these increasingly soulless franchises and the tools they use to ensure the longevity of the intellectual property. Worldbuilding is, admittedly, a huge part of this process. It is the continuity undergirded by an internally consistent fictional setting that allows consumers to accrue 'mastery' of the franchise and their emotional investment in it. Every offhand reference becomes a potential hook for a new project, extruded media product that exists not because any author felt they had a story to tell, but because this sort of reference-walking heuristic is an optimal way to generate content with a pre-included audience and some guarantee of success. And of course there are those who seek to ape the giant corporations, adopting these techniques for their own small media fiefdoms.
One major consideration is that this sort of perversion of worldbuilding is the sort of thing that only a big media franchise can reasonably hope to attain. A single author can generate but some fixed amount of work—Tolkien wrote the Lord of the Rings and its appendices, The Hobbit (which was retroactively canonized), and the notes that were later compiled into the Silmarillion by his son, as well as a host of letters. A large company can hire dozens or hundreds of writers and artists to churn out novels, comic books, movie and television show scripts, et cetera ad nauseam. This is how a single sentence in the opening crawl of Star Wars became a film (Rogue One), an offhand remark between Han Solo and Lando Calrissian became another disappointing film (Solo), a character who spent the few combined minutes he was on screen as the butt of a joke became a poorly received miniseries (The Book of Boba Fett), the story of the guy who died in the aforementioned film itself became a TV show (Andor), and so forth.
But wait! Let's look at Andor. Andor is a uncharacteristically good show—not just a good Star Wars show, but a good show, period. And nobody was expecting it to be a good show, given that its very premise seemed a ghastly parody of this corporate blood-from-a-stone worldbuilding-based content creation: "we made a movie about how the Death Star plans were stolen, now let's make a TV show about the nobody who was the main character of that movie!" But it was incredible. Tony Gilroy had a story he wanted to tell, and he was allowed to tell it, and it shows. You can see it when a previously-disaffected Andor shoots dead the scummy turncoat whose words echoed his own perspective just a few days prior, you can see it when Kino Loy rallies a whole complex of newly-freed prisoners to make good on the one way they can defy the Empire that took away their hope, you can see it when Luthien rails against the cosmic injustice, the sick cruel contradiction at the very heart of his private war against the Empire. Andor alone retroactively justifies Rogue One (itself not even a terrible film), and it goes to show that a story worth telling can indeed be told well, whether or not it was the connective tissue of worldbuilding that gave rise to that story.
Placing the blame upon 'worldbuilding' for the flaws of writers or corporate media is lazy and reductive. Worldbuilding is the author's prerogative, and theirs alone. If they want to keep everything vague and focus on the immediate story at hand, they are doing nothing wrong. If they want to build grand artifices of history and geography within which their characters can live and fight and suffer, they are still doing nothing wrong. But in the end, just as worldbuilding is the author's prerogative, telling a good story and telling it competently is their responsibility. If they fail to do so, it is their craft and theirs alone that they must work to improve, not 'worldbuilding' that must be tossed aside.