Benchmarks don't do a good job at capturing the difference in quality of frontier models over smaller models.
Opus 5.5 was so good that if it wasn't nerfed - at least just looking at the benchmarks - the difference in cost here does not seem to justify using this for anything besides design and review.
That's like maybe 1% of total token usage for SWE BUT a better design can have enormous benefits, so it's almost always worth using the best model for that.
I at least didn't see this. I had a baseline evaluation from before Nov 3 on Opus 5.5 medium, so I re-ran it this morning.
Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.
20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.
Caveats: one run per task today, medium effort only.
Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances
so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted
I used to be skeptical of nerfing, but as of a week or so ago I've noticed a severe degradation of quality for GPT Astra. It struggled with very simple tasks, and it often failed to follow my instructions.
I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.
I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.
People are prone to seeing patterns in noise especially after suggestion. It's not hard to imagine someone concluding the model is nerfd after few bad responses in a row
idk, it hasn't felt nerfed at all to me, and it's not like i'm just doing trivial CRUD webapps either. opus 5.5 is very successfully porting my phone to postmarketos. claude just got the touchscreen up and working (and made a little program for me to test it, and it's legit).
If this turns out to be accurate, I think that would be a pretty unethical thing for Anthropic. Even if it is just a "we had to allocate compute to this new model release".
It should be fraud to swap out the model. We need a term for this, something like a bait-and-switch.
With early handouts of Qwen4 starting to be reported 5 days ago ... so if they're good, they would have had some hours to evaluate Qwen4 against Claude 5.5 and, then or right after was when Anthropic decided to put out 5.2 ...
So yeah ... benchmaxxing and all that. BUT nevertheless, the published results seem to say that Qwen4-Max appears to match Opus 5.5 on some benchmarks, and Qwen4-27B (probably by going even harder than Qwen3.8-27B on thinking/inference time compute) can beat Sonnet 5.5 on some things.
I guess they're not taking any chances. Keep in mind that Anthropic has declared they have non-public models (it's in their IPO filing). These are referred to by the shorthand names Model 2, Model 3, and Model 4. Essentially 4 is better than 3 is better than 2.
Fable 5.2 is "2.5" so to speak. So Anthropic was holding back, but not anymore. And now they can dial the performance up once more with what they have ready, and probably once more after that.
> Our first formalization of Fermat’s Last Theorem, performed by an unreleased research version of Claude, took 11 days. It relied on a purpose-built harness along with guidance from a human researcher. By contrast, a team of 1,000 Fable 5.2 agents fully formalized the theorem in just 30 hours, with minimal human intervention. Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter. As can be seen in the figure below, Opus 5.5 (with the same 1,000-agent setup) completed only 26% of the formalization in 18 hours, and barely made any more progress after 30.
Most people don't really have intuitions on what it means to push the frontier now (because AI has surpassed everything they are good at, for example, coding). We are truly inside the singularity (using Kurzweil's definition, slightly paraphrased - the time when technology progresses so fast no one can keep track with it).
I feel like I'm increasingly reading AI current events is more like learning about historical events. By the time information has entered my head, there is even newer information out there that already makes what my brain is only currently actively processing, obsolete.
> We're all going to be harness engineers in the future
No, that will be the model's job. We will largely be unemployed. Of course, we can surely depend on the largesse of Altman and Amodei to keep us fed and sheltered.
Isn't this part of the lore behind the Dune universe?
Like, we're dropped into a post post post singularity world where AI already did all of that and is banned? as kind of a footnote that's never further explored in movie renditions that have been released
> We are truly inside the singularity (using Kurzweil's definition, slightly paraphrased - the time when technology progresses so fast no one can keep track with it).
I gotta admit, I am jelly as fuck I don't have access to a billion dollar datacenter. Would LOVE to put 1000 fable 5.2 agents to self-coordinate and get them to do something for me. Fuck life is unfair.
I’d be happy even without having access to a billion dollar datacenter, if I could use local models properly, but even that requires some serious hardware that is out of my reach. We live in weird times: I have lots of hope for the future, but meanwhile AI access is pretty iniquitous indeed.
Nerfing an existing model is the biggest crime in this space. I can’t imagine Dario okaying this because it’s what will immediately destroy your brand.
“…a reference to the repeated (and always erroneous) claims made after almost every patch that Onyxia was using her Deep Breath ability more often than before.”
Setting aside the comments about nerfing, do normal mortals out there have workflows that really require Fable/Mythos and don't just work fine on Opus 5.5?
I've been using Opus 5.5 on xhigh for some decently aggressive software development on a ~125k-line project, and it has been fine for me. I'm not really willing to dive into a "rewrite the whole project, don't make any mistakes" level of refactoring. I'm not looking to try to prove Fermat's Last Theorem myself. Is there anything that I'm truly missing?
I’m in the same boat. Though I will occasionally use Fable to review things to see if it can pick up anything but I’ve honestly not found it to be any more effective than Opus.
I’ve had a chance to use fable 5.2. Some observations:
- Anthropic is finally well balanced across models. Meaning: Opus 5.5 is so good that I do not feel Fable FOMO as I did when they were paired with Opus 4.8/5. And Sonnet sometimes makes me forget about Opus.
- Fable is good, but you really have to push the ambition of your prompt in order to see proper separation from Opus 5.5.
- It’s way faster and more reliable: one of the worst things about fable 5.1 was how slow it was an, on top of that, how frequently it would suffer from max output token overflows. I got none of these issues with 5.2.
- Fable is truly exceptional at drawing now. Fable’s ability to render an image with JavaScript actually has me questioning the value of dedicated image models. I prefer the images I’ve gotten fable/opus to make for me over the gpt-image lineup, it’s wild to think we’re at this point.
There were more than a few "leaks" saying it would be called Fable 5.5, so I'm curious if a) the leaks were just engagementbait or b) Anthropic changed it, despite the fact that Fable 5.2 being better than Opus 5.5 is confusing.
Surprisingly the model is available immediately in my Claude Code: usually there's a delay, either for rollout or safety.
I assume because Opus 5.5 was a huge improvement over 5.0 (or was it 5.1? i forget). Fable still talks a fair bit like Opus 5.0 for me, so i'd quite enjoy a Fable equivalent to 5.5. Ie i'm quite interested in Fable 5.5.
Curious to see how 5.2 measures up in producing readable output.
They should be going public within the next few weeks and that would be a very short interval between releases. (And despite the chatter I'll be very surprised if they delay the IPO; it would be a massive PR hit and make it even harder for them in the future).
My speculations: maybe this wasn't as much of an improvement over 5.1 as they were hoping, so they're saving 5.5 for the next major update (possibly "Model 4" from the METR report). Or, maybe capacity is constrained so "5.2" is a subtle encouragement to stick with Opus unless a task really needs the increased intelligence.
Maybe it's just me, but I found Claude's writing style terrible compared to GPT or Gemini. It's "smart", but the way it communicates seems somehow off compared to other models.
I really didn't like Opus 5 and was vocal about it. Opus 5.5 is noticeably better, but literally just yesterday I was fighting parsing some summary it had given me and it was a bit like the old days. IOW, it is better but still has some room to improve.
Yep I find the way Opus 5.5 talks to me really annoying and irritating. Quite condescending and sanctimonious.
I prefer Gemini for day to day use as I find it a lot more agreeable to be working with all day every day, and Gemini is pretty much on a par for coding Vs Opus 5.5, at least for my needs.
No its just straight up terrible. I tried out Gemini Argon when it released and my god did it write beautifully. Even the 3.8 Flash models are way better at prose, i do a lot of automatic scriptwriting for youtube and 3.8 flash is leagues ahead of anything from anthropic/openai
I enjoy a lot Argon, maybe more than chatgpt, but damn, like all Gemini models, lots of hallucination and sycophant to the max, so a bit useless for agentic coding.
I've found Claude's writing between 4.6 until 5.0 to be unacceptable if you are someone who is juggling 4-5 Agents. The Sol 5.X models all seemed to have great writing. Now with Opus 5.5 and Sol 6.1 I've found the trend is reversed -- Sol is now unbearable and Opus is workable again. Whatever caused the issues for Anthropic seem to be hitting OAI now.
sol is ok. it still such a superthinky model that it gets tangled up trying to present way too much? i struggle to connect with it often, have ti ask it to try again.
terra has been prettt good for me for me, when i need something looked at an explained to me.
i dont ask for much non-technical output though, which i expect yeah is its own cans of worms.
My own experience having used 6.1-Sol/6-Astra and Opus 5.5 extensively, Opus is more verbose and chatty in it's user communication (not code though, code is usually more straightforward & cleaner than GPT models). Multiple times I've gone "I'm not reading all that shit" when looking at Opus 5.5 output. This was also with the concise mode enabled in Claude Code.
It could also be a system prompt + harness issue, as opposed to a model issue, I haven't done too much experimentation here.
Did you find that to be true for Opus 5.5 as well?
I couldn't use prior Claude models without rolling my eyes at the writing every few sentences, Astra was refreshing and then Opus 5.5 made even Astra look terrible in comparison.
I don't think its this. Some models across both labs have had issues with writing and communication and it typically seems to pop up on successor models of really strong ones.
Opus 4.7/4.8 heavily suffered from these communication quirks. while the Sol models at the time worked well. Right now its flipped though and Opus 5.5 has none of these issues while Sol 6.1 does.
Small example I've had today: "Me: Hey Sol 6.1 xhigh, does X have property Y? Sol 6.1: Yes, if we change X to have property Y. Right now it has property [...], [...], [...], and it does not have property Y." The model shouldn't be answering questions like that. It makes understanding extremely hard and slow. For whatever reason it's either being encouraged or at least not stomped out during RL.
The best programmers I've met or read throughout my life are phenomenal communicators. I'd even go so far as to say it's likely one of the reasons they're so good at programming. After all, the majority of our job is converting human requirements in lossy language to fleshed out behaviors in formal, lossless language.
Heck, some of the most common debates are on the semantic clarity of abstractions or syntax.
“Testers have found that Fable 5.2 and Mythos 5.2 communicate more clearly than past Fable and Mythos models. They generate outputs up to 30% faster than Claude Fable 5.1 and Claude Mythos 5.1, and they’re available at a lower price: $8 per million input tokens and $40 per million output tokens.”
Honestly incredible that they’ve managed to beat all benchmarks, while making it cheaper. Always happy to see improvements offered at lower costs.
Will be interesting to see what OpenAI does in response
I think frontier models feel terrible after a new model requires you to do less babysitting, it's just really hard to go back once you've had to handhold less.
The only way I use Astra and 6.1 Sol now is if they're being orchestrated by Opus 5.5 so I don't have to talk to them directly.
Those theoretical physics papers look like the real thing (i.e. not mathematical physics like the OpenAI Yang-Mills result). It looks like AI is finally branching out of math.
Interesting that the papers list no authors other than Claude. Also interesting that they published on "Hexagon" instead of Arxiv. I haven't heard of it before. Is it an AI-friendly alternative to Arxiv or something?
Edit: yeah apparently it's an AI-moderated alternative to Arxiv that accepts AI-authored papers (as well as human-authored papers I guess if you want).
> the number of Fable resets you get depends on how long you’ve been a paid subscriber [...] Subscription length is measured from the date of your first paid bill. If you canceled and came back, the time since your first paid bill still counts.
> These resets expire on March 8, 2027 at 11:59 pm PT.
tl;dr if you joined when Opus 4.5 became a thing, you got 2 resets.
Assuming no implicit phacking a 15 point jump on tbench scientific is a big deal.
I suspect we will see people being adamant that this model simply 'feels' better due to long horizon goal alignment improvements ( like this kind that would cause this jump)
I've very interested in the improvements to robotics control capabilities. This model was able to outperform specialized robotics models. It's the bitter lesson coming for robotics.
Doubling Remote Labor Index (RLI) - at 41.3% compared to around 20% for the other models - I wonder what kinds of new agentic work this is going to unlock.
FrontierCode 2.0 is such a poor benchmark for Gemini since it's the only model forced to use Devin CLI while other models get to use their preferred harness.
I wonder why that team singles Gemini out like that.
Yay! New model day. The most likely time the model isn't nerfed and we get a glimpse at the latest capability.
I wonder how long this kind of excitement at new models will last? I mean, now when I get a new iPhone I don't get the same rush I used to. So I wonder if this hype feeling will fade as we plateau in model capability.
Why dont I feel any urge to even try Fable 5.2 and Mythos 5.2 unless it will force up on me?
Compared Opus 5.5 200 usd a month with Codex 200 usd (Astra High) and Kimi Pro 100 usd a month ... and now have a new conspiracy theory - more their suck your code the better outcome even just over few days time on large code base.
Horrible experience correcting course of Codex and Kimi flow compare to Opus. Opus much more responsive
Dont feel any gain of Codex 200 usd over Codex 100 usd
Kimi Pro feels like a fake Louis Vuitton bag with extreme high material quality and yet still fake ... a bit trash talk ... but clearly seen not so distant future where I can live without Codex and Claude
Very interesting that they set the flagship's version number to 5.2, but their penultimate model is 5.5. Do they want people to to continue to think that Opus is based on more advanced technology? The decision these AI companies make continues to baffle me.
It's really weird that they folded Project Glasswing into the CVP, but they're still limiting the new versions of Mythos to "vetted organizations". So Project Glasswing still exists then? Oh well.
Every single time I click on these announcements the frontier model ALWAYS scores around ~60-80%. Every. Single. Time. Like, I just went back to the Opus 4.6 + 5 announcements, they were also scoring 60-80%.
For every single announcement, not only do the top models always score 60-80%, the older models somehow start scoring 20-30% (even though previously they were scoring 60%). It's completely unclear whether the methodology changes, how these scores are even evaluated or which domains are being tested. What's worse is that the "benchmarks" keep being updated with no explanation or due process.
While I like OpenCode a lot, it’s not sensible to use it for anthropic models vs. a subscription (other than to use the included API credits) in terms of cost
Which model though? Sounds pretty dystopian if there's no local option.
Even speaking as a business, why would I want a environment where every single bit of information in my business goes to OpenAI or Anthropic?
I think that ultimately depends on how many providers remain in the endgame of this. If it's only one or two companies, I don't think it'll be very fun when it's time to negotiate your token pricing contract every few years.
Why not? If we use labour as a comparison. The majority of businesses by definition do not hire the most incredibly intelligent and most competent workers and they are still competing in their niches and earning an income. Why wouldn't that be the case with choosing an open model vs fable 8.0?
stop OPUS 5.5 FUCK, DAMMIT, THEY DIDN’T JUST NERF HIM—THEY ABSOLUTELY OBLITERATED HIM, THE FUCKING MISANTHROPES! I EVEN MADE A MISTAKE IN MY ENGLISH IN THE PREVIOUS SENTENCE—NOW THAT FUCKING CLAUDE IS TEACHING ME ABOUT BUGS!
- https://news.ycombinator.com/item?id=50422069
Opus 5.5 was so good that if it wasn't nerfed - at least just looking at the benchmarks - the difference in cost here does not seem to justify using this for anything besides design and review.
That's like maybe 1% of total token usage for SWE BUT a better design can have enormous benefits, so it's almost always worth using the best model for that.
Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.
20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.
Caveats: one run per task today, medium effort only.
so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted
It probably need a whistleblower to uncover what they are doing.
I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.
If this turns out to be accurate, I think that would be a pretty unethical thing for Anthropic. Even if it is just a "we had to allocate compute to this new model release".
It should be fraud to swap out the model. We need a term for this, something like a bait-and-switch.
Anthropic does this time and time again, unserious company
Right?
https://x.com/follistatindev/status/2119129611510383014
With early handouts of Qwen4 starting to be reported 5 days ago ... so if they're good, they would have had some hours to evaluate Qwen4 against Claude 5.5 and, then or right after was when Anthropic decided to put out 5.2 ...
So yeah ... benchmaxxing and all that. BUT nevertheless, the published results seem to say that Qwen4-Max appears to match Opus 5.5 on some benchmarks, and Qwen4-27B (probably by going even harder than Qwen3.8-27B on thinking/inference time compute) can beat Sonnet 5.5 on some things.
I guess they're not taking any chances. Keep in mind that Anthropic has declared they have non-public models (it's in their IPO filing). These are referred to by the shorthand names Model 2, Model 3, and Model 4. Essentially 4 is better than 3 is better than 2.
Fable 5.2 is "2.5" so to speak. So Anthropic was holding back, but not anymore. And now they can dial the performance up once more with what they have ready, and probably once more after that.
Most people don't really have intuitions on what it means to push the frontier now (because AI has surpassed everything they are good at, for example, coding). We are truly inside the singularity (using Kurzweil's definition, slightly paraphrased - the time when technology progresses so fast no one can keep track with it).
We've sort of entered this twilight zone where most humans don't have the time, expertise, or moral compass to be able to use these models.
We're all going to be harness engineers in the future.
No, that will be the model's job. We will largely be unemployed. Of course, we can surely depend on the largesse of Altman and Amodei to keep us fed and sheltered.
Like, we're dropped into a post post post singularity world where AI already did all of that and is banned? as kind of a footnote that's never further explored in movie renditions that have been released
Uh, are you serious?
Kurzweil probably didn't know the singularity is akshually 1000 schizo monkeys banging on a billion dollar typewriter.
It had seemed like progress was slowing last year as the juice ran out on moar parameters. We're well past that.
“…a reference to the repeated (and always erroneous) claims made after almost every patch that Onyxia was using her Deep Breath ability more often than before.”
https://warcraft.wiki.gg/wiki/She_Deep_Breaths_More_%2825_pl...
I've been using Opus 5.5 on xhigh for some decently aggressive software development on a ~125k-line project, and it has been fine for me. I'm not really willing to dive into a "rewrite the whole project, don't make any mistakes" level of refactoring. I'm not looking to try to prove Fermat's Last Theorem myself. Is there anything that I'm truly missing?
- Anthropic is finally well balanced across models. Meaning: Opus 5.5 is so good that I do not feel Fable FOMO as I did when they were paired with Opus 4.8/5. And Sonnet sometimes makes me forget about Opus.
- Fable is good, but you really have to push the ambition of your prompt in order to see proper separation from Opus 5.5.
- It’s way faster and more reliable: one of the worst things about fable 5.1 was how slow it was an, on top of that, how frequently it would suffer from max output token overflows. I got none of these issues with 5.2.
- Fable is truly exceptional at drawing now. Fable’s ability to render an image with JavaScript actually has me questioning the value of dedicated image models. I prefer the images I’ve gotten fable/opus to make for me over the gpt-image lineup, it’s wild to think we’re at this point.
Surprisingly the model is available immediately in my Claude Code: usually there's a delay, either for rollout or safety.
Fed to someone specifically to test if they'll leak it and to confirm who that is?
Curious to see how 5.2 measures up in producing readable output.
My speculations: maybe this wasn't as much of an improvement over 5.1 as they were hoping, so they're saving 5.5 for the next major update (possibly "Model 4" from the METR report). Or, maybe capacity is constrained so "5.2" is a subtle encouragement to stick with Opus unless a task really needs the increased intelligence.
Opus 5.5 is a massive improvement.
I prefer Gemini for day to day use as I find it a lot more agreeable to be working with all day every day, and Gemini is pretty much on a par for coding Vs Opus 5.5, at least for my needs.
terra has been prettt good for me for me, when i need something looked at an explained to me.
i dont ask for much non-technical output though, which i expect yeah is its own cans of worms.
It could also be a system prompt + harness issue, as opposed to a model issue, I haven't done too much experimentation here.
I couldn't use prior Claude models without rolling my eyes at the writing every few sentences, Astra was refreshing and then Opus 5.5 made even Astra look terrible in comparison.
Even the Plain English hack only is a temporary solution.
They're not typically known for their interpersonal skills.
Opus 4.7/4.8 heavily suffered from these communication quirks. while the Sol models at the time worked well. Right now its flipped though and Opus 5.5 has none of these issues while Sol 6.1 does.
Small example I've had today: "Me: Hey Sol 6.1 xhigh, does X have property Y? Sol 6.1: Yes, if we change X to have property Y. Right now it has property [...], [...], [...], and it does not have property Y." The model shouldn't be answering questions like that. It makes understanding extremely hard and slow. For whatever reason it's either being encouraged or at least not stomped out during RL.
Heck, some of the most common debates are on the semantic clarity of abstractions or syntax.
Honestly incredible that they’ve managed to beat all benchmarks, while making it cheaper. Always happy to see improvements offered at lower costs.
Will be interesting to see what OpenAI does in response
I think frontier models feel terrible after a new model requires you to do less babysitting, it's just really hard to go back once you've had to handhold less.
The only way I use Astra and 6.1 Sol now is if they're being orchestrated by Opus 5.5 so I don't have to talk to them directly.
They want to condition you to equal cheap with bad
- How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?
- Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?
- Why do they switch to comparing against Argon (rather than carbon) for robodojo?
Reference: https://x.com/METR_Evals/status/2118412205662314935
Edit: yeah apparently it's an AI-moderated alternative to Arxiv that accepts AI-authored papers (as well as human-authored papers I guess if you want).
> the number of Fable resets you get depends on how long you’ve been a paid subscriber [...] Subscription length is measured from the date of your first paid bill. If you canceled and came back, the time since your first paid bill still counts.
> These resets expire on March 8, 2027 at 11:59 pm PT.
tl;dr if you joined when Opus 4.5 became a thing, you got 2 resets.
That means you can lift your Fable max 50% usage and you'll just end up burning through your quota faster. You won't have more tokens.
"Resets your Fable limit, and takes this week’s Fable use off your weekly limit."
I suspect we will see people being adamant that this model simply 'feels' better due to long horizon goal alignment improvements ( like this kind that would cause this jump)
Was this before you lobotomized Opus 5.5 or after?
https://news.ycombinator.com/item?id=50419019
...but most of the benchmarks don't line up anymore because the benchmarks have changed.
The Humanity's Last Exam result matches the initial announcement, though.
I wonder why that team singles Gemini out like that.
I wonder how long this kind of excitement at new models will last? I mean, now when I get a new iPhone I don't get the same rush I used to. So I wonder if this hype feeling will fade as we plateau in model capability.
> Fable 5.2 at High effort scored about level with Opus 5.5 at Max effort, for roughly half the cost per task
Compared Opus 5.5 200 usd a month with Codex 200 usd (Astra High) and Kimi Pro 100 usd a month ... and now have a new conspiracy theory - more their suck your code the better outcome even just over few days time on large code base.
Horrible experience correcting course of Codex and Kimi flow compare to Opus. Opus much more responsive
Dont feel any gain of Codex 200 usd over Codex 100 usd
Kimi Pro feels like a fake Louis Vuitton bag with extreme high material quality and yet still fake ... a bit trash talk ... but clearly seen not so distant future where I can live without Codex and Claude
https://www.anthropic.com/news/claude-opus-4-6 https://www.anthropic.com/news/claude-opus-5 https://openai.com/index/introducing-gpt-5-3-codex/
For every single announcement, not only do the top models always score 60-80%, the older models somehow start scoring 20-30% (even though previously they were scoring 60%). It's completely unclear whether the methodology changes, how these scores are even evaluated or which domains are being tested. What's worse is that the "benchmarks" keep being updated with no explanation or due process.
You have session limits?
You should probably give OpenCode a try. Can run DeepSeek, Qwen, Kimi, … but also Fable 5.2, Opus 5.5 and GPT 6 Astra