A Slinky on an Escalator Is a Weirdly Good Test of LLM Intelligence

(morethanamachine.com)

22 points | by reddmachine 14 hours ago

3 comments

  • reddmachine 14 hours ago
    I want to preface that I've done LLM evals in a proper setting before. This is far sillier. It's derivative of the "pelican riding a bicycle" test, something simple that shows how capable an LLM is at perceiving intent, dealing with physics and general aesthetics. I run this for every new model now and it gives me a decent idea of what to expect.
  • Rodmine 8 hours ago
    The setup needs more description. Can it have pre-installed npm, npx, chrome, playright etc.? It turns out my agent did not have access to it and it is setting all of that itself. Will update when done.
    • reddmachine 5 hours ago
      Agreed. I need to specify the playwright setup in a clearer way. Other than that nothing custom should be needed.
  • cyanydeez 8 hours ago
    would be better if it compared to open weights models.
    • reddmachine 5 hours ago
      Yes. I need to set up the models and harnesses for this. Likely via open router.