This is very cool. We were thinking about doing something very similar with Skyvern

What was the reason you went down this path instead of extending selenium with AI features?

i partially addressed this in the "why vibium" section of the v1 announcement: https://github.com/VibiumDev/vibium/blob/main/docs/updates/2...

but why a new thing vs extending selenium? it's a little complicated, but neither selenium nor playwright were designed with ai in mind from day 1. with vibium, i'm optimizing for "vibe coding" and ai-driven workflows first.

This makes sense. I guess I wanted to understand why starting from scratch was better than "fixing" selenium, but perhaps "fixing" selenium isn't an option?

for the entire testing tools industry, in some ways, selenium was the "final boss" to beat. every new tool had to trash selenium in their marketing. eventually those "hit points" added up. "fixing selenium" is as much as of a branding problem as it is a technical problem. "oh, there's a new version of selenium? i heard selenium sucks!" is actually a problem that has to be dealt with. an entire new generation of coders only know "playwright rules, selenium drools".

of course, i have a new host of problems by going all in with "vibium"... i'm making a huge bet that "vibe coding" is a trend, not a fad. (it could still be a fad! we'll see if this post ages well soon enough!)

Also, as someone on the periphery of Selenium (mostly via WebDriver), some of the challenge is that Selenium has a huge amount of test code already written for it — and making radical API changes would break every test already written for it, and at that point you’re effectively a new library.

It’s gonna be very interesting to watch exactly how the adoption of WebDriver BiDi goes with Selenium, especially once WebDriver Classic starts to go away, and how API stability is balanced with exposing more and more async capabilities.

That makes a lot of sense. Sometimes it's easier to leave the baggage behind. It's too bad..selenium is a masterpiece. Thanks for sharing it with the world

maybe https://github.com/Skyvern-AI/Skyvern ?

What are some of the risks? This is a public web form available on the IRS website

If you're automating filling out the form, you aren't reading the instructions and you aren't checking what you're putting into it as much as you should be. And if you put in incorrect information, it tends to be considered fraud, even if it's downstream of a sloppy LLM rather than downstream of a particular fraudulent scheme.

You're right, but this is where the LLMs are especially useful. Our customers all prompt it to terminate if it doesn't have the right information / the pre submission confirmation doesn't match

Unrelated, but thoughtful gave us some very very helpful feedback early in our journey. We are big fans!

That's the dream

Definitely. What are your thoughts on the CloudFlare agent identity

It's funny, one time we had a customer that wanted to use us to test their website for bugs..

Skyvern kept suggesting improvements unrelated to the issue they were testing for

So how do clients process this sort of feedback? As a dev, “negative user feedback” gives me scares that “failed behavior testing” does not.

The AI isn’t mad, and won’t refuse to renew. Unless it’s being run by the client of course.

Are clients using your platform to assess vendors?

No, we don't have a lot of usage in that direction. People mainly use us to log into websites and either fill out forms or download files!

We do have them! We are HIPAA compliant, have soc-2 type 2 and offer self hosted deployments

Thank you for responding! Where is your compliance information? How do I sign a BAA?

Send me an email suchintan@skyvern.com - we can get you started

This is really cool. We might integrate this into Skyvern actually - we've been looking for a faster HTML extraction engine

Thanks for sharing!

Odd, I've been stuck by how different LLMs and kids learn the world.

You don’t get that whole uncanny valley disconnect do you?

How so? Your kid has a body that interacts with the physical world. An LLM is trained on terabytes of text, then modified by human feedback and rules to be a useful chatbot for all sorts of tasks. I don't see the similarity.

If you watch how agents attempt a task, fail, try to figure out what went wrong, try again, repeat a couple more times, then finally succeed -- you don't see the similarity?

no I see something resembling gradient descent which is fine but it's hardly a child

> try to figure out what went wrong

LLMs don't do this. They can't think. If you just one for like five minutes it's obvious that just because the text on the screen says "Sorry, I made I mistake, there are actually 5 r's in strawberry", doesn't mean there's any thought behind it.

I mean, you can literally watch their thought process. They try to figure out reasons why something went wrong, and then identify solutions. Often in ways that require real deduction and creativity. And have quite a high success rate.

If that's not thinking, then I don't know what is.

You're arguing in circles. They don't "try to figure out reasons" because there is no concept of "trying", "figuring out" or "reasons".

> If that's not thinking, then I don't know what is.

How about actual thinking, you know, what humans and to a lesser extent animals do?

You just haven't added the right tool together with the right system/developer prompt. Add a `add_memory` and `list_memory` (or automatically inject the right memories for the right prompts/LLM responses) and you have something that can learn.

You can also take it a step further and add automatic fine-tuning once you start gathering a ton of data, which will rewire the model somewhat.

Perhaps it can improve but it can't learn because that requires thought. Would you say that a PID regulator can "learn"?

I guess it depends on what you understand "learn" to mean.

But in my mind, if I tell the LLM to do something, and it did it wrong, then I ask it to fix it, and if in the future I ask the same thing and it avoids the mistake it did first, then I'd say it had learned to avoid that same pitfall, although I know very well it hasn't "learned" like a human would, I just added it to the right place, but for all intents and purposes, it "learned" how to avoid the same mistake.

This is a silly definition of learning, and any way, LLMs can't even do what you describe.

A person is not their body.

The person is the data that they have ingested and trained on through the senses that are exposed by their body. Body is just an interface to reality.

That is a very weird and fringe definition of what a person is.

If you have a different life experience than what you had so far, wouldn’t you be a different person?

If you had one fewer leg, wouldn't you be a different person? Implication is not equivalence.

> It deeply resembles how LLMs learn and think

What? LLMs don't think nor learn in the sense humans do. They have absolutely no resemblance to a human being. This must be the most ridiculous statement I've read this year

I am sorry, but you are scoffing at the humanity of your kid; you know that, right?