Google releases new Bard Gemini model that might be at GPT-4 level



summary
Summary

Google’s Bard chatbot is powered by a new Gemini model. Early users rate it as similar to GPT-4.

Google’s head of AI, Jeff Dean, announced the new Gemini model on X. It is a model from the Gemini Pro family with the suffix “scale”.

Thanks to the Gemini updates, Bard is “much better” and has “many more capabilities” compared to the launch in March, according to Dean.

Dean does not explain what “scale” means, but the name suggests that it could be a larger (scaled) version of the previous Pro model, which according to benchmarks does not even beat GPT-3.5 (free ChatGPT).

Ad

Ad

Pro is Google’s second-tier Gemini model, behind the top-of-the-line Gemini Ultra, which has yet to be released.

GPT-Pro “scale” tied with GPT-4 in human evaluation

Remarkably, the new Pro model immediately took second place in the neutral Chatbot arena benchmark, ahead of the two GPT-4 models 0314 (March 2023) and 0613 (Summer 2023), but behind GPT-4 Turbo (November 2023). The new Bard model is the first to break into the GPT-4 phalanx.

Image: Chatbot Arena Leaderboard screenshot

Chatbot Arena applies the Elo rating system used in chess and e-sports to evaluate and compare the performance of different language models. In the Arena, different models compete against each other in anonymous, randomly selected duels.

Users interact with the models and vote for their preferred responses. These votes are used to determine the ranking in the leaderboard. The platform collects all user interactions but only counts the votes cast if the names of the models are unknown, i.e., the user did not ask for the name.

Because these are user ratings or perceived quality, Chatbot Arena’s results may differ from the results of a typical synthetic benchmark.

Recommendation

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top