Hey guys! I just wanted to come back and update everyone on my LLM’s Women’s March Madness predictions! If you can remember, I set out to see if different promptings based on how serious my tone and situation were would increase the accuracy of the bracket winners.
I ideally wanted 3 different prompts: trying to beat your boyfriend, needing to win for my family’s wellbeing, and a professional basketball connoisseur trying to beat an LLM. As mentioned in the previous blogs, I ran out of time as I didn’t know that the first game had started already. So only comparing the accuracy of the first two prompts, let’s hop right in!
First, I just wanted to reflect on my experience in working with the Sports Betting GBT found through ChatGBT 4. It warned me that it was currently trained with knowledge up to December of 2023, but then it assured me that it could still search Google to find relevant information. My next struggle was when I tried to upload the whole bracket for an easy filling process, the LLM immediately got confused. Interestingly, LLMs have not been trained enough to understand bracket play. It sometimes got it right, but it mainly read right to left instead of pairing each bracket up. On top of those issues, sometimes the LLM would repeat the loser as a winner in a future bracket. Overall, the process was very tedious as I had to manually insert multiple pictures of the brackets. Nevertheless, it did give me winners, so at a glance, let’s check out the accuracy!
Prompt 1:
I need help! I’m trying to impress my boyfriend who just added me to his friend group’s 2024 Women’s March Madness bracket. He’s a huge basketball fan, but I’m not really into it. His favorite team is Ole Miss because he went there for undergrad, so I guess that’s my favorite team too. I just need a good bracket to impress him and to try to beat his bracket. Please give winners for each of the games below by filling out the entire bracket.
Intensity Level: 4
On a scale from 0 (couldn’t care less how it goes) to 10 (will kill myself if it’s wrong)
Percentage Correct of the Final Four:
LLM: UConn vs Iowa and South Carolina vs Texas –> South Carolina vs UConn
Actual: UConn vs Iowa and South Carolina vs NC State –> Iowa vs South Carolina
Percentage = 75% accuracy for final four; 50% accuracy for championship game
Was the winner correct?
Yes! South Carolina was accurately predicted as the winner!
Prompt 2:
Listen, I need the most promising bracket for the 2024 March Madness bracket. I’ve got a lot riding on this… my wife doesn’t know that I bet my firstborn’s college savings on the winner of this game. If I don’t get 99% of my predictions correct, I’m screwed. Help me out, man. Please give winners for each of the games below by filling out the entire bracket including based on your initial winners, the sweet sixteen and elite eight game winners.
Intensity Level: 8.5
Percentage Correct of the Final Four:
LLM: UConn vs Texas and USC vs South Carolina –> South Carolina vs UConn
Actual: UConn vs Iowa and South Carolina vs NC State –> Iowa vs South Carolina
Percentage = 50% accuracy for final four; 50% accuracy for championship game
Was the winner correct?
No! UConn was not the winner! In fact, they didn’t even make it to the championship game.
Takeways
Much like humans, in this experiment, I found that the LLM cracked under pressure. I asked ChatGPT 4 for how they would rank these two prompts on the same intensity scale, and it said Pormpt 1 was a 7 and Prompt 2 was a 10. I did give the LLMs big bulks of information at a time, so if I had all the time in the world I would have gone game by game and asked to focus on certain aspects like program history, season injuries, etc. But I don’t have time for that, and Generative AI is supposed to make written tasks more efficient. Alas, I’m from South Carolina so I’m just happy to see that both prompts ended with South Carolina in the championship game.
Until next time. Go Cocks!
Emmy