So if you've been following the LLM world some of the recent models that OpenAI has released have much larger context sizes than prior generations (8K → 128K tokens). And, like, that sounds like a lot and you can certainly stuff a lot of information into the prompt for the model (even entire codebases in some cases), but it's really not as big of a leap as some people think.
Like, what they want you to think is that their super-intelligent model is closely poring over each and every token in the prompt and assembling a carefully-considered response that incorporates all the information you put in there, but it's absolutely not. The more you start making use of the larger prompt size the more the model begins to skim (in the best case) or flat out ignore (in the worst case) stuff in the prompt.
