Think of it almost like a wage budget for a chain of stores. The company has X amount of money to distribute throughout the chain but needs to allocate them efficiently. In Google’s case, their crawl budgets are distributed primarily by PageRank, your sites online authority. Sites with a higher PageRank will see a larger Crawl Budget, and sites with a PR of above 5-6 tend to see an even larger jump in Crawl Budget.
The difficulty in truly defining Crawl Budget is the number of factors that go into determining it in the first place. Take Crawl Rate Limit for example. This refers to the limit put on the crawl rate for a given site, with the aim of remaining efficient but at the same time not having a negative impact on the experience for the site’s visitors. Now this itself is dictated by a couple of factors. Most notably your sites Crawl Health. Simply put, if your site responds quickly your crawl limit is increased. If response times start dropping, so does your crawl limit. It all aims to balance efficiency with experience.
On top of the previous factors, each site has a Crawl Demand. Thankfully this is pretty self-explanatory. Primarily determined by the popularity of your site and how stale the URLs within it are. The idea being that more popular sites are crawled more often to keep them relevant within an index.
Importantly, there are factors which can have a negative impact on your sites allocated crawl budget. Things such as on-site duplicate content, low-quality content and soft error pages can reduce the crawling activity on your site and hence it’s indexing within search results. Even just improving the response time and speed of your site can improve your crawl rate as it’s a sign of healthy servers and as such a higher crawl limit can be given without negatively affecting site user experiences.
The important fundamentals to take from this are important to all sites, however, if your site is small (roughly less than 4000 URLs) and pages are often crawled the same day as they are published Crawl Budget is not something you particularly need to fret about. Instead, focus on your content and site friendliness to optimise traffic and online presence.