FTB, also known as Feed the Beast, originated as a custom challenge map in Minecraft that made heavy use of multiple tech mods. It played very similar to a type of map known as Sky Block, where the player is started in a void world with only a small platform, and must overcome a series of challenges. Because FTB included tech mods, it was possible to use the various custom blocks and items to keep track of the challenges and the player’s progress, as well as automatically give awards after each challenge was met.
Feed the Beast is a “brand” of sorts. They have a custom launcher that hosts multiple different modpacks, all offering a different “flavor” of Minecraft mods. For example, there are modpacks centered around mods themed around magic; modpacks centered around “technology” mods, a modpack designed to be difficult and unforgiving, and so on. When MOST people say “feed the beast”, they are referring to the Big Pack; the name of it changes from time to time but it’s the pack with tons of mods.