How is security handled in a free market system

I assume that you are talking about security in the sense of police or national defense.

In a pure free market system, you would have no overall police force or army. These things (this kind of security) are called "public goods" because they are things that everyone gets use from. In other words, the army can't protect me without protecting my next-door neighbor. For that reason, there's no way an army could be profitable and so no private company would produce an army (it couldn't tell my neighbor "we'll stop protecting you from the Russians if you don't pay your army bill").

Public goods will only be produced if the government produces them. So security (except for things like private security guards) will not exist in a pure free market economy.