How do you tell if something is a combustion reaction given only the reactants?
for example, acetic acid + potassium hydroxide.
our notes say that it's a reaction with oxygen, but that doesn't make any sense to me.

A combustion reaction it refers at complete oxidation of, most commonly, organic compounds. One example is the burning of methane gas CH₄:

 CH₄ + 2 O₂ → CO₂ + 2 H₂O + heat

or the burning of ethanol:

C₂H₆O + 3 O₂ → 2 CO₂ + 3 H₂O + heat

The now we call this a combustion reaction because the compounds may be used as fuels which are combustible, but in the end there are oxidation reactions.

Now the oxidation of magnesium:

2Mg + O₂ → 2 MgO + heat

Now here I would not call this a combustion reaction because you we do not use magnesium as a fuel (not economically and practical reasons), but this reaction is for sure an oxidation reaction.

Reaction between acetic acid and potassium hydroxide is a double replacement  reaction or, because of the nature or reactants (acid and base), it may be called a neutralization reaction.

CH₃-COOH + KOH → CH₃-COOK + H₂O

Hope it helps.