I'll just post my method then:
Given that we have an equal number of balls m of each color from the machine, then let P_n(m) be the probability of this occurring and picking one of the normal machines, and P_s(m) be the analogous probability for the special machine.
Now let us consider the probability of each machine yielding a ball of each individual color. For the special machine, since each color has an equal chance of being given out, the probability of getting a ball of any color is 1/3, since there are three colors in total. Now consider the situation of getting m balls of each color. To find the probability of this happening, all we need to do is use the multiplicative law of probability for each ball.
That is, (1/3) ^ m * (1/3) ^ m * (1/3) ^ m
Which is (1/3 * 1/3 * 1/3) ^ m = (1/27) ^ m
We also need to take into account the number of machines. Since only 1 out of 11 of the machines are special, there is a 1/11 chance of picking the special machine at random.
Thus, we have P_s(m) = 1/11 * (1/27) ^ m
For the normal machine, we apply a similar principle. However, the individual probabilities are no longer equal. If we just apply the ratio given in the problem, then we have these probabilities:
blue = 1/10
yellow = 4/10 = 2/5
red = 5/10 = 1/2
Just like before, to find the probability of getting m balls of each color from the machine, we multiply the probabilities:
(1/10 * 2/5 * 1/2) ^ m = (1/50) ^ m
And since 10 out of the 11 machines are normal, P_n(m) = 10/11 * (1/50) ^ m
Finally, we need to determine when we are 99% certain that we picked from the special machine given our case of m balls. Using ultimifier's assumption, it is sufficient to find the smallest value of m such that:
P_s(m) >= 99 * P_n(m)
Or more simply,
P_s(m) / P_n(m) >= 99
Let's go ahead and simplify this using some algebra:
P_s(m) / P_n(m) = (1/11 * (1/27) ^ m) / (10/11 * (1/50) ^ m)
= [(1/11)/(10/11)] * [(1/27) ^ m / (1/50) ^ m]
= (1/10) * (50/27) ^ m
From here I simply used brute force (I wrote a program to check the value of this function starting at m = 1, stopping when it was greater than or equal to 99) and got the final result:
m = 12