During any period, a potential customer arrives at a certain facility with probability 1/2. If there aretwo people in the facility (including the one being served) the potential customer leaves the facilityimmediately and never returns. However, if there is one or fewer people, he enters the facility andbecomes an actual customer. The manager of the facility has two types of service rates available. Ifshe uses her slow service rate at a cost of $3 during a period, a customer will be served and leave thefacility with probability 3/5. If she uses her fast service rate at a cost of $9 during a period, a customerwill be served and leave the facility with probability 4/5. Note that the probability of more than onecustomer arriving or more than one customer being served in a period is 0. A profit of $50 is earnedwhen a customer is served. The manager wants to minimize the expected discounted cost of service over an infinite horizon β=0.9). Required:
a. Formulate the problem as a Markov decision process (e. g. specify the state space, decision set, transition probabilities, expected rewards) Hint: In computing the costs for services, do not forget the opportunity cost of losing a potential customer – e. g. 50$ is lost.
b. Formulate the problem as a linear program to determine the optimal stationary policy.
c. Implement the policy iteration algorithm starting with the stationary policy that always uses slow service. Iterate either until you find the optimal policy or implement 3 iterations.
d. Implement the value iteration algorithm for 3 iterations.
e. In the value iteration algorithm, what is the number of iterations necessary to achieve a desired precision of ε=0.1?