Is work done if you carry a book across the room at a constant velocity?

Now this question has many answers all over the internet, but this site gives me the power to pose my doubts in front of you all. Let's get straight to it: Suppose I move on a road while holding a book in my hand. I read that I am not doing any work to it, because the angle between the displacement and the motion is 90 degrees. But, I think that if the angle is 90 degrees, what is causing the book to get displaced? If the book is getting displaced, then there is some force which is causing it to move. So, work should be done.

So my question: Why is the book even getting displaced if force is not acting along the direction of motion? If it is, then why is the work done zero?

Another question(Edited): I have also read this: Suppose I hold a book and take it straight to a particular height. I do some work on it, which is transferred to it as potential energy. But if I take the same book first to the left, then to a little height; then to the right; then to a little height; and keep repeating it until I reach the same height, then same amount of potential energy is stored. How is this possible, if I run with an acceleration, while moving to the right and the left? If I am correct, then more potential energy should be transferred, isn't it?