Today I tried to mock up one of the rocker arms to check a few things on an engine that is under rebuild.

Started to put the nut of the rocker on the stud by hand and it was going on it smoothly. When the stud reached the top of the nut inside it got hard to turn. By hard I mean that I can't even turn it further by hand. With a wrench I can continue turning it.

My question would be whether this is normal and designed this way to keep the rocker nut in place while the engine is running or something is wrong here?

The rocker studs were changed on the cylinder heads to ARP screw-in studs.

 

It's normal. The tops of the nuts are oval shaped rather than round - this keeps them from loosening up on their own.

Nylon locking nuts won't work - the nylon will melt under the heat.

Locking washers won't work - no real room for them, and nothing really for them to catch on, and the valve train force will take them out anyway.

Thus, these nuts are squished into an oval at the tops, so that they can't back out on their own.