Since you are locked into a non-standard situation:
There may be better ways that I don't know of, but I would try testing the fasteners in a Skidmore with a torgue wrench. That is what is normally done for structural bolting. If the jam nuts will work at all, then you will have a torque value that correlates to a bolt tension value.
As you may already know, the testing is normally done with 3 fastener assemblies (FA= bolt,nut, washers) using the lot numbers that will actually be used.
If the bolts are too short for the Skidmore, then I would test the jam nuts with longer bolts and with a DTI washer (direct tension indicating washer). That should provide torque and tension values along with an average DTI gap. Then test the shorter bolts in a mock joint with the DTI. Apply the torque and measure the DTI gaps to see if they are equal to or less than what you tested on the Skidmore, which will be a way to measure bolt tension.
Because the jam nuts will be used in an unusual application, I would test more than 3 FA's. An arbitrary number that comes to my mind is to look for 10 FA's with consistent results. I don't believe you will see exact numbers from test to test, simply because there are fewer threads to work with so forces on the thread will be higher and more erratic. (It is hard enough to get consistent numbers when doing the tests with standard nuts.) But you should get a reasonable average to work with.
I would be concerned about the fasteners relaxing (threads yielding) some time after installation.
My other alternative, since you mentioned that the bolts would be in shear, would be to sub-drill the holes, assemble the joints, then ream to provide a hole that is no more that 1/32" over the bolt diameter. It seems that should limit the bolt/joint movement.
Those are just ust some thoughts, so please do not take any of this as authoratative information. And please let everyone know how everything works out.
Chet Guilford