I have to attend the client’s site to undertake the work but I can arrive when I like providing I make good progress within the scope of the project and have no direct supervisor. Would I be within IR35?

Attending the client’s site itself would not lead to being outside IR35. The fact that you seem to have control over when you are to undertake the work and the fact you have no supervisor, means the methodology of the work belongs to you. On these combined factors your IR35 status would therefore lend itself to being self-employed and outside the scope of the legislation.