I suggest you can first try with full s3 access. If there's still problems, then we first fix them. It was a bit of a long journey, but that's more to do with AWS roles and permissions story. There were also some bugs with in helm, but these got fixed. I managed to get the cluster running by including both arn:aws:s3:::my-state-store-bucket/* and arn:aws:s3:::my-state-store-bucket in the resource list of the IAM policy, but I wanted to reduce the scope of 'Actions' permitted by the policy. The final policy I used was:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my-state-store-bucket/*",
                "arn:aws:s3:::my-state-store-bucket"
            ]
        }
    ]
}