OK, I didn’t realise linear indices were always 1-based for ND-arrays. What is actually needed is to define get/setindex differently for the 1D and ND case, then:
Base.getindex{ScalarT}(v::PtrView{ScalarT,1,LayoutLeft}, i::Integer) = unsafe_load(v.ptr,i+1)
Base.setindex!{ScalarT}(v::PtrView{ScalarT,1,LayoutLeft}, value, i::Integer) = unsafe_store!(v.ptr,value,i+1)
Base.getindex{ScalarT,N}(v::PtrView{ScalarT,N,LayoutLeft}, i::Integer) = unsafe_load(v.ptr,i)
Base.setindex!{ScalarT,N}(v::PtrView{ScalarT,N,LayoutLeft}, value, i::Integer) = unsafe_store!(v.ptr,value,i)
This is necessary since e.g. A[0,0] is converted to A[1]. I wanted to specialize linearindices because I thought A[0] refers to the first matrix element, so indeed there is no need to specialize it.