How do I distinguish the first in pair and second in pair in pair end bam using XAM?

You’re on the right track, and Explain SAM Flags is a great resource.

You can query the flag properties using boolean algebra.
When a property is set, the flag value anded with the property value will equal the property value.

Using a flag of 163 as an example.

julia> flag = UInt8(163) # BAM.flag(record)
0xa3

julia> bitstring(flag)
"10100011"

The “read paired” property (0x1) can be checked by evaluating whether the bit in the first position (right most) is set to true.

julia> bitstring(0x1)
"00000001"

julia> flag & 0x1 |> bitstring
"00000001"

julia> flag & 0x1 == 0x1
true

The “read mapped in proper pair” property (0x2) can be checked by evaluating whether “10100011” & “01000000” == “01000000”.

julia> bitstring(0x2)
"00000010"

julia> flag & 0x2 |> bitstring
"00000010"

julia> flag & 0x2 == 0x2
true

The “read mapped in proper pair” property (0x40) can be check by evaluating whether “10100011” & “01000000” == “01000000”.

julia> bitstring(0x40)
"01000000"

julia> flag & 0x40 |> bitstring
"00000000"

julia> flag & 0x40 == 0x40
false

You may combine these properties with an ‘or’ operation to form one query.

julia> query = 0x1 | 0x2 | 0x40
0x43

julia> query |> string
"67"

julia> query |> bitstring
"01000011"

julia> flag & query == query
false