[ . . . ]
Why do you need a line plot and specific locations of the flux? If you only want the total integrated cross-shore transport then the divergence theorem is a better solution and not subject to these complications. (Though it will also require decisions about how to take the divergence derivatives at the edges ... then the difficulty will be at peninsulas, not indentations). The two methods would be a useful check on each other; the result will not be identical, and the difference would be a measure of uncertainty.
They will be identical if your derivative and integration are consistent. You are right that it will require decisions about how to take the derivatives. But, those decisions are strongly constrained for your results to satisfy Gauss's theorem. The key is to define your derivative (divergence) in such a way that Gauss's theorem holds for each grid box. This is what the programmers of OGCMs do, to ensure the conservation of volume, salt, etc.
In my calculations, I checked that Gauss's theorem exactly hold on my gridded flux field.
Best regards,
Ryo