But my guess is that, whatever the one piece is, it is physically inside Luffy's body because "X marks the Spot"

This is an excuse to talk about one of my favorite One Piece worldbuilding things.

The planet that One Piece is set on is mostly ocean, like ours, but it has only a single continent: the Red Line, that runs around its whole circumfrence, vaguely north-south. It is further divided by the Grand Line, running vaguely east-west, which is an especially dangerous part of the ocean with wild and unpredictable climate, protected by belts of sea monster-infested ocean with no wind on the top and bottom.

Thus, there are two gigantic Xes on either side of the planet.

One is a mysterious mountain that water flows up and down in different streams, which is how people get onto the Grand Line. The other is both the capital of the tyrannical World Government, and the underwater paradise of Fishman Island. Either one could hold other secrets....