[feature] Add support for write.data.path and write.metadata.path

[kevinjqliu]

Feature Request / Improvement

In the write properties section

#1452 adds support for LocationProvider #1452 (comment)
Similar issue, #161

[jiakai-li]

Sorry I raised a duplicated one by accident, which is closed now. Can I work on this feature?

[kevinjqliu]

@jiakai-li assigned to you!

[kevinjqliu]

BTW this is how java side is done https://grep.app/search?q=WRITE_DATA_LOCATION&filter[repo][0]=apache/iceberg&filter[path][0]=core/src/

[smaheshwar-pltr]

Thanks for volunteering @jiakai-li! Happy to review the LocationProvider-related changes for write.data.path if it'd help 😄

[jiakai-li]

Hey @smaheshwar-pltr , thanks for the offer! I hadn’t realized this feature is closely tied to LocationProvider. After looking into it, I think it could fit well with the changes you’re already working on. Would you like to handle it as part of your update, or would you prefer I continue working on it?

[smaheshwar-pltr]

I hadn’t realized this feature is closely tied to LocationProvider. After looking into it, I think it could fit well with the changes you’re already working on. Would you like to handle it as part of your update, or would you prefer I continue working on it?

Good suggestion @jiakai-li - I expressed my thoughts here: #1452 (comment). I'd prefer to keep #1452 as just what's needed to address the issue linked on it because:

• as is, there are important discussions to be had regarding location provision alone, e.g. Support Location Providers #1452 (comment) + Support Location Providers #1452 (review), and
• write.metadata.path, which feels like it should be supported along with write.data.path, is (as far as I understand) quite separate from those changes. IMO, we'd then doing too many things in that PR.

I'd prefer that both of these new write options be tackled together in a follow-up PR, that closes this issue - let me know how this sounds to you.

[jiakai-li]

Sure, that makes sense to me, thank you @smaheshwar-pltr , :-)

[kevinjqliu]

#1452 is merged, we can now work on adding support for the above