Skip to content

feat(io): Align FileStatus with the Java API.#45

Merged
SteNicholas merged 2 commits into
apache:mainfrom
Aitozi:file-list
Aug 13, 2024
Merged

feat(io): Align FileStatus with the Java API.#45
SteNicholas merged 2 commits into
apache:mainfrom
Aitozi:file-list

Conversation

@Aitozi

@Aitozi Aitozi commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

close #44

Comment thread crates/paimon/src/io/file_io.rs Outdated
.map(|meta| FileStatus {
size: meta.metadata().content_length(),
is_dir: meta.metadata().is_dir(),
last_modification_time: meta

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When testing, I found that the directory's last_modification_time = 0 for a dir. But for the LocalFileIO in java it's not.

image

Is this a bug for opendal Fs service ? CC @Xuanwo

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it's related to apache/opendal#4615. Would you like to take a look? It might be complex to resolve entirely, but we can start by fixing it for the fs backend first.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, will take a look

Comment thread crates/paimon/src/io/file_io.rs Outdated
pub size: u64,
pub is_dir: bool,
pub path: String,
pub last_modification_time: i64,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking about using chrono::DateTime to make it easier to use. What do you think?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense, updated.

@Aitozi

Aitozi commented Aug 12, 2024

Copy link
Copy Markdown
Contributor Author

CC @Xuanwo

Comment thread crates/paimon/src/io/file_io.rs Outdated
Ok(FileStatus {
size: meta.content_length(),
is_dir: meta.is_dir(),
last_modification_time: meta.last_modified(),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using last_modified or modified for short?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

/// TODO: Support building Operator from HashMap via options.
pub fn new(_: HashMap<String, String>) -> Result<Self> {
let op = Operator::from_config(MemoryConfig::default())
let op = Operator::new(Fs::default().root("/"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better not to hard-code fs here, but I think it's a good starting point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I think here should construct the fileIO according to the scheme. I'd like to do it in another pr.

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM!

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@SteNicholas SteNicholas merged commit 74a99f0 into apache:main Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Align FileStatus with the Java API.

3 participants