Is your feature request related to a problem? Please describe.
The op_store/views/ directory can get very large. For example, [one user] reported it at 2.6GB, which caused them to wonder why that's the case.
Even though disk space doesn't matter to some, this feels excessive.
Describe the solution you'd like The files contain protobuf-serialized data, which based on my sampling of one file is about 30% ASCII strings. They compress very well. My directory for one project has 5,493 files totaling 745M. With zstd, they compress down to 27% of their original size.
Is there a reason not compress the files before writing them?
Keeping around thousands of files in a single directory is also not great practice. Git, for example, shards its objects/ directory. I know it can slow down listing, but doesn't affect direct access, so maybe it's not a big issue.
Still, it might be helpful to also pack the files periodically into a single file, like Git does.
Activity
12
salty-horse10d agoWould you be open to a simple "compress each file" improvement, or will it have to be part of a larger redesign of the storage system?
martinvonz5d agoSome thoughts I've had about this in the past:
- Maybe there's little reason to even have view objects; we could embed it in the operation instead.
- Big parts of the view often don't change, such as
RemoteView. We should store those as separate objects in the op store and have the main view just refer to them by id. Or maybe we even want to expose the distinct remote view objects in theOpStoretrait, so we can even avoid loading them when they're not needed. - If we want to support hundreds of thousands of bookmarks like Git does - but I'm not sure we do - then we need to compress them somehow. That could be by storing deltas, or it could be by storing them as prolly trees.
salty-horse3d agoI think the compression (and directory sharding?) is a simple change to make with a large benefit for repositories with a lot of activity.
- Would this need a new storage type string? Something like
simple_op_store_compressed? While the change can be detected on-the-fly, it's better if old - Would there need to be a command to migrate back and forth, in case users want to use old versions of JJ?
PhilipMetzger3d ago2. Would there need to be a command to migrate back and forth, in case users want to use old versions of JJ?
No, I don't think its necessary since that can be done transparently by newer versions.
salty-horse3d agoNo, I don't think its necessary since that can be done transparently by newer versions.
I'm asking about migrating back because old versions won't be able to do anything with the new op store, and if it's not given a new name, they might even error in weird ways, and there was discussion about some developers keeping old versions around for experiments. Maybe not an important use-case.
What do you mean exactly by "done transparently"? I think that once a new version sees an op_store that's not compressed, it should migrate it automatically and never go back. Is that what you had in mind as well?
PhilipMetzger3d agoI think that once a new version sees an op_store that's not compressed, it should migrate it automatically and never go back. Is that what you had in mind as well?
Exactly, newer versions will be able to read/write both versions until the deprecation period is through.
No, I don't think its necessary since that can be done transparently by newer versions.
I'm asking about migrating back because old versions won't be able to do anything with the new op store, and if it's not given a new name, they might even error in weird ways, and there was discussion about some developers keeping old versions around for experiments. Maybe not an important use-case.
I mean the old versions will be able to read the op log for sometime until the old code path removed.
salty-horse3d agoSorry, just to clarify: So you think there should be a command to migrate to the new compressed op store, but no command to migrate back?
PhilipMetzger3d agoSorry, just to clarify: So you think there should be a command to migrate to the new compressed op store, but no command to migrate back?
No, the migration will be done transparently and we will have codepaths writing to both for sometime and after that we will deprecate and remove it so older versions will be no longer to read it though. This whole schpiel will take about 18 months per our deprecation guidelines.
salty-horse3d agoWhat's the benefit of having two op store representations active at the same time? Who is it for? That would increase the storage requirements even more, which I'm trying to avoid.
I'm thinking of how Git handles its ref formats. You can migrate between files and reftable, but not have both at the same time. So if you still use tools that don't support the new format, you can keep using the old one. (Or switch back to it.)
PhilipMetzger3d agoWhat's the benefit of having two op store representations active at the same time? Who is it for?
Its called backwards compatibility something this project really takes seriously.
That would increase the storage requirements even more, which I'm trying to avoid.
Only for a temporary time which is more than fine. Maybe we can provide an opt-in which would allow to only use one until the removal is done.
salty-horse3d agoI thought having an opt-in command to migrate to the new format, and to migrate back to the old, is providing backwards compatibility.
Two concurrent op-stores can cause problems. Suppose you run an old version of JJ on the repository, and it only writes a view file to the old op store. A new version of JJ would need to constantly check if both are in sync, possibly slowing down its operation.
The current format was just a way of getting something done quickly. Thus the name:
We're well aware that it wasn't a long-term solution, but it's actually worked well enough for longer than I expected; I'm a surprised it took this long before someone created a ticket for it :)