Skip to content

Try to avoid "...//state.txt" URLs for replication. - #320

Closed
amandasaurus wants to merge 1 commit into
osmcode:masterfrom
amandasaurus:repl-dont-double-slash
Closed

amandasaurus wants to merge 1 commit into
osmcode:masterfrom
amandasaurus:repl-dont-double-slash

Conversation

@amandasaurus

Copy link
Copy Markdown
Contributor

Previously, for a base URL of example.com/area-updates/ the code would look for example.com/area-updates//state.txt" etc. This change will make it request example.com/area-updates/state.txt`. It now doesn't “double slash”.

This will ensure providers of OSM data updates (e.g. Geofabrik) will see consistent URLs regardless of small typing changes.

This change is inspired by Postel's Law: “be conservative in what you send, be liberal in what you accept”.

@amandasaurus
amandasaurus force-pushed the repl-dont-double-slash branch from 9c83fb5 to 41267ea Compare October 5, 2026 13:25

@lonvia lonvia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine with me in general. Unfortunately, merging is currently stuck on rather exotic CI fails of the Windows builds, which I have yet no idea how to fix.

Comment thread src/osmium/replication/server.py Outdated
# Try to avoid calling `example.com/area-updates//state.txt`
baseurl_wo_slash = self.baseurl
while baseurl_wo_slash[-1] == "/":
baseurl_wo_slash = baseurl_wo_slash[:-1]

@lonvia lonvia Oct 12, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably better to do that in __init__ while assigning self.baseurl so it is only done once. (The variable could be overwritten later in theory but that would be unintended use, so safe to ignore.)

Also, you should use the endswith() function.

@clarisma

clarisma commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Looks like the Windows build system is keeping the build directory locked, which prevents it from being deleted during cleanup. Some leftover thread, possibly telemetry or other junk?
I'd recommend building using Ninja, which is generally faster and cleaner than the stock msvc builder.

I use cibuildwheel and have this:

      CMAKE_GENERATOR: Ninja
      CIBW_BEFORE_BUILD_WINDOWS: >
        python -m pip install "cmake>=4.2" "ninja"

Fix: #325

@amandasaurus
amandasaurus force-pushed the repl-dont-double-slash branch from 41267ea to 30c274e Compare October 12, 2026 09:49
Comment thread test/test_replication.py Outdated
(573923, 'https://text.org/000/573/923.state.txt'),
(3290012, 'https://text.org/003/290/012.state.txt'),
@pytest.mark.parametrize("baseurl,inp,outp", [
("https://text.org", None, 'https://text.org/state.txt'),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a tab here hidden somwehere which needs to go, see https://github.com/osmcode/pyosmium/actions/runs/38232342257/job/114754593730?pr=320

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@amandasaurus
amandasaurus force-pushed the repl-dont-double-slash branch from 30c274e to be011dc Compare October 12, 2026 12:33
@amandasaurus

Copy link
Copy Markdown
Contributor Author

Also, you should use the endswith() function.

Done.

Probably better to do that in __init__ while assigning self.baseurl

Done.
Turns out you need to do a sorta indirect way to remove all slashs, then add one. In order to cover all possibilities.
It means the URL will always be saved with a trailing /.

@lonvia

lonvia commented Oct 15, 2026

Copy link
Copy Markdown
Member

Do you mind rebasing this? CI should now run through again (except that the linter is still unhappy with your code ;) https://github.com/osmcode/pyosmium/actions/runs/38250018742/job/114813583694?pr=320)

@amandasaurus
amandasaurus force-pushed the repl-dont-double-slash branch from 6b334e3 to fd78ab7 Compare October 16, 2026 08:48
Previously, for a base URL of "example.com/area-updates/" the code
would look for "example.com/area-updates//state.txt" etc. This change
will make it request "example.com/area-updates/state.txt". It now
doesn't “double slash”.

This will ensure providers of OSM data updates (e.g. Geofabrik) will see
consistent URLs regardless of small typing changes.
@amandasaurus
amandasaurus force-pushed the repl-dont-double-slash branch from fd78ab7 to 8ff9ea5 Compare October 16, 2026 08:53
@lonvia

lonvia commented Oct 18, 2026

Copy link
Copy Markdown
Member

I've merged this now manually in 331784f with the suggested change and another fix for the tests.

@lonvia lonvia closed this Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants