Map download touchups #4185

orblivion · 2025-12-19T20:05:00Z

Fixes bug:

For map data downloading in ansible, I was concerned that the "to view progress" line did a line wrap in the ansible task name.

Also the log didn't have a starting timestamp.

Description of changes proposed in this pull request:

Output the "to view progress" instructions in a debug task right before the download task.
Rename the log file to use the md5sum of the file name. That way the file is of shorter, consistent length.
Output the date in the log, as well as the file name since the log file name no longer matches it due to the md5sum.

Smoke-tested on which OS or OS's:

RasPi OS Trixie

Mention a team member @username e.g. to help with code review:

@chapmanjacobd

chapmanjacobd · 2025-12-19T20:32:25Z

roles/maps/tasks/download_large_file.yml

+        #   fits within 80 characters on terminal (at least in the current
+        #   ansible version) in case users have small screens.
+        # * If the user mis-copies, it's less likely to do anything destructive.
+        log_file: "{{ item | hash('md5') }}.log"


Suggested change

log_file: "{{ item | hash('md5') }}.log"

log_file: "{{ item | hash('md5') | truncate(8, True, '') }}.log"

eh I guess it's fine either way

I was thinking about it but I don't know the properties of md5sum, at which point you start to get collisions. I suppose it's quite unlikely, plus somewhat unlikely that two such files would be present in this directory at once.

roles/maps/tasks/download_large_file.yml

chapmanjacobd · 2025-12-19T20:51:52Z

roles/maps/tasks/download_large_file.yml

        else:
          size_disp = str(round(size / GiB, 2)) + ' GiB'
-        print(' (' + size_disp + ') To check progress, run: "tail {{ working_dir }}/{{ log_file }}"')
+        print('(' + size_disp + ') To check download progress see above.')


This seems worse. Yes, it's not good to print redundant information but in this case it's probably fine to do so.

The directionality aspect can also make this confusing: is it the previous line, previous screen? how far above?

actually won't this print before the debug step anyway?

so it would actually be "see below" in which case I'd say just print the size and delete the extra text from this step

actually won't this print before the debug step anyway?

In this step, this text is printed to stdout but not displayed to the screen (unless there's an error in this task). It's captured by download_file_details.

The next step is the debug which outputs the instruction.

The step after that is the download, where I stick download_file_details into the name, and the user finally sees it.

My reason for removing this isn't that it's redundant. It's that it line wraps. I'm specifically trying to avoid them seeing it. I could try to word it to be more clear that it's immediately above.

I get it now... it's appended into the name of the download task.

It would be more clear to just print the size here and then in line 95 do something like

- name: "Download {{ item }} ({{ download_file_size.stdout_lines.0 ~ ' - see download progress in previous step' | default('done')}})"

ah that conflicts with the default('done') -- tricky

but does that ever get triggered anyway? because it will always have a size as long as the file is available remotely

Right but if the file exists locally I don't want to take the time to query the meta4 file.

(See where I exit the python script early)

okay I'll stop nitpicking--the change you made makes sense

chapmanjacobd · 2025-12-19T21:04:45Z

Since you're still getting Ansible lint errors anyway it might be worth trying the block level vars again... I still don't understand why something like this doesn't work:

- block:
    vars:
      dest_path: "{{ dest_base_path }}/{{ item }}"
      log_file: "{{ item | hash('md5') }}.log"
      working_dir: "/library/downloads/maps"
    
    - name: "Fetching file size for {{ iiab_map_host_url }}/{{ item }} via .meta4 file"
...

Co-authored-by: Jacob Chapman <7908073+chapmanjacobd@users.noreply.github.com>

orblivion · 2025-12-19T21:26:59Z

That doesn't parse, I tried it in a yaml parser. But in thinking about how to explain why, I think I figured out what the linter wanted from me. One sec...

vars: has to go first to satisfy the linter, I think

chapmanjacobd

looks good once those lint errors go away

orblivion · 2025-12-19T22:01:07Z

I should probably add, this is how it looks now:

TASK [maps : Fetching file size for https://iiab.switnet.org/maps/1/openstreetmap-openmaptiles.2025-12-10.zoom_0-09.pmtiles via .meta4 file] *************************************************
changed: [127.0.0.1]

TASK [maps : debug] **************************************************************************************************************************************************************************
ok: [127.0.0.1] => {
    "msg": [
        "",
        "",
        "   To check download progress, run:       ",
        "",
        "   tail /library/downloads/maps/ff516d8fd1cf5f0c97f2b8159dbb929c.log  ",
        "",
        ""
    ]
}

TASK [maps : Download openstreetmap-openmaptiles.2025-12-10.zoom_0-09.pmtiles (1.83 GiB) To check download progress, see previous step (above).] *********************************************

orblivion · 2025-12-19T22:04:28Z

roles/maps/tasks/download_large_file.yml


- block:
+
+- vars:


In the last PR it said to order it "vars, block, rescue" which was confusing until I realized that "block" was just another sibling and didn't need to come first.

chapmanjacobd · 2025-12-19T22:15:33Z

roles/maps/tasks/download_large_file.yml

+        msg:
+          - ""
+          - ""
+          - "   To check download progress, run:       "


I think I've tried this before but multiline debug messages don't work right?

msg: | like this

kinda annoying that doesn't work

Yeah, it string escapes the newlines

orblivion · 2025-12-19T22:23:25Z

@holta I'm ready as well to merge. Waiting to confirm that linting is happy.

orblivion added 3 commits December 19, 2025 14:21

Cleaner instructions for viewing logs for map file downloads

360b864

A bit more useful output in map data download log

b052071

Linting

4c76fc3

chapmanjacobd reviewed Dec 19, 2025

View reviewed changes

roles/maps/tasks/download_large_file.yml Outdated Show resolved Hide resolved

chapmanjacobd reviewed Dec 19, 2025

View reviewed changes

Update roles/maps/tasks/download_large_file.yml

871a44c

Co-authored-by: Jacob Chapman <7908073+chapmanjacobd@users.noreply.github.com>

orblivion added 2 commits December 19, 2025 16:36

Fix indentation

176e12a

Factor out vars: from the block's tasks

5f06902

vars: has to go first to satisfy the linter, I think

chapmanjacobd approved these changes Dec 19, 2025

View reviewed changes

Maybe clearer wording

f52cecc

orblivion commented Dec 19, 2025

View reviewed changes

chapmanjacobd reviewed Dec 19, 2025

View reviewed changes

chapmanjacobd approved these changes Dec 19, 2025

View reviewed changes

holta added the enhancement label Dec 20, 2025

holta added this to the 8.3 milestone Dec 20, 2025

holta merged commit 2aedb71 into iiab:master Dec 20, 2025
4 checks passed

	log_file: "{{ item \| hash('md5') }}.log"
	log_file: "{{ item \| hash('md5') \| truncate(8, True, '') }}.log"

Map download touchups #4185

Map download touchups #4185

Conversation

orblivion commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes bug:

Description of changes proposed in this pull request:

Smoke-tested on which OS or OS's:

Mention a team member @username e.g. to help with code review:

Uh oh!

chapmanjacobd Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

orblivion Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chapmanjacobd Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chapmanjacobd Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chapmanjacobd Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chapmanjacobd commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

orblivion commented Dec 19, 2025

Uh oh!

chapmanjacobd left a comment

Choose a reason for hiding this comment

Uh oh!

orblivion commented Dec 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

orblivion commented Dec 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

orblivion commented Dec 19, 2025 •

edited

Loading

chapmanjacobd Dec 19, 2025 •

edited

Loading

orblivion Dec 19, 2025 •

edited

Loading

chapmanjacobd Dec 19, 2025 •

edited

Loading

chapmanjacobd Dec 19, 2025 •

edited

Loading

chapmanjacobd Dec 19, 2025 •

edited

Loading

chapmanjacobd commented Dec 19, 2025 •

edited

Loading