Revert "GitHub Actions: fix ENOENT for jobs running in forks"

Revert b106cf2eef.
Sorry, it didn't actually fix the problem. See for example:
https://github.com/peterzhu2118/ruby/actions/runs/5659386206/job/15332833836

The plot thickens.

.github/actions/setup/directories/action.yml

@@ -42,7 +42,6 @@ runs:
       run: |
         mkdir -p ${{ inputs.srcdir }}
         mkdir -p ${{ inputs.builddir }}
-        mkdir -p ${{ inputs.srcdir }}/.downloaded-cache
 
     # Did you know that actions/checkout works without git(1)?  We are
     # checking that here.