In regards to Google Drive data QmeZLFkzycAL4nL7JdHMWibXhRCPzFjp2x35kaXRtzZ7mr ("low risk"):
* Downloaded: metadata - dirs, raws, per ID
* Partly downloaded: data
* Partly done: metadata - dirs, raws, per ID -> wbm
* Todo: get warc of dirs, per ID
* Commands:
** ran:
*** .../z8/text/3151ebd$ grep -P " https...web.archive.org.web.\d\d\d\d\d\d\d\d\d\d\d\d\d\d." logwget.txt | sed "s/^Location: //g" | sed "s/ \[following\]$//g" | sed "s/.* //g" | sort | uniq > logdone.txt
** running:
*** .../z8/text/3151ebd$ grep -o "/files/.................................?openDrive" logdone.txt | sed "s/?.*//g" | sed "s/.*\///g" | xargs -d "\n" sh -c 'for args do perl -p -i -e "s/$args\n//g" ./0/dirtodo.txt; done' _ # this command takes a while to finish
*** .../z8/text/3151ebd$ cat /z8/text/3151ebd/0/dir1.txt | xargs -d "\n" sh -c 'for args do echo "https://clients6.google.com/drive/v2beta/files/${args}?openDrive=false&reason=1001&syncType=0&errorRecovery=false&fields=kind%2CmodifiedDate%2CmodifiedByMeDate%2ClastViewedByMeDate%2CfileSize%2Cowners(kind%2CpermissionId%2Cid)%2ClastModifyingUser(kind%2CpermissionId%2Cid)%2ChasThumbnail%2CthumbnailVersion%2Ctitle%2Cid%2CresourceKey%2Cshared%2CsharedWithMeDate%2CuserPermission(role)%2CexplicitlyTrashed%2CmimeType%2CquotaBytesUsed%2Ccopyable%2CfileExtension%2CsharingUser(kind%2CpermissionId%2Cid)%2Cspaces%2Cversion%2CteamDriveId%2ChasAugmentedPermissions%2CcreatedDate%2CtrashingUser(kind%2CpermissionId%2Cid)%2CtrashedDate%2Cparents(id)%2CshortcutDetails(targetId%2CtargetMimeType%2CtargetLookupStatus)%2Ccapabilities(canCopy%2CcanDownload%2CcanEdit%2CcanAddChildren%2CcanDelete%2CcanRemoveChildren%2CcanShare%2CcanTrash%2CcanRename%2CcanReadTeamDrive%2CcanMoveTeamDriveItem)%2Clabels(starred%2Ctrashed%2Crestricted%2Cviewed)&supportsTeamDrives=true&retryCount=0&key=$(cat ~/ytapikey4.txt)" >> /z8/text/3151ebd/start_urls_1700324807.txt; done' _
** to run:
*** ...grab-site...start_urls_1700324807.txt... # dirs