Skip to main content

Bulk Download Automatic Course Archives

You can download several automatic course archives at once using the Automatic Archive Bulk Downloader script. With this script, you can select the specific courses you need or download the archives for all courses created in a specified date range. The script downloads the latest automatic archive of each course through the Learn REST API. It authenticates with a REST application that you set up once, so you don't need to use a Blackboard LMS username and password.

Download a single archive instead

If instead of downloading archives in bulk, you want to download a single archive, complete the following steps.

  1. Go to the /internal/autoArchive directory in the Content Collection.

  2. Open the subdirectory for the course you want. Each subdirectory is named with the course ID followed by the course creation date in YYYYMMDD format (for example, SampleCourseId01_20210125).

  3. Select Download Package on the archive you want.

Before you begin

You need a computer with Python 3.8 or later, a Developer Portal account, and a Blackboard LMS user for the integration with a dedicated system role.

Create a system role for the integration user with only these privileges. Do not use the System Administrator role; the integration only needs to read automatic archives.

  1. In the Administrator Panel, in the Users section, select System Roles.

  2. Select Create Role.

  3. Give the role a Name (for example, Auto Archive Integration), the select Submit.

  4. Open the menu of the new role and select Manage Privileges.

  5. Use Search to find each privilege below by the name shown in Blackboard LMS, select it, and select Permit Privileges.

  6. In the Administrator Panel, in the Users section, select Users.

  7. Search for the user you are using for this integration.

  8. Edit the integration user and add the new role you created to its System Roles.

Privilege as shown in Blackboard LMS

Privilege ID

Why it is needed

Read-only access to the Content System filesystem

bbcms.cs.fileSystem.REST.VIEW

Read Content Collection resources through the REST API

Course/Organization Control Panel (Packages and Utilities) > Archive Course/Organization

course.archive.EXECUTE

Access the /internal/autoArchive folder and every course folder in it

Course/Organization Control Panel (Packages and Utilities) > Archive Course/Organization, including user passwords

course.archive.user.password.EXECUTE

Access the archive packages, which include course users' passwords. Without it the folders appear empty

The privileges must be granted through system role. Granting them through a course role does not give access to /internal.

Register a REST application

You do this once in the Developer Portal; it give you the application ID, key, and secret.

  1. Go to the Developer Portal and sign up for a developer account, or sign in if you already have one. Refer to First steps with the Learn REST API for more information.

  2. Register a new REST Application, named, for example, Auto Archive Downloader - <your institution>.

  3. Copy the Application ID, Application Key, and Application Secret.

  4. Store the Application Secret in a password manager; the portal shows it only once.

Enable the application in Blackboard LMS

Once you have registered your application, you must enable it.

  1. In the Administrator Panel, in the Integrations section, select REST API Integrations.

  2. Select Create Integration.

  3. In Application ID, paste the Application ID you got from the Developer Portal.

  4. Next to Learn User, select Browse. Choose the integration user you set up in the Before you begin section that has the custom system role described.

  5. Set End User Access to No, so the application always acts with that user's privileges.

  6. Set Authorized to Act as User to No.

  7. Select Submit.

Refer to REST API 통합 for details.

Install Python and the dependencies

The script needs Python 3.8 or later and two packages, requests and boto3. If you don't have Python 3, download and install it. Open a Terminal (macOS or Linux) or PowerShell (Windows) for the commands below.

  1. Install pip. Python installers for Windows include pip. On macOS or Linux, check whether you have it:

    python3 -m pip --version

    If you don't, install or upgrade it:

    python3 -m pip install --user --upgrade pip

  2. Create a virtual environment, so the packages stay separate from the rest of your system:

    python3 -m venv env 
source env/bin/activate

    On Windows, activate it with env\Scripts\activate instead.

  3. Install the dependencies:

    python3 -m pip install requests boto3

    boto3 is only used when the output is an S3 bucket.

Download the script

Download auto_archive_bulk_download_v2.py. Note where you save it; you will run it from that folder.

Run the script

The script gets a token with your application key and secret, lists /internal/autoArchive, and downloads the latest archive of each course you select, with its log file. When it finishes, it prints a summary.

Set the secret as an environment variable so it does not end up in your shell history. If you don't set it, the script asks for it.

export AUTO_ARCHIVE_SECRET='<application secret>'

On Windows, use $env:AUTO_ARCHIVE_SECRET = '<application secret>'.

Parameter

Required

Description

--site

Yes

Base URL of Blackboard LMS site, for example, https://mysite.blackboard.com

--key

Yes

Application key from Developer Portal

--output

Yes

Where to save the archives: a local folder (for example ~/Downloads/autoArchives , created if missing) or an S3 bucket (for example s3://bucketname/path/). S3 uses the default AWS credential provider, which works with a profile or environment variables

--file

if no date range

Path to a text file with one course ID per line and no other delimiter. Course IDs are matched regardless of case

--beginDate, --endDate

if no --file

Download archives of courses created in this range, in YYYY/MM/DD format. Use both together. With --file , courses outside the range are listed in the summary

--debug

No

Print every request the script makes. Tokens, secrets and signed URLs are never printed

--connectTimeout, --readTimeout

No

Timeouts in seconds; the defaults are 30 and 300

Example with a course list:

python3 auto_archive_bulk_download.py --site https://mysite.blackboard.com -- key <application key> --output ~/Downloads/autoArchives --file ~/Documents/feedFile.txt

Example with a date range:

python3 auto_archive_bulk_download.py --site https://mysite.blackboard.com -- key <application key> --output ~/Downloads/autoArchives --beginDate 2020/01/20 --endDate 2021/01/20

Run only one instance of the script at a time to avoid performance degradation.

Output and summary

For each course the script saves the latest automatic archive ( .zip ) and its log ( .txt ) with their original names. Files are written to a temporary name and renamed when complete, so an interrupted run never leaves a partial archive. One course failing does not stop the others.

The summary lists:

  • How many archives were downloaded.

  • Courses with errors, with the reason.

  • Courses with an empty folder. This usually means the integration user is missing course.archive.user.password.EXECUTE.

  • Courses with a folder but no automatic archive package, with the files found in the folder. Only automatic archives are downloaded; other packages in the folder are skipped.

  • Archives downloaded without their log file.

  • Courses in --file that were not found, and courses found but outside the date range.

  • Whether the run stopped early, and how many courses were not processed.

Exit Code

Meaning

0

Every selected course was processed without errors

1

Some courses had errors, or the run stopped early

2

The run could not start: invalid key or secret, missing privileges, or the output folder or course file cannot be used

Use the exit code to detect failures when you schedule the script.

Limits

A new REST application can make 10,000 REST API calls every 24 hours (refer to limits). The script uses about one call per course, plus a few to find the folders, so a full run covers up to roughly 9,900 courses a day. If the limit is reached, the script stops, reports how many courses were not processed, and exits with code 1. Run it again the next day, or open a support ticket to raise the limit. In the ticket, include the email used to register the application and its production group name. Automatic archives are kept for 30 days. Schedule runs so that every archive you need is downloaded before it expires.

Troubleshooting

Message

What to check

Could not get a token: the application key or secret is not valid

The key and AUTO_ARCHIVE_SECRET match those from the Developer Portal, and the integration exists and is available in Blackboard LMS

The /internal/autoArchive folder is not visible to the integration user

The integration user's system role has bbcms.cs.fileSystem.REST.VIEW and course.archive.EXECUTE

Courses with an empty folder

The integration user's system role has course.archive.user.password.EXECUTE

Courses with a folder but no automatic archive package

The files listed next to the course are not automatic archives (for example, ArchiveExFile_ packages), so there is nothing to download. Automatic archives are named <course ID>…_<13-digit timestamp>.zip. If you expected one, check that the integrations user's system role has course.archive.user.password.EXECUTE

Too many requests (HTTP 429)

Another instance of the script, or another integration using the same application, is running; or the daily limit was reached

Download failed with HTTP 401, 403 or 404

The archive was deleted or expired between the listing and the download; run the script again for that course

There are no automatic archives on this site

Automatic archiving is enabled and courses had activity in the last 30 days