Archive Operating System (AOS 6.13) Updates

Google Workspace Email Crawler

Google Workspace email can now be imported into the Archiver through the “import via Google Workspace Crawler” tool. This crawler will discover and crawl all email from all user accounts under a given Google Workspace customer ID.

Before you can use this crawler you must set up a Google Service Account with the required permissions in order to grant the Archiver access to this email data.

• In order to use the Google Workspace email crawler, first visit the Maintenance -> Import Email page and select “Google Workspace Crawler” as the imported email source.

• Define a new server to crawl. Fill out the customer ID of your Google Workspace account, as well as the credentials.json text provided to you after the creation of your Google Service Account. You have the option of filling out a date range to get emails from specific dates.

• After you click “Run Migration” to start the email crawler, you should see a success message explaining that the process will start soon.

• Within a few seconds, you should see a new process on the System Status page called GmailCrawler*. Its output will tell you how many accounts are found and how many emails are being fetched from each account.

It is safe to rerun the Gmail crawler at any point. The process will detect which emails are already on the Archiver and will avoid retrieving them again.

Google Workspace Label Sync

Google Workspace labels can now be synced to the Archiver in the same manner that Microsoft Exchange folders can be. Synced labels will be referred to as “folders” on the Archiver system since the Archiver has its own internal labels. These synced labels (folders) can be used to search by folder or extend the data retention base on the folders emails reside in.

• In order to use the Google Workspace label sync service, first visit the Configuration -> Mailbox Services page.

• A Mail Server definition may already exist if you have already run a Gmail Crawler import. But if a server definition does not exist, you can create one by clicking “Add New Server” and filling out the following form.

• After defining a server, you need to define and schedule the label sync activity related to that server. Click on “Add New Activity” on the Mailbox Services page and fill out the following form.

• Your Archiver will start a label sync job on the days and time you specified. Check the System Status page for jobs names “GoogleLabelSync*” once the scheduled time has arrived. You will see detailed information on which user accounts and which labels are being synced.

Performance Improvements

1. The performance of indexing was significantly improved: an 80% to 1,000% speed improvement has been measured depending on the system.
2. The memory usage of search was improved to make it less likely to run out of memory and able to recover without intervention if it does.
3. Minor performance improvements to search have been made.

Disk space savings

• Added log rotation to database log files so that they will not unnecessarily fill up the disk.
• Retroactively clean up unneeded raw emails — remove any unneeded raw emails that may have accumulated prior to 6.12.

Security Improvements

1. For server-side errors, show a user-friendly error message, not a traceback.
2. Disable NTP monitoring (the monlist command) to prevent denial-of-service attacks
3. Restrict NTP mode 6 queries to prevent a denial of service condition.
4. Restrict the email address input on the email forwarding form from accepting line ending characters to prevent an email header injection vulnerability.

1. Fixed: Search on “From” address with “contains all of” was acting like “contains any of”.
2. Fixed: Deleted tags could still be applied to emails by a Scheduled Search.
3. Prevention for SMTP rejections due to SYN flooding was added.
4. Prevent LDAP connector name from containing a dash on copy (dash is disallowed elsewhere but was allowed on copy).
5. If a user tried to create a second mail services server with the same hostname as an existing one, they would see the success message “The Exchange Server entry was created” even though it was not. Now the user will see an error message instead: “Definition for the server already exists”.
6. Fixed: Import history reported incorrect counts for raw-type imports (raw, google, barracuda).
7. Fixed a very rare race condition that led to the backup uploader process hanging until restarted.
8. Fixed an error computing the amount of disk used in some cases.